DEV Community: Jason Barr

Create Your Own Programming Language 10: Optimization

Jason Barr — Sun, 09 Jul 2023 23:20:28 +0000

Welcome to the next article in the Create Your Own Programming Language series! In this one we're going to transform the abstract syntax tree (AST) to A Normal Form and also implement one of the coolest features of functional languages: tail call optimization.

As always, if you haven't read the previous article on iteration, do that first before continuing.

Ok, good? Let's do it.

A Normal Form

Unless you're a programming language geek, you've probably never heard of A Normal Form (ANF).

In a compiler, it's normal to have multiple passes that transform the AST into intermediary forms. ANF is one such intermediary form.

ANF is an intermediary form where we un-nest nested expressions (like call expressions inside of other call expressions) and flatten things out.

That means we'll take an expression like this one:

(slice (+ 1 3) (* 2 5) (concat (list 1 2 3) (list 4 5 6)))

And turn it into something like this:

(def _1 (+ 1 3))
(def _2 (* 2 5))
(def _3 (list 1 2 3))
(def _4 (list 4 5 6))
(def _5 (concat _3 _4))
(slice _1 _2 _5)

This simplification of nested expressions makes it possible to do all sorts of further transformations and optimizations on the program before emitting code.

Tail Call Optimization

Tail call optimization is a feature most often found in functional languages like Scheme that allows essentially infinite recursion as long as the recursive call is in tail position.

Tail position means the function call is the last expression in a function's control flow path, and the expression is just the call expression with no other operations being performed.

For example, here's a naive implementation of the factorial function:

(def fact (n)
  (if (= n 1)
    n
    (* n (fact (- n 1)))))

The recursive call to fact is not in tail position here, because it's part of the expression beginning with *.

Here's a tail recursive version of fact:

(def fact (n a)
  (if (= n 1)
    a
    (fact (- n 1) (* a n))))

As you can see, in the tail recursive version we keep the value of the multiplication operation as a parameter to the function. This kind of parameter is called an accumulator.

Normally, a recursive call adds another frame to the call stack, and when the stack grows large enough you get a stack overflow error.

In a language like Scheme, the interpreter is optimized so that tail recursive calls don't add new frames to the stack.

Instead, the frame for the new function call replaces the previous frame on the stack, allowing essentially infinite tail recursion.

The technique we use to optimize tail calls will allow us to have proper tail call optimization even though most JavaScript engines don't implement TCO natively (which is a shame, because it IS actually in the ECMAScript specification).

Fixing a Bug in The Parser

Before we begin implementing the A Normal Form transformation we need to quickly fix a bug in the parser.

I've gone back and fixed this in the original post on functions, but in case you read it before I edited it to include the fix I'll go ahead and share it here too.

In parseFunction in src/parser/parse.js there's a bug that prevents the 2nd expression in the body of a function from being properly parsed. In the code that checks to see if there is a return type annotation, we need to make a change. Starting from the top of the function change it to this:

  let retType, body;

  if (maybeArrow.type === TokenTypes.Symbol && maybeArrow.value === "->") {
    // has return type annotation
    retType = parseTypeAnnotation(maybeRetType);
    body = maybeBody;
  } else {
    retType = null;
    body = maybeRetType
      ? [maybeArrow, maybeRetType, ...maybeBody]
      : [maybeArrow, ...maybeBody];
  }

Then the function can continue from const variadic =... to the end.

Preparing for A Normal Form

We also need to make a couple of changes to the core library in preparation for the transformation to A Normal Form. We'll be constructing new AST nodes in the transformation, including some function calls, and we need to make sure those functions are correct in the standard library.

First, we need to make a minor change to the get function in lib/js/core.js. We'll make it so it also works to get an index from a vector:

    get: rt.makeFunction(
      (n, obj) => {
        let value =
          obj.get && typeof obj.get === "function" ? obj.get(n) : obj[n];

        if (value === undefined) {
          throw new Exception(`Value for index ${n} not found on object`);
        }

        return value;
      },
      { contract: "(((list any) || (vector any)) -> any)", name: "get" }
    ),

We're also going to add a new function slice at the bottom of the module:

    slice: rt.makeFunction(
      (start, end, obj = undefined) => {
        if (end < 0) {
          end = [...obj].length + end;
        } else if (obj === undefined) {
          // didn't pass in an end value, which is valid
          obj = end;
          end = [...obj].length;
        }

        if (isList(obj)) {
          let values = [];
          let i = start;

          while (i < end && obj.get(i) !== undefined) {
            values.push(obj.get(i));
          }

          return Cons.from(values);
        } else if (Array.isArray(obj)) {
          return obj.slice(start, end);
        } else {
          throw new Exception(
            "slice can only take a list or vector as its argument"
          );
        }
      },
      {
        name: "slice",
        contract:
          "(number, number, ((vector any) || (list any)) -> ((vector any) || (list any)))",
      }
    ),

As you'll see when we get to the ANF transformation there's a little white lie here: we're also going to use it on tuples. But since tuples are arrays under the hood and at the point of transformation we're already done with the type checker, it's not a big deal to do that. Just remember that slice won't type check for tuples, so you can't actually use it to slice tuples in your Wanda code.

You'll also notice that we've made the 3rd parameter default to undefined because if we didn't then makeFunction would curry it to 3 parameters. We need to be able to call it with only 2 arguments, as you'll see below when we get to transforming variable declarations.

Transforming to A Normal Form

An expression is said to be in A Normal Form if its complex and nested subexpressions have been un-nested and bound in let expressions (we'll use variable declarations).

Primitive values and symbols are already considered to be in A Normal Form.

All arguments to a function must be trivial, which is to say they should already be terminally evaluated and be passed into the function as either primitives or symbols. Another way of putting it is that arguments to a function need to be in a form where evaluating them halts immediately instead of calling out to another function.

You can have some nested expressions, i.e. the test expression for an if expression can still be a call expression as long as its arguments have been terminally evaluated.

You'll start to develop an intuition for it as we go through the conversion process.

Since we're unnesting expressions that are embedded within other expressions, a number of our conversion functions will have to return arrays of expression nodes. That means we're going to be doing a lot of checks for arrays in our conversion functions.

When a transformation produces an array of expressions, the last expression in the array will always be the target expression. So for example if we convert an if expression and it unnests additional expressions, the converted if expression will always be the last member of the array.

This makes array handling more straightforward, because we can always pop the last expression from the array and then concatenate the remainder of the array to our unnested expressions.

The Program node, do expressions, and when expressions all contain a body property that is an array of expressions, so when we get an array of unnested expressions in a function that processes one of those we'll need to use flatMap to join those arrays to the body properties of those nodes.

We're also going to handle destructuring at this stage and unnest all the separate variables that are parts of the vector and record patterns, which means we'll be able to remove the code in the emitter that handles destructured assignment.

It sounds more convoluted than it actually is, so let's start with some code and you can see how it works.

The Dispatcher Function

First, here are the imports for src/transform/anf.js:

import { AST, ASTTypes } from "../parser/ast.js";
import { isPrimitive } from "../parser/utils.js";
import { Exception } from "../shared/exceptions.js";
import { makeGenSym } from "../runtime/makeSymbol.js";
import { Token } from "../lexer/Token.js";
import { TokenTypes } from "../lexer/TokenTypes.js";

We also need these 2 helper functions:

const createFreshSymbol = (srcloc) => {
  return AST.Symbol(Token.new(TokenTypes.Symbol, makeGenSym(), srcloc));
};

const isPrimitiveOrSymbol = (node) => {
  return isPrimitive(node) || node.kind === ASTTypes.Symbol;
};

As usual, we need a dispatcher function that handles each of the different node kinds:

export const anf = (node) => {
  switch (node.kind) {
    case ASTTypes.Program:
      return transformProgram(node);
    case ASTTypes.NumberLiteral:
    case ASTTypes.StringLiteral:
    case ASTTypes.BooleanLiteral:
    case ASTTypes.KeywordLiteral:
    case ASTTypes.NilLiteral:
    case ASTTypes.Symbol:
      return node;
    case ASTTypes.CallExpression:
      return transformCallExpression(node);
    case ASTTypes.LambdaExpression:
      return transformLambdaExpression(node);
    case ASTTypes.VariableDeclaration:
      return transformVariableDeclaration(node);
    case ASTTypes.SetExpression:
      return transformSetExpression(node);
    case ASTTypes.TypeAlias:
      // ignore
      return node;
    case ASTTypes.VectorLiteral:
      return transformVectorLiteral(node);
    case ASTTypes.RecordLiteral:
      return transformRecordLiteral(node);
    case ASTTypes.MemberExpression:
      return transformMemberExpression(node);
    case ASTTypes.DoExpression:
      return transformDoExpression(node);
    case ASTTypes.AsExpression:
      return transformAsExpression(node);
    case ASTTypes.IfExpression:
      return transformIfExpression(node);
    case ASTTypes.WhenExpression:
      return transformWhenExpression(node);
    case ASTTypes.LogicalExpression:
      return transformLogicalExpression(node);
    default:
      throw new Exception(`Unhandled node kind: ${node.kind}`);
  }
};

You may wonder why we're writing separate functions instead of using our generic visitor. I decided there was no benefit to using the visitor since we'd need to write a method for every single node type that's left after desugaring, so I wrote the transformer as a series of functions.

As you can see, for primitives and symbols it simply returns the node. There's nothing to do with type aliases either, so it returns that node as well. The rest of the nodes all dispatch to separate transformer functions.

Transforming The Program

transformProgram is very simple: it just produces a new body by flatMapping over the original body with anf as the map function:

const transformProgram = (node) => {
  let body = node.body.flatMap(anf);

  return { ...node, body };
};

Transforming Call Expressions

transformCallExpression is the first function to return an array of expressions rather than a single expression.

We start by creating the array. Then we transform the function itself. If the result is an array, we pop the function off the array then concatenate the rest of the array to the unnested expressions array.

Then we need to handle arguments.

Arguments can be either simple or complex/nested expressions, so we need to do some work to handle arguments.

First, we create an array to hold the transformed arguments. Then we loop over the original node's arguments.

An argument to a function can be another call expression, in which case we need to process the subcall and unnest any of its arguments.

So if the argument is a call expression we loop over its args and unnest any nested expressions then push the bare arguments themselves onto a subargs array. We transform complex arguments by unnesting them and creating new variable assignments for the nested expressions, pushing the symbols for the new variables onto the subargs array and adding the unnested expressions and new variable declarations to the unnested expressions array.

We take the transformed subcall arguments and construct a new call expression, using it as the expression for a new variable declaration, then add that variable to the arguments array for the original call expression. The variable declaration gets added to the unnested expressions array.

Processing a subcall serves as a base case for recursively processing call expressions.

If an argument to the main call expression is not another call expression, it simply gets processed by the anf function and the transformed expression is handled by popping the main expression off any array that was returned and concatenating any unnested expressions onto the unnested expressions array.

It sounds convoluted, but the code is fairly straightforward. Here's the code for transformCallExpression, where you'll see I've annotated everything with comments so you can relate the code to the explanation I've given above:

const transformCallExpression = (node) => {
  // create an array for unnested expressions from the call expression
  let unnestedExprs = [];
  // transform the function
  let func = anf(node.func);

  // if func has been transformed into an array, get the actual function
  // which will be the last expression in the array from the transformer
  if (Array.isArray(func)) {
    func = func.pop();
    // add the unnested expressions to our unnested expressions array
    unnestedExprs.concat(func);
  }

  let args = [];

  for (let arg of node.args) {
    // if it's a call expression, we need to unnest any subexpressions from
    // the arguments to the sub-call expression and create a new call expr
    if (arg.kind === ASTTypes.CallExpression) {
      // we'll need unnested arguments for the subcall
      let subArgs = [];
      for (let a of arg.args) {
        // primitives and symbols are already in ANF
        if (isPrimitiveOrSymbol(a)) {
          subArgs.push(a);
          // otherwise, we need to unnest the expression and bind the result to a new
          // variable, then replace the expression in the call body with that variable
        } else {
          const freshLet = AST.VariableDeclaration(
            createFreshSymbol(a.srcloc),
            a,
            a.srcloc,
            null
          );
          const transformedLet = transformVariableDeclaration(freshLet);
          // the actual declaration will always be the last node in the array
          // we're going to need to add this to the unnested expressions
          // for the parent call expression, so we don't pop it
          const actualLet = transformedLet[transformedLet.length - 1];

          // add the unnested expressions from the VariableDeclaration
          // to the unnested expressions from the call expression
          unnestedExprs = unnestedExprs.concat(transformedLet);
          // add the variable that's been assigned to
          // its relative place in the subcall args
          subArgs.push(actualLet.lhv);
        }
      }

      // create a new CallExpression with unnested sub-arguments
      let subCall = AST.CallExpression(arg.func, subArgs, arg.srcloc);
      // create a fresh variable symbol
      const callSymbol = createFreshSymbol(subCall.srcloc);
      // assign the result of the unnested call expression to the fresh variable
      const callLet = AST.VariableDeclaration(
        callSymbol,
        subCall,
        subCall.srcloc
      );

      // add the assignment to the unnested expressions
      unnestedExprs.push(callLet);
      // the argument to the parent call expression should now be the fresh variable
      arg = callSymbol;
    } else {
      arg = anf(arg);
    }

    if (Array.isArray(arg)) {
      // the actual arg will always be the last element in this array, the rest
      // are all unnested expressions and should be concatenated to that array
      args.push(arg.pop());
      unnestedExprs = unnestedExprs.concat(arg);
    } else {
      // the anfed arg is a single node
      args.push(arg);
    }
  }

  const newCallExpr = AST.CallExpression(func, args, node.srcloc);

  return [...unnestedExprs, newCallExpr];
};

Transforming Lambda Expressions

Transforming lambda expressions is extremely simple. We don't need to worry about parameters because there's no nesting there, so all we need to do is flatMap over the body with anf as the mapping function:

const transformLambdaExpression = (node) => {
  const body = node.body.flatMap(anf);
  return { ...node, body };
};

Transforming Variable Declarations

Transforming variable declarations is complicated by the need to unnest variables used in destructuring.

First, like usual, we create an array for unnested expressions. Then we ANF transform the initializer expression for the declaration node.

If the ANFed expression is an array, we pop off the actual expression then concatenate the rest to the unnested expressions array.

Whether or not it's an array, we construct a new declaration using the transformed expression.

If it's a simple variable declaration using a single symbol as the variable name, we're done here and can just return an array containing the unnested expressions and ending with the newly constructed declaration.

const transformVariableDeclaration = (node) => {
  let unnestedExprs = [];
  const anfedExpr = anf(node.expression);
  let anfedDecl;
  let expression;

  if (Array.isArray(anfedExpr)) {
    expression = anfedExpr.pop();
    anfedDecl = { ...node, expression };

    unnestedExprs = unnestedExprs.concat(anfedExpr);
  } else {
    expression = anfedExpr;
    anfedDecl = { ...node, expression };
  }
  // rest of function which handles destructured declarations
  // If we get here, it's a simple variable declaration with a symbol as LHV
  return [...unnestedExprs, anfedDecl];
};

If it's a destructured variable declaration, we've got some work to do.

Transforming Vector Destructuring

If it's vector destructuring, we need to loop over the members of the vector pattern.

For each member of the vector pattern, we construct a new variable declaration with a call to the get function for the current index of the list/vector/tuple to get the correct value for the destructured variable.

Note that this function call will throw an error if the object being destructured doesn't have enough members to satisfy the number of destructured variables.

If the last member is a rest variable, then instead of using the get function we use slice, passing it the current index and the object being destructured. This is why we needed to be able to call slice with only 2 arguments, passing it only a start numeric value instead of requiring both start and stop.

  if (node.lhv.kind === ASTTypes.VectorPattern) {
    // is vector pattern destructuring
    /** @type {import("../parser/ast.js").VectorPattern} */
    const pattern = node.lhv;
    let i = 0;

    for (let mem of pattern.members) {
      if (i === pattern.members.length - 1 && pattern.rest) {
        // need to slice off the rest of the list/vector/tuple and assign it to the last member
        const destructuredDecl = AST.VariableDeclaration(
          mem,
          AST.CallExpression(
            AST.Symbol(Token.new(TokenTypes.Symbol, "slice", mem.srcloc)),
            [
              AST.NumberLiteral(
                Token.new(TokenTypes.Number, i.toString(), mem.srcloc)
              ),
              expression,
            ],
            mem.srcloc
          )
        );
        // and push it onto the unnestedExprs array
        unnestedExprs.push(destructuredDecl);
      } else {
        // need to get the value from the current index of the list/vector/tuple and assign it to the current pattern member
        const destructuredDecl = AST.VariableDeclaration(
          mem,
          AST.CallExpression(
            AST.Symbol(Token.new(TokenTypes.Symbol, "get", mem.srcloc)),
            [
              AST.NumberLiteral(
                Token.new(TokenTypes.Number, i.toString(), mem.srcloc)
              ),
              expression,
            ]
          )
        );
        // and push it onto the unnestedExprs array
        unnestedExprs.push(destructuredDecl);
      }
      i++;
    }
    return unnestedExprs;
  }
  // handle record destructuring...

Transforming Record Destructuring

If it's record destructuring, we start by assigning the actual object being destructured to a fresh variable name and pushing the assignment node onto the unnested expressions array. We save the symbol node for this fresh variable so we can use it in what follows.

Now we loop over the properties of the record pattern. We need to keep track of what properties have been used with an array, so we can use the unused properties to create the object assigned to the rest variable. Since this step comes after the type checker, we'll use the type information we got from the type checker to do that.

If it's not a rest variable, we simply create a new variable declaration node and construct a member expression node using the original object and the current property being looped over, then push that declaration onto the unnested expressions array.

Then we push the property name we've just used onto the used properties array.

If it's a rest variable, we create a new array of all the unused properties by filtering over the RHV's type properties then mapping those properties to new symbol nodes.

Then we get the properties for the remainder object by reducing the unused properties array and creating a new array of Property nodes, then use that array to construct a new RecordLiteral node.

Then we create a new variable declaration assigning the remainder object to the rest variable name.

Finally, we push the final declaration onto the unnested expressions array.

  // closing brace is the last one from the previous code block, do not duplicate
  } else if (node.lhv.kind === ASTTypes.RecordPattern) {
    // is record pattern destructuring
    // remember, we have the RHV's type at this point
    /** @type {import("../parser/ast.js").RecordPattern} */
    const pattern = node.lhv;
    // first we need to assign the actual object to a fresh variable name
    const objSymbol = createFreshSymbol(expression.srcloc);
    const objDecl = AST.VariableDeclaration(
      objSymbol,
      expression,
      expression.srcloc
    );

    unnestedExprs.push(objDecl);

    let i = 0;
    let used = [];

    for (let prop of pattern.properties) {
      if (i === pattern.properties.length - 1 && pattern.rest) {
        // need to get the rest of the object's properties and assign them to the rest variable
        // this maps the array of unused properties from the type to an array of Symbol
        // nodes with each property name as the node name
        const unusedProps = expression.type.properties
          .filter((p) => {
            return !used.includes(p.name);
          })
          .map((p) =>
            AST.Symbol(Token.new(TokenTypes.Symbol, p.name, prop.srcloc))
          );

        // now create an object using the properties
        const properties = unusedProps.reduce((props, p) => {
          return [
            ...props,
            AST.Property(
              p,
              AST.MemberExpression(objSymbol, p, p.srcloc),
              p.srcloc
            ),
          ];
        }, []);
        const remainingObject = AST.RecordLiteral(properties, prop.srcloc);
        // and a variable declaration using the remainder object assigning it to the rest variable
        const restDecl = AST.VariableDeclaration(
          prop,
          remainingObject,
          prop.srcloc
        );

        unnestedExprs.push(restDecl);
      } else {
        // need to assign the current variable's object property
        const currentDecl = AST.VariableDeclaration(
          prop,
          AST.MemberExpression(objSymbol, prop, prop.srcloc)
        );
        // and push it onto the unnestedExprs array
        unnestedExprs.push(currentDecl);
        used.push(prop.name);
      }
      i++;
    }

    return unnestedExprs;
  }

Here's the entire transformVariableDeclaration function:

const transformVariableDeclaration = (node) => {
  let unnestedExprs = [];
  const anfedExpr = anf(node.expression);
  let anfedDecl;
  let expression;

  if (Array.isArray(anfedExpr)) {
    expression = anfedExpr.pop();
    anfedDecl = { ...node, expression };

    unnestedExprs = unnestedExprs.concat(anfedExpr);
  } else {
    expression = anfedExpr;
    anfedDecl = { ...node, expression };
  }

  if (node.lhv.kind === ASTTypes.VectorPattern) {
    // is vector pattern destructuring
    /** @type {import("../parser/ast.js").VectorPattern} */
    const pattern = node.lhv;
    let i = 0;

    for (let mem of pattern.members) {
      if (i === pattern.members.length - 1 && pattern.rest) {
        // need to slice off the rest of the list/vector/tuple and assign it to the last member
        const destructuredDecl = AST.VariableDeclaration(
          mem,
          AST.CallExpression(
            AST.Symbol(Token.new(TokenTypes.Symbol, "slice", mem.srcloc)),
            [
              AST.NumberLiteral(
                Token.new(TokenTypes.Number, i.toString(), mem.srcloc)
              ),
              expression,
            ],
            mem.srcloc
          )
        );
        // and push it onto the unnestedExprs array
        unnestedExprs.push(destructuredDecl);
      } else {
        // need to get the value from the current index of the list/vector/tuple and assign it to the current pattern member
        const destructuredDecl = AST.VariableDeclaration(
          mem,
          AST.CallExpression(
            AST.Symbol(Token.new(TokenTypes.Symbol, "get", mem.srcloc)),
            [
              AST.NumberLiteral(
                Token.new(TokenTypes.Number, i.toString(), mem.srcloc)
              ),
              expression,
            ]
          )
        );
        // and push it onto the unnestedExprs array
        unnestedExprs.push(destructuredDecl);
      }
      i++;
    }
    return unnestedExprs;
  } else if (node.lhv.kind === ASTTypes.RecordPattern) {
    // is record pattern destructuring
    // remember, we have the RHV's type at this point
    /** @type {import("../parser/ast.js").RecordPattern} */
    const pattern = node.lhv;
    // first we need to assign the actual object to a fresh variable name
    const objSymbol = createFreshSymbol(expression.srcloc);
    const objDecl = AST.VariableDeclaration(
      objSymbol,
      expression,
      expression.srcloc
    );

    unnestedExprs.push(objDecl);

    let i = 0;
    let used = [];

    for (let prop of pattern.properties) {
      if (i === pattern.properties.length - 1 && pattern.rest) {
        // need to get the rest of the object's properties and assign them to the rest variable
        // this maps the array of unused properties from the type to an array of Symbol
        // nodes with each property name as the node name
        const unusedProps = expression.type.properties
          .filter((p) => {
            return !used.includes(p.name);
          })
          .map((p) =>
            AST.Symbol(Token.new(TokenTypes.Symbol, p.name, prop.srcloc))
          );

        // now create an object using the properties
        const properties = unusedProps.reduce((props, p) => {
          return [
            ...props,
            AST.Property(
              p,
              AST.MemberExpression(objSymbol, p, p.srcloc),
              p.srcloc
            ),
          ];
        }, []);
        const remainingObject = AST.RecordLiteral(properties, prop.srcloc);
        // and a variable declaration using the remainder object assigning it to the rest variable
        const restDecl = AST.VariableDeclaration(
          prop,
          remainingObject,
          prop.srcloc
        );

        unnestedExprs.push(restDecl);
      } else {
        // need to assign the current variable's object property
        const currentDecl = AST.VariableDeclaration(
          prop,
          AST.MemberExpression(objSymbol, prop, prop.srcloc)
        );
        // and push it onto the unnestedExprs array
        unnestedExprs.push(currentDecl);
        used.push(prop.name);
      }
      i++;
    }

    return unnestedExprs;
  }
  // If we get here, it's a simple variable declaration with a symbol as LHV
  return [...unnestedExprs, anfedDecl];
};

Remember, we're only using toplevel destructuring without nested patterns. Adding nested patterns would make this even more complex.

Transforming Set Expressions

The code for transformSetExpressions is similar to what just the simple symbol variable parts of transformVariableDeclaration look like:

const transformSetExpression = (node) => {
  const anfedExpr = anf(node.expression);

  if (Array.isArray(anfedExpr)) {
    let expression = anfedExpr.pop();
    return [...anfedExpr, { ...node, expression }];
  }

  return [{ ...node, expression: anfedExpr }];
};

Transforming Vector Literals

To transform a vector literal we just need to unnest any nested expressions that make up the vector's members:

const transformVectorLiteral = (node) => {
  let unnestedExprs = [];
  let members = [];

  for (let mem of node.members) {
    let anfed = anf(mem);

    if (Array.isArray(anfed)) {
      members.push(anfed.pop());
      unnestedExprs = unnestedExprs.concat(anfed);
    } else {
      members.push(anfed);
    }
  }

  return [...unnestedExprs, { ...node, members }];
};

Transforming Record Literals

Same goes for transforming record literals where we unnest any nested expressions used in creating the record's properties:

const transformRecordLiteral = (node) => {
  let unnestedExprs = [];
  let properties = [];

  for (let prop of node.properties) {
    let anfed = anf(prop.value);

    if (Array.isArray(anfed)) {
      let value = anfed.pop();
      properties.push({ ...prop, value });
      unnestedExprs = unnestedExprs.concat(anfed);
    } else {
      properties.push({ ...prop, value: anfed });
    }
  }

  return [...unnestedExprs, { ...node, properties }];
};

Transforming Member Expressions

The only complication in transforming a member expression is that the object can be practically any node type as long as the expression produces an object:

const transformMemberExpression = (node) => {
  let unnestedExprs = [];
  let anfedObject = anf(node.object);

  if (Array.isArray(anfedObject)) {
    const object = anfedObject.pop();
    unnestedExprs = unnestedExprs.concat(anfedObject);
    return [...unnestedExprs, { ...node, object }];
  }

  return [{ ...node, object: anfedObject }];
};

Transforming Do Expressions

Transforming a do expression is pretty much the same as transforming the Program:

const transformDoExpression = (node) => {
  const body = node.body.flatMap(anf);
  return { ...node, body };
};

Transforming As Expressions

In transforming as expressions, we're just going to transform the expression part and return it. That means we'll no longer need to handle as expressions in the emitter, just like we no longer need to handle destructuring in the emitter:

const transformAsExpression = (node) => {
  const anfed = anf(node.expression);

  if (Array.isArray(anfed)) {
    return anfed;
  }

  return [anfed];
};

Transforming If Expressions

If expressions are straightforward. Transform the test, then append any unnested expressions to the unnested expressions array. Then do the same with the consequent (then) and alternate (else) branches:

const transformIfExpression = (node) => {
  let unnestedExprs = [];
  let transformedCondition = anf(node.test);
  let test;

  if (Array.isArray(transformedCondition)) {
    test = transformedCondition.pop();
    unnestedExprs = unnestedExprs.concat(transformedCondition);
  } else {
    test = transformedCondition;
  }

  let transformedThen = anf(node.then);
  let then;

  if (Array.isArray(transformedThen)) {
    then = transformedThen.pop();
    unnestedExprs = unnestedExprs.concat(transformedThen);
  } else {
    then = transformedThen;
  }

  let transformedElse = anf(node.else);
  let elseBranch;

  if (Array.isArray(transformedElse)) {
    elseBranch = transformedElse.pop();
    unnestedExprs = unnestedExprs.concat(transformedElse);
  } else {
    elseBranch = transformedElse;
  }

  return [...unnestedExprs, { ...node, test, then, else: elseBranch }];
};

Transforming When Expressions

When expressions are like a combination between if and do expressions. Transform the test like an if expression, then transform the body like a do expression:

const transformWhenExpression = (node) => {
  let unnestedExprs = [];
  let transformedCondition = anf(node.test);
  let test;

  if (Array.isArray(transformedCondition)) {
    test = transformedCondition.pop();
    unnestedExprs = unnestedExprs.concat(transformedCondition);
  }

  let body = node.body.flatMap(anf);

  return [...unnestedExprs, { ...node, test, body }];
};

Transforming Logical Expression

Logical expressions are straightforward. Transform the left, transform the right, add anything that needs it to the unnested expressions array:

const transformLogicalExpression = (node) => {
  let unnestedExprs = [];
  let transformedLeft = anf(node.left);
  let left;

  if (Array.isArray(transformedLeft)) {
    left = transformedLeft.pop();
    unnestedExprs = unnestedExprs.concat(transformedLeft);
  } else {
    left = transformedLeft;
  }

  let transformedRight = anf(right);
  let right;

  if (Array.isArray(transformedRight)) {
    right = transformedRight.pop();
    unnestedExprs = unnestedExprs.concat(transformedRight);
  } else {
    right = transformedRight;
  }

  return [...unnestedExprs, { ...node, left, right }];
};

That's all the transformation functions!

Figuring out how to do AST transformations vexed me for a long time, so I really hope this helps you understand how transforming the tree works.

Tail Call Optimization

Now let's take the transformed tree and use it to detect recursive tail calls.

To detect recursive tail calls we traverse the whole transformed AST. When we come to a lambda expression, we check if it has a name property (which gets added when function declarations are desugared into variable declarations with lambdas).

If the last (tail position) expression in the lambda body is a call expression, or if it's an if, do, or logical expression that contains a call expression in tail position, we check to see if the func property of that call expression is a Symbol node with the same name. If it is, then it's a recursive tail call and we can mark it as such.

We'll mark the call expression, lambda expression, and if the tail call is part of an if expression the if expression with a boolean flag indicating if it's tail recursive.

We'll use a class and extend Visitor for this, because we only need functions for a couple of node types but we need to traverse the whole tree.

First the imports, in src/transform/tco.js:

import { ASTTypes } from "../parser/ast.js";
import { Visitor } from "../visitor/Visitor.js";

We need a couple of helper functions. First, we'll need one to swap the last node in an expression body. We need to replace tail call nodes with new nodes that mark them as tail calls. Note that the 1st argument to this is the node with the body property, not the body itself:

const swapLastExpr = (node, expr) => {
  node.body[node.body.length - 1] = expr;
};

We also need a function to check if expressions. We'll use a function for this instead of using the regular visitor method because we only need to check if expressions that are in tail position in the body of a lambda or do expression. All other if expressions can be handled as normal.

This function recursively checks if expressions that are the then or else branch of the if expression being checked:

const checkIfExpression = (node, name, visitor) => {
  let isTailRec = false;

  if (
    node.then.kind === ASTTypes.CallExpression &&
    node.then.func.name === name
  ) {
    node.then = visitor.visitCallExpression(node.then, true);
    isTailRec = true;
  }

  if (
    node.else.kind === ASTTypes.CallExpression &&
    node.else.func.name === name
  ) {
    node.else = visitor.visitCallExpression(node.else, true);
    isTailRec = true;
  }

  if (node.then.kind === ASTTypes.IfExpression) {
    node.then = checkIfExpression(node.then, name, visitor);
    if (node.then.isTailRec) {
      isTailRec = true;
    }
  }

  if (node.else.kind === ASTTypes.IfExpression) {
    node.else = checkIfExpression(node.else, name, visitor);
    if (node.else.isTailRec) {
      isTailRec = true;
    }
  }

  return { ...node, isTailRec };
};

Ok, first let's stub out the class for the TCO transformer:


class TCOTransformer extends Visitor {
  constructor(program) {
    super(program);
  }

  static new(program) {
    return new TCOTransformer(program);
  }
}

Now let's add an override for visitCallExpression that takes an extra, optional parameter isTailRec that defaults to false. It has to have a default because it will only get called with true from our override version of visitLambdaExpression:

  visitCallExpression(node, isTailRec = false) {
    return { ...node, isTailRec };
  }

Ok, now let's turn to visitLambdaExpression. It's a lot of code, but it's straightforward. If the lambda node has a name property, save it to a variable. Then get the last expression from the lambda body.

If the last expression is a call expression we check to see if its func property is a Symbol node with the same name as the lambda.

If it's a do expression, if expression, or logical expression we check to see if either the last expression (of a do expression) or either branch (of an if or logical expression) contains a call in tail position that calls a function with the same name.

If we detect a recursive tail call, we mark the call expression and lambda expression both with flags isTailRec set to true.

  visitLambdaExpression(node) {
    const name = node.name ?? "";
    const lastExpr = node.body[node.body.length - 1];

    // Could be tail recursive: CallExpression, DoExpression, IfExpression, LogicalExpression
    if (lastExpr.kind === ASTTypes.CallExpression) {
      // should only work if func is symbol
      if (lastExpr.func.name === name) {
        const newCall = this.visitCallExpression(lastExpr, true);
        swapLastExpr(node, newCall);
        return { ...node, isTailRec: true };
      }

      lastExpr.isTailRec = false;
      return { ...node, isTailRec: false };
    } else if (lastExpr.kind === ASTTypes.DoExpression) {
      const lastBodyExpr = lastExpr.body[lastExpr.body.length - 1];

      if (
        lastBodyExpr.kind === ASTTypes.CallExpression &&
        lastBodyExpr.func.name === name
      ) {
        const newCall = this.visitCallExpression(lastBodyExpr, true);
        swapLastExpr(lastExpr, newCall);
        return { ...node, isTailRec: true };
      } else if (lastBodyExpr.kind === ASTTypes.IfExpression) {
        const newIf = checkIfExpression(lastBodyExpr, name, this);
        swapLastExpr(lastExpr, newIf);
        return { ...node, isTailRec: newIf.isTailRec };
      } else if (lastBodyExpr.kind === ASTTypes.LogicalExpression) {
        let isTailRec = false;

        if (
          lastBodyExpr.left.kind === ASTTypes.CallExpression &&
          lastBodyExpr.left.func.name === name
        ) {
          isTailRec = true;
          lastBodyExpr.left = this.visitCallExpression(lastBodyExpr.left, true);
        }

        if (
          lastBodyExpr.right.kind === ASTTypes.CallExpression &&
          lastBodyExpr.right.func.name === name
        ) {
          isTailRec = true;
          lastBodyExpr.right = this.visitCallExpression(
            lastBodyExpr.right,
            true
          );
        }

        return { ...node, isTailRec };
      } else {
        lastExpr.isTailRec = false;
        return { ...node, isTailRec: false };
      }
    } else if (lastExpr.kind === ASTTypes.IfExpression) {
      let newIf = checkIfExpression(lastExpr, name, this);
      swapLastExpr(node, newIf);
      return { ...node, isTailRec: newIf.isTailRec };
    } else if (lastExpr.kind === ASTTypes.LogicalExpression) {
      let isTailRec = false;
      if (
        lastExpr.left.kind === ASTTypes.CallExpression &&
        lastExpr.left.func.name === name
      ) {
        lastExpr.left = this.visitCallExpression(lastExpr.left, true);
        isTailRec = true;
      }

      if (
        lastExpr.right.kind === ASTTypes.CallExpression &&
        lastExpr.right.func.name === name
      ) {
        lastExpr.right = this.visitCallExpression(lastExpr.right, true);
        isTailRec = true;
      }

      return { ...node, isTailRec };
    }

    return { ...node, isTailRec: false };
  }

Finally, we export a function that constructs the TCO transformer and runs it on a program:

export const tco = (program) => TCOTransformer.new(program).visit();

Now we need a function to handle all transformations, in src/transform/transform.js:

import { anf } from "./anf.js";
import { tco } from "./tco.js";

export const transform = (program) => tco(anf(program));

Changes to The Runtime

Now that we've added the ability to detect tail recursive calls, we need to optimize them.

The simplest way to do that in JavaScript is with a trampoline.

A trampoline is a loop that handles recursive calls. We're going to trampoline our tail recursive functions by making the tail call return a special object that includes the function and arguments to the tail call as properties on the object. You'll see how we rewrite the tail recursive call to achieve this when we get to the emitter.

The setup actually has 2 parts: a recur function that returns the special object and a trampoline function that returns the function that runs the loop.

It sounds complicated, but the two functions are actually very simple, in src/runtime/trampoline.js:

import { makeWandaValue } from "./conversion.js";

// based on this Stack Overflow answer: https://stackoverflow.com/a/50493099

export const recur = (f, ...args) => ({ tag: recur, f, args });

export const trampoline = (f) => {
  const trampolined = (...args) => {
    let t = f(...args);
    while (t && t.tag === recur) {
      t = t.f(...t.args);

      if (t && t.tag !== recur) {
        return makeWandaValue(t);
      }
    }
  };

  trampolined.f = f;

  return trampolined;
};

We also need to make a minor change to our makeFunction function in the runtime.

Currently the function we create in makeFunction returns a call to makeWandaValue, which means right now if we rewrite the recursive call to use rt.recur the object that returns will be transformed into a Wanda object, which will mess up the trampoline function's ability to process the special object.

So let's add a tailRec property to the options argument that defaults to false so when we detect a tail recursive function we can prevent it from transforming the object returned by the tail call. We'll let the trampoline function handle converting the final value to a Wanda value.

Here's our new makeFunction function in src/runtime/makeFunction.js:

import objectHash from "object-hash";
import { curryN } from "ramda";
import { makeWandaValue } from "./conversion.js";
import { addMetaField } from "./object.js";
import { parseContract } from "./parseContract.js";

export const makeFunction = (
  func,
  { contract = "", name = "", tailRec = false } = {}
) => {
  let fn = curryN(func.length, (...args) => {
    const val = tailRec ? func(...args) : makeWandaValue(func(...args));

    if (typeof val === "function") {
      return makeFunction(val);
    }

    return val;
  });
  const hash = objectHash(func);

  addMetaField(fn, "wanda", true);
  addMetaField(fn, "arity", func.length);
  addMetaField(fn, "name", name || hash);

  if (contract !== "") {
    Object.defineProperty(fn, "contract", {
      enumerable: false,
      configurable: false,
      writable: false,
      value: parseContract(contract),
    });
  }

  Object.defineProperty(fn, "name", {
    enumerable: false,
    configurable: false,
    writable: false,
    value: name || hash,
  });

  return fn;
};

Then we need to add trampoline and recur to makeRuntime in src/runtime/makeRuntime.js:

// other imports
import { trampoline, recur } from "./trampoline.js";

export const makeRuntime = () => {
  return {
    // other members
    trampoline,
    recur,
  };
};

Now that our trampoline is in place, we need to modify the emitter to use it.

Changes to The Emitter

The first thing we're going to do to our emitter is remove everything related to handling destructuring and as expressions, since thanks to our ANF transformation they won't be making it to the emitter anymore.

You can delete the emitAsExpression method and the case for it from the emit method. This leaves you with this for emit in src/emitter/Emitter.js:

  emit(node = this.program, ns = this.ns) {
    switch (node.kind) {
      case ASTTypes.Program:
        return this.emitProgram(node, ns);
      case ASTTypes.NumberLiteral:
        return this.emitNumber(node, ns);
      case ASTTypes.StringLiteral:
        return this.emitString(node, ns);
      case ASTTypes.BooleanLiteral:
        return this.emitBoolean(node, ns);
      case ASTTypes.KeywordLiteral:
        return this.emitKeyword(node, ns);
      case ASTTypes.NilLiteral:
        return this.emitNil(node, ns);
      case ASTTypes.Symbol:
        return this.emitSymbol(node, ns);
      case ASTTypes.CallExpression:
        return this.emitCallExpression(node, ns);
      case ASTTypes.VariableDeclaration:
        return this.emitVariableDeclaration(node, ns);
      case ASTTypes.SetExpression:
        return this.emitSetExpression(node, ns);
      case ASTTypes.DoExpression:
        return this.emitDoExpression(node, ns);
      case ASTTypes.TypeAlias:
        return this.emitTypeAlias(node, ns);
      case ASTTypes.MemberExpression:
        return this.emitMemberExpression(node, ns);
      case ASTTypes.RecordLiteral:
        return this.emitRecordLiteral(node, ns);
      case ASTTypes.RecordPattern:
        return this.emitRecordPattern(node, ns);
      case ASTTypes.VectorLiteral:
        return this.emitVectorLiteral(node, ns);
      case ASTTypes.VectorPattern:
        return this.emitVectorPattern(node, ns);
      case ASTTypes.LambdaExpression:
        return this.emitLambdaExpression(node, ns);
      case ASTTypes.LogicalExpression:
        return this.emitLogicalExpression(node, ns);
      case ASTTypes.IfExpression:
        return this.emitIfExpression(node, ns);
      case ASTTypes.WhenExpression:
        return this.emitWhenExpression(node, ns);
      default:
        throw new SyntaxException(node.kind, node.srcloc);
    }
  }

You can also delete the emitVariableDeclarationAssignment method since we won't be using it anymore.

Finally, you can vastly simplify emitVariableDeclaration so it looks like this:

  emitVariableDeclaration(node, ns) {
    const name = node.lhv.name;
    const translatedName = makeSymbol(name);

    if (ns.has(name)) {
      throw new ReferenceException(
        `Name ${name} has already been accessed in the current namespace; cannot access name before its definition`,
        node.srcloc
      );
    }

    ns.set(name, translatedName);

    return `var ${makeSymbol(name)} = ${this.emit(node.expression, ns)}`;
  }

Now to handle tail recursion and the trampoline.

In emitLambdaExpression change the last line beginning with code += to handle the new tailRec option for rt.makeFunction:

    code += `${
      node.name
        ? `, { name: "${node.name}"${node.isTailRec ? ", tailRec: true" : ""} }`
        : ""
    })`;

Also change the return statement to use the trampoline if node.isTailRec is true:

    return node.isTailRec ? `rt.trampoline(${code})` : code;

Now in emitCallExpression we're going to separate the call expression's function from its arguments since the rt.recur function needs them separated. Then if it's a tail call we call rt.recur and pass it the original function (NOT the trampolined version) as well as the arguments. We pass it the original function, which is stored in the f property added to the trampolined function, because if the trampoline function called itself we'd just run into the same recursion limits. Here's the new version of emitCallExpression:

  emitCallExpression(node, ns) {
    const func = `(${this.emit(node.func, ns)})`;
    const args = `${node.args.map((a) => this.emit(a, ns)).join(", ")}`;

    return node.isTailRec ? `rt.recur(${func}.f, ${args})` : `${func}(${args})`;
  }

That's it for the emitter! Now we just need to add the transformation to our compilation pipeline and it will be done.

Changes to The CLI

In src/cli/compile.js we need to import our transform function:

import { transform } from "../transform/transform.js";

And finally we need to add transform to the compile function. Here's the new compile function:

export const compile = (
  input,
  file = "stdin",
  ns = undefined,
  typeEnv = undefined
) =>
  emit(
    transform(
      desugar(typecheck(parse(expand(read(tokenize(input, file)))), typeEnv))
    ),
    ns
  );

And with that we have implemented TCO in our compiler. Were you expecting it to be more difficult? I was certainly surprised at how simple and straightforward it turned out to be.

Trying It Out

Ok, let's fire up a REPL and try it out. First, try the naive factorial function:

(def fact (n)
  (if (= n 1)
    n
    (* n (fact (- n 1)))))

Try it with (fact 1000). Ok, yeah, the answer is Infinity, so we're not going to get better numeric answers for bigger numbers, but that's not the point.

Try it with (fact 2000). If your Node instance is configured like mine, you just got a stack overflow error.

Close your REPL and then open it again (note to self: add a command to refresh the REPL state), then try the tail recursive version of fact:

(def fact (n a)
  (if (= n 1)
    a
    (fact (- n 1) (* a n))))

Start with (fact 1000 1). Now keep incrementing by 1000 and seeing what happens.

I got to 10,000 and then decided to do something crazy: (fact 100000 1).

Infinity.

No stack overflow error, even with 100,000 calls!

I'd say the trampoline works pretty damned well.

Conclusion

I'm thrilled that I was able to get this working the way I wanted to. Now we have true tail call optimization with virtually unlimited tail recursion! I'm excited with where we're at right now.

Like I said, AST transformations are something I struggled with conceptually for a long time before I finally figured out how to make this work.

As always, you can see the current state of the code at the relevant tag in the GitHub repo.

This is currently the last planned post in the series, though it's possible I'll come back and add more later. I hope you've had as much fun reading as I've had writing!

I also hope this inspires you to go out and create your own languages and language related tools. Let me know if you make something you think is cool!

Create Your Own Programming Language 9: Iteration

Jason Barr — Wed, 05 Jul 2023 21:34:23 +0000

In this installment of the Create Your Own Programming Language series we're going to add iteration to Wanda. We're also going to make some major improvements to the CLI at the end of the article.

As always, if you haven't read the previous article where we added conditionals, do that first and then continue below.

Ok, let's go!

Iteration in Wanda

The for form will handle iteration in Wanda, and it looks like this:

(for map ((i (range 5)))
  (+ i i))

The form takes an operator, a list of variables with their initializers, and a body with an arbitrary number of expressions.

Current operators in Wanda include each, map, filter, fold, and fold-r, but you can also define your own for operators by creating functions. They should be higher-order functions that take a function callback as the first argument, and the callback argument should take a parameter for each variable. Then the rest of the main functions' arguments should be the initializers for the callback's parameters.

For instance, here's an example implementation of a map operator:

(def map (fn lst)
  (if (nil? lst)
    lst
    (cons (fn (head lst)) (map fn (tail lst)))))

As you can see, the callback takes a single parameter and the higher-order function takes the callback and a list, which is the initializer for the callback parameter.

Here's how you'd use a for expression to sum up a list of numbers:

(for fold ((sum 0) (x (range 11)))
  (+ sum x))

A for expression desugars to a call expression that uses the operator as its function, constructs a lambda as the first argument to the operator, then passes in the initializers as the remaining arguments to the operator.

Inspiration for our for expressions comes chiefly from the Pyret language, which handles them similarly.

Pyret was inspired by Racket's list comprehensions, Ruby's blocks and iterators, and Smalltalk blocks.

New CLI Features

Here are the new features we're adding to the Wanda CLI:

History (up to 2000 lines)
Help info for the CLI, the wandac compiler, and the REPL
The ability to load a Wanda file from within the REPL
The ability to save a REPL session as a file

We'll get to those at the end of this article, but first let's implement for expressions!

An Easy Iterator

To make it easy to create an object to iterate over, let's add a range function to the core library.

In Python, a Range object doesn't actually contain all the numbers you iterate over. It computes them from the start, stop, and step values. We're not getting that fancy; our range function will just produce a list of numbers. It will work like Python's range function though, in that you can pass it 1, 2, or 3 arguments and it will calculate the range from them.

In lib/js/core.js, just below typeof, add an entry for range:

range: rt.makeFunction(
      (start, stop = undefined, step = 1) => {
        if (typeof stop === "undefined") {
          stop = start;
          start = 0;
        }

        let list = null;
        if (start < stop) {
          list = cons(start, list);
          for (let i = start + step; i < stop; i += step) {
            list.append(i);
          }
        } else if (stop < start) {
          list = cons(start, list);
          for (let i = start - step; i > stop; i -= step) {
            list.append(i);
          }
        }

        return list;
      },
      {
        // contract is variadic because language has no concept of default parameters
        contract: "(&(vector number) -> (list number))",
        name: "range",
      }
    ),

Changes to The Lexer and Reader

None. In fact, for the most part the lexer and reader are done. We may add some small things to them later, but there shouldn't be any major changes to either for the rest of this series.

Changes to The Parser

We need a ForExpression AST node and we need to parse them.

First, add a member to the ASTTypes enum in src/parser/ast.js:

export const ASTTypes = {
  // other members...
  ForExpression: "ForExpression",
};

Then add its constructor to the AST object:

export const AST = {
  // other constructors...
  ForExpression(op, vars, body, srcloc) {
    return {
      kind: ASTTypes.ForExpression,
      op,
      vars,
      body,
      srcloc,
    };
  },
}

In src/parser/parse.js we'll need a new case for the switch statement in parseList:

    // other cases...
    case "for":
      return parseForExpression(form);
    // default case

In parseForExpression we'll get the operator, the variable/initializer list, and the list of body expressions. The operator should be a symbol (though there's technically no reason why it couldn't be a lambda). Then we iterate over the list of variable/initializer pairs and parse them. Finally, we iterate over the body and parse each expression in it.

Here's parseForExpression:

const parseForExpression = (form) => {
  const [_, op, rawVars, ...body] = form;
  const srcloc = form.srcloc;
  const parsedOp = parseExpr(op);

  /** @type {import("./ast.js").ForVar[]} */
  let vars = [];

  for (let rawVar of rawVars) {
    const varName = parseExpr(rawVar.car);
    // need the head of the tail of the rawVar list
    const initializer = parseExpr(rawVar.cdr.car);

    vars.push({ var: varName, initializer });
  }

  /** @type {AST[]} */
  let parsedBody = [];

  for (let expr of body) {
    parsedBody.push(parseExpr(expr));
  }

  return AST.ForExpression(parsedOp, vars, parsedBody, srcloc);
};

That's it for changes to the parser. Now let's see how to type check a for expression.

Changes to The Type Checker

We actually only need to change 2 files in the type checker. First, we need to infer a type for the for expression.

We need to add a dispatch case to infer in src/typechecker/infer.js:

    // other cases...
    case ASTTypes.ForExpression:
      return inferForExpression(ast, env, constant);
    // default case

Now for inferForExpression.

We're actually going to deconstruct the for expression here and infer a type as if it were a function call. This means we need the operator as a function, a lambda as the first argument to the operator, and the list of arguments to follow the lambda.

We get the lambda's params by mapping over node.vars and constructing argument types from the initializers. If an initializer is a list or vector, we use the contained type.

Once we've got the lambda params we construct a LambdaExpression AST node using the params and the expression body. Then we construct an array of arguments for the call expression with the lambda as the first argument and mapping over node.vars again to get the initializers as the remaining arguments.

Then we construct a call expression from the operator and arguments and call infer on it.

Here's inferForExpression:

const inferForExpression = (node, env, constant) => {
  const lambdaArgs = node.vars.map((v) => {
    let varType = infer(v.initializer, env, constant);

    if (Type.isList(varType)) {
      varType = varType.listType;
    } else if (Type.isVector(varType)) {
      varType = varType.vectorType;
    }

    return { name: v.var, type: varType };
  });
  const lambda = AST.LambdaExpression(
    lambdaArgs,
    node.body,
    false,
    null,
    node.srcloc
  );
  const opArgs = [lambda, ...node.vars.map((v) => v.initializer)];

  return infer(AST.CallExpression(node.op, opArgs, node.srcloc), env, constant);
};

The process of constructing the CallExpression node in inferForExpression is similar to how we'll handle for expressions in the desugarer.

Next we need to handle the ForExpression node in src/typechecker/TypeChecker.js.

First, add a case to the switch statement in checkNode:

      // other cases...
      case ASTTypes.ForExpression:
        return this.checkForExpression(node, env);
      // default case

The checkForExpression method is pretty simple:

  checkForExpression(node, env) {
    const op = this.checkNode(node.op, env);
    return { ...node, op, type: infer(node, env) };
  }

That's it for changes to the type checker. It's a lot less than it's been in the past few articles where we've focused more on type checker features.

Now let's look at changes to the default visitor.

Changes to The Visitor

We need a dispatch case and default visitor for the ForExpression node in src/visitor/Visit.js.

First, add a case to the visit method's switch statement:

      // other cases...
      case ASTTypes.ForExpression:
        return this.visitForExpression(node);
      // default case

In visitForExpression we simply visit the operator, visit each variable and initializer in node.vars, and then visit each body expression.

Here's visitForExpression:

  visitForExpression(node) {
    const op = this.visit(node.op);

    let vars = [];

    for (let nodevar of node.vars) {
      const v = this.visit(nodevar.var);
      const initializer = this.visit(nodevar.initializer);

      vars.push({ var: v, initializer });
    }

    let body = [];

    for (let expr of node.body) {
      body.push(this.visit(expr));
    }

    return { ...node, op, vars, body };
  }

Now that there's a default visitor for the ForExpression node, we need to handle it in the desugarer.

Changes to The Desugarer

The process is similar to how we inferred a type for the expression: we construct a lambda, then construct a call expression with that lambda as its first argument.

Here's visitForExpression in src/desugarer/Desugarer.js:

  visitForExpression(node) {
    const op = this.visit(node.op);
    const lambdaArgs = node.vars.map((v) => ({ name: v.var }));
    const lambda = AST.LambdaExpression(
      lambdaArgs,
      node.body,
      false,
      null,
      node.srcloc
    );
    const callArgs = [lambda, ...node.vars.map((v) => v.initializer)];

    return AST.CallExpression(op, callArgs, node.srcloc);
  }

Now we don't need to worry about handling for expressions in the emitter! We do need to fix a bug I found, though.

Changes to The Emitter

The bug is in emitGlobalEnv in src/emitter/emitGlobalEnv.js.

When emitting the actual variable assignments, we currently don't emit the var keyword with them because we needed them to be globals in the REPL.

Now that we're emitting compiled files and have the option to emit a file that imports the global environment using ES2015 imports we need to change that slightly.

The reason is that ES2015 modules use strict mode by default, and in strict mode it throws an error if you assign to a variable without declaring it first with var, let, or const.

Since this is only an issue for compiled files, and not in the REPL or bundled files, we'll add var based on an optional boolean flag passed into the emitGlobalEnv function.

Here's the new version of emitGlobalEnv:

import path from "path";
import { ROOT_PATH } from "../../root.js";
import { makeGlobal } from "../runtime/makeGlobals.js";

export const emitGlobalEnv = (useVar = false) => {
  const globalEnv = makeGlobal();
  let code = `import { makeGlobal } from "${path.join(
    ROOT_PATH,
    "./src/runtime/makeGlobals.js"
  )}";
import { makeRuntime } from "${path.join(
    ROOT_PATH,
    "./src/runtime/makeRuntime.js"
  )}";

const globalEnv = makeGlobal();
${useVar ? "var " : ""}rt = makeRuntime();
`;

  for (let [k] of globalEnv) {
    code += `${useVar ? "var " : ""}${k} = globalEnv.get("${k}");\n`;
  }

  return code;
};

Now when we compile a file and run it as an ES2015 module we won't get an error because of undeclared variables.

Next we need to make a change to the AST printer to handle the new node type.

Changes to The Printer

We need to add the ForExpression node to the AST printer in src/printer/printAST.js.

First, let's add the case to the switch statement in the print method:

      // other cases...
      case ASTTypes.ForExpression:
        return this.printForExpression(node, indent);
      // default case

The printForExpression method is a little verbose because we have to iterate over both node.vars and node.body to print the subexpressions correctly.

Here's printForExpression:

  printForExpression(node, indent) {
    let prStr = `${prIndent(indent)}ForExpression:\n`;
    prStr += `${prIndent(indent + 2)}Operator:\n`;
    prStr += ` ${this.print(node.op, indent + 4)}\n`;
    prStr += `${prIndent(indent + 2)}Vars:\n`;

    for (let nodevar of node.vars) {
      prStr += `${prIndent(indent + 4)}Var: ${this.print(nodevar.var, 0)}\n`;
      prStr += `${prIndent(indent + 4)}Init: ${this.print(
        nodevar.initializer,
        0
      )}\n`;
    }

    prStr += `${prIndent(indent + 2)}Body:\n`;

    for (let expr of node.body) {
      prStr += this.print(expr, indent + 4) + "\n";
    }

    return prStr;
  }

That's it for changes to the printer! Now for expressions fully work within the language, so it's time to focus on our changes to the CLI.

Changes to The CLI

Remember, we're adding these new features to the CLI:

History (up to 2000 lines)
Help info for the CLI, the wandac compiler, and the REPL
The ability to load a Wanda file from within the REPL
The ability to save a REPL session as a file

In order to enable history, we're going to have to change how we get input in the REPL. Instead of using the readline-sync package, we're going to use Node's C++ API to directly access readline.

Don't worry, you won't have to write any C++ code to get this to work. There's a Node.js package that gives you access to the C++ API via JavaScript. Install the ffi-napi package with npm install ffi-napi.

The New Readline

We're going to put this in a new file in the CLI directory, src/cli/readline.js. This will handle both getting input and managing the history state when you fire up the REPL.

You'll need to import some dependencies:

import os from "os";
import fs from "fs";
import { join } from "path";
import ffi from "ffi-napi";

NOTE: The following code accesses libreadline directly, so I have no idea if it's cross platform. It probably isn't. I don't believe Windows includes libreadline by default, though I could be mistaken. I do my development, including the code for this series, on Linux so I haven't tried to make this run on Windows yet. If I do try it on Windows I'll post an update in a later article.

Ok, with that out of the way, first we use ffi to gain access to the readline and add_history primitives and store them in an object. We also set the path to the history file and a flag for if history has been loaded:

const rllib = ffi.Library("libreadline", {
  readline: ["string", ["string"]],
  add_history: ["int", ["string"]],
});

const HISTORY_FILE = join(os.homedir(), ".wanda-history");
let historyLoaded = false;

The readline function takes a prompt and returns the line given in response to the prompt. Most of what comes between those 2 things is managing history.

If the historyLoaded flag is false, we load the history. If the history file exists we read it, split on the end-of-line character, and filter out any blank lines.

If the history file doesn't yet exist or is blank, the array is empty.

Then we slice the remaining array so that a maximum of 2000 lines remain.

Next, loop over the array and add each line to the history.

After loading the history, we continue by prompting the user and getting text in reply to the prompt.

We add that line to the history, and append it to the end of the history file. Finally, return the text.

Here's the readline function:

export const readline = (prompt = ">") => {
  if (!historyLoaded) {
    let lines = [];

    if (fs.existsSync(HISTORY_FILE)) {
      lines = fs
        .readFileSync(HISTORY_FILE, { encoding: "utf-8" })
        .split(os.EOL)
        // remove blank lines
        .filter((line) => line !== "");
    }

    lines = lines.slice(Math.max(lines.length - 2000, 0));

    for (let line of lines) {
      rllib.add_history(line);
    }
  }

  const line = rllib.readline(prompt);

  if (line) {
    rllib.add_history(line);

    try {
      fs.appendFileSync(HISTORY_FILE, line + os.EOL, { encoding: "utf-8" });
    } catch (e) {
      // do nothing
    }
  }

  return line;
};

Getting The Version and Help Info

Next, in src/cli/utils.js, let's write 2 new functions: one to handle getting version info, and one to handle displaying help.

First, we need to import some values into the file:

import fs from "fs";
import { join } from "path";
import { ROOT_PATH } from "../../root.js";

getVersion retrieves the contents of package.json and parses them, then returns the version property.

export const getVersion = () => {
  const packageJson = JSON.parse(
    fs.readFileSync(join(ROOT_PATH, "./package.json"), {
      encoding: "utf-8",
    })
  );
  return packageJson.version;
};

printHelp shows a short introductory message, then loops over an object of commands and displays information based on the contents of each command's object.

A command's object looks like this:

{ alias?: string; description: string; usage?: string; }

Then after showing all the commands' info it shows a postscript message if one is included.

Here's printHelp:

export const printHelp = (commands, application, postscript = "") => {
  console.log(`**** ${application} v${getVersion()} help info ****`);
  console.log();
  console.log("Command:  |  Info:");
  console.log();

  for (let [name, command] of Object.entries(commands)) {
    console.log(`${name}`);
    command.alias && console.log(`             Alias: wanda ${command.alias}`);
    console.log(`             ${command.description}`);
    command.usage && console.log(`             Usage: ${command.usage}`);
  }

  console.log();
  postscript && console.log(postscript);
};

Commands

Ok, now we need help command objects for each of the REPL, the wanda CLI, and the wandac compiler.

Here are the commands for the REPL which should be at the top of src/cli/repl.js, including the new commands to load and save files in the REPL:

const COMMANDS = {
  ":quit": {
    description: "Quits the REPL with exit 0",
  },
  ":print-ast": {
    description:
      "Makes a printed representation of the AST show when you enter an expression",
  },
  ":print-ast -d": {
    description:
      "Like :print-ast, but shows the tree after the desugaring step, right before emitting code",
    usage: ":print-ast -d",
  },
  ":no-print-ast": {
    description: "Turns off AST printing if it's on",
  },
  ":save-file": {
    description: "Saves the current REPL session as a file",
    usage: "Prompts you for a path to save the file",
  },
  ":load-file": {
    description:
      "Loads the definitions from a file into the interactive session",
    usage: "Prompts you for a path to load the file from",
  },
  ":version": {
    description: "Prints the currently installed version of Wanda",
  },
  ":help": {
    description: "Shows this help message",
  },
};

Here are the commands for the wanda CLI, which you should put at the top of src/cli/run.js:

const COMMANDS = {
  load: {
    alias: "-l",
    description:
      "Loads a Wanda file into an interactive session so you can use its definitions",
    usage: "wanda load <filepath> or wanda -l <filepath>",
  },
  run: {
    alias: "-r",
    description: "Runs a Wanda file from the command line",
    usage: "wanda run <filepath> or wanda -r <filepath>",
  },
  repl: {
    alias: "-i",
    description: "Starts an interactive session with the Wanda REPL",
    usage: "wanda repl or wanda -i",
  },
  version: {
    alias: "-v",
    description: "Prints the current version number of your Wanda installation",
    usage: "wanda version or wanda -v",
  },
  help: {
    alias: "-h",
    description: "Prints this help message on the screen",
    usage: "wanda help or wanda -h",
  },
};

And here are the commands for the wandac compiler, which you should put at the top of src/cli/wandac.js:

const COMMANDS = {
  compile: {
    alias: "-c",
    description:
      "Compiles a single Wanda file to JavaScript that imports its dependencies",
    usage: "wandac compile <filepath> or wandac -c <filepath>",
  },
  build: {
    alias: "-b",
    description:
      "Compiles a Wanda file and builds it with bundled JavaScript dependencies",
    usage: "wandac build <filepath> or wandac -b <filepath>",
  },
  version: {
    alias: "-v",
    description:
      "Prints the current version number of your WandaC installation",
    usage: "wandac version or wandac -v",
  },
  help: {
    alias: "-h",
    description: "Prints this help message on the screen",
    usage: "wandac help or wandac -h",
  },
};

Changes to the REPL

Now let's go back to the REPL file.

First, make sure your imports look like this - there have been some changes:

import os from "os";
import vm from "vm";
import fs from "fs";
import { join } from "path";
import readlineSync from "readline-sync";
import { pprintAST, pprintDesugaredAST } from "./pprint.js";
import { println } from "../printer/println.js";
import { makeGlobalNameMap } from "../runtime/makeGlobals.js";
import { emitGlobalEnv } from "../emitter/emitGlobalEnv.js";
import { build } from "./build.js";
import { compile } from "./compile.js";
import { makeGlobalTypeEnv } from "../typechecker/makeGlobalTypeEnv.js";
import { countIndent, inputFinished } from "./utils.js";
import { readline } from "./readline.js";
import { getVersion, printHelp } from "./utils.js";

We're going to change the parameters the repl function takes. Instead of separate parameters for mode and file we're going to make them properties on a single options object and make them both optional.

Next, move these constants out of the repl function and put them at the top of the file:

// Create global compile environment
const compileEnv = makeGlobalNameMap();
const typeEnv = makeGlobalTypeEnv();

We need them to be at the module level now because we're going to use them in multiple functions.

You should now have this at the top of the repl function:

  // Build global module and instantiate in REPL context
  // This should make all compiled global symbols available
  const globalCode = build(emitGlobalEnv());
  vm.runInThisContext(globalCode);

Now we'll take all the code that was used to compile and run Wanda files when loading a file into an interactive session on invoking the wanda load <filename> command and move it to its own function, compileAndRunFromPath:

const compileAndRunFromPath = (path) => {
  const fileContents = fs.readFileSync(path, { encoding: "utf-8" });
  const compiledFile = compile(fileContents, path, compileEnv, typeEnv);
  vm.runInThisContext(compiledFile);
};

Then replace that code in the repl function with a call to the new function and pass it the path option:

  if (path) {
    // load file in REPL interactively
    compileAndRunFromPath(path);
  }

We need 2 additional new functions in src/cli/repl.js: one to get a file path from input in the REPL, and one to save a REPL session as its own file:

const saveAsFile = (session) => {
  const path = readlineSync.question("Enter the path to save the file at: ");
  const filePath = join(process.cwd(), path);

  try {
    fs.writeFileSync(filePath, session, { encoding: "utf-8" });
    console.log("File saved!");
  } catch (e) {
    console.log(
      `Error while saving file, please try again later: ${e.message}`
    );
  }
};

const getPathFromInput = () => {
  const path = readlineSync.question("Enter the path to load the file from: ");
  return join(process.cwd(), path);
};

Now, with the machinery in place, we turn back to the repl function.

Let's add a friendly welcome message with instructions on getting help that shows when you start a REPL session. Below the if (path) statement that calls compileAndRunFromPath, add this:

  console.log(
    `**** Welcome to the Wanda Programming Language v${getVersion()} interactive session ****`
  );
  console.log("Enter :help for more information");

Now we'll need to add a variable for the session contents to our series of variables just before the actual loop:

  let prompt = "> ";
  let input = "";
  let indent = 0;
  let session = "";

Next is the main loop with a nested try/catch. Here it is without the main contents:

  while (true) {
    try {
      // main contents
    } catch (e) {
      console.error(e.stack ? e.stack : e.message);
      input = "";
      indent = 0;
    }
  }

Now for the main contents. The first thing we need to do in the try block is read the input, indenting if it's multiline input:

      input += read(prompt + "  ".repeat(indent));

Now if the user enters a keyword that corresponds to a command we need to handle that. We'll extend our switch (input) statement to handle the new commands. Note that I've changed :print-desugared to just use a -d flag with the :print-ast command.

Here are the cases for the commands, switching on input:

        // If it's a command, execute it
        case ":quit":
          process.exit(0);
        case ":print-ast":
          mode = "printAST";
          input = "";
          break;
        case ":print-ast -d":
          mode = "printDesugared";
          input = "";
          break;
        case ":no-print-ast":
          mode = "repl";
          input = "";
          break;
        case ":save-file":
          saveAsFile(session);
          input = "";
          break;
        case ":load-file":
          compileAndRunFromPath(getPathFromInput());
          input = "";
          break;
        case ":version":
          console.log(getVersion());
          input = "";
          break;
        case ":help":
          printHelp(
            COMMANDS,
            "Wanda Interactive Session",
            "Enter an expression at the prompt for immediate evaluation"
          );
          input = "";
          break;

And the default case, which runs the code, remains the same except that we add completed input to the session variable and make sure incomplete input includes the indentation so the output files will have the same indentation as the REPL shows:

        // If it's code, compile and run it
        default:
          if (inputFinished(input)) {
            let compiled = compile(input, "stdin", compileEnv, typeEnv);
            let result = vm.runInThisContext(compiled);

            if (mode === "printAST") {
              console.log(pprintAST(input));
            } else if (mode === "printDesugared") {
              console.log(pprintDesugaredAST(input));
            }

            println(result);
            session += input + os.EOL + os.EOL;
            input = "";
            indent = 0;
          } else {
            indent = countIndent(input);
            input += os.EOL + "  ".repeat(indent);
          }

Or, if you need to see it all together, here's the complete repl function:

export const repl = ({ mode = "repl", path = "" } = {}) => {
  // Build global module and instantiate in REPL context
  // This should make all compiled global symbols available
  const globalCode = build(emitGlobalEnv());
  vm.runInThisContext(globalCode);

  if (path) {
    // load file in REPL interactively
    compileAndRunFromPath(path);
  }

  console.log(
    `**** Welcome to the Wanda Programming Language v${getVersion()} interactive session ****`
  );
  console.log("Enter :help for more information");

  let prompt = "> ";
  let input = "";
  let indent = 0;
  let session = "";

  while (true) {
    try {
      input += read(prompt + "  ".repeat(indent));

      switch (input) {
        // If it's a command, execute it
        case ":quit":
          process.exit(0);
        case ":print-ast":
          mode = "printAST";
          input = "";
          break;
        case ":print-ast -d":
          mode = "printDesugared";
          input = "";
          break;
        case ":no-print-ast":
          mode = "repl";
          input = "";
          break;
        case ":save-file":
          saveAsFile(session);
          input = "";
          break;
        case ":load-file":
          compileAndRunFromPath(getPathFromInput());
          input = "";
          break;
        case ":version":
          console.log(getVersion());
          input = "";
          break;
        case ":help":
          printHelp(
            COMMANDS,
            "Wanda Interactive Session",
            "Enter an expression at the prompt for immediate evaluation"
          );
          input = "";
          break;
        // If it's code, compile and run it
        default:
          if (inputFinished(input)) {
            let compiled = compile(input, "stdin", compileEnv, typeEnv);
            let result = vm.runInThisContext(compiled);

            if (mode === "printAST") {
              console.log(pprintAST(input));
            } else if (mode === "printDesugared") {
              console.log(pprintDesugaredAST(input));
            }

            println(result);
            session += input + os.EOL + os.EOL;
            input = "";
            indent = 0;
          } else {
            indent = countIndent(input);
            input += os.EOL + "  ".repeat(indent);
          }
      }
    } catch (e) {
      console.error(e.stack ? e.stack : e.message);
      input = "";
      indent = 0;
    }
  }
};

Now you can load and save files from inside the REPL.

Changes in Running The Wanda CLI

Since we're going to be running files from the command line now, we need to make sure we import everything needed for that. Here's the new list of imports in src/cli/run.js:

import vm from "vm";
import fs from "fs";
import { join } from "path";
import { repl } from "./repl.js";
import { makeGlobalNameMap } from "../runtime/makeGlobals.js";
import { emitGlobalEnv } from "../emitter/emitGlobalEnv.js";
import { build } from "./build.js";
import { compile } from "./compile.js";
import { makeGlobalTypeEnv } from "../typechecker/makeGlobalTypeEnv.js";
import { getVersion, printHelp } from "./utils.js";

We also need a new function runFile to compile and run the contents of a file passed to the CLI:

const runFile = (path) => {
  const fileContents = fs.readFileSync(path, { encoding: "utf-8" });
  const globalNs = makeGlobalNameMap();
  const typeEnv = makeGlobalTypeEnv();
  const globalCode = build(emitGlobalEnv());
  const compiledCode = compile(fileContents, path, globalNs, typeEnv);

  vm.runInThisContext(globalCode);
  return vm.runInThisContext(compiledCode);
};

In the run function, we'll have the command and alias cases fall through to a single handler for each command. I've also removed the option to start a REPL session with AST printing on because it makes more sense to me to have that just be an option you set when you're inside the interactive session.

Here's the new version of run. Note that I've also changed it from throwing exceptions on bad commands to just ending the process with an error code:

export const run = () => {
  switch (process.argv[2]) {
    case "-l":
    case "load":
      if (!process.argv[3]) {
        console.log(`load command requires file path as argument`);
        process.exit(1);
      }
      const path = join(process.cwd(), process.argv[3]);
      repl({ path });
      break;
    case "run":
    case "-r":
      if (!process.argv[3]) {
        console.log(`run command requires file path as argument`);
        process.exit(1);
      }
      return runFile(join(process.cwd(), process.argv[3]));
    case "-v":
    case "version":
      return console.log(getVersion());
    case "help":
    case "-h":
      return printHelp(
        COMMANDS,
        "Wanda Programming Language",
        "Just running wanda with no command also starts an interactive session"
      );
    case undefined:
    case "repl":
    case "-i":
      return repl();
    default:
      console.log("Invalid command specified");
      process.exit(1);
  }
};

Now you can run files directly from the command line.

Changes in Running The Compiler

First, we need to add an import for the new getVersion and printHelp functions:

import { getVersion, printHelp } from "./utils.js";

Most of the changes simply involve handling the new commands and aliases for the old commands, but we're also passing true into emitGlobalEnv in the default case because of the issue we fixed above. Here's the wandac function in src/cli/wandac.js:

export const wandac = () => {
  if (!process.argv[2]) {
    console.log(`wandac requires either a file path or command argument`);
    process.exit(1);
  }

  switch (process.argv[2]) {
    case "build":
    case "-b": {
      const pathname = join(process.cwd(), process.argv[3]);
      const compiledFile = compileFile(pathname);
      const globals = emitGlobalEnv();
      const code = globals + os.EOL + os.EOL + compiledFile;
      const bName = basename(pathname).split(".")[0];
      const outfile = bName + ".build" + ".js";
      const built = build(code, outfile, bName);

      fs.writeFileSync(outfile, built, { encoding: "utf-8" });
      break;
    }
    case "version":
    case "-v":
      getVersion();
      break;
    case "help":
    case "-h":
      printHelp(
        COMMANDS,
        "WandaC Compiler",
        "Just using wandac <filename> also compiles a single file"
      );
      break;
    default: {
      // should be a file path
      const pathname = join(process.cwd(), process.argv[2]);
      const compiledFile = compileFile(pathname);
      const globals = emitGlobalEnv(true);
      const code = globals + os.EOL + os.EOL + compiledFile;
      const outfile = basename(pathname).split(".")[0] + ".js";

      fs.writeFileSync(outfile, code, { encoding: "utf-8" });
      break;
    }
  }
};

Trying It Out

First, try out the new version and help commands with wanda and wandac. I think you'll agree they make the apps much more accessible than they used to be.

Now fire up a REPL session with wanda.

Enter this code into the REPL, one line at a time:

(for map ((i (range 5)))
  (+ i i))

Now enter :save-file and give it a filename at the prompt (I used examples/for.wanda). Your file should appear where you saved it!

You can also use the up arrow to cycle back through your history, and history is saved across sessions.

Try using the :load-file command and loading examples/inc.wanda from last time. Now you should be able to use the inc function just like if you'd defined it in your current session. Cool, right?

Conclusion

I hope this has been as fun for you as it's been for me.

As always, you can view the current state of the Wanda code as of the end of this article at the relevant tag in the GitHub repo.

Next time we're going to do something different. Instead of adding a new syntactic or type feature to Wanda, we're going to add an AST transformation and optimization.

We're going to implement an intermediate representation for the Wanda code between desugaring and code emitting called A Normal Form. I promise it's not as scary as it sounds.

We'll also add tail call optimization in the form of trampolining.

That will allow us to use essentially infinite recursion as long as the recursive call is in tail position. If you're not familiar with what that means, I'll explain all in the next article.

Stay tuned!

Create Your Own Programming Language 8: Conditionals

Jason Barr — Sat, 01 Jul 2023 22:58:06 +0000

It's time for the next installment in the Create Your Own Programming Language series. This time we're going to add conditionals to the Wanda language.

We're actually going to add 3 different kinds of conditionals: if, cond, and when.

if is a standard if expression, with 2 branches based on whether a test comes back truthy or falsy.

cond lets you chain together multiple conditions and expressions, with an :else branch at the end in case nothing matches. The :else branch is always required. cond expressions desugar to a chain of if expressions.

A cond expression looks like this. Note that each branch is enclosed within its own set of parentheses:

(cond
  ((= x 0) "zero")
  ((< x 10) "less than 10")
  ((< x 0) "less than zero")
  (:else "10 or greater"))

when is imperative and lets you execute a block of code if the test is truthy. It evaluates to nil either way, so you wouldn't want to use it to assign a toplevel variable. It's more of an imperative construct, ideally for containing side effects.

When also allows you to nest any number of expressions inside it, whereas each branch of an if expression can only take a single expression. And since when doesn't have an else branch, you can use it in cases where you don't need an alternate execution path if the main one isn't taken.

Here's an example of a when expression:

(when (= 1 1)
  (println "Wow, one equals one?")
  (println "Who woulda thunk it???"))

We're also going to add a new form, the LogicalExpression. This is for expressions with the and and or forms.

Some Lisps will let you use any number of arguments to the and and or forms, but we're going to restrict users to 2 arguments for each form to simplify things.

Finally, we'll add 2 new node types to help the type checker that will both desugar to call expressions: BinaryExpression which will include the equal? and not-equal? operations, and UnaryExpression which is only used with not.

These typechecker-only nodes will help with a process known as type narrowing, which allows the type checker to better understand things like which arm of a union to check against in branches of a conditional.

As always, if you haven't read the previous article in which we added several new types to the type checker, you should do that before reading ahead.

Ok? Let's go!

Changes to The Lexer and Reader

None! Both are fine just the way they are for today.

Changes to The Core Library

We need to add a not-equal? function to the core library to match up with the not-equal? binary expression.

Ordinarily in a Lisp when you want to check if a value is not equal you'd just prefix equal? with not, as in (not (equal? x y)), but to give the type checker some more help with narrowing we're adding this additional form.

In lib/js/core.js go down to the equal? function and add this between it and is?:

    "not-equal?": rt.makeFunction(
      (a, b) => {
        if (rt.hasDict(a) && rt.hasDict(b)) {
          return !equal(rt.getMetaField(a, "dict"), rt.getMetaField(b, "dict"));
        }
        return !equal(a, b);
      },
      { contract: "(any, any) -> boolean", name: "not-equal?" }
    ),

That's it for changes to the core library, so now we can move on to changes to the parser.

Changes to The Parser

We need to define new AST node types and constructors for those nodes.

In src/parser/ast.js add the following to the ASTTypes enum:

export const ASTTypes = {
  // other members...
  IfExpression: "IfExpression",
  CondExpression: "CondExpression",
  WhenExpression: "WhenExpression",
  BinaryExpression: "BinaryExpression",
  LogicalExpression: "LogicalExpression",
  UnaryExpression: "UnaryExpression",
};

And in the AST object add the following node constructors:

export const AST = {
  // other constructors...
  IfExpression(test, then, elseBranch, srcloc) {
    return {
      kind: ASTTypes.IfExpression,
      test,
      then,
      else: elseBranch,
      srcloc,
    };
  },

  CondExpression(clauses, elseBranch, srcloc) {
    return {
      kind: ASTTypes.CondExpression,
      clauses,
      else: elseBranch,
      srcloc,
    };
  },

  WhenExpression(test, body, srcloc) {
    return {
      kind: ASTTypes.WhenExpression,
      test,
      body,
      srcloc,
    };
  },

  BinaryExpression(left, op, right, srcloc) {
    return {
      kind: ASTTypes.BinaryExpression,
      left,
      op,
      right,
      srcloc,
    };
  },

  LogicalExpression(left, op, right, srcloc) {
    return {
      kind: ASTTypes.LogicalExpression,
      left,
      op,
      right,
      srcloc,
    };
  },

  UnaryExpression(op, operand, srcloc) {
    return {
      kind: ASTTypes.UnaryExpression,
      op,
      operand,
      srcloc,
    };
  },
}

We'll need 6 new parse functions in src/parser/parse.js:

parseIfExpression
parseCondExpression
parseWhenExpression
parseBinaryExpression
parseLogicalExpression
parseUnaryExpression

Add the following cases to the switch statement in parseList:

    // other cases
    case "if":
      return parseIfExpression(form);
    case "cond":
      return parseCondExpression(form);
    case "when":
      return parseWhenExpression(form);
    case "equal?":
    case "not-equal?":
      return parseBinaryExpression(form);
    case "and":
    case "or":
      return parseLogicalExpression(form);
    case "not":
      return parseUnaryExpression(form);
    // default case

parseIfExpression is pretty simple. Just grab the test, then, and elseBranch forms off the main form, parse each form, and pass them to the constructor.

const parseIfExpression = (form) => {
  const [_, test, then, elseBranch] = form;
  const srcloc = form.srcloc;
  const parsedTest = parseExpr(test);
  const parsedThen = parseExpr(then);
  const parsedElse = parseExpr(elseBranch);

  return AST.IfExpression(parsedTest, parsedThen, parsedElse, srcloc);
};

parseCondExpression is a little more complicated. We have to grab all the clauses off the main form, parse each one individually, and then when we get to the else branch handle that one separately since it gets its own property on the AST node object. Also, since the else branch is required, we throw an error if there isn't one.

const parseCondExpression = (form) => {
  const [_, ...clauses] = form;
  const srcloc = form.srcloc;
  /** @type {import("./ast.js").CondClause[]} */
  const parsedClauses = [];
  let hasElse = false;

  if (!(clauses.length >= 2)) {
    throw new Exception(
      `Cond expression must have at least 2 clauses; ${clauses.length} given at ${srcloc.file}, (${srcloc.line}:${srcloc.col})`
    );
  }

  for (let clause of clauses) {
    const [test, expr] = clause;

    if (test.type === TokenTypes.Keyword && test.value === ":else") {
      hasElse = true;
      break;
    }

    const parsedTest = parseExpr(test);
    const parsedExpr = parseExpr(expr);

    parsedClauses.push({ test: parsedTest, expression: parsedExpr });
  }

  if (!hasElse) {
    throw new Exception(
      `Cond expression expects an :else clause; none given at ${srcloc.file}, (${srcloc.line}:${srcloc.col})`
    );
  }

  /** @type {Cons} */
  const elseClause = clauses[clauses.length - 1];
  // we need the first element of the tail of the else clause
  const parsedElse = parseExpr(elseClause.cdr.car);

  return AST.CondExpression(parsedClauses, parsedElse, srcloc);
};

The other 4 parse functions, parseWhen, parseBinaryExpression, parseLogicalExpression, and parseUnaryExpression, are all simple:

const parseWhenExpression = (form) => {
  const [_, test, ...body] = form;
  const srcloc = form.srcloc;
  const parsedTest = parseExpr(test);
  const parsedBody = body.map(parseExpr);

  return AST.WhenExpression(parsedTest, parsedBody, srcloc);
};

const parseBinaryExpression = (form) => {
  const [op, left, right] = form;
  const srcloc = form.srcloc;
  const parsedLeft = parseExpr(left);
  const parsedRight = parseExpr(right);

  return AST.BinaryExpression(parsedLeft, op.value, parsedRight, srcloc);
};

const parseLogicalExpression = (form) => {
  const [op, left, right] = form;
  const srcloc = form.srcloc;
  const parsedLeft = parseExpr(left);
  const parsedRight = parseExpr(right);

  return AST.LogicalExpression(parsedLeft, op.value, parsedRight, srcloc);
};

const parseUnaryExpression = (form) => {
  const [op, operand] = form;
  const srcloc = form.srcloc;
  const parsedOperand = parseExpr(operand);

  return AST.UnaryExpression(op.value, parsedOperand, srcloc);
};

Changes to The Type Checker

Obviously we need to infer and check types for the new nodes, but the most significant change we'll be implementing in the type checker today is the addition of type narrowing.

Type narrowing allows the type checker to check certain expressions as if a certain condition were true or false, which allows it to make deductions about types it otherwise would not be able to make.

For example, let's say we have a discriminated union type:

(type Coord
  { type: "cartesian", x: number, y: number }
  || { type: "polar", radius: number, magnitude: number })

(def (point: Coord) { type: "cartesian", x: 5, y: 2.5 })

Given this when expression:

(when (equal? point.type "cartesian")
  ; assume point is the bottom-right point of
  ; a square that begins at the origin (0, 0)
  (def area (* point.x point.y))
  ; continue

Without type narrowing, the above when expression would have failed to type check because due to the union type we couldn't be certain the object would have x and y properties. Thanks to narrowing, since we know type is "cartesian" we know which arm of the union the object satisfies.

As Jake Donham says in his article about type narrowing, it can be helpful to think of the type environment as a logical statement that asserts facts about the types of values. Narrowing allows us to supplement the known facts with assumptions guaranteed to be correct if the right conditions are met.

Adding The Negation Type

To implement narrowing we're going to add a type that is only used in the type checker: the negation type, or not. The not type is only used in the type checker, it can't be annotated, and in fact it is only used in type narrowing and it's limited just to that functionality so we don't have to rewrite the entire checker just to account for it.

Let's add it in src/typechecker/types.js to the TypeTypes enum:

export const TypeTypes = {
  // other members...
  Not: "Not",
};

We'll also need a constructor in src/typechecker/constructors.js:

export const not = (base) => ({ kind: TypeTypes.Not, base });

and a validator in src/typechecker/validators.js:

export const isNot = (type) => {
  return type.kind === TypeTypes.Not;
};

We'll also need some utility functions and types for checking truthiness and falsiness of a value, when it can be known at compile time. We'll create those in src/typechecker/utils.js:

export const isTruthy = (type) => {
  switch (type.kind) {
    case TypeTypes.Nil:
      return false;
    case TypeTypes.Singleton:
      if (type.value === "false") {
        return false;
      }
    default:
      if (type.kind !== TypeTypes.Boolean) {
        return true;
      }
  }
};

export const isFalsy = (type) => {
  switch (type.kind) {
    case TypeTypes.Nil:
      return true;
    case TypeTypes.Singleton:
      if (type.value === "false") {
        return true;
      }
    default:
      if (type.kind !== TypeTypes.Boolean) {
        return false;
      }
  }
};

export const falsy = union(singleton("Boolean", "false"), nil);

export const truthy = intersection(
  not(singleton("Boolean", "false")),
  not(nil)
);

Remember that in Wanda all values except false and nil are truthy.

Let's add our utils to the Type module in src/typechecker/Type.js:

import { TypeTypes } from "./types.js";
import * as Validators from "./validators.js";
import * as Constructors from "./constructors.js";
import { typeToString } from "./typeToString.js";
import { union } from "./union.js";
import { map } from "./map.js";
import { intersection } from "./intersection.js";
import { isTruthy, isFalsy, truthy, falsy } from "./utils.js";

export const Type = {
  Type: TypeTypes,
  ...Constructors,
  ...Validators,
  toString: typeToString,
  union,
  map,
  intersection,
  isTruthy,
  isFalsy,
  truthy,
  falsy,
};

Just a quick note: I had a weird circular reference bug that popped up sometimes in Node when trying to use the type checker. The solution turned out to be removing the import of the Type module into src/typechecker/isSubtype.js and instead importing each of the type validators used in isSubtype one at a time and removing the Type. prefix for validators within isSubtype. If you have errors like "Cannot use Type before its initialization" after adding the rest of the code in this section that's how you fix them.

Narrowing

Now let's actually implement the narrowing functions. It's actually a lot of functions! But the thing most of them have in common is they restrict the applications of a type within an environment and return that modified environment to be used by the caller.

For a fuller explanation of what I'm about to show you, see Donham's post about type narrowing.

First is the narrow function.

When we type check a branch of a conditional, we call narrow with the test expression, the current environment, and a flag according to whether we're assuming the test to be true or false. It returns a new environment incorporating the information of the test and assumption.

When we deduce that a subexpression must satisfy a type, we call narrowUnary, narrowLogical, narrowBinary, or narrowPath to incorporate the appropriate expression type into the environment. When we reach a variable we call narrowType, which performs an operation similar to subtype checking between the current type from the environment and the deduced type in order to update the variable's type with the result.

Here are the narrowing functions, including the function widenNots that keeps not types from being returned in the environments produced by narrowing, in src/typechecker/narrow.js:

import { AST, ASTTypes } from "../parser/ast.js";
import { Exception, TypeException } from "../shared/exceptions.js";
import { Type } from "./Type.js";
import { TypeEnvironment } from "./TypeEnvironment.js";
import { infer } from "./infer.js";
import { isSubtype } from "./isSubtype.js";
import { propType } from "./propType.js";
import { TypeTypes } from "./types.js";

export const narrow = (ast, env, assume) => {
  switch (ast.kind) {
    case ASTTypes.UnaryExpression:
      return narrowUnary(ast, env, assume);

    case ASTTypes.LogicalExpression:
      return narrowLogical(ast, env, assume);

    case ASTTypes.BinaryExpression:
      return narrowBinary(ast, env, assume);

    default:
      return narrowPath(ast, env, assume ? Type.truthy : Type.falsy);
  }
};

const narrowUnary = (ast, env, assume) => {
  switch (ast.op) {
    case "not":
      return narrow(ast.operand, env, !assume);

    default:
      throw new Exception(`Unknown unary operator ${ast.op}`);
  }
};

const narrowLogical = (ast, env, assume) => {
  switch (ast.op) {
    case "and":
      if (assume) {
        env = narrow(ast.left, env, true);
        return narrow(ast.right, env, true);
      } else {
        if (Type.isTruthy(infer(ast.left, env))) {
          return narrow(ast.right, env, false);
        } else if (Type.isTruthy(infer(ast.right, env))) {
          return narrow(ast.left, env, false);
        } else {
          return env;
        }
      }

    case "or":
      if (!assume) {
        env = narrow(ast.left, env, false);
        return narrow(ast.right, env, false);
      } else {
        if (Type.isFalsy(infer(ast.left, env))) {
          return narrow(ast.right, env, true);
        } else if (Type.isFalsy(infer(ast.right, env))) {
          return narrow(ast.left, env, true);
        } else {
          return env;
        }
      }

    default:
      throw new Exception(`Unknown logical operator ${ast.op}`);
  }
};

const narrowBinary = (ast, env, assume) => {
  const left = infer(ast.left, env);
  const right = infer(ast.right, env);

  if ((ast.op === "equal?" && assume) || (ast.op === "not-equal?" && !assume)) {
    env = narrowPath(ast.left, env, right);
    return narrowPath(ast.right, env, left);
  } else if (
    (ast.op === "not-equal?" && assume) ||
    (ast.op === "equal?" && !assume)
  ) {
    if (Type.isSingleton(right)) {
      env = narrowPath(ast.left, env, Type.not(right));
    }

    if (Type.isSingleton(left)) {
      env = narrowPath(ast.right, env, Type.not(left));
    }

    return env;
  } else {
    return env;
  }
};

const narrowPath = (ast, env, type) => {
  switch (ast.kind) {
    case ASTTypes.Symbol:
      return narrowPathSymbol(ast, env, type);

    case ASTTypes.MemberExpression:
      return narrowPathMember(ast, env, type);

    default:
      return env;
  }
};

const narrowPathSymbol = (ast, env, type) => {
  const idType = env.get(ast.name);

  if (!idType) {
    throw new TypeException(
      `Expected bound identifier, got undefined`,
      ast.srcloc
    );
  }

  env.set(ast.name, narrowType(idType, type));
  return env;
};

const narrowPathMember = (ast, env, type) => {
  return narrowPath(
    ast.object,
    env,
    Type.object([{ name: ast.property.name, type: type }])
  );
};

export const narrowType = (x, y) => {
  if (Type.isAny(x) || Type.isAny(y)) return Type.any;
  if (Type.isUndefined(x) && Type.isUndefined(y)) return Type.undefinedType;
  if (Type.isUndefined(x)) return widenNots(y);
  if (Type.isUndefined(y)) return x;
  if (Type.isNever(x) || Type.isNever(y)) return Type.never;
  if (Type.isUnknown(x)) return widenNots(y);
  if (Type.isUnknown(y)) return x;

  if (Type.isTuple(x)) {
    return Type.tuple(x.types.map((a) => narrowType(a, y)));
  }

  if (Type.isTuple(y)) {
    return Type.tuple(y.types.map((b) => narrowType(x, b)));
  }

  if (Type.isUnion(x)) {
    return Type.union(...x.types.map((a) => narrowType(a, y)));
  }

  if (Type.isUnion(y)) {
    return Type.union(...y.types.map((b) => narrowType(x, b)));
  }

  if (Type.isIntersection(x)) {
    return Type.intersection(...x.types.map((a) => narrowType(a, y)));
  }

  if (Type.isIntersection(y)) {
    return Type.intersection(...y.types.map((b) => narrowType(x, b)));
  }

  if (Type.isNot(y)) {
    if (isSubtype(x, y.base)) {
      return Type.never;
    } else if (
      Type.isBoolean(x) &&
      Type.isSingleton(y.base) &&
      Type.isBoolean(y.base.base)
    ) {
      return Type.singleton(
        "Boolean",
        y.base.value === "true" ? "false" : "true"
      );
    } else {
      return x;
    }
  }

  if (Type.isSingleton(x) && Type.isSingleton(y)) {
    return x.value === y.value ? x : Type.never;
  }

  if (Type.isSingleton(x)) {
    return x.base === y.kind ? x : Type.never;
  }

  if (Type.isSingleton(y)) {
    return y.base === x.kind ? y : Type.never;
  }

  if (Type.isObject(x) && Type.isObject(y)) {
    const properties = x.properties.map(({ name, type: xType }) => {
      const yType = propType(y, name);
      const type = yType ? narrowType(xType, yType) : xType;
      return { name, type };
    });

    if (properties.some(({ type }) => Type.isNever(type))) {
      return Type.never;
    } else {
      return Type.object(properties);
    }
  }

  return Type.intersection(x, y);
};

const widenNots = (type) => {
  switch (type.kind) {
    case TypeTypes.Not:
      return Type.unknown;

    case TypeTypes.Union:
      return Type.union(...type.types.map(widenNots));

    case TypeTypes.Object:
      return Type.object(
        type.properties.map(({ name, type }) => ({
          name,
          type: widenNots(type),
        }))
      );

    default:
      return type;
  }
};

That's actually a lot of code! Please check out Jake's article if you don't understand anything—he explains the details quite well.

Inference

We need to make some changes to the map function in src/typechecker/map.js before we move on to the inference functions. Namely, we need to make it so map can accept either 2 or 3 arguments based on whether we're mapping over 1 type or 2. That way we can use map with binary expressions.

Rename map to map1, then add this new function in as map and export it:

export const map = (...args) => {
  switch (args.length) {
    case 2:
      return map1(args[0], args[1]);
    case 3:
      return map2(args[0], args[1], args[2]);
    default:
      throw new Exception(`Unexpected args length ${args.length}`);
  }
};

Here's the map2 function:

export const map2 = (t1, t2, fn) => {
  if (isUnion(t1) || isUnion(t2)) {
    const t1s = isUnion(t1) ? t1.types : [t1];
    const t2s = isUnion(t2) ? t2.types : [t2];
    const ts = [];
    for (const t1 of t1s) {
      for (const t2 of t2s) {
        ts.push(map(t1, t2, fn));
      }
    }
    return union(...ts);
  } else if (isIntersection(t1) || isIntersection(t2)) {
    const t1s = isIntersection(t1) ? t1.types : [t1];
    const t2s = isIntersection(t2) ? t2.types : [t2];
    const ts = [];
    let error = undefined;
    for (const t1 of t1s) {
      for (const t2 of t2s) {
        try {
          ts.push(fn(t1, t2));
        } catch (e) {
          if (!error) error = e;
        }
      }
    }
    if (ts.length === 0) {
      throw error;
    } else {
      return intersection(...ts);
    }
  } else {
    return fn(t1, t2);
  }
};

Now we need 6 new cases in the switch statement in infer in src/typechecker/infer.js:

    // other cases...
    case ASTTypes.IfExpression:
      return inferIfExpression(ast, env, constant);
    case ASTTypes.WhenExpression:
      return inferWhenExpression(ast, env, constant);
    case ASTTypes.CondExpression:
      return inferCondExpression(ast, env, constant);
    case ASTTypes.UnaryExpression:
      return inferUnaryExpression(ast, env, constant);
    case ASTTypes.BinaryExpression:
      return inferBinaryExpression(ast, env, constant);
    case ASTTypes.LogicalExpression:
      return inferLogicalExpression(ast, env, constant);
    // default case

For inferring if expressions, we first infer the test. Then we use the test to narrow for each of the consequent and alternate branches, wrapping each in a thunk so inference only happens if it's needed.

If the test is known to be truthy or falsy, we infer only the appropriate branch. Otherwise, we unify the two branches.

TypeScript constructs a union type of the two branches if they produce different types, but I want to enforce some degree of similarity between the branches so we're unifying the two types instead of going with a simple union. That means the type of one branch must be the same as or a subtype of the other.

Here's inferIfExpression:

const inferIfExpression = (node, env, constant) => {
  const test = infer(node.test, env, constant);
  const consequent = () => infer(node.then, narrow(node.test, env, true));
  const alternate = () => infer(node.else, narrow(node.test, env, false));

  if (Type.isTruthy(test)) {
    return consequent();
  } else if (Type.isFalsy(test)) {
    return alternate();
  } else {
    return unify(consequent(), alternate());
  }
};

inferWhenExpression is a little bit strange. It returns the nil type no matter what. But we still infer a type for the test then loop over the body block and infer a type for each expression to make sure there are no inference errors:

const inferWhenExpression = (node, env, constant) => {
  const test = infer(node.test, env, constant);

  // if the test is falsy, this will never run
  if (!Type.isFalsy(test)) {
    // check expressions in body
    for (let expr of node.body) {
      infer(expr, narrow(node.test, env, true), constant);
    }
  }

  return Type.nil;
};

inferCondExpression loops over all the clauses. If one of the clause tests produces a known truthy type, we return the type of its expression. Otherwise we collect the types of all clauses, including the :else clause, and unify them all. That means the common subtype is produced as the type for the expression so, like with an if expression, the types for each branch need to be subtypes of each other.

const inferCondExpression = (node, env, constant) => {
  let types = [];

  for (let clause of node.clauses) {
    const test = infer(clause.test, env, constant);
    const type = infer(
      clause.expression,
      narrow(clause.test, env, true),
      constant
    );

    if (Type.isTruthy(test)) {
      // the first truthy condition type will trigger the clause's expression
      return type;
    }

    types.push(type);
  }

  const elseType = infer(node.else, env, constant);

  return unifyAll(...types, elseType);
};

Since not is the only unary expression, that's the only case inferUnaryExpression handles. If the type of its operand is truthy or falsy it returns a singleton type of the opposite boolean. Otherwise, if it can't be determined at compile time, it returns the type boolean.

const inferUnaryExpression = (node, env, constant) => {
  const operand = infer(node.operand, env, constant);

  return Type.map(operand, (operand) => {
    switch (node.op) {
      case "not":
        if (Type.isTruthy(operand)) {
          return Type.singleton("Boolean", "false");
        } else if (Type.isFalsy(operand)) {
          return Type.singleton("Boolean", true);
        } else {
          return Type.boolean;
        }

      default:
        throw new Exception(`Unknown unary operator ${node.op}`);
    }
  });
};

inferBinaryExpression is similar, but it has 2 cases to handle and each case has 2 operands:

const inferBinaryExpression = (node, env, constant) => {
  const left = infer(node.left, env, constant);
  const right = infer(node.right, env, constant);

  return Type.map(left, right, (left, right) => {
    switch (node.op) {
      case "equal?":
        if (Type.isSingleton(left) && Type.isSingleton(right)) {
          return Type.singleton(left.value === right.value ? "true" : "false");
        } else {
          return Type.boolean;
        }

      case "not-equal?":
        if (Type.isSingleton(left) && Type.isSingleton(right)) {
          return Type.singleton(left.value !== right.value ? "true" : "false");
        } else {
          return Type.boolean;
        }

      default:
        throw new Exception(`Unknown binary operator ${node.op}`);
    }
  });
};

Finally, with inferLogicalExpression we can make assumptions to narrow the types based on the truthiness or falsiness of the left operand. For and, we unify the right with the result of narrowing the left type with the falsy type union, and for or we do the opposite:

const inferLogicalExpression = (node, env, constant) => {
  const left = infer(node.left, env, constant);
  const right = () => infer(node.right, env, constant);

  switch (node.op) {
    case "and":
      if (Type.isFalsy(left)) {
        return left;
      } else if (Type.isTruthy(left)) {
        return right();
      } else {
        return unify(narrowType(left, Type.falsy), right());
      }

    case "or":
      if (Type.isTruthy(left)) {
        return left;
      } else if (Type.isFalsy(left)) {
        return right();
      } else {
        return unify(narrowType(left, Type.truthy), right());
      }

    default:
      throw new Exception(`Unknown logical operator ${node.op}`);
  }
};

Checking Against If and Cond Expressions

We need to add cases to the check function in src/typechecker/check.js for if and cond expressions:

  // other cases
  if (ast.kind === ASTTypes.IfExpression) {
    return checkIfExpression(ast, type, env);
  }

  if (ast.kind === ASTTypes.CondExpression) {
    return checkCondExpression(ast, type, env);
  }
  // const inferredType etc...

The checking functions work similiarly to their respective inference functions:

const checkIfExpression = (ast, type, env) => {
  const test = infer(ast.test, env);
  const consequent = () => check(ast.then, type, narrow(ast.test, env, true));
  const alternate = () => check(ast.else, type, narrow(ast.test, env, false));

  if (Type.isTruthy(test)) {
    consequent();
  } else if (Type.isFalsy(test)) {
    alternate();
  } else {
    consequent();
    alternate();
  }
};

const checkCondExpression = (ast, type, env) => {
  for (let clause of ast.clauses) {
    const test = infer(clause.test, env);
    check(clause.expression, type, narrow(ast.test, env, true));

    if (Type.isTruthy(test)) {
      // we can stop here if test is truthy
      return;
    }
  }

  check(ast.else, type, env);
};

Checking with The Type Checker

First we need to add the relevant cases to the switch statement in checkNode in src/typechecker/TypeChecker.js:

      // other cases
      case ASTTypes.UnaryExpression:
        return this.checkUnaryExpression(node, env);
      case ASTTypes.BinaryExpression:
        return this.checkBinaryExpression(node, env);
      case ASTTypes.LogicalExpression:
        return this.checkLogicalExpression(node, env);
      case ASTTypes.IfExpression:
        return this.checkIfExpression(node, env);
      case ASTTypes.CondExpression:
        return this.checkCondExpression(node, env);
      case ASTTypes.WhenExpression:
        return this.checkWhenExpression(node, env);
      // default case

checkUnaryExpression checks the operand then returns a new node annotated with the inferred node type:

  checkUnaryExpression(node, env) {
    const operand = this.checkNode(node.operand, env);
    return { ...node, operand, type: infer(node, env) };
  }

checkBinaryExpression does the same thing except with left and right operands:

  checkBinaryExpression(node, env) {
    const left = this.checkNode(node.left, env);
    const right = this.checkNode(node.right, env);
    return { ...node, left, right, type: infer(node, env) };
  }

checkLogicalExpression is the same as checkBinaryExpression:

  checkLogicalExpression(node, env) {
    const left = this.checkNode(node.left, env);
    const right = this.checkNode(node.right, env);
    return { ...node, left, right, type: infer(node, env) };
  }

`checkIfExpression is similar, but with all 3 expressions:

`js
checkIfExpression(node, env) {
const test = this.checkNode(node.test, env);
const then = this.checkNode(node.then, env);
const elseBranch = this.checkNode(node.else, env);

return { ...node, test, then, else: elseBranch, type: infer(node, env) };

}
`

The checkCondExpression method loops over the node's clauses and checks each test and expression, then checks the else branch and returns the fully checked node:

`js
checkCondExpression(node, env) {
/** @type {TypedAST[]} */
let clauses = [];

for (let clause of node.clauses) {
  const test = this.checkNode(clause.test, env);
  const expression = this.checkNode(clause.expression, env);

  clauses.push({ test, expression });
}

const elseBranch = this.checkNode(node.else, env);

return { ...node, clauses, else: elseBranch, type: infer(node, env) };

}
`

Finally, checkWhenExpression checks the test, then checks each node of the body, then returns the annotated node:

`js
checkWhenExpression(node, env) {
if (!node.env) {
node.env = env.extend("WhenExpression");
}

const whenEnv = node.env;
const test = this.checkNode(node.test, env);
let body = [];

for (let expr of node.body) {
  body.push(this.checkNode(expr, whenEnv));
}

return { ...node, test, body, type: infer(node, whenEnv) };

}
`

And that's it for changes to the type checker!

Changes to The Visitor

We need to add new cases and methods for each of the new AST nodes in src/visitor/Visitor.js. First, let's add the cases to the switch statement in visit:

js // other cases... case ASTTypes.IfExpression: return this.visitIfExpression(node); case ASTTypes.CondExpression: return this.visitCondExpression(node); case ASTTypes.WhenExpression: return this.visitWhenExpression(node); case ASTTypes.UnaryExpression: return this.visitUnaryExpression(node); case ASTTypes.BinaryExpression: return this.visitBinaryExpression(node); case ASTTypes.LogicalExpression: return this.visitLogicalExpression(node); // default case

Here's visitIfExpression:

`js
visitIfExpression(node) {
const test = this.visit(node.test);
const then = this.visit(node.then);
const elseBranch = this.visit(node.else);

return { ...node, test, then, else: elseBranch };

}
`

and visitCondExpression:

`js
visitCondExpression(node) {
let clauses = [];

for (let clause of node.clauses) {
  const test = this.visit(clause.test);
  const expression = this.visit(clause.expression);

  clauses.push({ test, expression });
}

const elseBranch = this.visit(node.else);

return { ...node, clauses, else: elseBranch };

}
`

and visitWhenExpression:

`js
visitWhenExpression(node) {
/** @type {AST[]} */
let body = [];
const test = this.visit(node.test);

for (let expr of node.body) {
  body.push(this.visit(expr));
}

return { ...node, test, body };

}
`

and visitUnaryExpression:

js visitUnaryExpression(node) { const operand = this.visit(node.operand); return { ...node, operand }; }

and visitBinaryExpression:

`js
visitBinaryExpression(node) {
const left = this.visit(node.left);
const right = this.visit(node.right);

return { ...node, left, right };

}
`

and, finally, visitLogicalExpression:

`js
visitLogicalExpression(node) {
const left = this.visit(node.left);
const right = this.visit(node.right);

return { ...node, left, right };

}
`

With the new Visitor methods in place, let's look at the desugarer.

Changes to The Desugarer

We need to desugar binary expressions, unary expressions, and cond expressions, so we need methods for each of them in src/desugarer/Desugarer.js. The first two are simple.

Here's visitBinaryExpression:

`js
visitBinaryExpression(node) {
const left = this.visit(node.left);
const right = this.visit(node.right);
const func = AST.Symbol(Token.new(TokenTypes.Symbol, node.op, node.srcloc));

return AST.CallExpression(func, [left, right], node.srcloc);

}
`

and visitUnaryExpression:

`js
visitUnaryExpression(node) {
const operand = this.visit(node.operand);

return AST.CallExpression(
  AST.Symbol(Token.new(TokenTypes.Symbol, node.op, node.srcloc)),
  [operand],
  node.srcloc
);

}
`

visitCondExpression is a bit more complicated. We're going to create a recursive helper function called condVisitHelper that takes in an array of clauses and the accumulated nested IfExpression node.

If the clauses array has more than one element, we return the accumulated IfExpression (or, if it's the first time calling the function, we create it) with a recursive call to condVisitHelper in the else branch position.

Otherwise, if clauses has one element, we return the base case which is an IfExpression node with the else branch from the original CondExpression node in the else branch position.

Here's visitCondExpression:

`js
visitCondExpression(node) {
const srcloc = node.srcloc;

/**
 * Recursive helper to construct IfExpression from CondExpression
 * @param {import("../parser/ast.js").CondClause[]} clauses
 * @param {import("../parser/ast.js").IfExpression} accum
 */
const condVisitHelper = (clauses, accum) => {
  if (clauses.length > 1) {
    const clause = clauses[0];
    accum = AST.IfExpression(
      clause.test,
      clause.expression,
      condVisitHelper(clauses.slice(1), accum),
      srcloc
    );

    return accum;
  } else if (clauses.length === 1) {
    const clause = clauses[0];
    const elseBranch = node.else;

    return AST.IfExpression(
      clause.test,
      clause.expression,
      elseBranch,
      srcloc
    );
  }
};

return condVisitHelper(node.clauses, {});

}
`

That's it for changes to the desugarer. Now let's look at changes to the emitter.

Changes to The Emitter

Since we're desugaring away 3 of the new AST node types, we only need emitter methods for IfExpression, WhenExpression, and LogicalExpression nodes.

First let's add the cases to the switch statement in emit in src/emitter/Emitter.js:

js // other cases... case ASTTypes.LogicalExpression: return this.emitLogicalExpression(node, ns); case ASTTypes.IfExpression: return this.emitIfExpression(node, ns); case ASTTypes.WhenExpression: return this.emitWhenExpression(node, ns); // default case

emitLogicalExpression switches on the operator and calls a helper method depending on which operator the node has:

`js
emitLogicalExpression(node, ns) {
switch (node.op) {
case "and":
return this.emitLogicalAnd(node, ns);

  case "or":
    return this.emitLogicalOr(node, ns);

  default:
    throw new Exception(`Unknown logical operator ${node.op}`);
}

}

emitLogicalAnd(node, ns) {
const left = this.emit(node.left, ns);
const right = this.emit(node.right, ns);

return `rt.isTruthy(${left}) ? ${right} : ${left}`;

}

emitLogicalOr(node, ns) {
const left = this.emit(node.left, ns);
const right = this.emit(node.right, ns);

return `rt.isFalsy(${left}) ? ${left} : ${right}`;

}
`

We have to compile to a ternary expression and use rt.isTruthy and rt.isFalsy since the runtime semantics of Wanda are slightly different from those of JavaScript with regards to the truthiness and falsiness of values.

emitIfExpression emits a ternary expression:

js emitIfExpression(node, ns) { returnrt.isTruthy(${this.emit(node.test, ns)}) ? ${this.emit(
node.then,
ns
)} : ${this.emit(node.else, ns)}; }

Since a when expression is imperative, it returns nil instead of a type based on the expression itself. Because of that, we need to emit a little bit of ternary expression magic to cover the case where the test is falsy so the body code is not executed. We wrap the consequent branch of the ternary statement in an IIFE so we can execute its contents and then return nil from the expression body. Here's emitWhenExpression:

js emitWhenExpression(node, ns) { const whenNs = ns.extend("WhenExpression"); let code =(rt.isTruthy(${this.emit(node.test, ns)}))
? (() => {\n`;

for (let expr of node.body) {
  code += `    ${this.emit(expr, whenNs)};\n`;
}

code += "    return null";
code += "  })()\n";
code += "  : null\n";
return code;

}
`

That's all the changes to the emitter, so let's look at printing the new AST nodes.

Changes to The Printer

We don't have any new types of values to print, but we do have 6 new node types.

First, let's add them to be dispatched from the switch statement in print in src/printer/printAST.js:

js // other cases... case ASTTypes.IfExpression: return this.printIfExpression(node, indent); case ASTTypes.CondExpression: return this.printCondExpression(node, indent); case ASTTypes.WhenExpression: return this.printWhenExpression(node, indent); case ASTTypes.BinaryExpression: return this.printBinaryExpression(node, indent); case ASTTypes.LogicalExpression: return this.printLogicalExpression(node, indent); case ASTTypes.UnaryExpression: return this.printUnaryExpression(node, indent); // default case

The methods themselves are pretty simple, so I won't belabor the point. Here they are:

js printIfExpression(node, indent) { let prStr =${prIndent(indent)}IfExpression:\n; prStr +=${prIndent(indent + 2)}Test:\n; prStr += this.print(node.test, indent + 4) + "\n"; prStr +=${prIndent(indent + 2)}Then:\n+ "\n"; prStr += this.print(node.then, indent + 4) + "\n"; prStr +=${prIndent(indent + 2)}Else:\n`;
prStr += this.print(node.else, indent + 4) + "\n";

return prStr;

}

printCondExpression(node, indent) {
let prStr = ${prIndent(indent)}CondExpression:\n;

for (let clause of node.clauses) {
  prStr += `${prIndent(indent + 2)}Test:\n`;
  prStr += this.print(clause.test, indent + 4) + "\n";
  prStr += `${prIndent(indent + 2)}Expression:\n`;
  prStr += this.print(clause.expression, indent + 4) + "\n";
}

prStr += `${prIndent(indent + 2)}Else:\n`;
prStr += this.print(node.else, indent + 4) + "\n";

return prStr;

}

printWhenExpression(node, indent) {
let prStr = ${prIndent(indent)}WhenExpression:\n;
prStr += ${prIndent(indent + 2)}Test:\n;
prStr += this.print(node.test, indent + 4) + "\n";
prStr += ${prIndent(indent + 2)}Body:;

for (let expr of node.body) {
  prStr += this.print(expr, indent + 4) + "\n";
}

return prStr;

}

printBinaryExpression(node, indent) {
let prStr = ${prIndent(indent)}BinaryExpression:\n;
prStr += ${prIndent(indent + 2)}Left:\n;
prStr += this.print(node.left, indent + 4) + "\n";
prStr += ${prIndent(indent + 2)}Operator:\n;
prStr += ${prIndent(indent + 4)}${node.op}\n;
prStr += ${prIndent(indent + 2)}Right:\n;
prStr += this.print(node.right, indent + 4) + "\n";

return prStr;

}

printLogicalExpression(node, indent) {
let prStr = ${prIndent(indent)}LogicalExpression:\n;
prStr += ${prIndent(indent + 2)}Left:\n;
prStr += this.print(node.left, indent + 4) + "\n";
prStr += ${prIndent(indent + 2)}Operator:\n;
prStr += ${prIndent(indent + 4)}${node.op}\n;
prStr += ${prIndent(indent + 2)}Right:\n;
prStr += this.print(node.right, indent + 4) + "\n";

return prStr;

}

printUnaryExpression(node, indent) {
let prStr = ${prIndent(indent)}UnaryExpression:\n;
prStr += ${prIndent(indent + 2)}Operator:\n;
prStr += ${prIndent(indent + 4)}${node.op}\n;
prStr += ${prIndent(indent + 2)}Operand:\n;
prStr += this.print(node.operand, indent + 4) + "\n";

return prStr;

}
`

That's all the changes to the printer. Now we're going to make some changes to the CLI to make it more flexible.

Changes to The CLI

We're going to make 3 major changes to the CLI:

Allow running a Wanda file directly from the command line (compiling and executing it in one step)
Interactively load a Wanda file as a script and use its definitions in the REPL
Output a compiled file and optionally build the whole bundle

In the case of a compiled file, you have the option to either use ES2015 imports to import the global library or to use ESBuild to bundle everything into one giant JavaScript file.

Changes to The REPL

First, we're going to change the arguments the REPL function takes. Instead of a single string argument mode, we're going to give it an options object with properties mode (string) and path (string). Everything has defaults, so it can still be called simply by running the bare function with no arguments.

The way mode works isn't changing. We will, however, enable the ability to turn off printing the AST from within the REPL if you've enabled it. Not doing that sooner was a silly oversight on my part.

The big change is if there's a value for the path option it will grab the contents of that file, compile it, and execute the result in the global environment so you'll have access to any toplevel definitions in that file within the REPL.

Here's the new version of the repl function in src/cli/repl.js:

`js
import os from "os";
import vm from "vm";
import fs from "fs";
import readlineSync from "readline-sync";
import { pprintAST, pprintDesugaredAST } from "./pprint.js";
import { println } from "../printer/println.js";
import { makeGlobalNameMap } from "../runtime/makeGlobals.js";
import { emitGlobalEnv } from "../emitter/emitGlobalEnv.js";
import { build } from "./build.js";
import { compile } from "./compile.js";
import { makeGlobalTypeEnv } from "../typechecker/makeGlobalTypeEnv.js";
import { countIndent, inputFinished } from "./utils.js";

const read = (prompt) => readlineSync.question(prompt);

export const repl = ({ mode = "repl", path = "" } = {}) => {
// Create global compile environment
const compileEnv = makeGlobalNameMap();
const typeEnv = makeGlobalTypeEnv();

// Build global module and instantiate in REPL context
// This should make all compiled global symbols available
const globalCode = build(emitGlobalEnv());
vm.runInThisContext(globalCode);

if (path) {
// load file in REPL interactively
const fileContents = fs.readFileSync(path, { encoding: "utf-8" });
const compiledFile = compile(fileContents, path, compileEnv, typeEnv);
vm.runInThisContext(compiledFile);
}

let prompt = "wanda> ";
let input = "";
let indent = 0;

while (true) {
try {
input += read(prompt + " ".repeat(indent));

  switch (input) {
    // If it's a command, execute it
    case ":quit":
      process.exit(0);
    case ":print-ast":
      mode = "printAST";
      input = "";
      break;
    case ":print-desugared":
      mode = "printDesugared";
      input = "";
      break;
    case ":no-print-ast":
      mode = "repl";
      input = "";
      break;
    // If it's code, compile and run it
    default:
      if (inputFinished(input)) {
        let compiled = compile(input, "stdin", compileEnv, typeEnv);
        let result = vm.runInThisContext(compiled);

        if (mode === "printAST") {
          console.log(pprintAST(input));
        } else if (mode === "printDesugared") {
          console.log(pprintDesugaredAST(input));
        }

        println(result);
        input = "";
        indent = 0;
      } else {
        input += os.EOL;
        indent = countIndent(input);
      }
  }
} catch (e) {
  console.error(e.stack);
  input = "";
  indent = 0;
}

}
};
`

Changes to The `run` Function

I've removed the print command from the run function, now if you want to print the AST from within the REPL you have to enable it inside the REPL. You can decide whether or not you want to do the same—it's all the same to me either way.

We need to add a function that can compile and run a Wanda file, and parse the run command to do it. I've also replaced throwing exceptions with logging an error message and passing 1 to process.exit to fall more in line with UNIX shell script conventions. Here's the new run function and the runFile function that goes with it in src/cli/run.js:

`js
import vm from "vm";
import fs from "fs";
import { join } from "path";
import { repl } from "./repl.js";
import { makeGlobalNameMap } from "../runtime/makeGlobals.js";
import { emitGlobalEnv } from "../emitter/emitGlobalEnv.js";
import { build } from "./build.js";
import { compile } from "./compile.js";
import { makeGlobalTypeEnv } from "../typechecker/makeGlobalTypeEnv.js";

export const run = () => {
switch (process.argv[2]) {
case "-i":
if (!process.argv[3]) {
console.log(-i option requires file path as argument);
process.exit(1);
}
const path = join(process.cwd(), process.argv[3]);
repl({ path });
break;
case undefined:
case "repl":
return repl();
case "run":
if (!process.argv[3]) {
console.log(run command requires file path as argument);
process.exit(1);
}
return runFile(join(process.cwd(), process.argv[3]));
default:
console.log("Invalid command specified");
process.exit(1);
}
};

const runFile = (path) => {
const fileContents = fs.readFileSync(path, { encoding: "utf-8" });
const globalNs = makeGlobalNameMap();
const typeEnv = makeGlobalTypeEnv();
const globalCode = build(emitGlobalEnv());
const compiledCode = compile(fileContents, path, globalNs, typeEnv);

vm.runInThisContext(globalCode);
return vm.runInThisContext(compiledCode);
};
`

Compiling and Building Files

We also need the ability to compile Wanda files and output the results, with the optional step of building the whole program instead of just using ES2015 import statements for the global library.

We'll call the main function wandac after compilers like javac and rustc, and add an auxiliary compileFile function to handle the actual compilation. Here are those functions, in src/cli/wandac.js:

`js
import os from "os";
import fs from "fs";
import { join, basename } from "path";
import { compile } from "./compile.js";
import { build } from "./build.js";
import { makeGlobalNameMap } from "../runtime/makeGlobals.js";
import { makeGlobalTypeEnv } from "../typechecker/makeGlobalTypeEnv.js";
import { emitGlobalEnv } from "../emitter/emitGlobalEnv.js";

const compileFile = (path) => {
const contents = fs.readFileSync(path, { encoding: "utf-8" });
return compile(contents, path, makeGlobalNameMap(), makeGlobalTypeEnv());
};

export const wandac = () => {
if (!process.argv[2]) {
console.log(wandac requires either a file path or command argument);
process.exit(1);
}

switch (process.argv[2]) {
case "build": {
const pathname = join(process.cwd(), process.argv[3]);
const compiledFile = compileFile(pathname);
const globals = emitGlobalEnv();
const code = globals + os.EOL + os.EOL + compiledFile;
const bName = basename(pathname).split(".")[0];
const outfile = bName + ".build" + ".js";
const built = build(code, outfile, bName);

  fs.writeFileSync(outfile, built, { encoding: "utf-8" });
  break;
}
default: {
  // should be a file path
  const pathname = join(process.cwd(), process.argv[2]);
  const compiledFile = compileFile(pathname);
  const globals = emitGlobalEnv();
  const code = globals + os.EOL + os.EOL + compiledFile;
  const outfile = basename(pathname).split(".")[0] + ".js";

  fs.writeFileSync(outfile, code, { encoding: "utf-8" });
  break;
}

}
};
`

Now let's create a new Node shell script in the bin directory (not src/bin) and link it up to run as a global system command. First, add this to bin/wandac.js:

`js

!/usr/bin/env node

import { wandac } from "../src/cli/wandac.js";

wandac();
`

Next, we need to make a slight change to package.json to account for the fact that there are now 2 executable scripts. Change

json "bin": "bin/wanda.js",

to:

json "bin": { "wanda": "bin/wanda.js", "wandac": "bin/wandac.js" },

Now make sure you're in the compiler's root directory and run the command npm link. Now you should have both wanda and wandac available as global shell commands.

Example Files

Finally, we need a couple of example files to load into the REPL and play with! I've created a couple of simple ones just to get you started.

I'm using .wanda as the file extension.

Create a new directory in the compiler root called examples and add the following 2 files:

examples/inc.wanda:

clojure (def inc (x: number) -> number (+ x 1))

examples/fib.wanda:

clojure (def fib (n: number) -> number (if (= n 1) n (* n (fib (- n 1)))))

Now if you run the command wanda -i examples/inc.wanda you can run the inc function from within the REPL! You can also compile them if you want to see the output.

Conclusion

We've made some major strides today! While our language was technically Turing complete with just functions (it's amazing what you can do with just functions), it's a lot easier to consider it so now that we have branching enabled through conditional expressions. Recursion allows for repeating code, so the combination of conditionals and functions makes the language fully Turing complete.

The CLI is also now much more fully functional compared to just being able to run a simple REPL.

Please feel free to add your own examples. If you come up with anything cool, by all means send a pull request and I'll incorporate it into the examples I've got!

As always, you can find the code as of this point on the GitHub repo under the relevant tag.

Next time we're going to add 2 iterative constructs: loop and for. I'll see you then!

Create Your Own Programming Language 7: More Types

Jason Barr — Thu, 29 Jun 2023 20:32:22 +0000

Hello, again! It's time for another installment in the Create Your Own Programming Language series.

This time we're not going to add any significant new compiler features. Instead, we're going to add some interesting features to the type checker, including TypeScript-like union and intersection types.

As always, if you haven't read the previous article on adding functions to Wanda you should do that before continuing.

Ready? Ok, let's go.

What All We're Doing Today

Most of the work today is going to be concentrated on the type checker. We're going to add singleton types (which require us to implement constant declarations), tuple types, union types, intersection types, and type ascriptions using the :as keyword as an operator.

Singleton Types

Singleton types are types that have one and only one value. TypeScript calls them "literal types."

For example, you don't have a value of type string; you have a value of type "hello".

A singleton type is a subtype of its base, so "hello" is a subtype of string.

Singleton types require the values that have them not to change, so in order to have singleton type we're going to have to implement constant declarations.

Constants will only be constant in the type checker, though; ConstantDeclaration nodes will desugar to VariableDeclaration nodes so we won't need to add anything to the emitter.

Singleton types aren't terribly interesting by themselves, but you can do interesting things with them in conjunction with union types, below.

Tuple Types

Tuples are indexed collections that can hold values of different types, unlike a list or vector which is supposed to only hold 1 type (or its subtypes).

Tuples are going to be vectors under the hood (like TypeScript tuples, which are JavaScript arrays).

A tuple type could be something like [number, number, string].

Union Types

Union types are types where a value can have one of a number of specified types.

A union type is like logical OR for types. You can have simple unions like string || number or unions of objects.

A common pattern in TypeScript is to have unions of object types with a common field on all object types that indicates which type is being used based on its value. This is called a discriminated union. Here's an example:

type Coordinate =
  | { type: "cartesian"; x: number; y: number }
  | { type: "polar"; angle: number; magnitude: number };

Note that the 2 object types have type properties that are singleton types: "cartesian" for the first, and "polar" for the second. Not string types.

We're going to use || to set apart members of a union instead of | for symmetry with intersection types, for which we'll use && to avoid confusion with the & rest parameter operator.

In a later article we'll implement variant types, which are a kind of union type that also works like an enum in some ways.

To understand union types and how they work better, check out the article on union types in Jake Donham's Reconstructing TypeScript series.

Intersection Types

If union types are logical OR for types, intersection types are logical AND. A value of an intersection type must have all the properties of all members of the intersection.

A common use of intersection types is to add fields to an existing type. Here's an example in TypeScript:

type Coord2D = { x: number; y: number };

type Coord3D = Coord2D & { z: number };

An object of type Coord3D must have x, y, and z properties, all numbers.

To learn more about intersection types, once again you can check out the relevant article in Jake Donham's Reconstructing TypeScript series.

Type Ascriptions

Type ascriptions tell the type checker to treat an expression like a particular type. You use them when you have more information than the type checker about what type an expression is so you can get your program to correctly type check when it otherwise might not.

It's kind of an escape hatch when you need to give extra information to your type checker.

Here's what that looks like in TypeScript: let's say you have a union type AST that includes types for VariableDeclaration, NumberLiteral, and LambdaExpression AST nodes. You have a visitor class with a dispatch method based on what type of node it receives.

The dispatch method receives an argument of type AST, which will be the node's type inside that method. Your node processing methods expect arguments of their particular node type. A type ascription tells the type checker "This node, which is part of the AST union, is actually of type LambdaExpression. So you pass the node to the dispatched method like this:

enum ASTTypes {
  NumberLiteral,
  VariableDeclaration,
  LambdaExpression,
}

type NumberLiteral = { type: ASTTypes.NumberLiteral, value: number };
type VariableDeclaration = {
  type: ASTTypes.VariableDeclaration;
  name: string;
  value: NumberLiteral | LambdaExpression;
};
type LambdaExpression = {
  type: ASTTypes.LambdaExpression;
  args: string[];
  body: AST;
};

type AST = NumberLiteral | VariableDeclaration | LambdaExpression;

class Visitor {
  visit(node: AST): AST {
    switch (node.kind) {
      // other cases
      case ASTypes.LambdaExpression:
        return this.visitLambdaExpression(node as LambdaExpression);
    }
  }

  visitLambdaExpression(node: LambdaExpression): LambdaExpression {
    // method body
  }
}

Our type ascriptions will do the same thing, although obviously we don't have classes in the language yet.

Now, in TypeScript, you can't use a type ascription to say something like 5 as string. The type checker will still complain about that, as it should. Ours will too.

Changes to Shared Types

First, let's clean up something that's been bugging me about our in-language Exception types.

A couple of articles ago I added a :dict property to the base Exception class (as well as RuntimeException, though I later realized that's redundant because RuntimeException inherits from Exception), but I made the mistake of not making it a metafield. This caused it to show up in dumps when exceptions were thrown in debugging, which is annoying. Metafields are non-enumerable and so shouldn't show up in dumps by default.

Let's import addMetaField into src/shared/exceptions.js and fix that:

import { addMetaField } from "../runtime/object.js";

export class Exception extends Error {
  constructor(msg) {
    super(msg);

    addMetaField(this, "dict", { message: msg });
  }
}

You can also remove the :dict property defined in the constructor of RuntimeException if you have it there.

With that out of the way, now we can move on to the changes needed to implement the new features.

Changes to The Lexer

We need to make a very minor change to the lexer for intersection type annotations.

We're going to add support for an && token that will be used in intersection annotations.

First, in src/lexer/TokenTypes.js, add the new token to the TokenTypes enum:

export const TokenTypes = {
  // other types...
  AmpAmp: "AmpAmp",
};

Now in src/lexer/Lexer.js we need to tokenize the new token type, which requires modifying the else if case for isAmp(ch):

      // other cases...
      } else if (isAmp(ch)) {
        const { pos, line, col, file } = this.input;
        if (isAmp(this.input.lookahead(1))) {
          // skip over both ampersands
          this.input.next();
          this.input.next();
          tokens.push(
            Token.new(TokenTypes.AmpAmp, "&&", SrcLoc.new(pos, line, col, file))
          );
        } else {
          this.input.next(); // skip over punc
          tokens.push(
            Token.new(TokenTypes.Amp, ch, SrcLoc.new(pos, line, col, file))
          );
        }
      } // else...

That's it for changes to the lexer. Now we need to make changes to the reader so we can read type ascriptions with :as.

Changes to The Reader

Remember a couple of articles ago when I said we weren't adding binary expressions to a Lisp-like language? Turns out I lied. Type ascriptions are a binary expression and I sort of forgot about them.

We're going to make use of the Pratt algorithm once more and add the :as operator. Since the dot was a token type and :as is the value on a keyword token, we're going to need to change our PREC object and getPrec function to use the token's value property.

Here's the new PREC object in src/reader/read.js:

const PREC = {
  ".": 90,
  ":as": 50,
};

And getPrec:

const getPrec = (token) => PREC[token?.value] ?? 0;

As you can see, the dot and :as operators have different precedences so our Pratt algorithm might actually have some work to do besides checking if the precedence value isn't 0.

Now let's create a new readBinary function and replace the call to readMemberExpression in readExpr with it:

const readBinary = (reader, left) => {
  const tok = reader.peek();

  switch (tok.value) {
    case ".":
      return readMemberExpression(reader, left);
    case ":as":
      return readAsExpression(reader, left);
    default:
      throw new SyntaxException(tok.value, tok.srcloc);
  }
};

// readForm...

const readExpr = (reader, bp = 0) => {
  let left = readForm(reader);
  let tok = reader.peek();
  let prec = getPrec(tok);

  while (bp < prec) {
    left = readBinary(reader, left);
    tok = reader.peek();
    prec = getPrec(tok);
  }

  return left;
};

Finally, here's readAsExpression:

const readAsExpression = (reader, left) => {
  const tok = reader.peek();
  reader.expect(":as", tok.value);
  const prec = getPrec(tok);

  // skip :as keyword
  reader.skip();
  const typeAnnotation = readExpr(reader, prec);

  return {
    type: "AsExpression",
    expression: left,
    typeAnnotation,
    srcloc: left.srcloc,
  };
};

That should be the last bit of actual parsing we need to do in the reader for quite some time. I can picture my Twitter friend Shriram Krishnamurthi (a well-known figure in programming language research circles who has gracefully been very helpful in answering my probably asinine questions about languages and language development) rolling his eyes at me and pretending this article doesn't exist. 😂😂😂

Changes to The Parser

We need to add new nodes for AsExpression and ConstantDeclaration, plus we need to parse union and intersection type annotations. We're also going to add a utility function to check if a node is a primitive that will come in handy when type checking with singleton types.

First thing's first: we need to add a bunch of members to the TATypes enum at the top of parseTypeAnnotation:

export const TATypes = {
  // other members...
  Tuple: "Tuple",
  Singleton: "Singleton",
  Intersection: "Intersection",
  Union: "Union",
  Never: "Never",
  Unknown: "Unknown",
};

Never and Unknown are types that come along with union and intersection types. I'll explain those when we get to them in the type checker.

The Type Annotation Parser

The most substantial change is to the type annotation parser, so let's start with that.

Parsing singleton and tuple annotations is relatively straightforward.

Singleton annotations are just primitive literals, like "hello", true, or 7.

Tuple annotations are a comma-separated list of types inside square brackets, e.g. [number, number, string].

The reader will give singleton annotations to the parser as literal forms and tuple annotations as VectorLiteral forms.

For the former, let's create a function parseSingletonAnnotation in src/parser/parseTypeAnnotation.js:

const parseSingletonAnnotation = (annot) => ({
  kind: TATypes.Singleton,
  token: annot,
});

Then, above the section beginning with if (annot.typpe === TokenTypes.Symbol), check on annot.type and call parseSingletonAnnotation if the form is a Token that represents a literal:

  // previous cases
  if (
    annot.type === TokenTypes.Number ||
    annot.type === TokenTypes.String ||
    annot.type === TokenTypes.Boolean ||
    annot.type === TokenTypes.String
  ) {
    return parseSingletonAnnotation(annot);
  }
  // if containing switch case for symbol tokens

While we're doing things with parseTypeAnnotation, let's also refactor the guts for parsing a function annotation into its own function, called parseFunctionAnnotation:

const parseFunctionAnnotation = (annotation) => {
  // filter out arrow
  annotation = annotation.filter((item) => item.value !== "->");

  // get return type
  const retType = parseTypeAnnotation(annotation.pop());
  // get param types and if it's variadic
  let params = [];
  let variadic = false;

  for (let item of annotation) {
    if (item.type === TokenTypes.Amp) {
      variadic = true;
      continue;
    } else {
      params.push(parseTypeAnnotation(item));
    }
  }

  return { kind: TATypes.Function, params, retType, variadic };
};

Now replace everything in the block statement for if (hasArrow) with:

      return parseFunctionAnnotation(annotation);

For parseTypeAnnotation, you should be left with this (note that we're now also handling never and unknown in the function):

export const parseTypePrimitive = (annotation) => {
  if (annotation instanceof Cons) {
    // is function or generic annotation
    // flatten Cons to array
    annotation = [...annotation];

    // if it has an arrow, it's a function annotation
    const hasArrow = annotation.reduce((hasArrow, item) => {
      if (item.type === TokenTypes.Symbol && item.value === "->") {
        return true;
      }
      return hasArrow;
    }, false);

    if (hasArrow) {
      // is function annotation
      return parseFunctionAnnotation(annotation);
    }
  }

  let annot;
  if (Array.isArray(annotation)) {
    // is generic annotation
    // get container type
    annot = annotation[0];
  } else {
    // is simple annotation
    annot = annotation;
  }

  if (annot.type === "RecordLiteral") {
    return parseObjectAnnotation(annot);
  }

  if (annot.type === "VectorLiteral") {
    return parseTupleAnnotation(annot);
  }

  if (annot.type === TokenTypes.Nil) {
    return { kind: TATypes.NilLiteral };
  }

  if (
    annot.type === TokenTypes.Number ||
    annot.type === TokenTypes.String ||
    annot.type === TokenTypes.Boolean ||
    annot.type === TokenTypes.String
  ) {
    return parseSingletonAnnotation(annot);
  }

  if (annot.type === TokenTypes.Symbol) {
    switch (annot.value) {
      case "any":
        return { kind: TATypes.AnyLiteral };
      case "number":
        return { kind: TATypes.NumberLiteral };
      case "string":
        return { kind: TATypes.StringLiteral };
      case "boolean":
        return { kind: TATypes.BooleanLiteral };
      case "keyword":
        return { kind: TATypes.KeywordLiteral };
      case "never":
        return { kind: TATypes.Never };
      case "unknown":
        return { kind: TATypes.Unknown };
      case "list":
        // annotation is array with listType as 2nd member
        return parseListAnnotation(annotation[1]);
      case "vector":
        return parseVectorAnnotation(annotation[1]);
      default:
        // must be a named type
        return { kind: TATypes.Symbol, name: annot.value };
    }
  }

  throw new Exception(
    `Unknown type annotation kind ${JSON.stringify(annot, null, 2)}`
  );
};

Now for the most drastic change to the type annotation parser: rename the function parseTypeAnnotation to parseTypePrimitive and get rid of the export keyword before the const keyword for the function expression assignment.

Parsing Compound Type Annotations

Now stub out a new function named parseTypeAnnotation and for now just keep it simple while we figure out how to parse compound type annotations (i.e. union and intersection annotations):

export const parseTypeAnnotation = (annotation) => {
  return parseTypePrimitive(annotation);
};

We need to be able to figure out if an annotation is simple, in which case it's handled by parseTypePrimitive, or compound. A compound type annotation is contained within a cons list and contains multiple simple annotations separated by either a || or && token.

So the first thing we need to do is check for one of those tokens in the annotation.

We'll do something similar for checking if a simple type annotation has an arrow: flatten the annotation to an array and use reduce to see if it has at least one || or && token (the same type annotation will never have both – a type is either intersection or union, but not both).

Then if it has one of those tokens we get the first annotation in the list and parse it. Then we choose the type of the annotation based on which token it is and continue parsing annotations in a loop until we reach the end of the list. Then we return the compound annotation.

Here's the new parseTypeAnnotation:

export const parseTypeAnnotation = (annotation) => {
  const annot = annotation instanceof Cons ? [...annotation] : null;

  const isCompound =
    Array.isArray(annot) &&
    annot.reduce((isCompound, item) => {
      if (
        (item.type === TokenTypes.Symbol && item.value === "||") ||
        item.type === TokenTypes.AmpAmp
      ) {
        return true;
      }
      return isCompound;
    }, false);

  if (isCompound) {
    const first = parseTypeAnnotation(annot[0]);
    let sep = annot[1];
    const kind =
      sep.type === TokenTypes.AmpAmp ? TATypes.Intersection : TATypes.Union;
    const types = [first];

    let i = 2;
    while (sep) {
      types.push(parseTypeAnnotation(annot[i]));
      i++;
      sep = annot[i];
    }

    return { kind, types };
  }

  return parseTypePrimitive(annotation);
};

Adding New Nodes to The Parser

Now that we have parsing the new type annotations out of the way, let's move on to the regular parser. We need to add new nodes for and handle parsing :as expressions and constant declarations.

First, in src/parser/ast.js, we add new members to the ASTTypes enum:

export const ASTTypes = {
  // other members...
  AsExpression: "AsExpression",
  ConstantDeclaration: "ConstantDeclaration",
};

Now add new node constructors:

export const AST = {
  // other constructors...
  AsExpression(expression, type, srcloc) {
    return {
      kind: ASTTypes.AsExpression,
      expression,
      type,
      srcloc,
    };
  },

  ConstantDeclaration(lhv, expression, srcloc, typeAnnotation = null) {
    return {
      kind: ASTTypes.ConstantDeclaration,
      lhv,
      expression,
      srcloc,
      typeAnnotation,
    };
  },
};

In src/parser/parse.js, add a new case to the switch statement in parseComplexForm to handle :as expressions:

    // other cases...
    case "AsExpression": {
      const expression = parseExpr(form.expression);
      const type = parseTypeAnnotation(form.typeAnnotation);
      return AST.AsExpression(expression, type, form.srcloc);
    }
    // default case

Constant declarations, like function declarations, will begin with the def symbol. Since we have 2 forms that use the same symbol, we'll need a new function to disambiguate called parseMaybeFunctionDeclaration.

We call it from the "def" case in the switch statement in parseList:

    // other cases...
    case "def":
      return parseMaybeFunctionDeclaration(form);
    // case "fn"...

Here's parseMaybeFunctionDeclaration:

const parseMaybeFunctionDeclaration = (form) => {
  const length = [...form].length;

  if (length === 3) {
    // (def <name> <expression>) or
    // (def (<name>: <type>) <expression>)
    return parseConstantDeclaration(form);
  }

  return parseFunctionDeclaration(form);
};

Function declarations have length 4 because there's the def keyword, the function name, the function arguments, and the function body. Constant declarations just have the def keyword, the constant name, and the expression to initialize the constant's value.

parseConstantDeclaration delegates most of the work to parseVariableDeclaration since the 2 are exactly the same apart from the value produced for node.kind by the node constructor:

const parseConstantDeclaration = (form) => {
  const varDecl = parseVariableDeclaration(form);
  return AST.ConstantDeclaration(
    varDecl.lhv,
    varDecl.expression,
    varDecl.srcloc,
    varDecl.typeAnnotation
  );
};

Adding a Helper Function for Primitive Nodes

The last thing we need to do with the parser is add a utility function to tell us if an AST node is for a primitive literal. This will come in handy in the type checker when we check singleton types.

Add the isPrimitive function in src/parser/utils.js:

import { ASTTypes } from "./ast.js";

export const isPrimitive = (node) =>
  node.kind === ASTTypes.NumberLiteral ||
  node.kind === ASTTypes.StringLiteral ||
  node.kind === ASTTypes.BooleanLiteral ||
  node.kind === ASTTypes.KeywordLiteral ||
  node.kind === ASTTypes.NilLiteral;

That's it for changes to the parser! Now for the bulk of the work, which takes place in the type checker.

Changes to The Type Checker

First, add the new cases to the TypeTypes enum in src/typechecker/types.js:

export const TypeTypes = {
  // other values...
  Tuple: "Tuple",
  Singleton: "Singleton",
  Never: "Never",
  Union: "Union",
  Unknown: "Unknown",
  Intersection: "Intersection",
};

Now add type constructors for the simpler types in src/typechecker/constructors.js. Union and intersection types will take a bit more work, as I'll explain below.

// other constructors...
export const tuple = (types, constant = false) => ({
  kind: TypeTypes.Tuple,
  types,
  constant,
});

export const singleton = (base, value) => ({
  kind: TypeTypes.Singleton,
  base,
  value,
  constant: true,
});

export const never = { kind: TypeTypes.Never };

export const unknown = { kind: TypeTypes.Unknown };

Validators are simple in src/typechecker/validators.js:

export const isTuple = (type) => {
  return type.kind === TypeTypes.Tuple;
};

export const isSingleton = (type) => {
  return type.kind === TypeTypes.Singleton;
};

export const isNever = (type) => {
  return type.kind === TypeTypes.Never;
};

export const isUnion = (type) => {
  return type.kind === TypeTypes.Union;
};

export const isUnknown = (type) => {
  return type.kind === TypeTypes.Unknown;
};

export const isIntersection = (type) => {
  return type.kind === TypeTypes.Intersection;
};

Constructing Union Types

With unions, you can have different types that are equivalent. For example (in TypeScript):

'red' | 'blue' | 'green'
'blue' | 'red' | 'green'
'green' | 'red' | 'blue'

all contain the same values. If a union arm is a nested union, the arms of the inner union can be lifted up to the outer union (if a value satisfies the nested union, then it satisfies one of its arms); so for example:

'red' | ('green' | 'blue')
('red' | 'green') | 'blue'
'red' | 'green' | 'blue'

also contain the same values.

If one arm of a union is a subtype of another, it can be removed (all values that satisfy it also satisfy the other arm); so for example:

{ type: 'cartesian', x: number, y: number } | { x: number, y: number }
{ x: number, y: number } | never
{ x: number, y: number }

all contain the same values.

Finally, an object type with properties of union type supports the same operations as a union of object types; so for example:

{ foo: 1 | 2, bar: 3 | 4 }
{ foo: 1 | 2, bar: 3 } | { foo: 1 | 2, bar: 4 }
{ foo: 1, bar: 3 | 4 } | { foo: 2, bar: 3 | 4 }

all contain the same types.

To simplify the type checker and make its output more readable, we normalize union types in the constructor. To do that, we flatten nested unions and remove redundant arms. If there are no types left in the union, we return the never type.

Here's src/typechecker/union.js:

const collapseSubtypes = (ts) => {
  /**
   * An arm is kept if for every arm (except itself) it's not a subtype of the
   * other arm or it's equivalent to the other arm and this is the first
   * equivalent arm.
   */
  return ts.filter((t1, i1) => {
    return ts.every(
      (t2, i2) =>
        i1 === i2 || !isSubtype(t1, t2) || (isSubtype(t2, t1) && i1 < i2)
    );
  });
};

const flatten = (ts) => {
  return [].concat(...ts.map((t) => (isUnion(t) ? t.types : t)));
};

export const union = (...types) => {
  types = flatten(types);
  types = collapseSubtypes(types);

  if (types.length === 0) return never;
  if (types.length === 1) return types[0];
  return { kind: TypeTypes.Union, types };
};

For more information about what's going on here, check out Jake Donham's post on union types.

Constructing Intersection Types

Like union types, intersection types give us lots of ways to describe the same collection of values. The order of types in an intersection doesn't matter:

{ x: number } & { y: number } & { z: number }
{ y: number } & { x: number } & { z: number }
{ z: number } & { x: number } & { y: number }

all contain the same values.

If a part of an intersection is a nested intersection, the parts of the inner intersection can be lifted up to the outer intersection, so for example:

{ x: number } & ({ y: number } & { z: number })
({ x: number } & { y: number }) & { z: number }
{ x: number } & { y: number } & { z: number }

all contain the same values.

If one part of an intersection is a supertype of another, it can be removed; so for example:

{ type: 'cartesian', x: number, y: number } & { x: number, y: number }
{ type: 'cartesian', x: number, y: number } & unknown
{ type: 'cartesian', x: number, y: number }

all contain the same values.

As we do for union types, we normalize intersection types in the type checker. Normalizing intersections is more complicated than normalizing unions. We want to detect and eliminate empty intersections (this will be important for narrowing), but unions make it more difficult.

In addition to flattening nested intersections and removing redundant parts (as we do for unions), we also distribute intersections over unions and eliminate empty intersections. As with unions, we don't distribute intersections over object or function types (or objects / functions over intersections).

Here's a helper function for distributing over unions (in src/typechecker/union.js):

export const distributeUnion = (ts) => {
  let accum = [];

  const dist = (ts, i) => {
    if (i === ts.length) {
      accum.push(ts);
    } else {
      const ti = ts[i];

      if (isUnion(ti)) {
        for (let t of ti.types) {
          const ts2 = ts.slice(0, i).concat(t, ts.slice(i + 1));
          dist(ts2, i + 1);
        }
      } else {
        dist(ts, i + 1);
      }
    }
  };

  dist(ts, 0);
  return accum;
};

This function takes a list of types (which may contain unions) and returns roughly the Cartesian product over the unions.

Next we need a function to check if 2 types overlap, that is if their intersection is not empty. In src/typechecker/intersection.js:

const overlaps = (x, y) => {
  if (isNever(x) || isNever(y)) return false;
  if (isUnknown(x) || isUnknown(y)) return true;

  if (isUnion(x)) {
    return x.types.some((x) => overlaps(x, y));
  } else if (isUnion(y)) {
    return y.types.some((y) => overlaps(x, y));
  }

  if (isIntersection(x)) {
    return x.types.every((x) => overlaps(x, y));
  } else if (isIntersection(y)) {
    return y.types.every((y) => overlaps(x, y));
  }

  if (isSingleton(x) && isSingleton(y)) return x.value === y.value;
  if (isSingleton(x)) return x.base === y.kind;
  if (isSingleton(y)) return y.base === x.kind;

  if (isObject(x) && isObject(y)) {
    return x.properties.every(({ name, type: xType }) => {
      const yType = propType(y, name);
      if (!yType) return true;
      else return overlaps(xType, yType);
    });
  }

  return x.type === y.type;
};

Now, to normalize intersections we flatten nested intersections, distribute the intersection over unions, and check for emptiness/remove redundant parts. Continuing in intersection.js:

const collapseSupertypes = (ts) => {
  return ts.filter((t1, i1) =>
    ts.every(
      (t2, i2) =>
        i1 === i2 || !isSubtype(t2, t1) || (isSubtype(t1, t2) && i1 < i2)
    )
  );
};

const flatten = (ts) =>
  [].concat(...ts.map((t) => (isIntersection(t) ? t.types : t)));

const intersectionNoUnion = (ts) => {
  if (ts.some((t1, i1) => ts.some((t2, i2) => i1 < i2 && !overlaps(t1, t2)))) {
    return never;
  }

  ts = collapseSupertypes(ts);

  if (ts.length === 0) return unknown;
  if (ts.length === 1) return ts[0];

  return { type: TypeTypes.Intersection, types: ts };
};

export const intersection = (...ts) => {
  ts = flatten(ts);
  ts = distributeUnion(ts).map((ts) => intersectionNoUnion(ts));

  return union(...ts);
};

Again, for a better understanding see Jake Donham's post on intersection types.

Mapping Types

Whereas before we could just infer using simple types, now we need a map function to account for unions and intersections in our inference. In src/typechecker/map.js:

export const map = (t, fn) => {
  if (isUnion(t)) {
    return union(...t.types.map((t) => map(t, fn)));
  } else if (isIntersection(t)) {
    /** @type {import("./types.js").Type[]} */
    let ts = [];
    let error = null;

    for (let tt of t.types) {
      try {
        ts.push(fn(tt));
      } catch (e) {
        if (!error) error = e;
      }
    }

    if (ts.length === 0) {
      throw error;
    } else {
      return intersection(...ts);
    }
  } else {
    return fn(t);
  }
};

Now let's add union, intersection, and map to our Type module in src/typechecker/Type.js:

import { TypeTypes } from "./types.js";
import * as Validators from "./validators.js";
import * as Constructors from "./constructors.js";
import { typeToString } from "./typeToString.js";
import { union } from "./union.js";
import { map } from "./map.js";
import { intersection } from "./intersection.js";

export const Type = {
  Type: TypeTypes,
  ...Constructors,
  ...Validators,
  toString: typeToString,
  union,
  map,
  intersection,
};

Updated `fromTypeAnnotation`

Here's the updated fromTypeAnnotation function in src/typechecker/fromTypeAnnotation.js with cases for singleton, tuple, union, intersection, unknown, and never types:

export const fromTypeAnnotation = (
  typeAnnotation,
  typeEnv = TypeEnvironment.new()
) => {
  switch (typeAnnotation.kind) {
    case TATypes.AnyLiteral:
      return Type.any;
    case TATypes.NumberLiteral:
      return Type.number;
    case TATypes.StringLiteral:
      return Type.string;
    case TATypes.BooleanLiteral:
      return Type.boolean;
    case TATypes.KeywordLiteral:
      return Type.keyword;
    case TATypes.NilLiteral:
      return Type.nil;
    case TATypes.Never:
      return Type.never;
    case TATypes.Unknown:
      return Type.unknown;
    case TATypes.Symbol: {
      const name = typeAnnotation.name;
      const type = typeEnv.getType(name);

      if (!type) {
        throw new Exception(
          `Type ${name} not found in current type environment`
        );
      }

      return type;
    }
    case TATypes.List: {
      const listType = fromTypeAnnotation(typeAnnotation.listType, typeEnv);
      return Type.list(listType);
    }
    case TATypes.Vector: {
      const vectorType = fromTypeAnnotation(typeAnnotation.vectorType, typeEnv);
      return Type.vector(vectorType);
    }
    case TATypes.Object: {
      const propTypes = typeAnnotation.properties.map((prop) => ({
        name: prop.name,
        type: fromTypeAnnotation(prop.propType, typeEnv),
      }));
      return Type.object(propTypes);
    }
    case TATypes.Function: {
      const paramTypes = typeAnnotation.params.map((p) =>
        fromTypeAnnotation(p, typeEnv)
      );
      const retType = fromTypeAnnotation(typeAnnotation.retType, typeEnv);
      return Type.functionType(paramTypes, retType, typeAnnotation.variadic);
    }
    case TATypes.Tuple: {
      /** @type {import("./types.js").Type[]} */
      let types = [];

      for (let mem of typeAnnotation.types) {
        types.push(fromTypeAnnotation(mem, typeEnv));
      }

      return Type.tuple(types);
    }
    case TATypes.Singleton: {
      const tType = typeAnnotation.token.type;
      const base =
        tType === TokenTypes.Number
          ? "Number"
          : tType === TokenTypes.String
          ? "String"
          : tType === TokenTypes.Boolean
          ? "Boolean"
          : TokenTypes.Keyword
          ? "Keyword"
          : fail(
              `Invalid token type ${tType} when parsing type annotation ${JSON.stringify(
                typeAnnotation,
                null,
                2
              )}`
            );
      const value = typeAnnotation.token.value;

      return Type.singleton(base, value);
    }
    case TATypes.Union:
      return Type.union(
        ...typeAnnotation.types.map((t) => fromTypeAnnotation(t, typeEnv))
      );
    case TATypes.Intersection: {
      return Type.intersection(
        ...typeAnnotation.types.map((t) => fromTypeAnnotation(t, typeEnv))
      );
    }
    default:
      throw new Exception(
        `Type not found for type annotation ${JSON.parse(
          typeAnnotation,
          null,
          2
        )}`
      );
  }
};

Converting Types to String

Now we need cases for the new types in the switch statement in typeToString in src/typechecker/typeToString.js:

    // other cases...
    case TypeTypes.Tuple:
      return `[${type.types.map(typeToString).join(", ")}]`;
    case TypeTypes.Singleton:
      return type.value;
    case TypeTypes.Never:
      return "never";
    case TypeTypes.Union:
      return `(${type.types.map(typeToString).join(" | ")})`;
    case TypeTypes.Unknown:
      return "unknown";
    case TypeTypes.Intersection:
      return `(${type.types.map(typeToString).join(" && ")})`;
    // default case

Unifying Types

Now that we have unknown as a universal supertype, we need to modify the unification functions in src/typechecker/unify.js:

import { Type } from "./Type.js";
import { isSubtype } from "./isSubtype.js";

export const unify = (type1, type2) => {
  if (isSubtype(type1, type2)) {
    return type2;
  } else if (isSubtype(type2, type1)) {
    return type1;
  }

  return Type.unknown;
};

export const unifyAll = (...types) => {
  return types.reduce((unified, type) => {
    if (Type.isUnknown(unified) || Type.isAny(unified)) return unified;

    return unify(unified, type);
  });
};

Type Inference

Due to adding constants and singleton types, we actually need to make a change to nearly every single infer* function to accommodate constants.

We add a constant parameter to the infer function that defaults to false and pass it into the inference subfunctions in src/typechecker/infer.js:

export const infer = (ast, env, constant = false) => {
  switch (ast.kind) {
    case ASTTypes.NumberLiteral:
      return inferNumber(ast, constant);
    case ASTTypes.StringLiteral:
      return inferString(ast, constant);
    case ASTTypes.BooleanLiteral:
      return inferBoolean(ast, constant);
    case ASTTypes.KeywordLiteral:
      return inferKeyword(ast, constant);
    case ASTTypes.NilLiteral:
      return inferNil();
    case ASTTypes.Symbol:
      return inferSymbol(ast, env, constant);
    case ASTTypes.CallExpression:
      return inferCallExpression(ast, env, constant);
    case ASTTypes.VariableDeclaration:
      return inferVariableDeclaration(ast, env, constant);
    case ASTTypes.SetExpression:
      return inferSetExpression(ast, env, constant);
    case ASTTypes.DoExpression:
      return inferDoExpression(ast, env, constant);
    case ASTTypes.TypeAlias:
      return inferTypeAlias(ast, env, constant);
    case ASTTypes.VectorLiteral:
      return inferVectorLiteral(ast, env, constant);
    case ASTTypes.RecordLiteral:
      return inferRecordLiteral(ast, env, constant);
    case ASTTypes.MemberExpression:
      return inferMemberExpression(ast, env, constant);
    case ASTTypes.LambdaExpression:
      return inferFunction(ast, env, constant);
    case ASTTypes.FunctionDeclaration:
      return inferFunction(ast, env, constant);
    case ASTTypes.ConstantDeclaration:
      return inferVariableDeclaration(ast, env, true);
    case ASTTypes.AsExpression:
      return inferAsExpression(ast, env, constant);
    default:
      throw new Exception(`No type inferred for AST node type ${ast.kind}`);
  }
};

Note the new cases at the bottom of the switch statement for constant declarations (which just call inferVariableDeclaration with constant set to true) and the call to inferAsExpression. Here's that function:

const inferAsExpression = (node, env, constant) => {
  const type = fromTypeAnnotation(node.type);
  const exprType = infer(node.expression, env, constant);

  if (env.checkingOn) {
    if (!isSubtype(exprType, type)) {
      throw new TypeException(
        `${Type.toString(
          exprType
        )} is not a valid subtype of the given type ${Type.toString(
          type
        )} in :as expression`,
        node.srcloc
      );
    }
  }

  return type;
};

We'll need to modify the header for every subfunction except inferNil (which does not change).

We also need to modify the inference functions for primitives to handle singleton types. If constant is true, they now return singleton types:

const inferNumber = (ast, constant) =>
  constant ? Type.singleton("Number", ast.value) : Type.number;

const inferString = (ast, constant) =>
  constant ? Type.singleton("String", ast.value) : Type.string;

const inferBoolean = (ast, constant) =>
  constant ? Type.singleton("Boolean", ast.value) : Type.boolean;

const inferKeyword = (ast, constant) =>
  constant ? Type.singleton("Keyword", ast.value) : Type.keyword;

We also need to modify every recursive call to the infer function to make sure we pass in the constant parameter value.

Now let's modify inferCallExpression and inferMemberExpression to use the new Type.map function since those types could be unions or intersections:

const inferCallExpression = (node, env, constant) => {
  let func = infer(node.func, env, constant);

  if (Type.isAny(func)) {
    return Type.any;
  } else if (Type.isUndefined(func) || Type.isUndefined(func.ret)) {
    // this should only happen during first typechecker pass
    return Type.undefinedType;
  } else if (Type.isTypeAlias(func)) {
    func = getAliasBase(func.name, env);
  }

  return Type.map(func, (func) => {
    if (!func.variadic && node.args.length > func.params.length) {
      throw new TypeException(
        `Too many arguments for function: ${node.args.length} given; ${func.params.length} expected`,
        node.srcloc
      );
    }

    // handle partially applied functions
    if (
      node.args.length <
      (func.variadic ? func.params.length - 1 : func.params.length)
    ) {
      // is partially applied
      const params = func.params.slice(0, node.args.length);

      if (env.checkingOn) {
        checkArgTypes(node, params, env, func, constant);
      }

      const newParams = func.params.slice(node.args.length);
      return Type.functionType(newParams, func.ret, func.variadic);
    }

    if (env.checkingOn) {
      checkArgTypes(node, func.params, env, func, constant);
    }

    return func.ret;
  });
};

// ...

const inferMemberExpression = (node, env, constant) => {
  const prop = node.property;
  let object = infer(node.object, env, constant);

  if (Type.isTypeAlias(object)) {
    object = getAliasBase(object.name, env);
  }

  return Type.map(object, (object) => {
    if (!Type.isObject(object)) {
      if (env.checkingOn) {
        throw new TypeException(
          `Member expression expects object type; ${Type.toString(
            object
          )} given`,
          node.srcloc
        );
      } else {
        return Type.any;
      }
    }

    const type = propType(object, prop.name);

    if (!type && env.checkingOn) {
      throw new TypeException(
        `Property ${prop.name} not found on object of type ${Type.toString(
          object
        )}`,
        node.srcloc
      );
    }

    return type ?? Type.any;
  });
};

Let's also change inferVectorLiteral so that an empty vector has the vectorType of never instead of any:

const inferVectorLiteral = (node, env, constant) => {
  if (node.members.length === 0) {
    return Type.vector(Type.never);
  }

  const types = node.members.map((m) => infer(m, env, constant));
  const unified = unifyAll(...types);

  return Type.vector(unified, constant);
};

We're also going to add a check to inferFunction to make sure that if a function is annotated with the never type it doesn't return something else as its inferred type:

const inferFunction = (node, env, constant) => {
  const params = node.params.map((p) => {
    if (p.typeAnnotation) {
      env.checkingOn = true;
    }
    const type = p.typeAnnotation
      ? fromTypeAnnotation(p.typeAnnotation, env)
      : Type.any;
    env.set(p.name, type);
    return type;
  });

  if (node.retType) {
    env.checkingOn = true;
  }

  const retType = node.retType
    ? fromTypeAnnotation(node.retType, env)
    : Type.any;
  let inferredRetType;

  for (let expr of node.body) {
    // type of last expression "wins"
    inferredRetType = infer(expr, env, constant);
  }

  if (env.checkingOn && !isSubtype(inferredRetType, retType)) {
    throw new TypeException(
      `Inferred return type ${Type.toString(
        inferredRetType
      )} is not a subtype of annotated return type ${Type.toString(retType)}`,
      node.srcloc
    );
  } else if (env.checkingOn && Type.isNever(retType)) {
    if (!Type.isNever(inferredRetType)) {
      throw new TypeException(
        `Function with return type never cannot return inferred type of ${Type.toString(
          inferredRetType
        )}`,
        node.srcloc
      );
    }
  }

  return Type.functionType(
    params,
    Type.isAny(retType) || Type.isUndefined(retType)
      ? inferredRetType
      : retType,
    node.variadic
  );
};

Now go back and double check to make sure you didn't miss any function type signatures with the constant parameter and recursive calls to infer with the new constant parameter.

Subtyping Singleton, Tuple, Union, and Intersection Types

A singleton type is a subtype of its base type, whereas a type is only a subtype of a singleton type if the values match.

For a tuple type, make sure the types are the same length and that every type of type2 is a subtype of the corresponding type from type1.

A type is a subtype of a union type if it matches one arm of the union, and it's a subtype of an intersection type if it matches every intersection.

Here are the new cases in src/typechecker/isSubtype.js:

  // after checking function types
  if (Type.isTuple(type1) && Type.isTuple(type2)) {
    return type1.types.every((a, i) => isSubtype(a, type2.types[i]));
  }

  if (Type.isSingleton(type1)) {
    if (Type.isSingleton(type2)) return type1.value === type2.value;
    else return isSubtype(type1.base, type2);
  }

  if (Type.isUnion(type1)) {
    return type1.types.every((t1) => isSubtype(t1, type2));
  }

  if (Type.isUnion(type2)) {
    return type2.types.some((t2) => isSubtype(type1, t2));
  }

  if (Type.isIntersection(type1)) {
    return type1.types.some((a) => isSubtype(a, type2));
  }

  if (Type.isIntersection(type2)) {
    return type2.types.every((b) => isSubtype(type1, b));
  }
  // default return false

Checking Singleton, Tuple, and Union Types

We don't need to add anything to our check function for intersection types, but we do for singletons, tuples, and unions.

With singleton types, we need to check if the literal value matches the singleton's value. We also need to make this check for properties of an object type.

In check in src/typechecker/check.js:

  if (Type.isSingleton(type) && isPrimitive(ast) && ast.value === type.value) {
    // primitive matches singleton type
    return;
  }

And the new checkObject:

const checkObject = (ast, type, env) => {
  const astProps = ast.properties.map((prop) => ({
    name: prop.key.name,
    expr: prop.value,
  }));

  type.properties.forEach(({ name }) => {
    const astProp = astProps.find(({ name: astName }) => astName === name);

    if (!astProp) {
      throw new TypeException(
        `Property ${name} not found on record literal`,
        ast.srcloc
      );
    }
  });

  astProps.forEach(({ name, expr }) => {
    const pType = propType(type, name);

    if (!pType) {
      throw new TypeException(
        `Property ${name} not found on object of type ${Type.toString(type)}`,
        ast.srcloc
      );
    }

    if (
      Type.isSingleton(pType) &&
      isPrimitive(expr) &&
      pType.value === expr.value
    ) {
      // continue
    } else {
      check(expr, pType, env);
    }
  });
};

We need to check each member of a tuple against the corresponding node of the VectorLiteral it's matched against.

In check:

  if (ast.kind === ASTTypes.VectorLiteral && Type.isTuple(type)) {
    return checkTuple(ast, type, env);
  }

And checkTuple:

const checkTuple = (ast, type, env) => {
  let i = 0;
  for (let t of type.types) {
    check(ast.members[i], t, env);
    i++;
  }
};

Finally, for unions, we need to check against each arm of the union. If nothing matches, we need to construct an error message based on the whole union type and throw that as a TypeException. See Jake Donham's post on union types for why it's good to do this.

In check:

  if (Type.isUnion(type)) {
    return checkUnion(ast, type, env);
  }

And in checkUnion:

const checkUnion = (ast, type, env) => {
  for (let t of type.types) {
    try {
      check(ast, t, env);
      return;
    } catch (_) {
      // do nothing
    }
  }
  // Nothing matched, so construct error message based on the whole union
  const inferredType = infer(ast, env);

  if (!isSubtype(inferredType, type)) {
    throw new TypeException(
      `Type ${Type.toString(
        inferredType
      )} is not a valid subtype of ${Type.toString(type)}`,
      ast.srcloc
    );
  }
};

A Slight Change to The Type Environment

I forgot when creating the type environment class to make sure checking propagates to child environments, so in the extend method we want to make sure that if checkingOn is set for the parent it's also set for the child in src/typechecker/TypeEnvironment.js:

  extend(name) {
    let env = new TypeEnvironment(this, { name });
    env.checkingOn = this.checkingOn;
    return env;
  }

The TypeChecker

First, I found a bug in the TypeChecker class in src/typechecker/TypeChecker.js between articles. In the checkVariableDeclaration method, change the return statement to this:

    return {
      ...node,
      expression: this.checkNode(node.expression, env),
      type,
    };

This fixes it so any type annotation given is still present during the second pass of the type checker. Previously it was being discarded prior to the second check. Whoops!

I also found a second bug that was causing an error when processing type aliases on the second pass because I had named the type annotation property on the AST node type instead of something else. Since the AST node is already defined, I decided just to check if it's the 2nd pass and, if so, return the value of the type property which is the type defined for the alias:

  checkTypeAlias(node, env) {
    const type = isSecondPass ? node.type : fromTypeAnnotation(node.type, env);
    env.setType(node.name, type);
    return { ...node, type };
  }

Now for the current changes to the TypeChecker: in the checkNode method, we need to add cases for the new ConstantDeclaration and AsExpression nodes to the switch statement:

      // other cases...
      case ASTTypes.ConstantDeclaration:
        return this.checkConstantDeclaration(node, env);
      case ASTTypes.AsExpression:
        return this.checkAsExpression(node, env);
      // default case

Now let's make a quick edit to checkVariableDeclaration so that it takes a constant parameter that defaults to false and passes that into the infer function:

  checkVariableDeclaration(node, env, constant = false) {
    let type;

    if (node.typeAnnotation) {
      type = fromTypeAnnotation(node.typeAnnotation, env);
      check(node.expression, type, env);
      env.checkingOn = true;
    } else {
      type = infer(node.expression, env, constant);
    }
  // rest of method body
  }

Tuples can also be destructured in variable assignment, so we have to account for that too. We're also going to throw an error if someone tries to assign a value to a never type, though hopefully no one would ever try to do that in a variable declaration because... why would you do that?:

    // continuing from above
    if (env.checkingOn && Type.isNever(type)) {
      throw new TypeException(
        `Type never cannot be assigned a value`,
        node.srcloc
      );
    }

    if (node.lhv.kind === ASTTypes.Symbol) {
      env.set(node.lhv.name, type);
    } else if (node.lhv.kind === ASTTypes.VectorPattern) {
      if (!Type.isVector(type) && !Type.isList(type) && !Type.isTuple(type)) {
        throw new TypeException(
          `Vector pattern destructuring must take a vector, list, or tuple type`,
          node.srcloc
        );
      } else {
        let i = 0;
        for (let mem of node.lhv.members) {
          if (Type.isVector(type) || Type.isList(type)) {
            if (node.lhv.rest && i === node.lhv.members.length - 1) {
              env.set(mem.name, type);
            } else {
              env.set(
                mem.name,
                type.vectorType ? type.vectorType : type.listType
              );
            }
          } else {
            // is tuple type
            if (node.lhv.rest && i === node.lhv.members.length - 1) {
              // is rest variable
              env.set(mem.name, type.types.slice(i));
            } else {
              env.set(mem.name, type.types[i]);
            }
          }
          i++;
        }
      }
    } // continue with RecordPattern case

Now let's add the checkConstantDeclaration method:

  checkConstantDeclaration(node, env) {
    return this.checkVariableDeclaration(node, env, true);
  }

And a method for checkAsExpression:

  checkAsExpression(node, env) {
    env.checkingOn = true;
    const type = infer(node, env);
    return { ...node.expression, type };
  }

We also need to modify checkSetExpression to make sure we're not trying to set a value for a constant, which is illegal:

  checkSetExpression(node, env) {
    if (node.lhv.kind !== ASTTypes.Symbol) {
      throw new TypeException(
        `Cannot use destructuring with set! assignment`,
        node.srcloc
      );
    }

    const nameType = env.get(node.lhv.name);

    if (Type.isSingleton(nameType)) {
      const exprType =
        node.expression.kind === ASTTypes.Symbol
          ? env.get(node.expression.name)
          : infer(node.expression, env);
      if (
        (Type.isSingleton(exprType) && exprType.value !== nameType.value) ||
        (isPrimitive(node.expression) &&
          node.expression.value !== nameType.value)
      ) {
        throw new TypeException(
          `Cannot assign different value to variable of singleton type ${Type.toString(
            nameType
          )}`,
          node.expression.srcloc
        );
      }
    }

    if (nameType.constant) {
      throw new TypeException(
        `Cannot assign to constant value ${node.lhv.name}`,
        node.srcloc
      );
    }

    if (env.checkingOn) {
      check(node.expression, nameType, env);
      return {
        kind: node.kind,
        lhv: node.lhv,
        expression: this.checkNode(node.expression, env),
        srcloc: node.srcloc,
        type: nameType,
      };
    }

    return {
      kind: node.kind,
      lhv: node.lhv,
      expression: this.checkNode(node.expression, env),
      srcloc: node.srcloc,
      type: infer(node, env),
    };
  }

Changes to The Visitor

Now we need to add methods to the Visitor class for constant declarations and as expressions in src/visitor/Visitor.js. In the switch expression in the visit method:

      // other cases...
      case ASTTypes.ConstantDeclaration:
        return this.visitConstantDeclaration(node);
      case ASTTypes.AsExpression:
        return this.visitAsExpression(node);
      // default case

Here's the visitConstantDeclaration method:

  visitConstantDeclaration(node) {
    const lhv = this.visit(node.lhv);
    const expression = this.visit(node.expression);
    return { ...node, lhv, expression };
  }

And here's visitAsExpression:

  visitAsExpression(node) {
    const expression = this.visit(node.expression);
    return { ...node, expression };
  }

Changes to The Desugarer

Since constant declarations desugar to variable declarations for the emitter, we need a method for that in the Desugarer class. It's pretty simple in src/desugarer/Desugarer.js:

  visitConstantDeclaration(node) {
    return { ...node, kind: ASTTypes.VariableDeclaration };
  }

In visitFunctionDeclaration I've also added a name property to the AST node output, since a function declaration will always have a name. This lets us recover the names for our functions since passing the function into rt.makeFunction in the emitter was knocking out the function name.

Just add this line somewhere below the definition of lambda in the visitFunctionDeclaration method:

lambda.name = node.name.name;

Changes to The Runtime

There's a new version of makeFunction in src/runtime/makeFunction.js to handle adding names to functions:

import objectHash from "object-hash";
import { curryN } from "ramda";
import { makeWandaValue } from "./conversion.js";
import { addMetaField } from "./object.js";
import { parseContract } from "./parseContract.js";

export const makeFunction = (func, { contract = "", name = "" } = {}) => {
  let fn = curryN(func.length, (...args) => {
    const val = makeWandaValue(func(...args));

    if (typeof val === "function") {
      return makeFunction(val);
    }

    return val;
  });
  const hash = objectHash(fn);

  addMetaField(fn, "wanda", true);
  addMetaField(fn, "arity", func.length);
  addMetaField(fn, "name", name || hash);

  if (contract !== "") {
    Object.defineProperty(fn, "contract", {
      enumerable: false,
      configurable: false,
      writable: false,
      value: parseContract(contract),
    });
  }

  Object.defineProperty(fn, "name", {
    enumerable: false,
    configurable: false,
    writable: false,
    value: name || hash,
  });

  return fn;
};

I've added some metadata in makeObject in src/runtime/object.js:

export const makeObject = (obj) => {
  let newObj = {};
  addMetaField(newObj, "dict", obj);
  addMetaField(newObj, "constructor", function (...args) {
    return new obj.constructor(...args);
  });
  addMetaField(newObj, "type", "object");
  addMetaField(
    newObj[makeKeyword("constructor")],
    "name",
    obj.constructor?.name ?? "WandaObject"
  );

  // to allow destructuring
  for (let [k, v] of Object.entries(obj)) {
    newObj[makeSymbol(k)] = v;
  }

  return newObj;
};

Finally, make sure you have imported the isNil function into src/runtime/makeRuntime.js and have it added to the runtime object:

import { isNullish as isNil } from "../shared/utils.js";

export const makeRuntime = () => {
  return {
    // other values...
    isNil,
  };
};

Changes to The Emitter

There's only one change to the emitter: handling re-adding the function name when it exists on the lambda node we're receiving from the desugarer in src/emitter/Emitter.js by passing it as a parameter to rt.makeFunction:

  emitLambdaExpression(node, ns) {
    const funcNs = ns.extend("Lambda");
    let params = [];
    let i = 0;

    for (let p of node.params) {
      funcNs.set(p.name.name, makeSymbol(p.name));

      if (node.variadic && i === node.params.length - 1) {
        params.push(`...${this.emit(p.name, funcNs)}`);
      } else {
        params.push(this.emit(p.name, funcNs));
      }
      i++;
    }

    let code = `rt.makeFunction((${params.join(", ")}) => {\n`;

    let j = 0;
    for (let expr of node.body) {
      if (j === node.body.length - 1) {
        code += `return ${this.emit(expr, funcNs)};`;
      } else {
        code += this.emit(expr, funcNs) + ";\n";
      }
      j++;
    }

    code += "\n}";
    code += `${node.name ? `, { name: "${node.name}" }` : ""})`;

    return code;
  }

Changes to Shared Types

I've added an isList function to src/shared/cons.js to check for if a Cons cell is a valid list (that is, it ends in null):

export const isList = (obj) => {
  if (obj != null && obj.constructor?.name !== "Cons") {
    return false;
  } else if (obj == null) {
    return true;
  }

  // only option left is cons
  return isList(obj.cdr);
};

I've used it in the printer, which you'll see below.

Changes to The Printer

In src/printer/printString.js I've rewritten printList using the isList function and some recursive magic to detect if the list is a proper list or degenerate. Add isList to your import from shared/cons.js and here's the new printList function:

const printList = (list) => {
  const printListElems = (next, str = "") => {
    if (next == null) {
      return str;
    } else if (next.constructor?.name === "Cons") {
      return printListElems(next.cdr, str + printString(next.car) + " ");
    } else {
      return str + ". " + printString(next);
    }
  };

  let prStr = printListElems(list);
  // if it's a list, get rid of the extra space at the end
  prStr = isList(list) ? prStr.slice(0, -1) : prStr;
  return "(" + prStr + ")";
};

We also need to print AST nodes for ConstantDeclaration and AsExpression nodes. First, add the cases to the switch statement in the print method:

      // other cases...
      case ASTTypes.ConstantDeclaration:
        return this.printConstantDeclaration(node, indent);
      case ASTTypes.AsExpression:
        return this.printAsExpression(node, indent);
      // default case

Here's printConstantDeclaration:

  printConstantDeclaration(node, indent) {
    let prStr = `${prIndent(indent)}ConstantDeclaration:\n`;
    prStr += `${this.print(node.lhv, indent + 2)}\n`;
    prStr += `${this.print(node.expression, indent + 2)}`;
    return prStr;
  }

And here's printAsExpression:

  printAsExpression(node, indent) {
    return this.print(node.expression, indent);
  }

Updating The Core Library

Finally, I've added names to all the functions in our core library. I've also fixed the append function for vectors, since there was a bug in it earlier.

Here's the entire updated version of lib/js/core.js:

import equal from "fast-deep-equal/es6/index.js";
import { makeModule } from "../../src/runtime/Module.js";
import { Cons, cons } from "../../src/shared/cons.js";
import { print } from "../../src/printer/print.js";
import { println } from "../../src/printer/println.js";
import { printString } from "../../src/printer/printString.js";
import { isTruthy, makeKeyword } from "../../src/runtime/utils.js";
import { Exception } from "../../src/shared/exceptions.js";
import { hasMetaField } from "../../src/runtime/object.js";

export const theModule = makeModule("Core", (rt, ns) => {
  const isList = (obj) => {
    if (!rt.isNil(obj) && !(obj instanceof Cons)) {
      return false;
    } else if (rt.isNil(obj)) {
      return true;
    }

    // only option left is cons
    return isList(obj.cdr);
  };

  return {
    print: rt.makeFunction(print, {
      contract: "(any -> string)",
      name: "print",
    }),
    println: rt.makeFunction(println, {
      contract: "(any -> string)",
      name: "println",
    }),
    cons: rt.makeFunction(cons, {
      contract: "(any, any -> (list any))",
      name: "cons",
    }),
    car: rt.makeFunction((pair) => pair.car, {
      contract: "((list any) -> any)",
      name: "car",
    }),
    cdr: rt.makeFunction((pair) => pair.cdr, {
      contract: "((list any) -> any)",
      name: "cdr",
    }),
    string: rt.makeFunction(printString, {
      contract: "(any -> string)",
      name: "string",
    }),
    number: rt.makeFunction(Number, {
      contract: "(string -> number)",
      name: "number",
    }),
    boolean: rt.makeFunction((val) => isTruthy(val), {
      contract: "(any -> boolean)",
      name: "boolean",
    }),
    keyword: rt.makeFunction((str) => Symbol.for(":" + str), {
      contract: "(string -> keyword)",
      name: "keyword",
    }),
    "+": rt.makeFunction(
      (a, b, ...nums) => nums.reduce((sum, n) => sum + n, a + b),
      { contract: "(number, number, &(vector number) -> number)", name: "+" }
    ),
    "-": rt.makeFunction(
      (a, b, ...nums) => nums.reduce((diff, n) => diff - n, a - b),
      { contract: "(number, number, &(vector number) -> number)", name: "-" }
    ),
    "*": rt.makeFunction(
      (a, b, ...nums) => nums.reduce((prod, n) => prod * n, a * b),
      { contract: "(number, number, &(vector number) -> number)", name: "*" }
    ),
    "/": rt.makeFunction(
      (a, b, ...nums) => nums.reduce((quot, n) => quot / n, a / b),
      { contract: "(number, number, &(vector number) -> number)", name: "/" }
    ),
    "%": rt.makeFunction(
      (a, b, ...nums) => nums.reduce((quot, n) => quot % n, a % b),
      { contract: "(number, number, &(vector number) -> number)", name: "%" }
    ),
    "=": rt.makeFunction((a, b) => a === b, {
      contract: "(number, number -> boolean)",
      name: "=",
    }),
    ">": rt.makeFunction((a, b) => a > b, {
      contract: "(number, number -> boolean)",
      name: ">",
    }),
    ">=": rt.makeFunction((a, b) => a >= b, {
      contract: "(number, number -> boolean)",
      name: ">=",
    }),
    "<": rt.makeFunction((a, b) => a < b, {
      contract: "(number, number -> boolean)",
      name: "<",
    }),
    "<=": rt.makeFunction((a, b) => a <= b, {
      contract: "(number, number -> boolean)",
      name: "<=",
    }),
    not: rt.makeFunction((x) => !x, {
      contract: "(any -> boolean)",
      name: "not",
    }),
    list: rt.makeFunction((...args) => Cons.from(args), {
      contract: "(&(vector any) -> (list any))",
      name: "list",
    }),
    length: rt.makeFunction(
      (obj) => {
        if (obj instanceof Cons) {
          let i = 0;
          for (let _ of obj) {
            i++;
          }
          return i;
        }

        return obj.length;
      },
      { contract: "((list any) -> number)", name: "length" }
    ),
    get: rt.makeFunction(
      (n, obj) => {
        const value = obj.get(n);

        if (value === undefined) {
          throw new Exception(`Value for index ${n} not found on object`);
        }

        return value;
      },
      { contract: "((list any) -> any)", name: "get" }
    ),
    "list?": rt.makeFunction(isList, {
      contract: "((list any) -> boolean)",
      name: "list?",
    }),
    "pair?": rt.makeFunction((obj) => obj instanceof Cons, {
      contract: "((list any) -> boolean)",
      name: "pair?",
    }),
    "number?": rt.makeFunction((obj) => typeof obj === "number", {
      contract: "(any -> boolean)",
      name: "number?",
    }),
    "string?": rt.makeFunction((obj) => typeof obj === "string", {
      contract: "(any -> boolean)",
      name: "string?",
    }),
    "boolean?": rt.makeFunction((obj) => typeof obj === "boolean", {
      contract: "(any -> boolean)",
      name: "boolean?",
    }),
    "nil?": rt.makeFunction((obj) => obj == null, {
      contract: "(any -> boolean)",
      name: "nil?",
    }),
    "keyword?": rt.makeFunction(
      (obj) => typeof obj === "symbol" && obj.description.startsWith(":"),
      { contract: "(any -> boolean)", name: "keyword?" }
    ),
    "equal?": rt.makeFunction(
      (a, b) => {
        if (rt.hasDict(a) && rt.hasDict(b)) {
          return equal(rt.getMetaField(a, "dict"), rt.getMetaField(b, "dict"));
        }
        return equal(a, b);
      },
      { contract: "(any, any) -> boolean", name: "equal?" }
    ),
    "is?": rt.makeFunction((a, b) => Object.is(a, b), {
      contract: "(any, any -> boolean)",
      name: "is?",
    }),
    append: rt.makeFunction(
      (obj1, obj2) => {
        if (typeof obj1 === "string" && typeof obj2 === "string") {
          return obj1 + obj2;
        } else if (obj1 instanceof Cons) {
          return Cons.from(obj1).append(obj2);
        } else if (Array.isArray(obj1)) {
          let v = [...obj1];
          v.push(obj2);
          return v;
        } else {
          rt.failRuntime(`Value of type ${typeof obj1} cannot be appended`);
        }
      },
      { contract: "(any, any -> any)", name: "append" }
    ),
    with: rt.makeFunction(
      (rec1, rec2) => {
        const newDict = Object.assign(
          {},
          rt.getMetaField(rec1, "dict"),
          rt.getMetaField(rec2, "dict")
        );
        return rt.makeObject(newDict);
      },
      { contract: "(any, any -> any)", name: "with" }
    ),
    prop: rt.makeFunction((prop, obj) => rt.getField(obj, prop), {
      contract: "(string, any -> any)",
      name: "prop",
    }),
    each: rt.makeFunction(
      (fn, lst) => {
        for (let item of lst) {
          fn(item);
        }
      },
      {
        contract: "((any -> nil), (list any) -> nil)",
        name: "each",
      }
    ),
    map: rt.makeFunction(
      (fn, lst) => {
        let mapped = [];
        for (let item of lst) {
          mapped.push(fn(item));
        }
        return Cons.from(mapped);
      },
      {
        contract: "((any -> any), (list any) -> (list any))",
        name: "map",
      }
    ),
    filter: rt.makeFunction(
      (fn, lst) => {
        let filtered = [];
        for (let item of lst) {
          if (rt.isTruthy(fn(item))) {
            filtered.push(item);
          }
        }
        return Cons.from(filtered);
      },
      {
        contract: "((any -> boolean), (list any) -> (list any))",
        name: "filter",
      }
    ),
    fold: rt.makeFunction(
      (fn, init, lst) => {
        let acc = init;
        for (let item of lst) {
          acc = fn(acc, item);
        }
        return acc;
      },
      {
        contract: "((any, any -> any), any, (list any) -> any)",
        name: "fold",
      }
    ),
    "fold-r": rt.makeFunction(
      (fn, init, lst) => {
        let acc = init;
        const reversed = [...lst].reverse();
        for (let item of reversed) {
          acc = fn(acc, item);
        }
        return acc;
      },
      {
        contract: "((any, any -> any), any, (list any) -> any)",
        name: "fold-r",
      }
    ),
    typeof: rt.makeFunction((obj) => {
      if (obj == null) {
        return "nil";
      }

      if (hasMetaField(obj, "type")) {
        return obj[makeKeyword("type")];
      }

      if (obj instanceof Cons) {
        return isList(obj) ? "list" : "pair";
      }

      if (Array.isArray(obj)) {
        return "vector";
      }

      return typeof obj;
    }),
  };
});

Trying Out The New Functionality

Here's an example of how to use the new type. First, enter the wanda command, then at the prompt:

(type Coord ({ type: "cartesian", x: number, y: number } || { type: "polar", angle: number, magnitude: number }))
(var (c: Coord) { type: "cartesian", x: 42, y: 17.5 })

Here's an example of an intersection type:

(type Point2D { x: number, y: number })
(type Point3D (Point2D && { z: number }))

Have fun with it!

Conclusion

I hope that was as much fun for you as it was for me! Now we've greatly increased the expressivity of our type system.

As always, you can check out the code for this post at the relevant tag at the GitHub repo.

Next time we're going to add branching with if expressions, which will also allow us to perform type narrowing in the checker so we can do better things with union types. For instance, if you have a function that takes the type number || string you can define branches of the function that work with it in each case in a type safe manner.

I'll see you there!

Create Your Own Programming Language 6: Functions

Jason Barr — Tue, 20 Jun 2023 03:30:42 +0000

Welcome to the next installment in the Create Your Own Programming Language series! In this article we're going to add functions to the Wanda language.

As always, if you haven't read the previous article in the series on vectors and records go ahead and do that first.

Ok, let's go!

What We're Going To Do

The most important thing we're going to do today is add functions to our language. Functions are extremely important. In fact, when you have functions you can literally do anything with your language. You might not believe it, but literally everything in your programming language can be functions if you're dedicated enough.

That's one of my all-time favorite computer science talks and I strongly recommend you take the time to watch it (it's split into 2 parts in the playlist).

Adding functions is the most important thing we're doing today, but we'll also do a few other things to support the addition of functions to our language.

First, we'll improve our type annotations and how we parse them. We hadn't really thought through how type annotations were supposed to work, so we'll solidify that and implement a more uniform method of parsing them.

We'll also add desugaring for the first time.

As mentioned before, "desugaring" means converting more complex language forms into simpler core forms that make things easier for the compiler to handle.

To do that, we'll create a base Visitor from which the desugarer will inherit so we only have to implement the methods we need on the Desugarer class. We're going to make it so that function declarations, which use the keyword def, desugar to the var variable declaration form with a Lambda Expression as the value being assigned.

We're also going to make it so that all functions created in the language are curried. If you're not familiar with curried functions, a curried function is one that can take in one argument at a time. If you partially apply a curried function, i.e. give it fewer arguments than it needs to execute, you get back another function that only needs the remaining arguments before it will execute. That means we'll need to make some changes to how call expressions are checked in the type checker.

We're also going to allow functions to be variadic, which is just a fancy way of saying you can allow a function to take any number of arguments.

Finally, we'll add contracts for the functions in our core library. A contract is basically a type signature. It's a promise that this function will receive these types of arguments and produce that type of return value.

There are no changes to the lexer and reader for this article, so we'll start with the parser.

Changes To The Parser

We need to do 2 things with the parser: enable it to parse functions and fix how we handle type annotations. Let's do the latter first.

Fixing Type Annotations

Before now, it's been possible for type annotations to be 2 things: either a primitive or named (Symbol) value type (i.e. number) or what we could call a "generic" type (even though we don't have generics yet) where you have a main type and that type has one or more types it contains.

The two kinds of "generic" types we've used so far are list and vector types. We've been writing them like list (number).

This actually complicates parsing type annotations quite a bit, because we could have to parse 2 things from the list of forms or only 1 thing depending on which kind of annotation it is. I don't want to have that kind of complexity in parsing type annotations, so I'm going to change how we handle generic type annotations.

From now on, instead of writing list (number) we're going to use (list number). That way regardless of whether it's a named type or a generic type the parser only has to handle a single form to parse the type annotation.

When we get generic types in full, we'll continue the same scheme so, i.e., a Result type that can be either a string or an Exception will get the annotation (Result string Exception). If you're not familiar with Result types, don't worry about it for now; it's just an example of a generic type that has more than one type parameter.

We're also going to add type annotations for functions, which will be written like (argtype1 argtype2 ... -> returnType). Again, the type annotation parser only has to handle one form from the reader because the entire annotation will be inside a cons list.

For variadic functions with type annotations, we'll use the & operator and make the rest parameter a vector.

A variadic function annotation could look something like this:

(number &(vector number) -> number)

This will greatly streamline parsing when it comes to handling type annotations, which will simplify things like parsing function parameters.

Parsing Type Annotations

Let's take a look at src/parser/parseTypeAnnotation.js. We need to check if the annotation is a Cons or a Symbol. If it's a Cons we'll flatten it into an array.

Next we need to see if it's a function or generic annotation. If it's a function annotation it will have an arrow. If it has an arrow, we'll handle it as a function annotation.

We'll filter the arrow out because once we know it's a function annotation the arrow has done its job. We'll parse the last item in the array as the return type and all the rest as parameter types. Then construct the annotation and it's ready to go.

We've also changed the way we handle a nil type annotation. I found during further testing that the reader passes nil back as a Nil token, not as a Symbol token.

Everything else stays the same as it was: if the annotation was generic we get the first item in the array as the container type then handle the 2nd item as the contained type. Otherwise we handle the annotation as a named type like number or a type alias.

export const parseTypeAnnotation = (annotation) => {
  if (annotation instanceof Cons) {
    // is function or generic annotation
    // flatten Cons to array
    annotation = [...annotation];

    // if it has an arrow, it's a function annotation
    const hasArrow = annotation.reduce((hasArrow, item) => {
      if (item.type === TokenTypes.Symbol && item.value === "->") {
        return true;
      }
      return hasArrow;
    }, false);

    if (hasArrow) {
      // is function annotation
      // filter out arrow
      annotation = annotation.filter((item) => item.value !== "->");

      // get return type
      const retType = parseTypeAnnotation(annotation.pop());
      // get param types and if it's variadic
      let params = [];
      let variadic = false;

      for (let item of annotation) {
        if (item.type === TokenTypes.Amp) {
          variadic = true;
          continue;
        } else {
          params.push(parseTypeAnnotation(item));
        }
      }

      return { kind: TATypes.Function, params, retType, variadic };
    }
  }

  let annot;
  if (Array.isArray(annotation)) {
    // is generic annotation
    // get container type
    annot = annotation[0];
  } else {
    // is simple annotation
    annot = annotation;
  }

  if (annot.type === "RecordLiteral") {
    return parseObjectAnnotation(annot);
  }

  if (annot.type === TokenTypes.Nil) {
    return { kind: TATypes.NilLiteral };
  }

  if (annot.type === TokenTypes.Symbol) {
    switch (annot.value) {
      case "any":
        return { kind: TATypes.AnyLiteral };
      case "number":
        return { kind: TATypes.NumberLiteral };
      case "string":
        return { kind: TATypes.StringLiteral };
      case "boolean":
        return { kind: TATypes.BooleanLiteral };
      case "keyword":
        return { kind: TATypes.KeywordLiteral };
      case "list":
        // annotation is array with listType as 2nd member
        return parseListAnnotation(annotation[1]);
      case "vector":
        return parseVectorAnnotation(annotation[1]);
      default:
        // must be a named type
        return { kind: TATypes.Symbol, name: annot.value };
    }
  }

  throw new Exception(
    `Unknown type annotation kind ${JSON.stringify(annot, null, 2)}`
  );
};

Parsing Functions

First, let's add AST nodes for the new forms: FunctionDeclaration and LambdaExpression.

We'll add 3 members (including one for function parameters) to the ASTTypes enum in src/parser/ast.js:

export const ASTTypes = {
  // other nodes
  Param: "Param",
  FunctionDeclaration: "FunctionDeclaration",
  LambdaExpression: "LambdaExpression",
};

And we'll add constructors for the nodes:

export const AST = {
  // other constructors...
  FunctionDeclaration(name, params, body, variadic, retType, srcloc) {
    return {
      kind: ASTTypes.FunctionDeclaration,
      name,
      params,
      body,
      variadic,
      retType,
      srcloc,
    };
  },

  LambdaExpression(params, body, variadic, retType, srcloc) {
    return {
      kind: ASTTypes.LambdaExpression,
      params,
      body,
      variadic,
      retType,
      srcloc,
    };
  },
};

There are 2 new nodes, so 2 different function forms we need to parse: function declarations and lambda expressions. Function declarations start with the def symbol and lambdas start with fn.

Let's add those cases to the switch statement in parseList in src/parser/parse.js:

    // other cases...
    case "def":
      return parseFunctionDeclaration(form);
    case "fn":
      return parseLambdaExpression(form);
    // default case

We just delegate to the parser functions for these nodes, which are very similar. The biggest difference is that parseFunctionDeclaration needs to handle the name of the function. We can delegate most of the work to a single parseFunction function. We'll also handle parsing parameters with a separate parseParams function since that will be a little bit of work.

Both function declarations and lambda expressions can have annotations both for parameter types and return types, so we'll need to handle both the cases where these are included and when they are omitted.

We'll use "maybe" variable names for the relevant parts of a function form.

Here are parseFunctionDeclaration and parseLambdaExpression:

const parseFunctionDeclaration = (form) => {
  const [_, name, params, maybeArrow, maybeRetType, ...maybeBody] = form;
  const srcloc = form.srcloc;
  const parsedName = parseExpr(name);
  const { parsedParams, parsedBody, variadic, retType } = parseFunction(
    params,
    maybeArrow,
    maybeRetType,
    maybeBody
  );

  return AST.FunctionDeclaration(
    parsedName,
    parsedParams,
    parsedBody,
    variadic,
    retType,
    srcloc
  );
};

const parseLambdaExpression = (form) => {
  const [_, params, maybeArrow, maybeRetType, ...maybeBody] = form;
  const srcloc = form.srcloc;
  const { parsedParams, parsedBody, variadic, retType } = parseFunction(
    params,
    maybeArrow,
    maybeRetType,
    maybeBody
  );

  return AST.LambdaExpression(
    parsedParams,
    parsedBody,
    variadic,
    retType,
    srcloc
  );
};

Then parseFunction handles most of the real work:

const parseFunction = (params, maybeArrow, maybeRetType, maybeBody) => {
  let retType, body;

  if (maybeArrow.type === TokenTypes.Symbol && maybeArrow.value === "->") {
    // has return type annotation
    retType = parseTypeAnnotation(maybeRetType);
    body = maybeBody;
  } else {
    retType = null;
    body = maybeRetType
      ? [maybeArrow, maybeRetType, ...maybeBody]
      : [maybeArrow, ...maybeBody];
  }

  const variadic = [...params].reduce((isVar, param) => {
    if (param.type === TokenTypes.Amp) {
      return true;
    }

    return isVar;
  }, false);
  const parsedParams = parseParams(params);
  /** @type {AST[]} */
  const parsedBody = body.map(parseExpr);

  return {
    parsedParams,
    parsedBody,
    variadic,
    retType,
  };
};

And parseParams uses a for loop over the parameter forms and handles the cases where they do and do not have type annotations:

const parseParams = (forms) => {
  forms = [...forms];
  /** @type {import("./ast.js").Param[]} */
  let params = [];
  for (let i = 0; i < forms.length; i++) {
    const form = forms[i];
    if (form.type === TokenTypes.Symbol) {
      const name = parseExpr(form);
      let typeAnnotation = null;

      if (
        forms[i + 1]?.type === TokenTypes.Keyword &&
        forms[i + 1].value === ":"
      ) {
        // has type annotation
        typeAnnotation = parseTypeAnnotation(forms[i + 2]);
        i += 2;
      }

      params.push({ kind: ASTTypes.Param, name, typeAnnotation });
    } else if (form.type === TokenTypes.Amp) {
      continue;
    }
  }

  return params;
};

See how simple it is now that we only have to worry about a single form for a type annotation? This would be much more complicated to parse if we'd kept the old list (number) type of annotation.

We also need to make a slight edit to parseVariableDeclaration to handle the new type annotation paradigm:

const parseVariableDeclaration = (decl) => {
  let [_, lhv, expression] = decl;

  let parsedLhv,
    typeAnnotation = null;
  if (lhv instanceof Cons) {
    // has type annotation
    const realLhv = lhv.get(0);
    // skip over : and get type annotation
    typeAnnotation = lhv.cdr.cdr;

    if (typeAnnotation.cdr === null) {
      // is a simple annotation, otherwise it's a Cons type annotation
      typeAnnotation = typeAnnotation.car;
    }
    // parse type annotation
    typeAnnotation = parseTypeAnnotation(typeAnnotation);
    parsedLhv = parseExpr(realLhv);
  } else {
    parsedLhv = parseExpr(lhv);
  }

  if (parsedLhv.kind === ASTTypes.VectorLiteral) {
    parsedLhv = convertVectorLiteralToVectorPattern(parsedLhv);
  }

  const parsedExpression = parseExpr(expression);

  return AST.VariableDeclaration(
    parsedLhv,
    parsedExpression,
    decl.srcloc,
    typeAnnotation
  );
};

Changes to The Type Checker

Adding functions to our language presents a particular problem for the type checker: recursive functions won't have a type inferred for the return type. Even with type annotations, the function's name won't be set yet in the type environment because its body hasn't finished being processed yet.

That also means you can't define mutually recursive functions. Trying to call the 2nd function in the body of the 1st function will throw an error because the type of the 2nd function hasn't been defined yet.

To handle this issue, we're going to implement 2 passes in the type checker.

You'll see that when we get to the changes to the TypeChecker class, but first let's implement the machinery we need to check function types.

As a nod towards handling the issue with the function name not being set in the environment, we're going to add a special internal type to the type checker called undefined.

You'll remember that we actually already have function types because we needed them to implement call expressions a few articles back.

Add the undefined case to the TypeTypes enum in src/typechecker/types.js:

export const TypeTypes = {
  // other members
  Undefined: "Undefined",
};

Also add a constructor in src/typechecker/constructors.js:

export const undefinedType = { kind: TypeTypes.Undefined };

And a validator in src/typechecker/validators.js:

export const isUndefined = (type) => {
  return type.kind === TypeTypes.Undefined;
};

A type should only ever be undefined during the first pass of the type checker, and we need undefined to always pass a subtype check so we'll treat it similarly to any in src/typechecker/isSubtype.js:

  // other cases
  if (Type.isUndefined(type1) || Type.isUndefined(type2)) return true;
  // rest of function

We'll also need to handle subtyping functions. A function is a subtype of another function if all the parameters of one are subtypes of all the parameters of the other and the return type of the first is a subtype of the return type of the second.

  // undefined case
  if (Type.isFunctionType(type1) && Type.isFunctionType(type2)) {
    return (
      type1.params.length === type2.params.length &&
      type1.params.every((a, i) => isSubtype(type2.params[i], a)) &&
      isSubtype(type1.ret, type2.ret)
    );
  }
  // rest of function

We also need to be able to parse a function type from a type annotation in fromTypeAnnotation, in src/typechecker/fromTypeAnnotation.js:

    // other cases
    case TATypes.Function: {
      const paramTypes = typeAnnotation.params.map((p) =>
        fromTypeAnnotation(p, typeEnv)
      );
      const retType = fromTypeAnnotation(typeAnnotation.retType, typeEnv);
      return Type.functionType(paramTypes, retType, typeAnnotation.variadic);
    }
    // default case

We need to be able to infer types for functions. We can use the same inferFunction function to handle both function declarations and lambda expressions, since the type is what matters here and not the function name.

In src/typechecker/infer.js we add cases for LambdaExpression and FunctionDeclaration to the switch statement in the infer function:

    // other cases
    case ASTTypes.LambdaExpression:
      return inferFunction(ast, env);
    case ASTTypes.FunctionDeclaration:
      return inferFunction(ast, env);
    // default case

Note that in inferFunction the environment that will be passed into infer will be the extended function environment, not the global environment.

We map over the function node's parameters. If the parameters have type annotations, we turn checking on.

If the parameters have type annotations we construct the type from the annotation; otherwise we set them to any.

We set each parameter name with its type in the environment and return the type from the map callback to get an array of types for the parameters.

If the node has an annotated return type we make sure type checking is on.

If there's a return type annotation we construct that type, otherwise we assign any. Then we loop over the function body and infer the return type from the function's body using the extended function environment.

If the inferred body type is a subtype of the annotated return type, everything is good and we construct and return the function type.

If the function wasn't annotated (in which case its return annotation type is any) or has been inferred as undefined, we use the inferred return type. Otherwise, if there was a function annotation, we use the return annotation type.

Here's inferFunction:

const inferFunction = (node, env) => {
  const params = node.params.map((p) => {
    if (p.typeAnnotation) {
      env.checkingOn = true;
    }
    const type = p.typeAnnotation
      ? fromTypeAnnotation(p.typeAnnotation, env)
      : Type.any;
    env.set(p.name.name, type);
    return type;
  });

  if (node.retType) {
    env.checkingOn = true;
  }

  const retType = node.retType
    ? fromTypeAnnotation(node.retType, env)
    : Type.any;
  let inferredRetType;

  for (let expr of node.body) {
    // type of last expression "wins"
    inferredRetType = infer(expr, env);
  }

  if (env.checkingOn && !isSubtype(inferredRetType, retType)) {
    throw new TypeException(
      `Inferred return type ${Type.toString(
        inferredRetType
      )} is not a subtype of annotated return type ${Type.toString(retType)}`,
      node.srcloc
    );
  }

  return Type.functionType(
    params,
    Type.isAny(retType) || Type.isUndefined(retType)
      ? inferredRetType
      : retType,
    node.variadic
  );
};

We also need to make some changes to inferCallExpression.

Since we now will have curried functions that can be partially applied, we don't need to throw an error if the call expression has fewer arguments than the type of the function. Instead, we'll construct a new function type using the remaining parameter types and the function's return type and return that.

Let's also do some refactoring and delegate the actual argument checking to a function called checkArgTypes.

Here's the new version of inferCallExpression with checkArgTypes:

const inferCallExpression = (node, env) => {
  const func = infer(node.func, env);

  if (Type.isAny(func)) {
    return Type.any;
  } else if (Type.isUndefined(func) || Type.isUndefined(func.ret)) {
    // this should only happen during first typechecker pass
    return Type.undefinedType;
  }

  if (!func.variadic && node.args.length > func.params.length) {
    throw new TypeException(
      `Too many arguments for function: ${node.args.length} given; ${func.params.length} expected`,
      node.srcloc
    );
  }

  // handle partially applied functions
  if (
    node.args.length <
    (func.variadic ? func.params.length - 1 : func.params.length)
  ) {
    // is partially applied
    const params = func.params.slice(0, node.args.length);

    if (env.checkingOn) {
      checkArgTypes(node, params, env, func);
    }

    const newParams = func.params.slice(node.args.length);
    return Type.functionType(newParams, func.ret, func.variadic);
  }

  if (env.checkingOn) {
    checkArgTypes(node, func.params, env, func);
  }

  return func.ret;
};

const checkArgTypes = (node, params, env, func) => {
  node.args.forEach((arg, i) => {
    const argType = infer(arg, env);

    if (func.variadic && i >= params.length - 1) {
      // is part of rest args
      let p = params[params.length - 1];
      if (!p.vectorType) {
        throw new TypeException(
          `Rest parameter type must be vector; ${Type.toString(p)} given`,
          arg.srcloc
        );
      }
      p = p.vectorType;
      if (!isSubtype(argType, p)) {
        throw new TypeException(
          `${Type.toString(argType)} is not a subtype of ${Type.toString(p)}`,
          arg.srcloc
        );
      }
    } else {
      const p = params[i];
      if (!isSubtype(argType, p)) {
        throw new TypeException(
          `${Type.toString(argType)} is not a subtype of ${Type.toString(p)}`,
          arg.srcloc
        );
      }
    }
  });
};

Note that checkArgTypes has to handle the case of a variadic function where the call expression can have more arguments than the number of parameters in the function type.

In src/typechecker/check.js we need a case for function types. Add this check to check above const inferredType = infer(ast, env);:

  if (
    (ast.kind === ASTTypes.FunctionDeclaration ||
      ast.kind === ASTTypes.LambdaExpression) &&
    Type.isFunctionType(type)
  ) {
    return checkFunction(ast, type, env);
  }

checkFunction checks the type of each parameter in the function type against each parameter in the AST node, then checks the return types. Here's checkFunction:

const checkFunction = (ast, type, env) => {
  if (type.params.length !== ast.params.length) {
    throw new TypeException(
      `Expected ${type.params.length} args; got ${ast.params.length}`,
      ast.srcloc
    );
  }

  const maybeType = env.get(ast.name?.name);
  const funcType = maybeType ? maybeType : infer(ast, env);

  type.params.forEach((p, i) => {
    const pType = funcType.params[i];
    if (!isSubtype(pType, p)) {
      throw new TypeException(
        `${Type.toString(pType)} is not a valid subtype of ${Type.toString(p)}`,
        ast.srcloc
      );
    }
  });

  if (!isSubtype(funcType.ret, type.ret)) {
    throw new TypeException(
      `${Type.toString(funcType.ret)} is not a valid subtype of ${Type.toString(
        type.ret
      )}`,
      ast.srcloc
    );
  }
};

We already have a string conversion for function types, but we also need one for undefined.

In the switch statement in src/typechecker/typeToString.js:

    // other cases
    case TypeTypes.Undefined:
      return "undefined";
    // default case

The biggest changes to the type checker will be in the TypeChecker class as we implement a two-pass typechecker.

The first pass is just to populate the environment, including getting all function names added to the environment (even if the function types return undefined).

Then the 2nd pass actually infers and checks all types now that the environment has been set up.

First, at the top of the src/typechecker/TypeChecker.js file (above the TypeChecker class declaration), add this:

let isSecondPass = false;

Next, rename the check method on the TypeChecker class to checkNode.

If you're using VS Code you can do this easily by highlighting the name of the check method and pressing F2. Other IDEs should have similar refactoring capabilities, so you shouldn't have to search and replace the text.

Now add a new check method that implements the 2-pass checking:

  check(env = this.env) {
    // first pass is to populate environments so valid forward references will resolve
    const firstPassProgram = this.checkNode(this.program, env);
    isSecondPass = true;
    return this.checkNode(firstPassProgram, env);
  }

To keep track of all the environments, we're actually going to add the extended environments to the nodes they're there for. That means with DoExpression, FunctionDeclaration, and LambdaExpression nodes we attach the extended environment to the node itself.

Here's the edited version of checkDoExpression:

  checkDoExpression(node, env) {
    if (!node.env) {
      // during the first pass this will create the child env
      node.env = env.extend("DoExpression");
    }

    const doEnv = node.env;

    /** @type {TypedAST[]} */
    let body = [];
    for (let expr of node.body) {
      const node = this.checkNode(expr, doEnv);
      body.push(node);
    }

    return {
      kind: node.kind,
      body,
      srcloc: node.srcloc,
      type: infer(node, doEnv),
    };
  }

Next let's add the checkFunctionDeclaration method. Similarly to checkDoExpression, we check the node for the existence of an env property and, if we don't find it, we add it.

Next we infer the type of the function using the new function environment.

We have a nested environment, so if checking has been turned on in the nested environment we need to be sure to propagate that change to the parent environment. Also, if checking is on, we need to check the node.

Finally, we set the function type to its name in the parent environment and return the type-annotated AST node.

  checkFunctionDeclaration(node, env) {
    if (!node.env) {
      node.env = env.extend(node.name.name);
    }

    const funcEnv = node.env;
    const type = infer(node, funcEnv);

    if (funcEnv.checkingOn && isSecondPass) {
      env.checkingOn = funcEnv.checkingOn;
      check(node, type, funcEnv);
    }

    env.set(node.name.name, type);
    return { ...node, type };
  }

checkLambdaExpression is almost exactly the same except without setting the function type to its name in the environment:

  checkLambdaExpression(node, env) {
    if (!node.env) {
      node.env = env.extend("Lambda");
    }

    const funcEnv = node.env;
    const type = infer(node, funcEnv);

    if (funcEnv.checkingOn) {
      env.checkingOn = funcEnv.checkingOn;
      check(node, type, funcEnv);
    }

    return { ...node, type };
  }

We also need to make sure in checkCallExpression that we don't return undefined as the type from a function on the 2nd checker pass:

  checkCallExpression(node, env) {
    // infer handles checking for argument types
    let type = infer(node, env);

    if (Type.isUndefined(type) && isSecondPass) {
      type = Type.any;
    }

    return { ...node, type };
  }

That shouldn't ever happen, but just in case it does we'll handle it there.

Finally, we need to handle the new two-pass checking in checkSymbol:

  checkSymbol(node, env) {
    try {
      let type = infer(node, env);

      if (Type.isUndefined(type)) {
        // this should never happen on the 2nd pass
        type = Type.any;
        env.set(node.name, type);
      }

      return { ...node, type };
    } catch (e) {
      if (!isSecondPass) {
        env.set(node.name, Type.undefinedType);
      }
    }
  }

Adding a Generic Visitor

You know by now that for compiler passes over the AST we've been using the Visitor Pattern to execute a postorder traversal over all the nodes of the syntax tree.

We haven't been able to do this until now, because we've needed compile-time environments in both the traversal passes we've implemented thus far, but now we can actually implement a generic Visitor with a method for each node that we use to transform and return new versions of the syntax tree.

We're going to use it for desugaring the typechecked AST into some core forms used by the emitter, which will allow us to simplify the emitter and not have to implement emitter methods for every kind of AST node.

The generic Visitor simply takes in a node with each method, visits any child nodes, and then returns a node exactly like the original.

Then in the Desugarer class we only have to implement the methods we need.

Here's the Visitor class in src/visitor/Visitor.js:

import { AST, ASTTypes } from "../parser/ast.js";
import { SyntaxException } from "../shared/exceptions.js";

export class Visitor {
  constructor(program) {
    this.program = program;
  }

  static new(program) {
    return new Visitor(program);
  }

  visit(node = this.program) {
    switch (node.kind) {
      case ASTTypes.Program:
        return this.visitProgram(node);
      case ASTTypes.NumberLiteral:
        return this.visitNumberLiteral(node);
      case ASTTypes.StringLiteral:
        return this.visitStringLiteral(node);
      case ASTTypes.BooleanLiteral:
        return this.visitBooleanLiteral(node);
      case ASTTypes.KeywordLiteral:
        return this.visitKeywordLiteral(node);
      case ASTTypes.NilLiteral:
        return this.visitNilLiteral(node);
      case ASTTypes.Symbol:
        return this.visitSymbol(node);
      case ASTTypes.CallExpression:
        return this.visitCallExpression(node);
      case ASTTypes.VariableDeclaration:
        return this.visitVariableDeclaration(node);
      case ASTTypes.SetExpression:
        return this.visitSetExpression(node);
      case ASTTypes.DoExpression:
        return this.visitDoExpression(node);
      case ASTTypes.TypeAlias:
        return this.visitTypeAlias(node);
      case ASTTypes.VectorLiteral:
        return this.visitVectorLiteral(node);
      case ASTTypes.VectorPattern:
        return this.visitVectorPattern(node);
      case ASTTypes.RecordLiteral:
        return this.visitRecordLiteral(node);
      case ASTTypes.RecordPattern:
        return this.visitRecordPattern(node);
      case ASTTypes.MemberExpression:
        return this.visitMemberExpression(node);
      case ASTTypes.Param:
        return this.visitParam(node);
      case ASTTypes.FunctionDeclaration:
        return this.visitFunctionDeclaration(node);
      case ASTTypes.LambdaExpression:
        return this.visitLambdaExpression(node);
      default:
        throw new SyntaxException(node.kind, node.srcloc);
    }
  }

  visitBooleanLiteral(node) {
    return node;
  }

  visitCallExpression(node) {
    const func = this.visit(node.func);
    const args = node.args.map(this.visit.bind(this));
    return { ...node, func, args };
  }

  visitDoExpression(node) {
    let body = [];

    for (let n of node.body) {
      body.push(this.visit(n));
    }

    return { ...node, body };
  }

  visitFunctionDeclaration(node) {
    const params = node.params.map(this.visit.bind(this));
    let body = [];

    for (let expr of node.body) {
      body.push(this.visit(expr));
    }

    return { ...node, params, body };
  }

  visitKeywordLiteral(node) {
    return node;
  }

  visitLambdaExpression(node) {
    const params = node.params.map(this.visit.bind(this));
    let body = [];

    for (let expr of node.body) {
      body.push(this.visit(expr));
    }

    return { ...node, params, body };
  }

  visitMemberExpression(node) {
    const object = this.visit(node.object);
    const property = this.visit(node.property);
    return { ...node, object, property };
  }

  visitNilLiteral(node) {
    return node;
  }

  visitNumberLiteral(node) {
    return node;
  }

  visitParam(node) {
    return node;
  }

  visitProgram(node) {
    let body = [];
    for (let n of node.body) {
      body.push(this.visit(n));
    }

    return { ...node, body };
  }

  visitRecordLiteral(node) {
    return node;
  }

  visitRecordPattern(node) {
    return node;
  }

  visitSetExpression(node) {
    const lhv = this.visit(node.lhv);
    const expression = this.visit(node.expression);
    return { ...node, lhv, expression };
  }

  visitStringLiteral(node) {
    return node;
  }

  visitSymbol(node) {
    return node;
  }

  visitTypeAlias(node) {
    return node;
  }

  visitVariableDeclaration(node) {
    const lhv = this.visit(node.lhv);
    const expression = this.visit(node.expression);
    return { ...node, lhv, expression };
  }

  visitVectorLiteral(node) {
    return node;
  }

  visitVectorPattern(node) {
    return node;
  }
}

Note that some methods map over arrays of nodes, which means they need to use the bind method on the method they pass into map.

Now it's time to desugar FunctionDeclaration nodes into VariableDeclaration nodes with LambdaExpression values.

We'll create a new file src/desugarer/Desugarer.js and stub out the Desugarer class:

import { AST } from "../parser/ast.js";
import { TATypes } from "../parser/parseTypeAnnotation.js";
import { Visitor } from "../visitor/Visitor.js";

export class Desugarer extends Visitor {
  constructor(program) {
    super(program);
  }

  static new(program) {
    return new Desugarer(program);
  }
}

Since the base class handles all other nodes, the only method we need to implement is visitFunctionDeclaration. Remember that this is after type checking, so the AST we're getting in this case is already annotated with type information.

The emitter expects to have that type information, so we need to make sure our visitFunctionDeclaration node handles it.

We construct a LambdaExpression node from the FunctionDeclaration node, use the node's name property to create the variable name, and handle type information accordingly.

  visitFunctionDeclaration(node) {
    const variadic = node.variadic;
    // since it's TypedAST it has a type property
    const type = node.type;
    const lambda = AST.LambdaExpression(
      node.params,
      node.body,
      variadic,
      node.retType,
      node.srcloc
    );
    lambda.type = type;
    const paramTypes = node.params.map(
      (p) => p.typeAnnotation ?? { kind: TATypes.AnyLiteral }
    );
    const retType = node.retType ?? { kind: TATypes.AnyLiteral };
    const funcType = {
      kind: TATypes.Function,
      params: paramTypes,
      retType,
      variadic,
    };

    const varDecl = AST.VariableDeclaration(
      node.name,
      lambda,
      node.srcloc,
      funcType
    );

    varDecl.type = type;
    return varDecl;
  }

I bet that's way simpler than you thought it would be at first. I was amazed myself at how straightforward it is the first time I did this.

Now we need to use the Desugarer class. In src/desugarer/desugar.js we rewrite the desugar function:

import { Desugarer } from "./Desugarer.js";

export const desugar = (ast) => Desugarer.new().visit(ast);

Changes to The Runtime

First, we're going to use a package to handle currying functions. Ramda is a good choice, and it also has a lot of other goodies if we decide we want to use them.

npm install ramda

We also need a function to parse contracts so we can implement them on library functions defined in JavaScript.

Let's add the parseContract function to the new file src/runtime/parseContract.js:

import { tokenize } from "../lexer/tokenize.js";
import { read } from "../reader/read.js";
import { parseTypeAnnotation } from "../parser/parseTypeAnnotation.js";
import { fromTypeAnnotation } from "../typechecker/fromTypeAnnotation.js";

export const parseContract = (code) =>
  fromTypeAnnotation(parseTypeAnnotation(read(tokenize(code)).car));

Since we're only ever going to read in one contract at a time, we can take the car property off the output from read and use it as the input to parseTypeAnnotation.

Now we need to rewrite makeFunction in src/runtime/makeFunction.js to handle currying everything, as well as creating contracts for library functions.

We'll pass an optional 2nd parameter into makeFunction that's an options object.

Right now the only option is contract, but that could change in the future.

We'll also add a hash of the function object as a name property since the compiler loses the name.

Here's the new version of makeFunction:

import objectHash from "object-hash";
import { curryN } from "ramda";
import { makeWandaValue } from "./conversion.js";
import { addMetaField } from "./object.js";
import { parseContract } from "./parseContract.js";

export const makeFunction = (func, { contract = "" } = {}) => {
  let fn = curryN(func.length, (...args) => {
    const val = makeWandaValue(func(...args));

    if (typeof val === "function") {
      return makeFunction(val);
    }

    return val;
  });
  const hash = objectHash(fn);

  addMetaField(fn, "wanda", true);
  addMetaField(fn, "arity", func.length);
  addMetaField(fn, "name", hash);

  if (contract !== "") {
    Object.defineProperty(fn, "contract", {
      enumerable: false,
      configurable: false,
      writable: false,
      value: parseContract(contract),
    });
  }

  Object.defineProperty(fn, "name", {
    enumerable: false,
    configurable: false,
    writable: false,
    value: hash,
  });

  return fn;
};

Now we can add contracts to functions from the core library, which we'll do in a minute. First, though, I want to handle emitting code for functions.

Since we're desugaring away FunctionDeclaration nodes, we only need to handle the case for LambdaExpression in the emitter.

Add it to the switch statement in the emit method in src/emitter/Emitter.js:

      // other cases
      case ASTTypes.LambdaExpression:
        return this.emitLambdaExpression(node, ns);
      // default case

We're going to emit the parameters into an array of strings so we can easily join them with ", " in the final output.

We need to handle the case of a variadic function by outputting ... before the rest parameter.

For the function body we loop over the body and emit code for each node.

Finally, we wrap the whole thing in rt.makeFunction.

Here's emitLambdaExpression in its entirety:

  emitLambdaExpression(node, ns) {
    const funcNs = ns.extend("Lambda");
    /** @type {string[]} */
    let params = [];
    let i = 0;

    for (let p of node.params) {
      funcNs.set(p.name.name, makeSymbol(p.name.name));

      if (node.variadic && i === node.params.length - 1) {
        params.push(`...${this.emit(p.name, funcNs)}`);
      } else {
        params.push(this.emit(p.name, funcNs));
      }
      i++;
    }

    let code = `rt.makeFunction((${params.join(", ")}) => {\n`;

    let j = 0;
    for (let expr of node.body) {
      if (j === node.body.length - 1) {
        code += `return ${this.emit(expr, funcNs)};`;
      } else {
        code += this.emit(expr, funcNs) + ";\n";
      }
      j++;
    }

    code += "\n})";

    return code;
  }

That's it for changes to the emitter!

Changes to The Printer

We need to add a case for functions to the switch statement in the printString function in src/printer/printString.js:

    // other cases...
    case "function":
      return `Function: ${value.name || "lambda"}`;
    // case "object"...

We need to add cases for both nodes to the print method in src/printer/printAST.js:

      // other cases...
      case ASTTypes.FunctionDeclaration:
        return this.printFunctionDeclaration(node, indent);
      case ASTTypes.LambdaExpression:
        return this.printLambdaExpression(node, indent);
      // default case

And a printFunctionDeclaration method:

  printFunctionDeclaration(node, indent) {
    let prStr = `${prIndent(indent)}FunctionDeclaration:\n`;
    prStr += `${prIndent(indent + 2)}Name: ${node.name.name}\n`;
    prStr += `${prIndent(indent + 2)}Params:\n`;

    for (let param of node.params) {
      prStr += this.print(param.name, indent + 4) + "\n";
    }

    prStr += `${prIndent(indent + 2)}Body:\n`;

    for (let expr of node.body) {
      prStr += this.print(expr, indent + 4) + "\n";
    }

    return prStr;
  }

And a printLambdaExpression method:

  printLambdaExpression(node, indent) {
    let prStr = `${prIndent(indent)}LambdaExpression:\n`;
    prStr += `${prIndent(indent + 2)}Params:\n`;

    for (let param of node.params) {
      prStr += this.print(param.name, indent + 4) + "\n";
    }

    prStr += `${prIndent(indent + 2)}Body:\n`;

    for (let expr of node.body) {
      prStr += this.print(expr, indent + 4) + "\n";
    }

    return prStr;
  }

Changes to The Core Library

You'll see with all these function contracts that there are still a lot of any annotations.

That will start to change as we add new and better types to our type system, but for now everything is pretty basic. We have to use any in a lot of cases because right now there isn't a better way to do it.

Adding generics will go a long way towards improving this, as will union and intersection types.

Also note that I've added 4 higher-order functions that work on lists to the core library. Eventually I'd like them to work on any iterable type, like vectors, but it will take some time to be able to make that work with the type system.

I'm just going to include the whole core library here because as you'll notice I've made some small edits to a few functions here and there.

If you read closely, you'll note that I've used a from method on the Cons class that you've not seen before. Here's the definition for that in src/shared/cons.js:

  static from(iter) {
    return Cons.of(...iter);
  }

Now for the core library in lib/js/core.js:

import equal from "fast-deep-equal/es6/index.js";
import { makeModule } from "../../src/runtime/Module.js";
import { Cons, cons } from "../../src/shared/cons.js";
import { print } from "../../src/printer/print.js";
import { println } from "../../src/printer/println.js";
import { printString } from "../../src/printer/printString.js";
import { isTruthy } from "../../src/runtime/utils.js";
import { Exception } from "../../src/shared/exceptions.js";

export const theModule = makeModule("Core", (rt, ns) => {
  const isList = (obj) => {
    if (!rt.isNil(obj) && !(obj instanceof Cons)) {
      return false;
    } else if (rt.isNil(obj)) {
      return true;
    }

    // only option left is cons
    return isList(obj.cdr);
  };

  return {
    print: rt.makeFunction(print, { contract: "(any -> string)" }),
    println: rt.makeFunction(println, { contract: "(any -> string)" }),
    cons: rt.makeFunction(cons, { contract: "(any, any -> (list any))" }),
    car: rt.makeFunction((pair) => pair.car, {
      contract: "((list any) -> any)",
    }),
    cdr: rt.makeFunction((pair) => pair.cdr, {
      contract: "((list any) -> any)",
    }),
    string: rt.makeFunction(printString, { contract: "(any -> string)" }),
    number: rt.makeFunction(Number, { contract: "(string -> number)" }),
    boolean: rt.makeFunction((val) => isTruthy(val), {
      contract: "(any -> boolean)",
    }),
    keyword: rt.makeFunction((str) => Symbol.for(":" + str), {
      contract: "(string -> keyword)",
    }),
    "+": rt.makeFunction(
      (a, b, ...nums) => nums.reduce((sum, n) => sum + n, a + b),
      { contract: "(number, number, &(vector number) -> number)" }
    ),
    "-": rt.makeFunction(
      (a, b, ...nums) => nums.reduce((diff, n) => diff - n, a - b),
      { contract: "(number, number, &(vector number) -> number)" }
    ),
    "*": rt.makeFunction(
      (a, b, ...nums) => nums.reduce((prod, n) => prod * n, a * b),
      { contract: "(number, number, &(vector number) -> number)" }
    ),
    "/": rt.makeFunction(
      (a, b, ...nums) => nums.reduce((quot, n) => quot / n, a / b),
      { contract: "(number, number, &(vector number) -> number)" }
    ),
    "%": rt.makeFunction(
      (a, b, ...nums) => nums.reduce((quot, n) => quot % n, a % b),
      { contract: "(number, number, &(vector number) -> number)" }
    ),
    "=": rt.makeFunction((a, b) => a === b, {
      contract: "(number, number -> boolean)",
    }),
    ">": rt.makeFunction((a, b) => a > b, {
      contract: "(number, number -> boolean)",
    }),
    ">=": rt.makeFunction((a, b) => a >= b, {
      contract: "(number, number -> boolean)",
    }),
    "<": rt.makeFunction((a, b) => a < b, {
      contract: "(number, number -> boolean)",
    }),
    "<=": rt.makeFunction((a, b) => a <= b, {
      contract: "(number, number -> boolean)",
    }),
    not: rt.makeFunction((x) => !x, { contract: "(any -> boolean)" }),
    list: rt.makeFunction((...args) => Cons.from(args), {
      contract: "(&(vector any) -> (list any))",
    }),
    length: rt.makeFunction(
      (obj) => {
        if (obj instanceof Cons) {
          let i = 0;
          for (let _ of obj) {
            i++;
          }
          return i;
        }

        return obj.length;
      },
      { contract: "((list any) -> number)" }
    ),
    get: rt.makeFunction(
      (n, obj) => {
        const value = obj.get(n);

        if (value === undefined) {
          throw new Exception(`Value for index ${n} not found on object`);
        }

        return value;
      },
      { contract: "((list any) -> any)" }
    ),
    "list?": rt.makeFunction(isList, { contract: "((list any) -> boolean)" }),
    "pair?": rt.makeFunction((obj) => obj instanceof Cons, {
      contract: "((list any) -> boolean)",
    }),
    "number?": rt.makeFunction((obj) => typeof obj === "number", {
      contract: "(any -> boolean)",
    }),
    "string?": rt.makeFunction((obj) => typeof obj === "string", {
      contract: "(any -> boolean)",
    }),
    "boolean?": rt.makeFunction((obj) => typeof obj === "boolean", {
      contract: "(any -> boolean)",
    }),
    "nil?": rt.makeFunction((obj) => obj == null, {
      contract: "(any -> boolean)",
    }),
    "keyword?": rt.makeFunction(
      (obj) => typeof obj === "symbol" && obj.description.startsWith(":"),
      { contract: "(any -> boolean)" }
    ),
    "equal?": rt.makeFunction(
      (a, b) => {
        if (rt.hasDict(a) && rt.hasDict(b)) {
          return equal(rt.getMetaField(a, "dict"), rt.getMetaField(b, "dict"));
        }
        return equal(a, b);
      },
      { contract: "(any, any) -> boolean" }
    ),
    "is?": rt.makeFunction((a, b) => Object.is(a, b), {
      contract: "(any, any -> boolean)",
    }),
    append: rt.makeFunction(
      (obj1, obj2) => {
        if (typeof obj1 === "string" && typeof obj2 === "string") {
          return obj1 + obj2;
        } else if (obj1 instanceof Cons) {
          return Cons.from(obj1).append(obj2);
        } else if (Array.isArray(obj1)) {
          return [...obj1].push(obj2);
        } else {
          rt.failRuntime(`Value of type ${typeof obj1} cannot be appended`);
        }
      },
      { contract: "(any, any -> any)" }
    ),
    with: rt.makeFunction(
      (rec1, rec2) => {
        const newDict = Object.assign(
          {},
          rt.getMetaField(rec1, "dict"),
          rt.getMetaField(rec2, "dict")
        );
        return rt.makeObject(newDict);
      },
      { contract: "(any, any -> any)" }
    ),
    prop: rt.makeFunction((prop, obj) => rt.getField(obj, prop), {
      contract: "(string, any -> any)",
    }),
    each: rt.makeFunction(
      (fn, lst) => {
        for (let item of lst) {
          fn(item);
        }
      },
      {
        contract: "((any -> nil), (list any) -> nil)",
      }
    ),
    map: rt.makeFunction(
      (fn, lst) => {
        let mapped = [];
        for (let item of lst) {
          mapped.push(fn(item));
        }
        return Cons.from(mapped);
      },
      {
        contract: "((any -> any), (list any) -> (list any))",
      }
    ),
    filter: rt.makeFunction(
      (fn, lst) => {
        let filtered = [];
        for (let item of lst) {
          if (rt.isTruthy(fn(item))) {
            filtered.push(item);
          }
        }
        return Cons.from(filtered);
      },
      {
        contract: "((any -> boolean), (list any) -> (list any))",
      }
    ),
    fold: rt.makeFunction(
      (fn, init, lst) => {
        let acc = init;
        for (let item of lst) {
          acc = fn(acc, item);
        }
        return acc;
      },
      {
        contract: "((any, any -> any), any, (list any) -> any)",
      }
    ),
    "fold-r": rt.makeFunction(
      (fn, init, lst) => {
        let acc = init;
        const reversed = [...lst].reverse();
        for (let item of reversed) {
          acc = fn(acc, item);
        }
        return acc;
      },
      {
        contract: "((any, any -> any), any, (list any) -> any)",
      }
    ),
  };
});

Conclusion

That's it for adding functions to the language! Fire up the REPL with the wanda command and take them for a spin.

(def add-3 (a: number, b: number, c: number) -> number
  (+ a b c))

(add-3 1 2 3) ;-> 6

As always you can see the state of the code as of the end of this article on the relevant tag at the Wanda repo.

Next time we're going to do a lot of work with type checking. We'll add tuple, union, and intersection types to Wanda as well as :as expressions that will tell the type checker to treat a value as a specific type for times when it needs a little help.

I'll see you then!

Create Your Own Programming Language 5: Vectors and Records

Jason Barr — Fri, 16 Jun 2023 20:32:32 +0000

Welcome to the next entry in the "Create Your Own Programming Language" series. In this article we're going to add vectors and records to the Wanda Programming Language, plus member expressions with dot syntax.

I know, I know, if you're a hardcore Lisper that's heresy. It's not my first heresy and it will be far from my last one, so it is what it is. It does complicate reading and parsing a bit, but that's the price we pay for convenience.

We're also going to add some destructuring in variable assignment, but only one level deep. No nested destructuring because I didn't want to make things overly complex. Type checking for destructuring was an interesting exercise, though, so I wanted to at least show you how to implement some destructuring even if it's not as fully-featured as what you can do in JavaScript or Clojure. We're also only allowing destructuring in variable declarations, not in set expressions.

As always, if you haven't read the last post in the series about variables and building the type checker you'll want to make sure to do that before continuing on with this one.

We good? Good. Let's go.

Fixing a Mistake from Last Time

The first thing we need to do is fix a mistake I made in the previous article. I've fixed the code there, but there's a good chance you read that article before I made the fixes in it so I'm going to mention them here as well.

What happened is I didn't adequately make sure the TypeChecker class returns type-annotated nodes based on the original AST nodes.

In its original state some AST nodes got decorated with types, but others did not. Here are the updated methods to make sure all nodes in the AST get decorated with types:

  checkProgram(node, env) {
    let body = [];
    let type;
    let i = 0;
    for (let expr of node.body) {
      if (i === node.body.length - 1) {
        const node = this.check(expr, env);
        type = node.type;
        body.push(node);
      } else {
        const node = this.check(expr, env);
        body.push(node);
      }
    }

    return { kind: node.kind, body, srcloc: node.srcloc, type };
  }

  checkDoExpression(node, env) {
    let body = [];
    for (let expr of node.body) {
      const node = this.check(expr, env);
      body.push(node);
    }

    return {
      kind: node.kind,
      body,
      srcloc: node.srcloc,
      type: infer(node, env),
    };
  }

  checkVariableDeclaration(node, env) {
    if (node.typeAnnotation) {
      const annotType = fromTypeAnnotation(node.typeAnnotation);
      check(node.expression, annotType, env);
      env.checkingOn = true;
      env.set(node.lhv.name, annotType);
      return { ...node, type: annotType };
    }

    const type = infer(node, env);
    env.set(node.lhv.name, type);
    return {
      kind: node.kind,
      lhv: node.lhv,
      expression: this.check(node.expression, env),
      srcloc: node.srcloc,
      type,
    };
  }

  checkSetExpression(node, env) {
    if (env.checkingOn) {
      const nameType = env.getType(node.lhv.name);
      check(node.expression, nameType, env);
      return { ...node, type: nameType };
    }

    return {
      kind: node.kind,
      lhv: node.lhv,
      expression: this.check(node.expression, env),
      srcloc: node.srcloc,
      type: infer(node, env),
    };;
  }

Everything else is ok as it was originally published. Now let's move on to the new changes!

Preliminary Changes

First, I had to remove colons as valid symbol characters in src/lexer/utils.js. Make sure your isSymbolChar function looks like this:

export const isSymbolChar = (ch) =>
  /[=@~<>%&|?\\/^*&#'\p{L}\p{N}_$!+-]/u.test(ch);

I had accidentally included them originally, so I want to make sure you remove the colon from the symbol character regex like I did.

Note that this means you can now write variable type annotations with the colon adjacent to the symbol instead of having to put a space in between, e.g.:

(var (x: number) 7)

I also changed the constructor for SyntaxException in src/shared/exceptions.js so it can take an optional 3rd parameter expected:

constructor(value, srcloc, expected = "") {
    super(
      `Syntax Exception: invalid syntax ${value} found at ${srcloc.file} (${
        srcloc.line
      }:${srcloc.col})${expected ? ` (expected ${expected})` : ""}`
    );
  }

Changes to The Lexer

We need to add a few tokens to the lexer. Vector literals will be written by putting a list of expressions between square brackets ([ and ]), and record literals will be written by putting entries between curly braces ({ and }). We'll also include the ampersand (&) for use as a rest variable indicator during destructuring. We'll also add a token for the dot for use with member expressions.

Let's add character matchers for braces, brackets, and the ampersand in src/lexer/utils.js:

// after isRParen
export const isLBrack = (ch) => /\[/.test(ch);
export const isRBrack = (ch) => /\]/.test(ch);
export const isLBrace = (ch) => /\{/.test(ch);
export const isRBrace = (ch) => /\}/.test(ch);
export const isAmp = (ch) => /&/.test(ch);

Now let's add the token types to the TokenTypes enum in src/lexer/TokenTypes.js:

export const TokenTypes = {
  // after RParen
  LBrack: "LBrack",
  RBrack: "RBrack",
  LBrace: "LBrace",
  RBrace: "RBrace",
  Dot: "Dot",
  Amp: "Amp",
};

Finally, the changes to the actual lexer. We'll need to add the new functions to the imports in src/lexer/Lexer.js:

import {
  isAmp,
  // ...
  isLBrace,
  isLBrack,
  //...
  isRBrace,
  isRBrack,
  //...
} from "./utils.js";

To keep it simple I've just added additional else if cases in the main loop in the tokenize method:

// after else if (isRParen(ch)) ...
      } else if (isLBrack(ch)) {
        const { pos, line, col, file } = this.input;
        this.input.next(); // skip over punc
        tokens.push(
          Token.new(TokenTypes.LBrack, ch, SrcLoc.new(pos, line, col, file))
        );
      } else if (isRBrack(ch)) {
        const { pos, line, col, file } = this.input;
        this.input.next(); // skip over punc
        tokens.push(
          Token.new(TokenTypes.RBrack, ch, SrcLoc.new(pos, line, col, file))
        );
      } else if (isLBrace(ch)) {
        const { pos, line, col, file } = this.input;
        this.input.next(); // skip over punc
        tokens.push(
          Token.new(TokenTypes.LBrace, ch, SrcLoc.new(pos, line, col, file))
        );
      } else if (isRBrace(ch)) {
        const { pos, line, col, file } = this.input;
        this.input.next(); // skip over punc
        tokens.push(
          Token.new(TokenTypes.RBrace, ch, SrcLoc.new(pos, line, col, file))
        );
      } else if (isDot(ch)) {
        const { pos, line, col, file } = this.input;
        this.input.next(); // skip over punc
        tokens.push(
          Token.new(TokenTypes.Dot, ch, SrcLoc.new(pos, line, col, file))
        );
      } else if (isAmp(ch)) {
        const { pos, line, col, file } = this.input;
        this.input.next(); // skip over punc
        tokens.push(
          Token.new(TokenTypes.Amp, ch, SrcLoc.new(pos, line, col, file))
        );
      } else {
// error case...

If you'd prefer to refactor this into a dedicated method for lexing punctuation tokens, that's fine too. You'd probably need to add a character matching function for punctuation tokens in src/lexer/utils.js and add it to your import.

Changes to The Reader

We need to add two methods to the Reader class. First, update your imports in src/reader/Reader.js to import the Exception class:

import { Exception } from "../shared/exceptions.js";

The two methods we're going to add are expect, which looks for a certain kind of token and throws an error if it doesn't get it, and lookahead, which looks ahead n tokens in the stream and returns that token:

  expect(expected, actual) {
    if (expected !== actual) {
      throw new Exception(`Expected ${expected} token; got ${actual}`);
    }
  }

  lookahead(n = 1) {
    return this.tokens[this.pos + n];
  }

Now we need to make some changes in src/reader/read.js to handle the new forms. We'll need to handle vector literals, record literals, member expressions with dot syntax, and both vector and record patterns for destructuring.

We're going to keep vector and record patterns very simple: each is just a list of symbols. No renaming in record patterns like you can do when destructuring objects in JavaScript, so no need to worry about colons in record patterns.

[a b c]
{a b c}

Indicate a rest variable with an ampersand, e.g.:

(var [a &b] [1 2 3])

A vector literal is just a list of expressions between square brackets, e.g.:

[1 2 3]

Note that you can put commas between the expressions and the lexer will read them as whitespace:

[1, 2, 3]

A record literal is a list of key/value pairs (keys and values separated by colons) between curly braces:

{a: "hi"}

A member expression is an object and property with a dot between them. An object can be any expression, but a property will always be a valid symbol:

object.property

Reading vector/record literals and patterns is fairly straight forward: we look for [ or { and then read accordingly.

Reading member expressions requires a little more work.

Pratt Parsing

We're actually going to use a simplified version of Vaughan Pratt's precedence algorithm to read member expressions. Pratt's algorithm can be used to read just about any expression type, including ternary, binary, and unary expressions, but we're just going to use it for member expressions. We won't be introducing binary expression syntax into Wanda or anything like that.

Pratt's algorithm works by parsing in a loop. You assign all operations a numeric precedence. The default is 0, so any operator will override the default precedence.

Start by parsing the left hand expression. Then you peek at the next token. If it's an operator, get its precedence.

Then you enter a loop. While the precedence for the current operator is greater than the precedence of the just-parsed expression, you continue parsing expressions until you either run out of expressions to parse or you run into an operator with lower precedence.

Since we only have one operator, the dot for member expressions, we just keep parsing member expressions until we run out of dots.

First, we'll need a helper function to get the current token's precedence:

const getPrec = (token) => PREC[token?.type] ?? 0;

Then at the top of read.js add this object:

const PREC = {
  [TokenTypes.Dot]: 90,
};

The object is there so getPrec can get the precedence for the dot operator, which is the only token that has a precedence value.

Next, a function to read member expressions. In src/reader/read.js add the readMemberExpression function (I have it just above readForm in my code):

const readMemberExpression = (reader, left) => {
  const tok = reader.next();
  const prec = getPrec(tok);
  reader.expect(TokenTypes.Dot, tok.type);
  const property = readExpr(reader, prec);
  return {
    type: "MemberExpression",
    object: left,
    property,
    srcloc: left.srcloc,
  };
};

Here's the new version of readExpr that implements Pratt's algorithm:

const readExpr = (reader, bp = 0) => {
  let left = readForm(reader);
  let tok = reader.peek();
  let prec = getPrec(tok);

  while (bp < prec) {
    left = readMemberExpression(reader, left);
    tok = reader.peek();
    prec = getPrec(tok);
  }

  return left;
};

Believe it or not, you can use Pratt's algorithm to parse just about any kind of expression. I have a toy project that can parse basically any kind of expression as long as you feed it a table with precedence, arity, and associativity if you want to see what's possible with a Pratt parser (obviously since it's a toy it's not for production use - there are definitely bugs).

I've added some checks to readForm to throw SyntaxExceptions if it runs into a closing brace or bracket in the wrong position, and made it dispatch to readVector and readMaybeRecord functions on encountering an opening bracket or brace:

const readForm = (reader) => {
  const tok = reader.peek();

  switch (tok.type) {
    case TokenTypes.RParen:
      // there shouldn't be an RParen here
      throw new SyntaxException(tok.value, tok.srcloc);
    case TokenTypes.RBrack:
      // there shouln't be an RBrack here
      throw new SyntaxException(tok.value, tok.srcloc);
    case TokenTypes.RBrace:
      // there shouldn't be an RBrace here
      throw new SyntaxException(tok.value, tok.srcloc);
    case TokenTypes.LParen:
      return readList(reader);
    case TokenTypes.LBrack:
      return readVector(reader);
    case TokenTypes.LBrace:
      return readMaybeRecord(reader);
    default:
      return readAtom(reader);
  }
};

The readVector function is straightforward:

const readVector = (reader) => {
  // Get srcloc info from opening bracket and skip it
  let tok = reader.next();
  const srcloc = tok.srcloc;
  let lastTok = tok;
  tok = reader.peek();
  /** @type {Form[]} */
  let members = [];

  while (tok?.type !== TokenTypes.RBrack) {
    if (!tok) {
      throw new SyntaxException("EOF", lastTok.srcloc, "]");
    }

    members.push(readExpr(reader));
    lastTok = tok;
    tok = reader.peek();
  }

  // skip closing bracket
  reader.skip();

  return {
    type: "VectorLiteral",
    members,
    srcloc,
  };
};

The readMaybeRecord function dispatches to readRecordLiteral and readRecordPattern. readRecordLiteral uses readProperty as a helper. Here are all 4 functions:

const readMaybeRecord = (reader) => {
  let tok = reader.next();
  const srcloc = tok.srcloc;
  // First token after brace should always be a symbol
  tok = reader.peek();
  reader.expect(TokenTypes.Symbol, tok.type);
  tok = reader.lookahead(1);

  if (tok.type === TokenTypes.Keyword && tok.value === ":") {
    // record literal = { prop : value, prop2: value2 }
    return readRecordLiteral(reader, srcloc);
  } else {
    // record pattern = { prop, prop2 }
    return readRecordPattern(reader, srcloc);
  }
};

const readProperty = (reader) => {
  let tok = reader.peek();
  let srcloc = tok.srcloc;

  reader.expect(TokenTypes.Symbol, tok.type);

  const key = readExpr(reader);

  tok = reader.peek();
  reader.expect(":", tok.value);
  reader.skip();

  const value = readExpr(reader);

  return {
    type: "Property",
    key,
    value,
    srcloc,
  };
};

const readRecordLiteral = (reader, srcloc) => {
  let tok = reader.peek();
  /** @type {Property[]} */
  let properties = [];
  let lastTok = tok;

  while (tok?.type !== TokenTypes.RBrace) {
    if (!tok) {
      throw new SyntaxException("EOF", lastTok.srcloc, "}");
    }

    reader.expect(TokenTypes.Symbol, tok.type);
    properties.push(readProperty(reader));
    lastTok = tok;
    tok = reader.peek();
  }

  // skip closing brace
  reader.skip();

  return {
    type: "RecordLiteral",
    properties,
    srcloc,
  };
};

const readRecordPattern = (reader, srcloc) => {
  /** @type {Token[]} */
  let properties = [];
  let tok = reader.peek();
  let lastTok = tok;

  while (tok?.type !== TokenTypes.RBrace) {
    if (!tok) {
      if (!tok) {
        throw new SyntaxException("EOF", lastTok.srcloc, "}");
      }
    }

    tok = reader.peek();
    properties.push(readExpr(reader));
    lastTok = tok;
    tok = reader.peek();
  }

  // skip closing brace
  reader.skip();

  return {
    type: "RecordPattern",
    properties,
    srcloc,
  };
};

That's it for changes to the reader, and hopefully now you understand parsing a little better!

Changes to The Parser

First, we need to add some new node types to the ASTTypes enum in src/parser/ast.js:

export const ASTTypes = {
  // additional node types
  VectorLiteral: "VectorLiteral",
  VectorPattern: "VectorPattern",
  Property: "Property",
  RecordLiteral: "RecordLiteral",
  RecordPattern: "RecordPattern",
  MemberExpression: "MemberExpression",
};

Let's also add node constructors:

export const AST = {
  // additional node constructors...
  VectorPattern(members, srcloc, rest = false) {
    return {
      kind: ASTTypes.VectorPattern,
      members,
      srcloc,
      rest,
    };
  },

  VectorLiteral(members, srcloc) {
    return {
      kind: ASTTypes.VectorLiteral,
      members,
      srcloc,
    };
  },

  Property(key, value, srcloc) {
    return {
      kind: ASTTypes.Property,
      key,
      value,
      srcloc,
    };
  },

  RecordPattern(properties, srcloc, rest = false) {
    return {
      kind: ASTTypes.RecordPattern,
      properties,
      srcloc,
      rest,
    };
  },

  RecordLiteral(properties, srcloc) {
    return {
      kind: ASTTypes.RecordLiteral,
      properties,
      srcloc,
    };
  },

  MemberExpression(object, property, srcloc) {
    return {
      kind: ASTTypes.MemberExpression,
      object,
      property,
      srcloc,
    };
  },
};

In src/parser/parse.js we need to handle parsing the new forms. In parseExpr we'll delegate to a new parseComplexForm function to handle the new forms:

const parseExpr = (form) => {
  if (form instanceof Cons) {
    return parseList(form);
  }

  if (form instanceof Token && form.type === TokenTypes.Amp) {
    // Will be handled when parsing vector or record pattern
    return form;
  }

  if (form instanceof Token) {
    return parsePrimitive(form);
  }

  return parseComplexForm(form);
};

We'll need a parseProperty helper function for parsing record literals:

const parseProperty = (form) => {
  const key = parseExpr(form.key);
  const value = parseExpr(form.value);
  return AST.Property(key, value, form.srcloc);
};

The parseComplexForm function handles all the forms that aren't primitives or lists, so all the new forms added today:

const parseComplexForm = (form) => {
  switch (form.type) {
    case "VectorLiteral": {
      const members = form.members.map(parseExpr);
      return AST.VectorLiteral(members, form.srcloc);
    }
    case "RecordLiteral": {
      const properties = form.properties.map(parseProperty);
      return AST.RecordLiteral(properties, form.srcloc);
    }
    case "RecordPattern": {
      let properties = [];
      let rest = false;

      for (let prop of form.properties) {
        if (prop instanceof Token && prop.type === TokenTypes.Amp) {
          rest = true;
          continue;
        }

        if (prop instanceof Token && prop.type !== TokenTypes.Symbol) {
          throw new SyntaxException(prop.type, prop.srcloc, TokenTypes.Symbol);
        }

        properties.push(parseExpr(prop));
      }

      return AST.RecordPattern(properties, form.srcloc, rest);
    }
    case "MemberExpression": {
      const object = parseExpr(form.object);
      const property = parseExpr(form.property);
      return AST.MemberExpression(object, property, form.srcloc);
    }
    default:
      // this should never happen
      throw new SyntaxException(form.type, form.srcloc);
  }
};

You probably noticed that vector pattern is missing. That's because in the reader phase a vector pattern is indistinguishable from a vector literal made up only of symbols. We'll have to convert vector literals to vector patterns if they're in the left hand position of a variable declaration.

Here's the function to do that:

const convertVectorLiteralToVectorPattern = (parsedLhv) => {
  let members = [];
  let rest = false;
  for (let mem of parsedLhv.members) {
    if (mem instanceof Token && mem.type === TokenTypes.Amp) {
      rest = true;
      continue;
    }

    if (mem.kind === ASTTypes.VectorLiteral) {
      mem = convertVectorLiteralToVectorPattern(mem);
    } else if (
      mem.kind !== ASTTypes.Symbol &&
      mem.kind !== ASTTypes.RecordPattern
    ) {
      throw new SyntaxException(
        mem.kind,
        mem.srcloc,
        `${ASTTypes.Symbol} or ${ASTTypes.RecordPattern}`
      );
    }

    members.push(mem);
  }

  return AST.VectorPattern(members, parsedLhv.srcloc, rest);
};

And here's the new version of parseVariableDeclaration that handles vector and record patterns:

const parseVariableDeclaration = (decl) => {
  let [_, lhv, expression] = decl;

  let parsedLhv,
    typeAnnotation = null;
  if (lhv instanceof Cons) {
    // has type annotation
    const realLhv = lhv.get(0);
    // convert to array and get rid of ":" when passing into parseTypeAnnotation
    typeAnnotation = parseTypeAnnotation([...lhv.cdr].slice(1));
    parsedLhv = parseExpr(realLhv);
  } else {
    parsedLhv = parseExpr(lhv);
  }

  if (parsedLhv.kind === ASTTypes.VectorLiteral) {
    parsedLhv = convertVectorLiteralToVectorPattern(parsedLhv);
  }

  const parsedExpression = parseExpr(expression);

  return AST.VariableDeclaration(
    parsedLhv,
    parsedExpression,
    decl.srcloc,
    typeAnnotation
  );
};

Finally, we need to be able to parse type annotations for vector and record types. A vector type annotation is similar to that for a list: you simply write vector and put the type in a list, e.g. vector (number).

The type annotation for a record type is like an object type literal in TypeScript. It looks like an object literal, but with a type instead of a value for the object property:

type Person = {
  name: string
  age: number
}

Note that we don't put semicolons after property types because in Wanda a semicolon starts a comment.

First, add the new type annotation types to the TATypes enum in src/parser/parseTypeAnnotation.js:

export const TATypes = {
  // additional TA types
  Vector: "Vector",
  Object: "Object",
};

Parsing vector annotations is just like parsing list annotations:

const parseVectorAnnotation = (type) => {
  const vectorType = parseTypeAnnotation(type);
  return { kind: TATypes.Vector, vectorType };
};

Don't forget to add the vector case to the switch statement in parseTypeAnnotation:

      // other cases
      case "vector":
        return parseVectorAnnotation(annotation[1]);
      // default case

We find a quirk when we go to parse object type literals: the reader has already processed them as RecordLiteral forms! That's to be expected because the reader has no way to disambiguate between a form and a type annotation. So we'll check for a type of "RecordLiteral" on the annotation and dispatch that to parseObjectAnnotation in parseTypeAnnotation:

export const parseTypeAnnotation = (annotation) => {
  // checking for Cons and array
  if (annot.type === "RecordLiteral") {
    return parseObjectAnnotation(annot);
  }
  // switch statement inside if (annot.type === TokenTypes.Symbol)
};

Finally, here's the parseObjectAnnotation function:

const parseObjectAnnotation = (annot) => {
  let properties = [];
  for (let prop of annot.properties) {
    const name = prop.key.value;
    const propType = parseTypeAnnotation(prop.value);
    properties.push({ name, propType });
  }

  return { kind: TATypes.Object, properties };
};

That's it for changes to the parser! Now we need to add some shared types and functions before moving on to the type checker.

Changes to Shared Functionality

I've added 3 new exception types to the compiler. I didn't add a TypeException last time when first making the type checker, but I did for this article. I've also added ReferenceException and RuntimeException. In src/shared/exceptions.js:

export class TypeException extends Exception {
  constructor(msg, srcloc) {
    super(`${msg} at ${srcloc.file} ${srcloc.line}:${srcloc.col}`);
  }
}

export class ReferenceException extends Exception {
  constructor(msg, srcloc) {
    super(`${msg} at ${srcloc.file} ${srcloc.line}:${srcloc.col}`);
  }
}

export class RuntimeException extends Exception {
  constructor(msg) {
    super(msg);
    this[Symbol.for(":dict")] = { message: msg };
  }
}

RuntimeException won't have access to AST nodes, so there's no parameter for srcloc info in its constructor.

Also, since RuntimeException may be accessible from inside Wanda code, the exception object gets the same :dict property all Wanda objects will get. I've also added it to Exception. I'll talk more about the :dict property below.

There are also a few utils we'll need to eventually share between different compiler modules, so let's go ahead and add them to src/shared/utils.js:

export const isNullish = (obj) => obj == null;

export const hasProperty = (obj, prop) => !isNullish(obj?.[prop]);

export const hasMethod = (obj, methodName) =>
  hasProperty(obj, methodName) && typeof obj[methodName] === "function";

Changes to The Type Checker

We've got a fair bit of work to do to add new types to the type checker.

Note that we're just going to consider records to be object types, and we'll use the same object types when we add classes and objects to the language later.

Object types are structural, so the type checker cares more about what shape an object has and the types of its properties than it does about what class was used to construct an object. This greatly simplifies object type checking, but it will result in some less helpful error messages when debugging object types unless we do a little extra work when it comes to handling object classes.

First, we need to add new members to the TypeTypes enum in src/typechecker/types.js:

export const TypeTypes = {
  // other types...
  Vector: "Vector",
  Property: "Property",
  Object: "Object",
};

In src/typechecker/constructors.js I've renamed listType to list and added constructors for vector and object:

export const list = (listType) => ({ kind: TypeTypes.List, listType });

export const vector = (vectorType) => ({
  kind: TypeTypes.Vector,
  vectorType,
});

export const object = (properties) => ({ kind: TypeTypes.Object, properties });

We also need isVector and isObject validators in src/typechecker/validators.js:

export const isVector = (type) => {
  return type.kind === TypeTypes.Vector;
};

export const isObject = (type) => {
  return type.kind === TypeTypes.Object;
};

We need to be able to construct concrete types from vector and object type annotations in src/typechecker/fromTypeAnnotation.js. I've also added the ability to annotate a type as any, which is currently necessary in some cases because we don't yet have generics in the language:

    case TATypes.AnyLiteral:
      return Type.any;
    // additional switch cases from NumberLiteral to List...
    case TATypes.Vector: {
      const vectorType = fromTypeAnnotation(typeAnnotation.vectorType, typeEnv);
      return Type.vector(vectorType);
    }
    case TATypes.Object: {
      const propTypes = typeAnnotation.properties.map((prop) => ({
        name: prop.name,
        type: fromTypeAnnotation(prop.propType, typeEnv),
      }));
      return Type.object(propTypes);
    }
    // default case...

We'll add a propType helper function to easily get the type of an object property in src/typechecker/propType.js:

export const propType = (type, name) => {
  const prop = type.properties.find(({ name: propName }) => propName === name);
  return prop ? prop.type : null;
};

This function returns null instead of throwing an error if the property type isn't found so we can instead throw the error from a point in the type checker closer to the actual error and thus provide better error messages.

Now we need to be able to check if an object is a subtype of another. An object is a subtype of another if it has all the same properties as the supertype and the corresponding properties are all subtypes of the same property on the supertype object. A subtype object can also have additional properties, but those aren't checked because they're not relevant to checking against the supertype.

We also add a case for vector types in src/typechecker/isSubtype.js:

  // additional cases
  if (Type.isVector(type1) && Type.isVector(type2)) {
    return isSubtype(type1.vectorType, type2.vectorType);
  }

  if (Type.isObject(type1) && Type.isObject(type2)) {
    return type2.properties.every(({ name: type2name, type: type2type }) => {
      const type1type = propType(type1, type2name);
      if (!type1type) return false;
      else return isSubtype(type1type, type2type);
    });
  }
  // default return

We also need to be able to convert our new types to strings in src/typechecker/typeToString.js:

    // additional cases...
    case TypeTypes.Vector:
      return `vector (${typeToString(type.vectorType)})`;
    case TypeTypes.Object:
      return `{ ${type.properties
        .map((p) => `${p.name} : ${typeToString(p.type)}`)
        .join(", ")} }`;
    // default case

We're also going to add functions to unify types. Unification is something you do when attempting to infer types. You unify types within a set of constraints and produce the most general type that satisfies all constraints for all types being unified.

It's especially important in Hindley-Damas-Milner type inference, which is used in ML-family languages like OCaml, but it's useful in bidirectional type checking as well.

Unification algorithms can get pretty complex, but ours is simple: we just return the common supertype of all types being unified or null if there is no common supertype. If any of the types being unified is the any type, we return that and effectively turn type checking off.

Note that in the future we'll add a new kind of type called intersection types that will introduce a common supertype for all types. When we do that, we'll return that type instead of null.

We'll add a function to unify 2 types and another to unify a list of types in src/typechecker/unify.js:

export const unify = (type1, type2) => {
  if (isSubtype(type1, type2)) {
    return type2;
  } else if (isSubtype(type2, type1)) {
    return type1;
  }

  return null;
};

export const unifyAll = (...types) => {
  return types.reduce((unified, type) => {
    if (unified === null) return null;
    if (Type.isAny(unified)) return unified;

    return unify(unified, type);
  });
};

Next we'll add inference for our new types in src/typechecker/infer.js. First, update exports to include the new SyntaxException and new unifyAll and propType functions:

import { Exception, TypeException } from "../shared/exceptions.js";
import { unifyAll } from "./unify.js";
import { propType } from "./propType.js";

Now in the infer function we need new dispatch cases for the new AST nodes:

    // previous cases
    case ASTTypes.VectorLiteral:
      return inferVectorLiteral(ast, env);
    case ASTTypes.RecordLiteral:
      return inferRecordLiteral(ast, env);
    case ASTTypes.MemberExpression:
      return inferMemberExpression(ast, env);
    // default case

And we need to infer types for vector and record literals and member expressions:

const inferVectorLiteral = (node, env) => {
  if (node.members.length === 0) {
    // change this to never when we add union types
    return Type.vector(Type.any);
  }

  const types = node.members.map((m) => infer(m, env));
  const unified = unifyAll(...types);

  if (unified === null && env.checkingOn) {
    throw new TypeException(
      `Incompatible types in Vector literal`,
      node.srcloc
    );
  } else if (unified === null) {
    return Type.any;
  }

  return Type.vector(unified);
};

const inferRecordLiteral = (node, env) => {
  const properties = node.properties.map((prop) => ({
    kind: Type.Type.Property,
    name: prop.key.name,
    type: infer(prop.value, env),
  }));
  return Type.object(properties);
};

const inferMemberExpression = (node, env) => {
  const prop = node.property;
  const object = infer(node.object, env);

  if (!Type.isObject(object)) {
    if (env.checkingOn) {
      throw new TypeException(
        `Member expression expects object type; ${Type.toString(object)} given`,
        node.srcloc
      );
    } else {
      return Type.any;
    }
  }

  const type = propType(object, prop.name);

  if (!type && env.checkingOn) {
    throw new TypeException(
      `Property ${prop.name} not found on object of type ${Type.toString(
        object
      )}`,
      node.srcloc
    );
  }

  return type ?? Type.any;
};

Note that in vector literals we unify all types included in the vector, which returns a common subtype. If unification fails (returns null) and type checking is on, we throw an error.

When inferring a member expression, we require the object's type to be an object type. We'll need to do some work here later on if we decide to allow method calls on non-object types (i.e. calling a method like String.prototype.toUpperCase).

When inferring a record literal we simply collect the inferred type of all properties and use them to construct an object type.

In src/typechecker/check.js we need to update our imports:

import { TypeException } from "../shared/exceptions.js";
import { isSubtype } from "./isSubtype.js";
import { propType } from "./propType.js";

We'll update check to use the new TypeException and dispatch to our checkObject function for checking object types:

export const check = (ast, type, env) => {
  if (ast.kind === ASTTypes.RecordLiteral && Type.isObject(type)) {
    return checkObject(ast, type, env);
  }

  const inferredType = infer(ast, env);

  if (!isSubtype(inferredType, type)) {
    throw new TypeException(
      `Type ${Type.toString(
        inferredType
      )} is not a valid subtype of ${Type.toString(type)}`,
      ast.srcloc
    );
  }
};

Here's checkObject. Note that it gets property names from the node, checks if every property name on the type is present in the node's property names, then checks the type of each property. Note that for record literals we require the node to have all properties specified on the type, without extra properties. This effectively disallows using record literals as subtypes of an object type on the assumption that since it's a literal object adding any extra properties is a mistake (the same assumption TypeScript makes). We can revisit this later if we decide we need to allow record literals to be subtypes of the types we check them against:

const checkObject = (ast, type, env) => {
  const astProps = ast.properties.map((prop) => ({
    name: prop.key.name,
    expr: prop.value,
  }));

  type.properties.forEach(({ name }) => {
    const astProp = astProps.find(({ name: astName }) => astName === name);

    if (!astProp) {
      throw new TypeException(
        `Property ${name} not found on record literal`,
        ast.srcloc
      );
    }
  });

  astProps.forEach(({ name, expr }) => {
    const pType = propType(type, name);

    if (!pType) {
      throw new TypeException(
        `Property ${name} not found on object of type ${Type.toString(type)}`,
        ast.srcloc
      );
    }

    check(expr, pType, env);
  });
};

The TypeChecker class needs edits both to handle checking types for the new AST nodes and for handling destructuring in variable assignment.

First, we have several new imports so I'm just going to give you the whole list:

import { ASTTypes } from "../parser/ast.js";
import { Exception, TypeException } from "../shared/exceptions.js";
import { TypeEnvironment } from "./TypeEnvironment.js";
import { check } from "./check.js";
import { infer } from "./infer.js";
import { fromTypeAnnotation } from "./fromTypeAnnotation.js";
import { Type } from "./Type.js";
import { propType } from "./propType.js";

Now here are the new cases to dispatch in the switch statement in the check method:

      // other cases...
      case ASTTypes.MemberExpression:
        return this.checkMemberExpression(node, env);
      case ASTTypes.RecordLiteral:
        return this.checkRecordLiteral(node, env);
      case ASTTypes.VectorLiteral:
        return this.checkVectorLiteral(node, env);
      // default case

The methods for these cases are actually pretty simple. Here's checkMemberExpression:

  checkMemberExpression(node, env) {
    return { ...node, type: infer(node, env) };
  }

checkRecordLiteral:

  checkRecordLiteral(node, env) {
    return { ...node, type: infer(node, env) };
  }

and checkVectorLiteral:

  checkVectorLiteral(node, env) {
    return { ...node, type: infer(node, env) };
  }

Since we're disallowing destructuring in set expressions, add this to the top of checkSetExpression:

    if (node.lhv.kind !== ASTTypes.Symbol) {
      throw new TypeException(
        `Cannot use destructuring with set! assignment`,
        node.srcloc
      );
    }
    // rest of method

The big change comes to checkVariableDeclaration with the addition of destructuring.

If we have a vector destructuring pattern, we need to make sure the type being assigned is either a vector or a list.

If we have a record destructuring pattern, we need to be sure the type being assigned is an object.

We also need to handle rest variables. In the case of a vector pattern, this is simple: just assign the vector or list type to the rest variable.

With a record it gets more complicated. We have to keep track of which properties have already been used, then construct a new object type of the remaining properties and assign that to the rest variable.

Here's the new version of checkVariableDeclaration:

  checkVariableDeclaration(node, env) {
    let type;

    if (node.typeAnnotation) {
      type = fromTypeAnnotation(node.typeAnnotation, env);
      check(node.expression, type, env);
      env.checkingOn = true;
    } else {
      type = infer(node, env);
    }

    if (node.lhv.kind === ASTTypes.Symbol) {
      env.set(node.lhv.name, type);
    } else if (node.lhv.kind === ASTTypes.VectorPattern) {
      if (!Type.isVector(type) && !Type.isList(type)) {
        throw new TypeException(
          `Vector pattern destructuring must take a vector or list type`,
          node.srcloc
        );
      } else {
        let i = 0;
        for (let mem of node.lhv.members) {
          if (node.lhv.rest && i === node.lhv.members.length - 1) {
            env.set(mem.name, type);
          } else {
            env.set(mem.name, type.vectorType ? type.vectorType : type.listType);
          }
          i++;
        }
      }
    } else if (node.lhv.kind === ASTTypes.RecordPattern) {
      if (!Type.isObject(type)) {
        throw new TypeException(
          `Cannot destructure non-object type with record pattern`,
          node.srcloc
        );
      } else {
        let i = 0;
        /** @type {string[]} */
        let used = [];
        for (let prop of node.lhv.properties) {
          if (node.lhv.rest && i === node.lhv.properties.length - 1) {
            const unusedProps = type.properties.filter(
              (p) => !used.includes(p.name)
            );

            env.set(prop.name, Type.object(unusedProps));
          } else {
            const pType = propType(type, prop.name);

            if (!pType) {
              throw new TypeException(
                `Property ${
                  prop.name
                } not found on object of type ${Type.toString(type)}`,
                node.srcloc
              );
            }

            env.set(prop.name, pType);
            used.push(prop.name);
          }

          i++;
        }
      }
    }

    return {
      kind: node.kind,
      lhv: node.lhv,
      expression: this.check(node.expression, env),
      srcloc: node.srcloc,
      type,
    };
  }

That's it for changes to the type checker. As you can see, checking types can be a lot of work. IMO It's worth it, though, when you consider how many bugs type checking can catch.

Changes to The Runtime

We need to make some updates to the runtime to handle our new expressions.

The biggest change is that we need to actually make the runtime object available from within compiled code. The reason we need to do that is because we're converting raw JS objects to a special object structure for Wanda.

Wanda objects have a :dict keyword property under the hood, and the :dict property is what actually holds its key/value pairs. The base object is for metadata. That means we need a getField function in the runtime that handles getting values from the :dict property of a Wanda object and can also get values from a raw JS object if it needs to.

We'll make some other changes as well, but that's the biggest one.

First, let's add a function to src/runtime/utils.js:

export const makeKeyword = (str) => Symbol.for(`:${str}`);

Next we're going to add some functions to convert back and forth between Wanda and JS values. Add these 2 functions to src/runtime/conversion.js:

import { makeFunction } from "./makeFunction.js";
import { hasDict, hasMetaField, makeObject } from "./object.js";
import { makeKeyword } from "./utils.js";

export const makeWandaValue = (val) => {
  switch (typeof val) {
    case "undefined":
      return null;
    case "object":
      if (val === null) {
        return null;
      }

      if (Array.isArray(val)) {
        return val;
      }

      if (val.constructor?.name === "Cons") {
        return val;
      }

      return makeObject(val);
    default:
      return val;
  }
};

export const makeJSValue = (val) => {
  if (val === null) {
    return null;
  }

  switch (typeof val) {
    case "object":
      if (hasDict(val)) {
        return val[makeKeyword("dict")];
      }
    default:
      return val;
  }
};

As you can see, this file requires 3 functions from src/runtime/object.js, which doesn't exist yet. Let's create it and add the following functions:

import { RuntimeException } from "../shared/exceptions.js";
import { hasProperty, hasMethod as hM } from "../shared/utils.js";
import { makeSymbol } from "./makeSymbol.js";
import { makeKeyword } from "./utils.js";

const DICT = makeKeyword("dict");

export const hasDict = (obj) => hasProperty(obj, DICT);

export const hasField = (obj, field) => {
  if (hasDict(obj)) {
    return hasProperty(obj[DICT], field);
  }

  return hasProperty(obj[field]);
};

export const hasMethod = (obj, method) => {
  if (hasDict(obj)) {
    return hM(obj[DICT], method);
  }

  return hM(obj, method);
};

export const getField = (obj, field) => {
  const value = hasDict(obj) ? obj[DICT][field] : obj?.[field];

  if (value === undefined) {
    failRuntime(
      `Field ${
        typeof field === "symbol" ? field.description : field
      } not found on object`
    );
  }

  if (typeof value === "function") {
    return hasDict(obj) ? value.bind(obj[DICT]) : value.bind(obj);
  }

  return value;
};

export const hasMetaField = (obj, field) => {
  const kw = makeKeyword(field);
  return hasProperty(obj[kw]);
};

export const getMetaField = (obj, field) => {
  const kw = makeKeyword(field);
  const value = obj[kw];

  if (value === undefined) {
    failRuntime(
      `Field ${
        typeof field === "symbol" ? field.description : field
      } not found on object`
    );
  }

  return value;
};

export const addMetaField = (obj, field, value) => {
  const meta = makeKeyword(field);
  Object.defineProperty(obj, meta, {
    configurable: false,
    enumerable: false,
    writable: false,
    value,
  });
};

export const makeObject = (obj) => {
  let newObj = {};
  addMetaField(newObj, "dict", obj);
  addMetaField(newObj, "constructor", function (...args) {
    return new obj.constructor(...args);
  });

  Object.defineProperty(newObj[makeKeyword("constructor")], "name", {
    configurable: false,
    enumerable: false,
    writable: false,
    value: obj.constructor?.name ?? "WandaObject",
  });

  // to allow destructuring
  for (let [k, v] of Object.entries(obj)) {
    newObj[makeSymbol(k)] = v;
  }

  return newObj;
};

export const failRuntime = (msg) => {
  throw new RuntimeException(msg);
};

Now we're finally going to use the makeFunction function in src/runtime/makeFunction.js (although we'll be adding more to it in the future):

import { makeWandaValue } from "./conversion.js";
import { addMetaField } from "./object.js";

export const makeFunction = (func) => {
  const fn = (...args) => makeWandaValue(func(...args));
  addMetaField(fn, "wanda", true);

  return fn;
};

We'll need a function to convert a string to a Wanda number in src/runtime/makeNumber.js. This will do more if I decide to add things like arbitrary-precision decimals, rational numbers, or other kinds of numbers to Wanda, but this will do fine for now:

export const makeNumber = (str) => Number(str);

Now we need to add a package. We'll need the ability to generate unique, valid JavaScript identifiers in the compiler. You'll see one reason why when we get to the changes to the emitter, but it has to do with handling destructuring. We're going to use the cuid2 package that generates unique identifiers.

npm install @paralleldrive/cuid2

Now import it into src/runtime/makeSymbol.js:

import { createId } from "@paralleldrive/cuid2";

And add this function at the bottom of the file:

export const makeGenSym = () => PREFIX + createId();

Finally, let's update src/runtime/makeRuntime.js:

import { fail } from "../shared/fail.js";
import { makeFunction } from "./makeFunction.js";
import * as utils from "./utils.js";
import * as obj from "./object.js";
import { makeSymbol, makeGenSym } from "./makeSymbol.js";
import { makeWandaValue, makeJSValue } from "./conversion.js";
import { makeNumber } from "./number.js";

export const makeRuntime = () => {
  return {
    ...utils,
    makeFunction,
    ...obj,
    makeSymbol,
    makeGenSym,
    makeWandaValue,
    makeJSValue,
    fail,
    makeNumber,
  };
};

That's it for changes to the runtime! Now let's look at changes to the emitter.

Changes to The Emitter

The first change to the emitter is we need to add the runtime object in the global environment so it's available to compiled code, in src/emitter/emitGlobalEnv.js:

import path from "path";
import { ROOT_PATH } from "../../root.js";
import { makeGlobal } from "../runtime/makeGlobals.js";

export const emitGlobalEnv = () => {
  const globalEnv = makeGlobal();
  let code = `import { makeGlobal } from "${path.join(
    ROOT_PATH,
    "./src/runtime/makeGlobals.js"
  )}";
import { makeRuntime } from "${path.join(
    ROOT_PATH,
    "./src/runtime/makeRuntime.js"
  )}";

const globalEnv = makeGlobal();
rt = makeRuntime();
`;

  for (let [k] of globalEnv) {
    code += `${k} = globalEnv.get("${k}");\n`;
  }

  return code;
};

Now in the emitter we can call the runtime functions in emitted code as long as we prefix them with rt.

Now for changes to the Emitter class. First, let's update the imports into src/emitter/Emitter.js:

import { ASTTypes } from "../parser/ast.js";
import { ReferenceException, SyntaxException } from "../shared/exceptions.js";
import { Namespace } from "../shared/Namespace.js";
import { makeGenSym, makeSymbol } from "../runtime/makeSymbol.js";

We need to add cases in the emit method for all our new AST nodes:

      // other cases...
      case ASTTypes.MemberExpression:
        return this.emitMemberExpression(node, ns);
      case ASTTypes.RecordLiteral:
        return this.emitRecordLiteral(node, ns);
      case ASTTypes.RecordPattern:
        return this.emitRecordPattern(node, ns);
      case ASTTypes.VectorLiteral:
        return this.emitVectorLiteral(node, ns);
      case ASTTypes.VectorPattern:
        return this.emitVectorPattern(node, ns);
      // default case...

We're going to edit some methods that emit primitive values to use the new makeWandaValue and makeNumber runtime functions:

  emitBoolean(node, ns) {
    return `rt.makeWandaValue(${node.value})`;
  }

  emitNil(node, ns) {
    return `rt.makeWandaValue(${null})`;
  }

  emitNumber(node, ns) {
    return `rt.makeNumber("${node.value}")`;
  }

  emitString(node, ns) {
    return `rt.makeWandaValue(${"`" + node.value.slice(1, -1) + "`"})`;
  }

In emitting member expressions we'll use the rt.getField function:

  emitMemberExpression(node, ns) {
    return `rt.getField(${this.emit(node.object, ns)}, "${
      node.property.name
    }")`;
  }

In emitting record literals we'll use the rt.makeObject function:

  emitRecordLiteral(node, ns) {
    let code = "rt.makeObject({";

    for (let prop of node.properties) {
      code += `"${prop.key.name}": ${this.emit(prop.value, ns)}, `;
    }

    code += "})";
    return code;
  }

We'll use rt.makeWandaValue for emitting vector literals:

  emitVectorLiteral(node, ns) {
    let code = "rt.makeWandaValue([";

    for (let mem of node.members) {
      code += `${this.emit(mem, ns)}, `;
    }

    code += "])";
    return code;
  }

Emitting record patterns is fairly straightforward except for one thing: handling rest variables. Since we're using the :dict property on Wanda variables, that greatly complicates destructuring objects.

For most properties it's simple, because we're making the JavaScript identifier versions of the property names and putting them on the base objects, but handling the rest variable is a whole ordeal and we're just not going to mess with it in this method. We'll handle it when emitting variable declarations instead. So we just break out of the loop when we get to the rest variable here.

  emitRecordPattern(node, ns) {
    let code = "{";

    let i = 0;
    for (let prop of node.properties) {
      if (node.rest && i === node.properties.length - 1) {
        // this is the rest variable, which requires extra work
        // see: this.emitVariableDeclarationAssignment
        break;
      } else {
        code += `${this.emit(prop, ns)}, `;
      }

      i++;
    }

    code += "}";
    return code;
  }

Emitting vector patterns, on the other hand, is just what you'd expect:

  emitVectorPattern(node, ns) {
    let code = "[";

    let i = 0;
    for (let mem of node.members) {
      if (node.rest && i === node.members.length - 1) {
        code += `...${this.emit(mem, ns)}`;
      } else {
        code += `${this.emit(mem, ns)}, `;
      }
      i++;
    }

    code += "]";
    return code;
  }

There are big changes to emitVariableDeclaration to handle destructuring. First, we need to make sure to populate the namespace with the mappings from Wanda identifiers to JavaScript identifiers. Then we'll delegate emitting the actual code to the new emitVariableDeclarationAssignment method.

  emitVariableDeclaration(node, ns) {
    if (node.lhv.kind === ASTTypes.Symbol) {
      const name = node.lhv.name;

      if (ns.has(name)) {
        throw new ReferenceException(
          `Name ${name} has already been accessed in the current namespace; cannot access name before its definition`,
          node.srcloc
        );
      }

      const translatedName = makeSymbol(name);
      ns.set(name, translatedName);
    } else if (
      node.lhv.kind === ASTTypes.VectorPattern ||
      node.lhv.kind === ASTTypes.RecordPattern
    ) {
      const members =
        node.lhv.kind === ASTTypes.RecordPattern
          ? node.lhv.properties
          : node.lhv.members;

      for (let mem of members) {
        if (ns.has(mem.name)) {
          throw new ReferenceException(
            `Name ${mem.name} has already been accessed in the current namespace; cannot access name before its definition`,
            mem.srcloc
          );
        }
        ns.set(mem.name, makeSymbol(mem.name));
      }
    }

    return this.emitVariableDeclarationAssignment(
      node.lhv,
      node.expression,
      ns
    );
  }

In emitVariableDeclarationAssignment, handling cases where the LHV is a symbol or a vector pattern are straightforward: just emit both sides of the assignment in the context of a variable declaration.

  emitVariableDeclarationAssignment(lhv, rhv, ns) {
    if (lhv.kind === ASTTypes.Symbol) {
      return `var ${makeSymbol(lhv.name)} = ${this.emit(rhv, ns)}`;
    } else if (lhv.kind === ASTTypes.VectorPattern) {
      return `var ${this.emit(lhv, ns)} = ${this.emit(rhv, ns)}`;
    } // handling record pattern...
  }

Handling the record pattern is a bit of a mess. We generate a unique identifier and assign the actual object to it. Then we destructure the record pattern off the unique identifier, skipping over the rest variable.

We use the type the AST has been decorated with to construct a new object containing the properties that haven't yet been used, then emit another variable declaration that assigns that object to the rest variable's identifier.

And that's how we do it.

This is a case where inferring types helps us even if the type checker is turned off.

Here's the emitVariableDeclarationAssignment method in its entirety:

  emitVariableDeclarationAssignment(lhv, rhv, ns) {
    if (lhv.kind === ASTTypes.Symbol) {
      return `var ${makeSymbol(lhv.name)} = ${this.emit(rhv, ns)}`;
    } else if (lhv.kind === ASTTypes.VectorPattern) {
      return `var ${this.emit(lhv, ns)} = ${this.emit(rhv, ns)}`;
    } else if (lhv.kind === ASTTypes.RecordPattern) {
      /* Note that this DOES NOT WORK on rest variables in nested record patterns */
      // create random variable to hold object being destructured
      const gensym = makeGenSym();
      let origObjCode = `var ${gensym} = ${this.emit(rhv, ns)}`;
      let code = `${origObjCode};\n`;

      code += `var ${this.emit(lhv, ns)} = ${gensym};\n`;

      if (lhv.rest) {
        /** @type {string[]} */
        let used = [];
        let i = 0;
        for (let prop of lhv.properties) {
          if (i !== lhv.properties.length - 1) {
            used.push(prop.name);
          }
          i++;
        }

        /** @type {import("../parser/ast.js").Property[]} */
        const unusedProps = rhv.type.properties.filter((p) => {
          return !used.includes(p.name);
        });

        const restVarName = lhv.properties[lhv.properties.length - 1].name;
        let restObjCode = "{ ";

        for (let prop of unusedProps) {
          restObjCode += `"${prop.name}": rt.getField(${gensym}, "${prop.name}"), `;
        }

        restObjCode += "}";

        code += `var ${makeSymbol(restVarName)} = ${restObjCode}`;
      }

      return code;
    }
  }

That's all for changes to the emitter. Now for changes to the printer.

Changes to The Printer

All the changes in src/printer/printString.js are to the "object" case in the switch statement:

    // other cases
    case "object":
      if (value === null) {
        return "nil";
      } else if (value.constructor?.name === "Cons") {
        return printList(value);
      } else if (Array.isArray(value)) {
        return `[${value.map(printString).join(", ")}]`;
      }

      return hasDict(value)
        ? JSON.stringify(value[Symbol.for(":dict")], null, 2)
        : JSON.stringify(value, null, 2);
    // default case

We're punting a bit by just using JSON.stringify, but we'll probably clean that up later.

We've also added methods to the ASTPrinter class to print the new nodes. First, the new cases for the switch statement in print:

      // other cases...
case ASTTypes.RecordLiteral:
        return this.printRecordLiteral(node, indent);
      case ASTTypes.RecordPattern:
        return this.printRecordPattern(node, indent);
      case ASTTypes.VectorLiteral:
        return this.printVectorLiteral(node, indent);
      case ASTTypes.VectorPattern:
        return this.printVectorPattern(node, indent);
      case ASTTypes.MemberExpression:
        return this.printMemberExpression(node, indent);
      // default case

And the methods:

  printRecordLiteral(node, indent) {
    const prStr = `${prIndent(indent)}RecordLiteral:`
    prStr += `${prIndent(indent + 2)}Properties:`

    for (let prop of node.properties) {
      prStr += `${this.print(prop.key, indent + 4)}`;
      prStr += `${this.print(prop.value, indent + 4)}`;
    }

    return prStr
  }

  printRecordPattern(node, indent) {
    return `${prIndent(indent)}${node.properties.map(p => p.name).join(", ")}`;
  }

  printVectorLiteral(node, indent) {
    let prStr = `${prIndent(indent)}VectorLiteral:`

    for (let mem of node.members) {
      prStr += `${this.print(mem, indent + 2)}`;
    }

    return prStr;
  }

  printVectorPattern(node, indent) {
    return `${prIndent(indent)}${node.members.map (p => p.name).join(", ")}`;
  }

  printMemberExpression(node, indent) {
    let prStr = `${prIndent(indent)}MemberExpression:`
    prStr += `${prIndent(indent + 2)}Object:`
    prStr += `${this.print(node.object, indent + 4)}`;
    prStr += `${prIndent(indent + 2)}Property:`
    prStr += `${this.print(node.property, indent + 4)}`

    return prStr;
  }

That's it for changes to the printer, and that does it for changes to the compiler to add these new features.

Go ahead and fire up a REPL using your wanda command and check them out!

Conclusion

We added 5 new node types, 2 new kinds of types, the ability to access record members with dot notation, and toplevel destructuring to our compiler. You're doing a fantastic job if you're keeping up with everything!

As always, feel free to let me know in the comments if you're having any difficulties. If there are bugs I'll try to fix them and update the article accordingly.

You can see the state of the code as of the end of this article in the Wanda repo on the relevant tag.

There are tests in the repo, but I must confess I haven't updated tests for this article so there's nothing new there since the last time I published. I need to do better at keeping up with tests.

There are also JSDoc blocks for nearly all functions and types defined in the repo to help with Intellisense if you want to hack on the compiler a bit. If you come up with any awesome new features, feel free to submit a pull request!

In the next article we'll implement functions and the ability to define your own functions, which will include the ability to define function contracts in native JS modules. Stay tuned!

Create Your Own Programming Language 4: Variables and Types

Jason Barr — Sun, 11 Jun 2023 04:17:17 +0000

Wanda has come a long way since we started. We have the ability to call functions and work with basic data types. The reader also works with linked lists like a real Lisp.

Now it's time to add variables to the language.

In this article we'll add 3 new syntactic forms: variable declarations, set expressions, and do blocks.

Variable declarations are exactly what they sound like. Set expressions allow you to change the value of a variable. Do blocks are a list of expressions you can use when you want to fit multiple expressions into a space made for just a single expression.

The Perils of Mutable State

Note that by introducing both variable declarations and a means to modify the value of variables we're opening the door to having mutable state in our programs. Mutable state means values your program relies on to calculate its result that can change throughout the course of the program.

I've decided to allow mutable state in Wanda as a kind of "escape hatch" for when it's really necessary, but the reality is it's actually rare that you really need mutable state. It's even more rare that using mutable state is actually a good idea.

By avoiding mutable state in your programs you go a long way towards making your programs easier to reason about and understand. Just because I'm giving us set expressions in Wanda does not mean it's a good idea to actually use them unless it's absolutely necessary.

Ok, end rant.

Semantic Analysis

We'll also dip our toes into the water of semantic analysis and start working on a proper type checker for Wanda.¹

Wanda is a gradually typed language, which means you can use it either like a statically typed language or a dynamically typed one. With type annotations, you turn on the type checker for an extra layer of safety. If you use the language without annotations, you can work with it just like any other dynamically typed language; the type checker won't get in your way.

When we're done, Wanda will have a very expressive type system including records, objects and classes, algebraic data types, and generics.

As always, if you haven't read the previous post on call expressions you should go ahead and do that first. Really, it's ok. I'll wait right here for you.

Ok, ready? Awesome, let's go!

Fixing A Mistake from Last Time

First, we need to undo something we did in the last article. I didn't think through things well enough, and I neglected to consider the impact it would have to wrap the compiled output of a Program node in an immediately-invoked function expression (IIFE).

The problem is when we try to declare variables: declaring a variable inside an IIFE makes the variable local to that function, which means we can never use it again. That's obviously not what we want.

So open src/emitter/Emitter.js and modify the emitProgram method like this:

  emitProgram(node, ns) {
    let code = "";
    for (let n of node.body) {
      code += `${this.emit(n, ns)};\n`;
    }

    return code;
  }

That takes care of the problem.

What We Don't Need To Change Today

The lexer and reader have occupied a large amount of our time in the first few articles in this series, but for today's article we can leave them just as they are.

What We Do Need To Change Today

We'll make some changes to the parser to accommodate the new syntactic forms and to parse type annotations. We'll need to update the emitter to handle the new forms, and we'll also need to implement the guts of the type checker.

The type checker will take the most time and effort to implement, so we'll save it for last.

We'll also update the CLI so the type checker is part of our compilation pipeline and allow multiline input in the REPL. A common convention with do blocks is to put each expression on its own line, which is hard to do when all your input has to be on the same line in the REPL.

Updates To The Parser

The first big update to the parser we're going to make is to change a property on all the AST nodes.

Of Types and Kinds

I've decided that since we're doing type checking in the compiler it makes more sense to rename the type property on AST nodes to kind and save things related to the word "type" for type checking.

In src/parser/ast.js update the node creators like so:

export const AST = {
  Program(exprs) {
    return {
      kind: ASTTypes.Program,
      body: exprs,
      srcloc: exprs[0]?.srcloc ?? SrcLoc.new(0, 0, 0, "none"),
    };
  },
  NumberLiteral(token) {
    return {
      kind: ASTTypes.NumberLiteral,
      value: token.value,
      srcloc: token.srcloc,
    };
  },
  StringLiteral(token) {
    return {
      kind: ASTTypes.StringLiteral,
      value: token.value,
      srcloc: token.srcloc,
    };
  },
  BooleanLiteral(token) {
    return {
      kind: ASTTypes.BooleanLiteral,
      value: token.value,
      srcloc: token.srcloc,
    };
  },
  KeywordLiteral(token) {
    return {
      kind: ASTTypes.KeywordLiteral,
      value: token.value,
      srcloc: token.srcloc,
    };
  },
  NilLiteral(token) {
    return {
      kind: ASTTypes.NilLiteral,
      value: token.value,
      srcloc: token.srcloc,
    };
  },
  Symbol(token) {
    return {
      kind: ASTTypes.Symbol,
      name: token.value,
      srcloc: token.srcloc,
    };
  },
  CallExpression(func, args, srcloc) {
    return {
      kind: ASTTypes.CallExpression,
      func,
      args,
      srcloc,
    };
  },
};

In src/emitter/Emitter.js switch on node.kind instead of node.type in the emit method:

  emit(node = this.program, ns = this.ns) {
    switch (node.kind) {
    // and so on...

Then, also in the emit method, in the default block change the first argument to SyntaxException from node.type to node.kind.

Finally, in src/printer/printAST.js change all references to node.type to node.kind including in the print and printPrimitive methods.

Parsing The New Forms

Variable declarations, set expressions, and do blocks have the following syntax:

(var x 10)
(set! x 15)
(do
  (var a 10)
  (println "About to increment a")
  (+ a 1))

Set expressions use the set! keyword, with the exclamation point as an indicator that this form mutates a value.

You can also use a type annotation with a variable declaration to turn on the type checker:

(var (x : number) 10)

Note that there must be a space both between the variable name and the colon and the colon and the type. That's to prevent the lexer from reading the colon as either part of the variable name or the beginning of a keyword.

First, let's add the AST node types to the node constructors object. In src/parser/ast.js:

export const AST = {
  // existing constructors...
  VariableDeclaration(lhv, expression, srcloc, typeAnnotation = null) {
    return {
      kind: ASTTypes.VariableDeclaration,
      lhv,
      expression,
      srcloc,
      typeAnnotation,
    };
  },
  SetExpression(lhv, expression, srcloc) {
    return {
      kind: ASTTypes.SetExpression,
      lhv,
      expression,
      srcloc,
    };
  },
  DoExpression(body, srcloc) {
    return {
      kind: ASTTypes.DoExpression,
      body,
      srcloc,
    };
  },
};

We're just going to stub a function for parsing type annotations for right now; we'll come back to it when it's time to work on the type checker. In src/parser/parseTypeAnnotation.js add this stub:

export const parseTypeAnnotation = (annotation) => {};

Now we get to add our first non-call expression forms to the parseList function. First, let's write the parse functions for variable declarations, set expressions, and do blocks.

For variable declarations and set expressions, we'll need left hand values (LHVs) and initializer expressions. For do blocks, we'll just need a list of expressions.

I've gone ahead and included everything you'll need in this file to parse type annotations. Note that we're skipping over the colon and converting the tail of the Cons list to a JavaScript array.

In src/parser/parse.js:

const parseVariableDeclaration = (decl) => {
  let [_, lhv, expression] = decl;

  let parsedLhv,
    typeAnnotation = null;
  if (lhv instanceof Cons) {
    // has type annotation
    const realLhv = lhv.get(0);
    // convert to array and get rid of ":" when passing into parseTypeAnnotation
    typeAnnotation = parseTypeAnnotation([...lhv.cdr].slice(1));
    parsedLhv = parseExpr(realLhv);
  } else {
    parsedLhv = parseExpr(lhv);
  }

  const parsedExpression = parseExpr(expression);

  return AST.VariableDeclaration(
    parsedLhv,
    parsedExpression,
    decl.srcloc,
    typeAnnotation
  );
};

const parseSetExpression = (expr) => {
  const [_, lhv, expression] = expr;
  const parsedLhv = parseExpr(lhv);
  const parsedExpression = parseExpr(expression);

  return AST.SetExpression(parsedLhv, parsedExpression, expr.srcloc);
};

const parseDoExpression = (expr) => {
  const [_, ...exprs] = expr;
  let body = [];

  for (let ex of exprs) {
    body.push(parseExpr(ex));
  }

  return AST.DoExpression(body, expr.srcloc);
};

Now, in the parseList function, add the following 3 cases to the switch statement:

  switch (first.value) {
    case "var":
      return parseVariableDeclaration(form);
    case "set!":
      return parseSetExpression(form);
    case "do":
      return parseDoExpression(form);
    // default case...

That's it for the parser until we come back to parse type annotations. We'll also add parsing type aliases, which is super simple.

Changes To The Emitter

Obviously we need to add methods for each of the 3 new AST nodes to the emitter, as well as adding clauses to the switch statement in the emit method.

We're also going to make it so it does a little bit of semantic checking in the emitter.

We're going to make the emitter check to see if a variable has been accessed prior to declaration in its scope, instead of adding a separate pass to the semantic analysis phase.

This isn't necessarily the best way to do it. Ideally, we would create a separate visitor to check that or do it in the type checker. However, it's the simplest way to do it in our case. Doing it like this prevents us from having to implement 2 passes during semantic analysis (and thus from having to maintain 2 environments in the REPL) or make the type checker more complicated.

First, let's update the imports for the emitter in src/emitter/Emitter.js:

import { makeSymbol } from "../runtime/makeSymbol.js";

Now let's write the 3 emitter methods emitDoExpression, emitSetExpression, and emitVariableDeclaration:

  emitDoExpression(node, ns) {
    const childNs = ns.extend("doExpression");
    let code = "(() => {";

    let i = 0;
    for (let expr of node.body) {
      if (i === node.body.length - 1) {
        code += "return " + this.emit(expr, childNs) + "\n";
      } else {
        code += this.emit(expr, childNs) + "\n";
      }
      i++;
    }

    code += "})();";
    return code;
  }

  // more methods...
  emitSetExpression(node, ns) {
    return `${this.emit(node.lhv, ns)} = ${this.emit(node.expression, ns)};`;
  }

  // more methods...
  emitVariableDeclaration(node, ns) {
    const name = node.lhv.name;

    if (ns.has(name)) {
      throw new Exception(
        `Name ${name} defined at ${node.srcloc.file} ${node.srcloc.line}:${node.srcloc.col} has already been accessed in the current namespace; cannot access identifier before defining it`
      );
    }

    const translatedName = makeSymbol(name);

    ns.set(name, translatedName);

    return `var ${translatedName} = ${this.emit(node.expression, ns)};`;
  }

Then in the switch statement in the emit method add the following cases just above the default case:

      case ASTTypes.VariableDeclaration:
        return this.emitVariableDeclaration(node, ns);
      case ASTTypes.SetExpression:
        return this.emitSetExpression(node, ns);
      case ASTTypes.DoExpression:
        return this.emitDoExpression(node, ns);

Finally in the emitSymbol method, just before the return statement, add:

    // To make sure when compiling a variable definition the variable name hasn't already been accessed in the same scope
    if (!ns.has(name)) {
      ns.set(name, emittedName);
    }

Changes to The AST Printer

Finally, let's wrap this up by making it so the AST Printer can print do blocks, variable declarations, and set expressions.

In src/printer/printAST.js, add these cases to the switch statement in the print method:

      // additional cases
      case ASTTypes.VariableDeclaration:
        return this.printVariableDeclaration(node, indent);
      case ASTTypes.SetExpression:
        return this.printSetExpression(node, indent);
      case ASTTypes.DoExpression:
        return this.printDoExpression(node, indent);
      // default case

Then add the methods printDoExpression, printSetExpression, and printVariableDeclaration:

  printDoExpression(node, indent) {
    let prStr = `${prIndent(indent)}DoExpression:\n`;

    let i = 0;
    for (let expr of node.body) {
      prStr += `${this.print(expr, indent + 2)}`;
      prStr += i === node.body.length - 1 ? "" : "\n";
      i++;
    }

    return prStr;
  }

  printSetExpression(node, indent) {
    let prStr = `${prIndent(indent)}SetExpression:\n`;
    prStr += `${this.print(node.lhv, indent + 2)}\n`;
    prStr += `${this.print(node.expression, indent + 2)}`;
    return prStr;
  }

  printVariableDeclaration(node, indent) {
    let prStr = `${prIndent(indent)}VariableDeclaration:\n`;
    prStr += `${this.print(node.lhv, indent + 2)}\n`;
    prStr += `${this.print(node.expression, indent + 2)}`;
    return prStr;
  }

Believe it or not, that's all it takes to add variable declarations, set expressions, and do blocks to Wanda! We'll make some changes to the CLI later to include multiline input in the REPL, but first let's turn our attention to the type checker.

Parsing and The Type Checker

The first thing we need to do is parse type annotations. We stubbed out the function earlier, but now we're going to fill it in.

We're also going to add type aliases to Wanda. This allows us to give names to types. This can make things more expressive for programmers. Let's say, for instance, that you have a function where age is represented by a number. If you want to refer to a value as age instead of number, just create a type alias:

(type age number)

This is a simple example, but when types get more complex this feature will be more useful than it currently appears.

Let's look at src/parser/parseTypeAnnotation.js.

First, we'll construct an enum of Type Annotation Types:

export const TATypes = {
  NumberLiteral: "NumberLiteral",
  StringLiteral: "StringLiteral",
  BooleanLiteral: "BooleanLiteral",
  KeywordLiteral: "KeywordLiteral",
  NilLiteral: "NilLiteral",
  Symbol: "Symbol",
  List: "List",
};

If an annotation is coming into the function as a Cons list we're just going to convert it to an array for simplicity's sake.

Then, if an annotation is an array, we'll pluck the 1st item for testing. Otherwise, the annotation is a simple form and we'll just use it as is.

Currently, the only values we have for testing are Tokens of type "Symbol." So we'll switch on the token's value property to see what kind of annotation it is.

The annotation will be either a literal type or a named type. Literal types include primitives like number, as well as lists and type aliases. A list has a list type, for example it could be list (string) with string as the list type.

Here's the parseTypeAnnotation function:

export const parseTypeAnnotation = (annotation) => {
  let annot;

  if (annotation instanceof Cons) {
    annotation = [...annotation];
  }

  if (Array.isArray(annotation)) {
    annot = annotation[0];
  } else {
    annot = annotation;
  }

  if (annot.type === TokenTypes.Symbol) {
    switch (annot.value) {
      case "number":
        return { kind: TATypes.NumberLiteral };
      case "string":
        return { kind: TATypes.StringLiteral };
      case "boolean":
        return { kind: TATypes.BooleanLiteral };
      case "keyword":
        return { kind: TATypes.KeywordLiteral };
      case "nil":
        return { kind: TATypes.NilLiteral };
      case "list":
        // annotation is array with listType as 2nd member
        // i.e. list (string) annotation == [list, string]
        return parseListAnnotation(annotation[1]);
      default:
        // must be a named type
        return { kind: TATypes.Symbol, name: annot.value };
    }
  }
};

The parseListAnnotation function is simple:

const parseListAnnotation = (type) => {
  const listType = parseTypeAnnotation(type);
  return { kind: TATypes.List, listType };
};

We'll add one more node to the parser for type aliases. In src/parser/ast.js add a member to the ASTTypes enum:

export const ASTTypes = {
  // members...
  TypeAlias: "TypeAlias",
};

Then add the TypeAlias node constructor:

export const AST = {
  // additional constructors...
  TypeAlias(name, type, srcloc) {
    return {
      kind: ASTTypes.TypeAlias,
      name,
      type,
      srcloc,
    };
  },
};

Add a parseTypeAlias function to src/parser/parse.js:

const parseTypeAlias = (form) => {
  let [_, name, type] = form;
  name = name.value;
  const parsedType = parseTypeAnnotation(type);

  return AST.TypeAlias(name, parsedType, form.srcloc);
};

And add the following case to the switch statement in parseList:

    // additional cases...
    case "type":
      return parseTypeAlias(form);
    // default case

Now the parser is ready to handle type annotations and type aliases!

Emitting and Printing Type Aliases

The emitter and AST printer methods for type aliases both just return empty strings. JavaScript has no equivalent for a type alias, so there's no code to emit.

In src/emitter/Emitter.js add the type alias case to the switch in the emit method:

      // other cases
      case ASTTypes.TypeAlias:
        return this.emitTypeAlias(node, ns);
      // default case

Then the emitTypeAlias method:

  emitTypeAlias(node, ns) {
    return "";
  }

Then do the same in the switch statement in the print method in src/printer/printAST.js:

      // other cases...
      case ASTTypes.TypeAlias:
        return this.printTypeAlias(node, indent);
      // default case

And the printTypeAlias method:

  printTypeAlias(node, indent) {
    return "";
  }

Implementing The Type Checker

Here's a high-level overview of how the type checker works.

When the checker encounters an expression, it first tries to infer a type from the expression. If the expression uses a type annotation, it checks the inferred type against the annotated type to make sure the inferred type is a proper subtype of the annotated type. If it is, then we're all good. Otherwise, it throws an Exception.

The presence of a single type annotation will turn on checking for the entire program. A more sophisticated gradual type checker might only check variables and functions that were specifically annotated and leave the rest alone, but that would be much more complicated.

Types, Constructors, and Validators

The first thing we need to do is define our basic types. Then we'll create constructors and validators for the types so we can do the work of checking and returning types from the inference and checking functions.

Here's the src/typechecker/types.js file, which contains an enum of TypeTypes as well as JSDoc definitions for different type objects:

/**
 * @enum {string}
 */
export const TypeTypes = {
  Any: "Any",
  Number: "Number",
  String: "String",
  Boolean: "Boolean",
  Keyword: "Keyword",
  Nil: "Nil",
  FunctionType: "FunctionType",
  TypeAlias: "TypeAlias",
  List: "List",
};
/**
 * @typedef Any
 * @prop {TypeTypes.Any} kind
 */
/**
 * @typedef Number
 * @prop {TypeTypes.Number} kind
 */
/**
 * @typedef String
 * @prop {TypeTypes.String} kind
 */
/**
 * @typedef Boolean
 * @prop {TypeTypes.Boolean} kind
 */
/**
 * @typedef Keyword
 * @prop {TypeTypes.Keyword} kind
 */
/**
 * @typedef Nil
 * @prop {TypeTypes.Nil} kind
 */
/**
 * @typedef TypeAlias
 * @prop {TypeTypes.TypeAlias} kind
 * @prop {string} name
 * @prop {Type} base
 */
/**
 * @typedef FunctionType
 * @prop {TypeTypes.FunctionType} kind
 * @prop {Type[]} params
 * @prop {Type} ret
 * @prop {boolean} variadic
 */
/**
 * @typedef List
 * @prop {TypeTypes.List} kind
 * @prop {Type} listType
 */
/**
 * @typedef {Number|String|Boolean|Keyword|Nil} PrimitiveTypes
 */
/**
 * @typedef {Any|PrimitiveTypes|FunctionType|TypeAlias|List} Type
 */

Note the existence of an Any type. Since Wanda is gradually typed, we need a case to handle types when type checking is turned off. The Any type is compatible with every other type and every other type is compatible with it, meaning any time you check against an Any type the check will pass. This has the effect of basically turning off the type checker.

Obviously as we add new kinds of types (like object types) this file will grow. Notice that we have a type for FunctionType even though we haven't added functions to the language yet. That's because we need it for checking call expressions.

Next we have type constructors. These return objects shaped like the type definitions in types.js. In src/typechecker/constructors.js:

import { TypeTypes } from "./types.js";

export const any = { kind: TypeTypes.Any };

export const number = { kind: TypeTypes.Number };

export const string = { kind: TypeTypes.String };

export const boolean = { kind: TypeTypes.Boolean };

export const keyword = { kind: TypeTypes.Keyword };

export const nil = { kind: TypeTypes.Nil };

export const functionType = (params, ret, variadic = false) => {
  return { kind: TypeTypes.FunctionType, params, ret, variadic };
};

export const typeAlias = (name, base) => ({
  kind: TypeTypes.TypeAlias,
  name,
  base,
});

export const listType = (listType) => ({ kind: TypeTypes.List, listType });

Now let's combine our TypeType enum, constructors, and validators together into a single Type module. We'll add more to this module later, but this will do for now. In src/typechecker/Type.js:

import { TypeTypes } from "./types.js";
import * as Validators from "./validators.js";
import * as Constructors from "./constructors.js";

export const Type = {
  Type: TypeTypes,
  ...Constructors,
  ...Validators,
};

From Annotations to Types

Next we need to be able to convert type annotations into concrete types. In src/typechecker/fromTypeAnnotation.js:

import { TATypes } from "../parser/parseTypeAnnotation.js";
import { Type } from "./Type.js";
import { Exception } from "../shared/exceptions.js";

export const fromTypeAnnotation = (typeAnnotation, typeEnv) => {
  switch (typeAnnotation.kind) {
    case TATypes.NumberLiteral:
      return Type.number;
    case TATypes.StringLiteral:
      return Type.string;
    case TATypes.BooleanLiteral:
      return Type.boolean;
    case TATypes.KeywordLiteral:
      return Type.keyword;
    case TATypes.NilLiteral:
      return Type.nil;
    case TATypes.Symbol: {
      const name = typeAnnotation.name;
      const type = typeEnv.getType(name);

      if (!type) {
        throw new Exception(
          `Type ${name} not found in current type environment`
        );
      }

      return type;
    }
    case TATypes.List: {
      const listType = fromTypeAnnotation(typeAnnotation.listType, typeEnv);
      return Type.listType(listType);
    }
    default:
      throw new Exception(
        `Type not found for type annotation ${JSON.parse(
          typeAnnotation,
          null,
          2
        )}`
      );
  }
};

Converting Types to Strings

We'll also need a function to convert types to strings for things like error messages. In src/typechecker/typeToString.js:

import { Exception } from "../shared/exceptions.js";
import { TypeTypes } from "./types.js";

export const typeToString = (type) => {
  switch (type.kind) {
    case TypeTypes.Any:
      return "any";
    case TypeTypes.Number:
      return "number";
    case TypeTypes.String:
      return "string";
    case TypeTypes.Boolean:
      return "boolean";
    case TypeTypes.Keyword:
      return "keyword";
    case TypeTypes.Nil:
      return "nil";
    case TypeTypes.FunctionType:
      return `(${type.params.map(typeToString).join(", ")}) -> ${typeToString(
        type.ret
      )}`;
    case TypeTypes.TypeAlias:
      return `TypeAlias: ${type.name}, base: ${typeToString(type.base)}`;
    case TypeTypes.List:
      return `list (${typeToString(type.listType)})`;
    default:
      throw new Exception(`String conversion not implemented for ${type.kind}`);
  }
};

Now add typeToString to the Type module in src/typechecker/Type.js:

import { typeToString } from "./typeToString.js";

export const Type = {
  // existing members...
  toString: typeToString,
};

Subtypes

The main purpose of the check function we're going to write in a minute is to compare a type to a node and see if the node is a subtype of the provided type. To that end, we need a function to check if one type is a subtype of another.

Add this function in src/typechecker/isSubtype.js to check if type1 is a subtype of type2:

import { Type } from "./Type.js";

export const isSubtype = (type1, type2) => {
  if (Type.isAny(type1) || Type.isAny(type2)) return true;
  if (Type.isNumber(type1) && Type.isNumber(type2)) return true;
  if (Type.isString(type1) && Type.isString(type2)) return true;
  if (Type.isBoolean(type1) && Type.isBoolean(type2)) return true;
  if (Type.isKeyword(type1) && Type.isKeyword(type2)) return true;
  if (Type.isNil(type1) && Type.isNil(type2)) return true;

  if (Type.isTypeAlias(type1) && Type.isTypeAlias(type2)) {
    return isSubtype(type1.base, type2.base);
  }

  if (Type.isTypeAlias(type1) && !Type.isTypeAlias(type2)) {
    return isSubtype(type1.base, type2);
  }

  if (Type.isTypeAlias(type2) && !Type.isTypeAlias(type1)) {
    return isSubtype(type1, type2.base);
  }

  if (Type.isList(type1) && Type.isList(type2)) {
    return isSubtype(type1.listType, type2.listType);
  }

  return false;
};

The Type Environment

Similar to the Namespace we use in the emitter, we also need a compile-time environment for the type checker. We'll call this class TypeEnvironment and have it inherit from Namespace.

Note that I've moved the Namespace class from src/runtime to src/shared. If you do the same, you'll need to update your imports accordingly.

In src/typechecker/TypeEnvironment.js:

import { Namespace } from "../shared/Namespace.js";

export class TypeEnvironment extends Namespace {
  constructor(parent = null, { name = "global" } = {}) {
    super(parent, { name });
    this.types = new Map();
    this.checkingOn = false;
  }

  static new(parent = null, { name = "global" } = {}) {
    return new TypeEnvironment(parent, { name });
  }

  existsType(name) {
    return this.lookupType(name) !== null;
  }

  extend(name) {
    return new TypeEnvironment(this, { name });
  }

  getType(name) {
    const scope = this.lookupType(name);

    if (scope) {
      return scope.types.get(name);
    }

    return null;
  }

  hasType(name) {
    return this.types.has(name);
  }

  lookupType(name) {
    let scope = this;

    while (scope) {
      if (scope.types.has(name)) {
        return scope;
      }

      scope = scope.parent;
    }

    return null;
  }

  setType(name, type) {
    this.types.set(name, type);
  }
}

Note the checkingOn property set in the constructor. We'll set that property to true if we encounter a type annotation so the checker knows to check the types associated with variables and, when we have functions, function parameter and return types against the types it infers from expressions.

Getting The Base Type of A Type Alias

We also need to be able to get the base type of a type alias so we can properly type check annotations that name an alias. In src/typechecker/utils.js create the getAliasBase function:

import { Type } from "./Type.js";

export const getAliasBase = (name, env) => {
  let baseType = env.getType(name);

  while (Type.isTypeAlias(baseType)) {
    baseType = env.getType(baseType.name);
  }

  return baseType;
};

Inferring Types from Expressions

Now we're getting into the real meat of the type checking implementation.

Our type checker is going to be bidirectional, which means it can get types either from annotations or from the expressions themselves, which allows us to drastically reduce the number of type annotations required to properly type check a program.

We don't have functions in the language yet, but when we do we'll be able to type check most programs based solely on the annotations on functions and their parameters.

This is how the TypeScript type checker works, and our type system is heavily influenced by TypeScript's.

To that end, we need an infer function that infers types from expressions.

In src/typechecker/infer.js you can see that the main infer function dispatches to sub-functions:

import { ASTTypes } from "../parser/ast.js";
import { Exception } from "../shared/exceptions.js";
import { Type } from "./Type.js";
import { isSubtype } from "./isSubtype.js";
import { getAliasBase } from "./utils.js";
import { fromTypeAnnotation } from "./fromTypeAnnotation.js";

export const infer = (ast, env) => {
  switch (ast.kind) {
    case ASTTypes.NumberLiteral:
      return inferNumber();
    case ASTTypes.StringLiteral:
      return inferString();
    case ASTTypes.BooleanLiteral:
      return inferBoolean();
    case ASTTypes.KeywordLiteral:
      return inferKeyword();
    case ASTTypes.NilLiteral:
      return inferNil();
    case ASTTypes.Symbol:
      return inferSymbol(ast, env);
    case ASTTypes.CallExpression:
      return inferCallExpression(ast, env);
    case ASTTypes.VariableDeclaration:
      return inferVariableDeclaration(ast, env);
    case ASTTypes.SetExpression:
      return inferSetExpression(ast, env);
    case ASTTypes.DoExpression:
      return inferDoExpression(ast, env);
    case ASTTypes.TypeAlias:
      return inferTypeAlias(ast, env);
    default:
      throw new Exception(`No type inferred for AST node type ${ast.kind}`);
  }
};

Inferring for literal expressions is simple. We just return the type constructor for the literal node:

// Infer types from literal nodes
const inferNumber = () => Type.number;
const inferString = () => Type.string;
const inferBoolean = () => Type.boolean;
const inferKeyword = () => Type.keyword;
const inferNil = () => Type.nil;

For inferSymbol we need to look up the symbol in the type environment, then get the base type if the symbol resolves to a type alias:

const inferSymbol = (node, env) => {
  const name = node.name;
  const namedType = env.get(name);
  const baseType = Type.isTypeAlias(namedType)
    ? getAliasBase(namedType.name)
    : namedType;

  if (!namedType) {
    throw new Exception(
      `Type not found in current environment for ${name} at ${node.srcloc.file} ${node.srcloc.col}:${node.srcloc.col}`
    );
  }

  return baseType;
};

For inferCallExpression we infer the type for the function. If the function is of type Any we just return the Any type.

Otherwise we check to make sure the number of arguments is correct. For variadic functions this means a number of arguments equal to or greater than the function's arity (number of arguments it takes). For all other functions, this means the same number of arguments as the function's arity.

If type checking is on, we check the types of the arguments against the types of the formal parameters. This includes checking variadic arguments against the type of the variadic parameter.

Finally, if everything checks out, we return the function's return type.

const inferCallExpression = (node, env) => {
  const func = infer(node.func, env);

  if (Type.isAny(func)) {
    return Type.any;
  }

  if (
    node.args.length !== func.params.length ||
    (func.variadic && node.args.length >= func.params.length - 1)
  ) {
    throw new Exception(
      `Expected${func.variadic ? " at least " : " "}arguments; ${
        node.args.length
      } given at ${node.srcloc.file} ${node.srcloc.line}:${node.srcloc.col}`
    );
  }

  if (env.checkingOn) {
    func.params.forEach((p, i, a) => {
      const argType = infer(node.args[i], env);
      if (!isSubtype(argType, p)) {
        const node = node.args[i];
        throw new Exception(
          `${Type.toString(argType)} is not a subtype of ${Type.toString(
            p
          )} at ${node.srcloc.file} ${node.srcloc.line}:${node.srcloc.col}`
        );
      }

      if (i === a.length - 1) {
        for (let arg of node.args.slice(i)) {
          const argType = infer(arg, env);

          if (!isSubtype(argType, p)) {
            const node = node.args[i];
            throw new Exception(
              `${Type.toString(argType)} is not a subtype of ${Type.toString(
                p
              )} at ${node.srcloc.file} ${node.srcloc.line}:${node.srcloc.col}`
            );
          }
        }
      }
    });
  }

  return func.ret;
};

We need a function inferVariableDeclaration because we need to be able to infer a type for every expression that could be in a program or do block, but primary checking of variable declarations will happen in the TypeChecker class so we'll just infer the type of node.expression and call it good.

const inferVariableDeclaration = (node, env) => {
  return infer(node.expression, env);
};

inferSetExpression is the same:

const inferSetExpression = (node, env) => {
  return infer(node.expression, env);
};

inferDoExpression extends the type environment and uses that to infer each expression it contains, then returns the type of the last expression.

const inferDoExpression = (node, env) => {
  let doType = Type.any;
  const doEnv = env.extend("doExpression");

  for (let expr of node.body) {
    doType = infer(expr, doEnv);
  }

  return doType;
};

Lastly, inferTypeAlias returns the base type of a type alias declaration:

const inferTypeAlias = (node, env) => {
  return fromTypeAnnotation(node.type, env);
};

We'll revisit this function when we add generics to Wanda, but this will work for now.

Checking Types

The last bit of machinery we need before building the type checker proper is a check function that takes a node, a type, infers the type from the node, and checks that type against the provided type. In src/typechecker/check.js:

export const check = (ast, type, env) => {
  const inferredType = infer(ast, env);

  if (!isSubtype(inferredType, type)) {
    throw new Exception(
      `Type ${Type.toString(inferredType)} is not a subtype of ${Type.toString(
        type
      )} at ${ast.srcloc.file} ${ast.srcloc.line}:${ast.srcloc.col}`
    );
  }
};

The `TypeChecker` Class

Now we'll create a class that implements the visitor pattern, visits each node of the AST, and type checks it. Each method will return a modified AST node that includes type information.

For most of the nodes we have so far, this is as simple as calling infer on the provided node. One of the nice things about bidirectional type checking is that you can get your inference to go a long way, which saves the programmer the need to provide copious type annotations.

Let's construct the TypeChecker class in src/typechecker/TypeChecker.js:

import { ASTTypes } from "../parser/ast.js";
import { Exception } from "../shared/exceptions.js";
import { check } from "./check.js";
import { infer } from "./infer.js";
import { Type } from "./Type.js";
import { fromTypeAnnotation } from "./fromTypeAnnotation.js";

export class TypeChecker {
  constructor(program, env) {
    this.program = program;
    this.env = env;
  }

  static new(program, env = TypeEnvironment.new()) {
    return new TypeChecker(program, env);
  }
}

Next we'll write the check method, which dispatches to other methods based on the AST node's kind property:

  check(node = this.program, env = this.env) {
    switch (node.kind) {
      case ASTTypes.Program:
        return this.checkProgram(node, env);
      case ASTTypes.NumberLiteral:
        return this.checkNumber(node, env);
      case ASTTypes.StringLiteral:
        return this.checkString(node, env);
      case ASTTypes.BooleanLiteral:
        return this.checkBoolean(node, env);
      case ASTTypes.KeywordLiteral:
        return this.checkKeyword(node, env);
      case ASTTypes.NilLiteral:
        return this.checkNil(node, env);
      case ASTTypes.Symbol:
        return this.checkSymbol(node, env);
      case ASTTypes.CallExpression:
        return this.checkCallExpression(node, env);
      case ASTTypes.VariableDeclaration:
        return this.checkVariableDeclaration(node, env);
      case ASTTypes.SetExpression:
        return this.checkSetExpression(node, env);
      case ASTTypes.DoExpression:
        return this.checkDoExpression(node, env);
      case ASTTypes.TypeAlias:
        return this.checkTypeAlias(node, env);
      default:
        throw new Exception(`Type checking not implemented for ${node.kind}`);
    }
  }

The checkProgram method loops over node.body and checks each expression:

  checkProgram(node, env) {
    /** @type {TypedAST[]} */
    let body = [];
    let type;
    let i = 0;
    for (let expr of node.body) {
      if (i === node.body.length - 1) {
        const node = this.check(expr, env);
        type = node.type;
        body.push(node);
      } else {
        const node = this.check(expr, env);
        body.push(node);
      }
    }

    return { kind: node.kind, body, srcloc: node.srcloc, type };
  }

The primitive check methods simply infer the type from the node:

  checkBoolean(node, env) {
    return { ...node, type: infer(node, env) };
  }

  checkKeyword(node, env) {
    return { ...node, type: infer(node, env) };
  }

  checkNil(node, env) {
    return { ...node, type: infer(node, env) };
  }

  checkNumber(node, env) {
    return { ...node, type: infer(node, env) };
  }

  checkString(node, env) {
    return { ...node, type: infer(node, env) };
  }

The checkSymbol method does the same, since infer looks up the type for the symbol from the type environment already:

  checkSymbol(node, env) {
    return { ...node, type: infer(node, env) };
  }

The checkCallExpression does the same since infer handles all the heavy lifting:

  checkCallExpression(node, env) {
    // infer handles checking for argument types
    return { ...node, type: infer(node, env) };
  }

The checkDoExpression method loops over each expression in the body and checks them one at a time:

  checkDoExpression(node, env) {
    /** @type {TypedAST[]} */
    let body = [];
    for (let expr of node.body) {
      const node = this.check(expr, env);
      body.push(node);
    }

    return {
      kind: node.kind,
      body,
      srcloc: node.srcloc,
      type: infer(node, env),
    };
  }

The checkVariableDeclaration method checks if the node has a type annotation. If so, it derives the type from the annotation and uses that to check the type of the expression. Then it sets the annotation type in the environment and turns on type checking for the whole program.

If there's no annotation, it infers the type of the expression and sets that in the environment.

  checkVariableDeclaration(node, env) {
    if (node.typeAnnotation) {
      const annotType = fromTypeAnnotation(node.typeAnnotation);
      check(node.expression, annotType, env);
      env.checkingOn = true;
      env.set(node.lhv.name, annotType);
      return { ...node, type: annotType };
    }

    const type = infer(node, env);
    env.set(node.lhv.name, type);
    return {
      kind: node.kind,
      lhv: node.lhv,
      expression: this.check(node.expression, env),
      srcloc: node.srcloc,
      type,
    };
  }

checkSetExpression checks to see if checking is turned on. If so, it checks the type of its expression against the type set in the environment for the symbol. Otherwise it infers the type from the expression.

  checkSetExpression(node, env) {
    if (env.checkingOn) {
      const nameType = env.getType(node.lhv.name);
      check(node.expression, nameType, env);
      return { ...node, type: nameType };
    }

    return {
      kind: node.kind,
      lhv: node.lhv,
      expression: this.check(node.expression, env),
      srcloc: node.srcloc,
      type: infer(node, env),
    };;
  }

Finally, checkTypeAlias gets the type from the annotation on the TypeAlias node and sets it in the type environment:

  checkTypeAlias(node, env) {
    const type = fromTypeAnnotation(node.type, env);
    env.setType(node.name, type);
    return { ...node, type };
  }

Constructing The Global Type Environment

Now we need to make a way to construct the global type environment, including types for core library functions.

We're going to loop over the items of the core library and set the name to the value of the contract property on the core library function, defaulting to the Any type if there is no contract.

We're not going to go back and add contracts to all the core library functions because we don't actually have the ability to parse function type annotations yet, so for now that means everything gets the Any type. Obviously we'll change that in a future article.

Write the makeGlobalTypeEnv function in src/typechecker/makeGlobalTypeEnv.js:

import { theModule as Core } from "../../lib/js/core.js";
import { makeRuntime } from "../runtime/makeRuntime.js";
import { Type } from "./Type.js";
import { TypeEnvironment } from "./TypeEnvironment.js";

export const makeGlobalTypeEnv = () => {
  const mod = Core.module(makeRuntime());
  const typeEnv = TypeEnvironment.new(null, { name: "global" });

  for (let [k, v] of Object.entries(mod)) {
    typeEnv.set(k, v.contract ?? Type.any);
  }

  return typeEnv;
};

Now when we do add contracts for core library functions they'll be automatically added to the global type environment.

A Functional Wrapper

The last thing we're going to do with the type checker is create a functional wrapper like we have with the other stages of the compilation pipeline. In src/typechecker/typecheck.js:

import { TypeChecker } from "./TypeChecker.js";

export const typecheck = (ast, env = undefined) =>
  TypeChecker.new(ast, env).check();

Changes To The CLI

Now that we have everything set up for our type checker, let's make it so we can use it in our CLI.

First, we need to update our compile function to add type checking to the pipeline. In src/cli/compile.js:

import { tokenize } from "../lexer/tokenize.js";
import { read } from "../reader/read.js";
import { expand } from "../expander/expand.js";
import { parse } from "../parser/parse.js";
import { desugar } from "../desugarer/desugar.js";
import { emit } from "../emitter/emit.js";
import { typecheck } from "../typechecker/typecheck.js";

export const compile = (
  input,
  file = "stdin",
  ns = undefined,
  typeEnv = undefined
) =>
  emit(
    desugar(typecheck(parse(expand(read(tokenize(input, file)))), typeEnv)),
    ns
  );

We also need to add a type environment creator param for our compileAndBuild function, in src/cli/compileAndBuild.js:

import { makeGlobalNameMap } from "../runtime/makeGlobals.js";
import { emitGlobalEnv } from "../emitter/emitGlobalEnv.js";
import { build } from "./build.js";
import { compile } from "./compile.js";
import { makeGlobalTypeEnv } from "../typechecker/makeGlobalTypeEnv.js";

export const compileAndBuild = (
  wandaCode,
  {
    fileName = "stdin",
    contextCreator = emitGlobalEnv,
    nsCreator = makeGlobalNameMap,
    outName = "main.js",
    moduleName = "main",
    typeEnvCreator = makeGlobalTypeEnv,
  } = {}
) => {
  const contextCode = contextCreator();
  const compiledCode = `${contextCode}
  ${moduleName}.result = ${compile(
    wandaCode,
    fileName,
    nsCreator(),
    typeEnvCreator()
  )}`;

  return build(compiledCode, outName, moduleName);
};

We're also going to make some changes so the REPL can take multiline input to make it easier to write do blocks. First, add the file src/cli/utils.js and add these 2 functions:

export const countIndent = (str) => {
  let indentCount = 0;

  for (let char of str) {
    if (char === "(" || char === "[" || char === "{") {
      indentCount++;
    } else if (char === ")" || char === "]" || char === "}") {
      indentCount--;
    }
  }

  return indentCount;
};

export const inputFinished = (input) => countIndent(input) === 0;

We don't have tokens in the language yet for square brackets and curly braces, but we will when we add Vectors and Records in the next article, so I'm just going ahead and putting them in this function now. That way we don't have to remember to edit it later.

Now we'll move the input variable outside of the while loop, add a variable for indentation, and handle the case where the programmer has more open parens than closing ones.

We also need to create a type environment to store type information in the REPL.

Here's the new src/cli/repl.js:

import os from "os";
import vm from "vm";
import readlineSync from "readline-sync";
import { pprintAST, pprintDesugaredAST } from "./pprint.js";
import { println } from "../printer/println.js";
import { makeGlobalNameMap } from "../runtime/makeGlobals.js";
import { emitGlobalEnv } from "../emitter/emitGlobalEnv.js";
import { build } from "./build.js";
import { compile } from "./compile.js";
import { makeGlobalTypeEnv } from "../typechecker/makeGlobalTypeEnv.js";
import { countIndent, inputFinished } from "./utils.js";

const read = (prompt) => readlineSync.question(prompt);

export const repl = (mode = "repl") => {
  // Create global compile environment
  const compileEnv = makeGlobalNameMap();
  const typeEnv = makeGlobalTypeEnv();

  // Build global module and instantiate in REPL context
  // This should make all compiled global symbols available
  const globalCode = build(emitGlobalEnv());
  vm.runInThisContext(globalCode);

  let prompt = "wanda> ";
  let input = "";
  let indent = 0;

  while (true) {
    try {
      input += read(prompt + "  ".repeat(indent));

      switch (input) {
        // If it's a command, execute it
        case ":quit":
          process.exit(0);
        case ":print-ast":
          mode = "printAST";
          break;
        case ":print-desugared":
          mode = "printDesugared";
          break;
        // If it's code, compile and run it
        default:
          if (inputFinished(input)) {
            let compiled = compile(input, "stdin", compileEnv, typeEnv);
            let result = vm.runInThisContext(compiled);

            if (mode === "printAST") {
              console.log(pprintAST(input));
            } else if (mode === "printDesugared") {
              console.log(pprintDesugaredAST(input));
            }

            println(result);
            input = "";
            indent = 0;
          } else {
            input += os.EOL;
            indent = countIndent(input);
          }
      }
    } catch (e) {
      console.error(e.message);
    }
  }
};

If you have a bin directive in your package.json file that points to bin/wanda.js and you haven't already used npm link in the directory, you can do so now. This will make wanda available as a global system command.

Now let's see the type checker in action. Run wanda in your terminal, then try to create a variable with a type annotation. Something like this:

(var (x : number) 10)

It should work, and if you input x and hit enter in the CLI now you should get back the value 10.

If you try with a mismatched annotation and expression type, it should throw an error. Try it with something like:

(var (s : string) 7)

You should get an error. Neat, eh?

Conclusion

Now we've got variables, do blocks, and type checking. This language is really coming along! Next time we'll add Vectors and Records to the language, as well as extending the type checker to handle union types.

As always, you can see the current state of the code on the current tag in the Wanda repo. There are also tests you can run with npm run test as well as JSDoc comments for nearly all functions and types (I can't promise I've remembered them all).

I'll see you in the next one!

The type checker implementation owes much to Jake Donham's Reconstructing TypeScript series ↩

Create Your Own Programming Language 3: Call Expressions

Jason Barr — Wed, 07 Jun 2023 04:15:00 +0000

Every computer program has 2 parts: data and logic. Thus far in our series we've seen some data, in the form of primitive data types like numbers and strings, but there's no logic yet.

That changes today with the addition of call expressions and the beginnings of a standard library.

We're also going to make some changes to the reader (and subsequent compilation phases) to make it produce a linked list instead of an array.

That will make it easier down the road to add the king of all Lisp language features... macros!

That's because Lisp code, with all its parentheses (a syntax known as s-expressions), is an exact match for the data structure representation produced by the reader. This allows you to treat code as data and vice versa, which allows for some pretty mind-bendy stuff.

This property, where code maps to data and vice versa, is called homoiconicity and will make it much easier to implement the advanced macro feature when we get to it.

Implementing call expressions also requires us to implement symbols (identifiers) and deal with the fact that Wanda allows characters in its identifiers that are invalid in JavaScript, which means we get to use compile-time environments for the first (but not the last) time.

Since I've made the design decision to output compiled Wanda modules using standard ES2015 imports, this will also require us to implement a build step before our compiled code will work with our REPL.

If it sounds like a lot, that's because it is. The code for this post actually ended up taking way more work than I had anticipated. That's just how it goes sometimes!

As always, if you haven't read the previous post on adding primitives to the language, do that before continuing.

Ok? Good, let's get started.

The Cons Structure and Linked Lists

Lisp famously represents its code in the compiler as a series of cons cells that form singly-linked lists.

The Cons and Primitive Operations

Constructing a cons cell, also known as a pair, is a primitive operation in Lisp. The name "cons" is short for "constructor," because the operation was so common it was both safe to assume using "a constructor" meant creating a pair and that it was desirable to have a shorthand name for "constructor." Thus the "cons" was born.

There are 2 primitive operations on a cons cell: car and cdr. You'll often see these referred to as "head" and "tail," respectively. They simply mean getting the first (car) and second (cdr) elements of a pair. The names car and cdr come from obscure characteristics related to the registers used to implement pairs in the architecture of a computer from the late 1950s, and continue to be used for historical reasons (except in Clojure, because Rich Hickey hates history 😉). I've retained them as an historical curiosity, but you're free to use the more colloquial "head" and "tail" if you like.

Lists and Recursion

The definition of a list in Lisp is recursive: a list is either nil or a pair whose second item is a list. A list made up of pairs is singly-linked.

Ordinarily all operations on a list are recursive in Lisp, but we're going to implement an ES2015 iterator using the Symbol.iterator method both to make some things easier and because JavaScript unfortunately doesn't have certain interpreter features that make using recursion more efficient like tail call elimination, which makes it so that calls in tail position don't add a new frame to the call stack, thus preventing stack overflow.

Tail call elimination is technically in the TC39 ECMAScript specification, but as far as I'm aware no major JavaScript engine actually implements it. More's the pity, because in many ways JavaScript is extremely well-suited to a more functional style of programming (which often uses recursion instead of iteration).

Coding The `Cons` Class

I've chosen to use 2-element JavaScript arrays as the implementation type for cons because it's not uncommon to use arrays to represent tuples (that's how TypeScript does it) and a pair is essentially a 2-tuple. I'll create a class then export a function that constructs instances of the class so the interface to Cons will be more like how it's done in Lisp with the cons function.

In src/shared/cons.js, first import our Exception class because we'll need it:

import { Exception } from "./exceptions.js";

The class itself will extend the native JavaScript Array class:

export class Cons extends Array {
  constructor(car, cdr) {
    super(car, cdr);
  }

  get car() {
    return this[0];
  }

  get cdr() {
    return this[1];
  }

  set cdr(value) {
    this[1] = value;
  }
}

We're going to allow setting the cdr value for internal use, namely while appending to the list, but we're not going to use anything like pair.cdr = value anywhere in the compiler's code because we prefer to make our data structures immutable whenever possible.

Note that in some Lisps (like Scheme) you can mutate the elements of a pair directly, whereas in others (like Racket and Clojure) you cannot. Actually you can in Racket, but you have to do some extra work to make it happen – which they actively discourage.

Now let's add an append method to make it easier to add items to a list (remembering that one definition for a list is a pair whose 2nd item is a list). We'll check to see if the cdr element is a Cons or null, continue traversing the list until we finally reach the end, and then if the end is null append a new Cons with the value as its car and null as its cdr. Note that a list whose last pair is not nil, but a regular value instead, is known as an "improper list." Since there's no nil at the end of an improper list, trying to append to it causes an error:

  append(value) {
    let list = this;
    let cdr = this.cdr;

    while (cdr !== undefined) {
      if (cdr instanceof Cons) {
        if (cdr.cdr === null) {
          // is the end of the list
          cdr.cdr = cons(value, null);
          return list;
        }

        // is not yet at the end of the list; keep going
        cdr = cdr.cdr;
      } else if (cdr === null) {
        // this is a single-item list
        this.cdr = cons(value, null);
        return list;
      } else {
        // this is an improper list and cannot be appended to
        throw new Exception(
          "Cannot append item to improper list or pair whose tail is not nil"
        );
      }
    }

    // if we're out of the loop and haven't returned or errored yet, something bad happened and I don't know what
    throw new Exception(
      `Error trying to append ${
        typeof value === "object" ? JSON.stringify(value, null, 2) : value
      } to list`
    );
  }

Now with the append method in place, we can add a static of method to make it easier to construct a list from an arbitrary number of arguments:

  static of(first, ...args) {
    let list = cons(first, null);

    for (let arg of args) {
      list.append(arg);
    }

    return list;
  }

Next, let's write the Symbol.iterator method to make Cons iterable. Similar to append, we'll continue traversing the list as long as there is a cons in the cdr position. We'll use a generator for the method to eliminate some boilerplate with the method's return type:

  *[Symbol.iterator]() {
    let value = this.car;
    let tail = this.cdr;

    while (tail !== undefined) {
      if (tail instanceof Cons) {
        yield value;
        value = tail.car;
        tail = tail.cdr;
      } else if (tail === null) {
        yield value;
        tail = undefined;
      } else {
        // is a pair or improper list
        yield value;
        yield tail;
        tail = undefined;
      }
    }
  }

Finally for now, let's add a get method to make it simpler to retrieve an item from a list based on its numeric index. This will come in handy in the parser:

  get(n) {
    let i = 0;

    for (let value of this) {
      if (i === n) {
        return value;
      }
      i++;
    }

    return undefined;
  }

Then at the bottom of the file we export the cons function:

export const cons = (car, cdr) => new Cons(car, cdr);

Next we'll add new tokens for symbols and parentheses to the lexer, then change the reader to produce a linked list. We'll also make the reader process our first list-based form for call expressions.

Lists and Call Expressions

It's no accident that our reader will produce linked lists. In fact, syntactic forms in a Lisp symbolically correspond to lists themselves. Different forms have different configurations of lists and symbols that differentiate them.

The simplest list-based form is the call expression. Here are some examples:

(+ 1 2)
(* 2 5 6)
(append "Hello, " "world!")
(* 2 (+ 1 3))

As you can see, a call expression is just a list with the function as its first element followed by the arguments. This is known as prefix notation, and it's used for every function and operation including mathematical operations.

Of course, like with any programming language, an argument to a call expression can be another call expression.

Changes to the Lexer

First, let's add a couple of character matching functions to src/lexer/utils.js:

// after isSymbolChar
export const isLParen = (ch) => /\(/.test(ch);
export const isRParen = (ch) => /\)/.test(ch);
// ...

While we're at it, let's do something heretical in the Lisp world and add commas to our whitespace characters. This will allow us to separate function arguments, list parameters, etc. with commas if we wish, for readability.

Also in src/lexer/utils.js, rewrite the isWhitespace function like so:

// ...functions
export const isWhitespace = (ch) => /[\s,]/.test(ch);
// ...functions

Next, we add 3 new types to the TokenTypes enum in src/lexer/TokenTypes.js:

export const TokenTypes = {
  // existing tokens
  Symbol: "Symbol",
  LParen: "LParen",
  RParen: "RParen",
};

The next change we need to make is in the dispatch loop in Lexer#tokenize in src/lexer/Lexer.js. After the clause including isSymbolStart(ch), add this:

      // clauses ending with reading symbols
      } else if (isLParen(ch)) {
        const { pos, line, col, file } = this.input;
        this.input.next(); // skip over punc
        tokens.push(
          Token.new(TokenTypes.LParen, ch, SrcLoc.new(pos, line, col, file))
        );
      } else if (isRParen(ch)) {
        const { pos, line, col, file } = this.input;
        this.input.next(); // skip over punc
        tokens.push(
          Token.new(TokenTypes.RParen, ch, SrcLoc.new(pos, line, col, file))
        );
      } else {
      // error cases

Finally, we need to amend the readSymbol method to output a symbol when it's not a boolean or nil by replacing the error with returning a symbol token:

  readSymbol() {
    let { pos, line, col, file } = this.input;
    const srcloc = SrcLoc.new(pos, line, col, file);
    const sym = this.input.readWhile(isSymbolChar);

    if (isBoolean(sym)) {
      return Token.new(TokenTypes.Boolean, sym, srcloc);
    } else if (isNil(sym)) {
      return Token.new(TokenTypes.Nil, sym, srcloc);
    }

    return Token.new(TokenTypes.Symbol, sym, srcloc);
  }

Changes to the Reader

We need to make 2 major changes to the reader:

We need to change the read function to produce a list instead of an array
We need to add a function to read list forms from the token stream and call it from readForm

First, let's update the imports in src/reader/read.js:

import { Token } from "../lexer/Token.js";
import { TokenTypes } from "../lexer/TokenTypes.js";
import { Exception, SyntaxException } from "../shared/exceptions.js";
import { Reader } from "./Reader.js";
import { cons } from "../shared/cons.js";
import { SrcLoc } from "../lexer/SrcLoc.js";

Now let's change the read function.

Since it's possible the input could be of length 0, we'll need to handle that case since the cons function requires 2 arguments. Let's make it so we always have a form even when the input is empty. Then we'll cons this form and null to make a list, then continue appending new forms to the list until the reader has reached the end of the token stream.

In src/reader/read.js, here's the new read function:

export const read = (tokens) => {
  const reader = Reader.new(tokens);

  const form =
    reader.length === 0
      ? Token.new(TokenTypes.Nil, "nil", SrcLoc.new(0, 1, 1, "reader"))
      : readExpr(reader);
  let parseTree = cons(form, null);

  while (!reader.eof()) {
    parseTree.append(readExpr(reader));
  }

  return parseTree;
};

The read function calls a new function, readExpr, which for now will just delegate to readForm but in a future article we'll use it to read some expression types (like objects with dot notation). Here's what that function looks like for now:

const readExpr = (reader) => {
  return readForm(reader);
};

Now, in readForm, let's add cases for the left and right paren tokens. Obviously if the reader comes across a right paren in this position it's an error. If it's a left paren, we'll delegate to readList to read a list form. Here's the new readForm:

const readForm = (reader) => {
  const tok = reader.peek();

  switch (tok.type) {
    case TokenTypes.RParen:
      // there shouldn't be an RParen here
      throw new SyntaxException(tok.value, tok.srcloc);
    case TokenTypes.LParen:
      return readList(reader);
    default:
      return readAtom(reader);
  }
};

Finally, we need to write the readList function. We're going to deviate slightly from our pure array-based linked list structure and add a srcloc property to the forms that use readList to make it easier for the parser to include srcloc info on AST nodes.

First, the function will consume the left paren token and get the srcloc info from it. Then it will peek at the next token.

If it's a right paren, it's an empty list so instead of producing a list it will return a nil token. This is different from how other Lisp readers handle empty lists, but it's good enough for our purposes. If a form is expecting a list of items rather than nil it will cause an error, which is what we need.

Assuming there are tokens before the right paren, we'll cons the first form and null to create a list. Then we loop over the remaining tokens, consuming and appending forms, until we get to the closing paren. Then we skip over it and return the list.

Here's the readList function:

const readList = (reader) => {
  // just selects the LParen and advances the token stream
  let tok = reader.next();
  let srcloc = tok.srcloc;

  // this should never happen
  if (tok.type !== TokenTypes.LParen) {
    throw new SyntaxException(tok.value, tok.srcloc);
  }

  tok = reader.peek();

  if (tok.type === TokenTypes.RParen) {
    // is empty list, which is nil
    reader.skip();
    return Token.new(TokenTypes.Nil, "nil", tok.srcloc);
  }

  let list = cons(readExpr(reader), null);
  list.srcloc = srcloc;

  let lastTok = tok;
  tok = reader.peek();

  while (tok?.type !== TokenTypes.RParen) {
    if (!tok) {
      // Whoops, we're past the end of the token stream without ending the list
      throw new Exception(
        `Expected ); got EOF at ${lastTok.srcloc.line}:${lastTok.srcloc.col}`
      );
    }

    list.append(readExpr(reader));
    lastTok = tok;
    tok = reader.peek();
  }

  // skip the closing RParen
  reader.skip();

  return list;
};

That takes care of the changes to the reader! These are actually the last changes we'll need to make for a while to both the lexer and reader. After a few more articles, we'll come back to the reader and implement some reader macros, which are transformations done during the reading process.

The expander is unchanged, so now let's look at the parser.

Changes to the Parser

First, we need to implement a new class to replace Reader in the parser because now that the reader produces lists we can't use that one anymore. We'll call this class ConsReader and give it the same API as the original Reader class.

In src/parser/ConsReader.js:

export class ConsReader {
  constructor(forms) {
    this.forms = forms;
    this.pos = 0;
  }

  static new(forms) {
    return new ConsReader(forms);
  }

  get length() {
    return [...this.forms].length;
  }
}

I said we'd give it the same API as the original Reader class, but that's actually not quite true. We only need the eof and next methods for now. Here they are:

  eof() {
    return this.pos >= this.length;
  }

  next() {
    return this.forms.get(this.pos++);
  }

See how the next method uses the get method we implemented for our linked list? It really simplifies things, which shows the importance of taking some time to think through your abstractions before you implement them.

Now let's add AST node types and constructors for the Symbol and CallExpression nodes. Add cases for Symbol and CallExpression to the ASTTypes enum in src/parser/ast.js:

export const ASTTypes = {
  // other node types
  Symbol: "Symbol",
  CallExpression: "CallExpression",
};

Then add node constructors to the AST object. The CallExpression node will need properties for the function being called and the arguments to the expression:

export const AST = {
  // existing constructors
  Symbol(token) {
    return {
      type: ASTTypes.Symbol,
      name: token.value,
      srcloc: token.srcloc,
    };
  },

  CallExpression(func, args, srcloc) {
    return {
      type: ASTTypes.CallExpression,
      func,
      args,
      srcloc,
    };
  },
};

As you might expect since we completely changed the type of data structure it takes, we need to make major changes to the parser.

First, we need to import the new ConsReader class into src/parser/parse.js:

// other imports
import { ConsReader } from "./ConsReader.js";
import { AST } from "./ast.js";

We need to rethink the whole parser architecture. Originally, since it largely mirrored the reader, we were passing the Reader object around between the parse functions. This was fine for just parsing primitives, but we'll need to change it now that we're going to parse more complex forms.

What we'll need to do is create mutually recursive functions that process forms from the reader and advance the list of forms as they parse them.

The parse function itself is the one least affected by this, in src/parser/parse.js:

export const parse = (readTree) => {
  let body = [];
  const reader = ConsReader.new(readTree);

  while (!reader.eof()) {
    body.push(parseExpr(reader.next()));
  }

  return AST.Program(body);
};

The parse function calls parseExpr, which will delegate based on whether the current form is a primitive or a list:

const parseExpr = (form) => {
  if (form instanceof Cons) {
    return parseList(form);
  }

  return parsePrimitive(form);
};

The parsePrimitive function is similar to what it was before, only now it takes a token instead of the reader:

const parsePrimitive = (token) => {
  switch (token.type) {
    case TokenTypes.Number:
      return AST.NumberLiteral(token);
    case TokenTypes.String:
      return AST.StringLiteral(token);
    case TokenTypes.Boolean:
      return AST.BooleanLiteral(token);
    case TokenTypes.Keyword:
      return AST.KeywordLiteral(token);
    case TokenTypes.Nil:
      return AST.NilLiteral(token);
    case TokenTypes.Symbol:
      return AST.Symbol(token);
    default:
      throw new SyntaxException(token.value, token.srcloc);
  }
};

The parseList function only has one form to parse at present, though it will eventually have more. Forms that aren't call expressions will be distinguished by the symbol in the first position of the list, so we can switch on the symbol token value to delegate parsing those forms and have parsing call expressions as the default case since any list form that isn't a special form is a call expression:

const parseList = (form) => {
  const [first] = form;

  switch (first.value) {
    default:
      return parseCall(form);
  }
};

Finally, the parseCall function destructures the function (the first form of its list) from the arguments (all remaining forms in the list). We'll use the srcloc property we appended to the lists from the reader as well. Then we use parseExpr in a mutually recursive manner to properly parse both the function and argument forms, and return a call to the node constructor.

const parseCall = (callExpression) => {
  const [func, ...args] = callExpression;
  const srcloc = callExpression.srcloc;
  const parsedFunc = parseExpr(func);
  const parsedArgs = args.map(parseExpr);

  return AST.CallExpression(parsedFunc, parsedArgs, srcloc);
};

Emitting Code for Symbols and Call Expressions

This is where we run into the first time in which things are significantly different in Wanda than in JavaScript. Wanda allows many characters in identifiers that are illegal in JavaScript, like +, *, and *.

Obviously if we're going to be calling functions then we'll need identifiers for those functions.

The way we're going to handle this is by constructing a compile-time environment that maps in-language symbol names to valid JavaScript identifiers.

Hashing

Since Wanda identifiers aren't necessarily valid JavaScript identifiers, we've got a little work to do.

We'll hash the Wanda symbol names, which converts them to a seemingly-random (but not really random) string of letters and numbers.

Then, since that string could start with a number, we'll prepend it with "$W_" to make sure those strings are valid JavaScript identifiers.

Hashing is a one-way operation, which means theoretically there's no way to work backwards from a hash to its original value, but it's also deterministic so every time you hash the same value you get the same result.

If hashing "h" gives you "123" one time, it gives you "123" every time.

Because of this, if a programmer does something like shadowing a Wanda variable name with a function parameter name, the variable name will actually be shadowed.

First, we'll need a hash function. We'll get this by installing the Object Hash package from NPM with npm install object-hash.

A Compile-Time Environment

Next, we'll need an environment or, as we'll be calling it, Namespace. We'll also use this Namespace class in the next article when we start doing semantic analysis.

Create a new directory runtime in the src folder, and create Namespace.js in that directory.

A Namespace object has 3 properties: parent, vars, and name. The parent property is either a containing Namespace or null, if it's the top-level environment.

It needs the following methods:

exists, checks if a name exists anywhere in the scope chain
extend, extends the current namespace with a child namespace
get, gets the value for a name anywhere in the scope chain
has, checks if the current namespace only has a name
lookup, finds the scope in which a name exists
set, sets a value for a name in the current scope

Plus we'll also make it an iterator to make some tasks easier.

Here's the Namespace class, in src/runtime/Namespace.js:

export class Namespace {
  constructor(parent = null, { name = "global" } = {}) {
    this.parent = parent;
    this.vars = new Map();
    this.name = name;
  }

  static new(parent = null, { name = "global" } = {}) {
    return new Namespace(parent, { name });
  }

  exists(name) {
    return this.lookup(name) !== null;
  }

  extend(name) {
    return new Namespace(this, { name });
  }

  get(name) {
    const scope = this.lookup(name);

    if (scope) {
      return scope.vars.get(name);
    }

    return undefined;
  }

  has(name) {
    return this.vars.has(name);
  }

  lookup(name) {
    let scope = this;

    while (scope !== null) {
      if (scope.vars.has(name)) {
        return scope;
      }

      scope = scope.parent;
    }

    return null;
  }

  set(key, value) {
    this.vars.set(key, value);
  }

  *[Symbol.iterator]() {
    for (let [k, v] of this.vars) {
      yield [k, v];
    }
  }
}

Converting Wanda Symbols to Valid JavaScript Identifiers

Now let's go ahead and create a function using Object Hash to convert a Wanda symbol into a valid JavaScript identifier, in src/runtime/makeSymbol.js:

import objectHash from "object-hash";

export const PREFIX = "$W_";

export const makeSymbol = (str) => PREFIX + objectHash(str);

Changes to the Emitter Itself

Now we've got to make some changes to the emitter. Obviously we need to handle the new Symbol and CallExpression nodes, but we also need to handle using a compile-time environment.

We're going to add a property to the object for the namespace, plus a parameter for one to every method.

Adding a property to the object is mostly so we have a default namespace if we're performing a task where we don't need to define a namespace separately from the emitter.

Additionally, we're going to make some changes to how a Program node is emitted, namely by wrapping the output in an IIFE (Immediately Invoked Function Expression).

Older JavaScript programmers will be familiar with IIFEs, but if you've only been using JavaScript since we got all the new goodies (including modules) in 2015 you may not recognize them. For our purposes it will suffice to say that an IIFE is a function expression that executes immediately, wrapping the contents in its own scope. Before 2015, it was common to see IIFEs used to simulate modules in JavaScript programs because there was no standard module construct.

We'll use an IIFE to make it simpler for the program to return a value to the interpreter in the case of using the REPL or running tests.

We'll also make it so the value of the last expression in the Program is returned from the IIFE.

For the emitSymbol method, we'll first lookup the node's name property in the namespace. If the name has not been set, we'll throw an error. Otherwise, we'll emit the name returned by the lookup.

For emitCallExpression, we'll simply translate the function and arguments into the appropriate emitted code.

Here's the new version of the Emitter class in src/emitter/Emitter.js:

import { ASTTypes } from "../parser/ast.js";
import { Exception, SyntaxException } from "../shared/exceptions.js";
import { Namespace } from "../runtime/Namespace.js";

export class Emitter {
  constructor(program, ns) {
    this.program = program;
    this.ns = ns;
  }

  static new(program, ns = new Namespace()) {
    return new Emitter(program, ns);
  }

  emit(node = this.program, ns = this.ns) {
    switch (node.type) {
      case ASTTypes.Program:
        return this.emitProgram(node, ns);
      case ASTTypes.NumberLiteral:
        return this.emitNumber(node, ns);
      case ASTTypes.StringLiteral:
        return this.emitString(node, ns);
      case ASTTypes.BooleanLiteral:
        return this.emitBoolean(node, ns);
      case ASTTypes.KeywordLiteral:
        return this.emitKeyword(node, ns);
      case ASTTypes.NilLiteral:
        return this.emitNil(node, ns);
      case ASTTypes.Symbol:
        return this.emitSymbol(node, ns);
      case ASTTypes.CallExpression:
        return this.emitCallExpression(node, ns);
      default:
        throw new SyntaxException(node.type, node.srcloc);
    }
  }

  emitBoolean(node, ns) {
    return node.value;
  }

  emitCallExpression(node, ns) {
    return `(${this.emit(node.func, ns)})(${node.args
      .map((a) => this.emit(a, ns))
      .join(", ")})`;
  }

  emitKeyword(node, ns) {
    return `Symbol.for("${node.value}")`;
  }

  emitNil(node, ns) {
    return "null";
  }

  emitNumber(node, ns) {
    return node.value;
  }

  emitProgram(node, ns) {
    let code = "(() => {\n";

    let i = 0;
    for (let n of node.body) {
      if (i === node.body.length - 1) {
        code += `  return ${this.emit(n, ns)}`;
      } else {
        code += `  ${this.emit(n, ns)};`;
      }

      i++;
    }

    code += "\n})();";

    return code;
  }

  emitString(node, ns) {
    return "`" + node.value.slice(1, -1) + "`";
  }

  emitSymbol(node, ns) {
    const name = node.name;
    const emittedName = ns.get(name);

    if (!emittedName) {
      throw new Exception(
        `The name ${name} has not been defined in ${node.srcloc.file} at ${node.srcloc.line}:${node.srcloc.col}`
      );
    }

    return emittedName;
  }
}

We'll also need to make sure the emit function has the compile-time environment as a parameter in src/emitter/emit.js:

import { Emitter } from "./Emitter.js";

export const emit = (ast, ns = undefined) => Emitter.new(ast, ns).emit();

Changes to the Printer and AST Printer

Now that we have lists in our language, we'll need to add a case to the printString function to handle them. We'll also need to handle Symbol and CallExpression nodes in the AST pretty printer.

First, the changes to printString, in src/printer/printString.js. We need to import the Cons class:

import { Cons } from "../shared/cons.js";

We'll add a clause in the case where typeof value is "object" for instances of the Cons class:

export const printString = (value, withQuotes) => {
  switch (typeof value) {
    case "number":
      return String(value);
    case "string":
      return withQuotes ? `"${value}"` : value;
    case "symbol":
      return value.description.startsWith(":")
        ? value.description
        : `'${value.description}`;
    case "boolean":
      return String(value);
    case "undefined":
      return "nil";
    case "object":
      if (value === null) {
        return "nil";
      } else if (value.constructor?.name === "Cons") {
        return printList(value);
      }
    default:
      throw new Exception(`Invalid print value ${value}`);
  }
};

We've also slightly modified the case for "symbol" to print quoted symbols if it's a symbol that's not a keyword. That's not important now, but it will be in the future.

The printList function iterates over the list and recursively calls printString for each list item, inserting a comma between printed values (which, as a Lisp programmer, might actually be heresy, but I'm doing it anyway because it's easier to read):

const printList = (list) => {
  let prStr = "'(";

  let i = 0;
  let length = [...list].length;
  for (let item of list) {
    if (i === length - 1) {
      prStr += printString(item);
    } else {
      prStr += `${printString(item)}, `;
    }
    i++;
  }

  prStr += ")";
  return prStr;
};

In src/printer/printAST.js, I've factored out a simple little function for printing indent spaces to cut down on repetition:

const prIndent = (indent) => " ".repeat(indent);

We also need to import the Exception class, as we'll be using it in the ASTPrinter class now:

import { Exception } from "../shared/exceptions.js";

We need to add cases to the dispatcher for Symbol and CallExpression nodes, and methods for printing them. We also need to refactor the existing methods to use the new prIndent function. Here's the new version of the ASTPrinter class:

class ASTPrinter {
  constructor(program) {
    this.program = program;
  }

  static new(program) {
    return new ASTPrinter(program);
  }

  print(node = this.program, indent = 0) {
    switch (node.type) {
      case ASTTypes.Program:
        return this.printProgram(node, indent);
      case ASTTypes.NumberLiteral:
      case ASTTypes.StringLiteral:
      case ASTTypes.BooleanLiteral:
      case ASTTypes.KeywordLiteral:
      case ASTTypes.NilLiteral:
        return this.printPrimitive(node, indent);
      case ASTTypes.Symbol:
        return this.printSymbol(node, indent);
      case ASTTypes.CallExpression:
        return this.printCallExpression(node, indent);
      default:
        throw new Exception(`Unknown AST type ${node.type} to print`);
    }
  }

  printCallExpression(node, indent) {
    let prStr = `${prIndent(indent)}CallExpression\n`;
    prStr += `${prIndent(indent + 2)}Func:\n${this.print(
      node.func,
      indent + 4
    )}\n`;
    prStr += `${prIndent(indent + 2)}Args:\n`;

    for (let arg of node.args) {
      prStr += this.print(arg, indent + 4) + "\n";
    }

    return prStr;
  }

  printPrimitive(node, indent) {
    return `${prIndent(indent)}${node.type}: ${
      node.type === "NilLiteral" ? "nil" : node.value
    }`;
  }

  printProgram(node, indent) {
    let pStr = "";

    for (let n of node.body) {
      pStr += this.print(n, indent);
      +"\n";
    }

    return pStr;
  }

  printSymbol(node, indent) {
    return `${prIndent(indent)}Symbol: ${node.name}`;
  }
}

Note that with the addition of printing CallExpression nodes we finally get to use that indent parameter.

I'm also renaming the print function to println and changing its filename from src/printer/print.js to src/printer/println.js to distinguish it from a new print function that writes to stdout without a trailing newline.

Add this function to src/printer/print.js:

import { printString } from "./printString.js";

export const print = (str) => {
  process.stdout.write(printString(str, true));
};

The Beginning of a Standard Library for Wanda

Now we have the machinery for compiling call expressions in place, but something is still missing: functions to call!

To that end, we'll start building a core library for Wanda. These are the global names that will be available in all Wanda code, including modules, without needing to be explicitly imported into your program.

Wanda Modules

I'll add a few functions that will be used in the core library as well as other parts of the compiler, but first we need to think about what a Wanda module should look like.

This includes the module for the core library as well as other modules that may be imported into Wanda programs.

The most important part of a module is the names/values it provides, so the simplest way to do this would just be to define an object filled with values and functions and use the property names as their in-language names.

Thinking ahead a little, though, when we add the ability to define modules in the language we'll have dependencies for those modules.

A module should probably also have a name.

Plus we'll want the ability to use other modules in module code, which will require some way to access them.

The simplest way to provide a module namespace while also allowing other modules to be used inside a new one is to have modules compile to a constructor function that returns its provided values (from now on, provides).

The dependencies can be injected into the constructor function and used in its body. This is the approach Pyret takes with its modules, and we're going to use it too (albeit in a much simplified fashion).

We're going to create a Module class with 4 properties:

name, the module name
module, the module constructor
requires, an array of strings of locations of Wanda-defined modules
nativeRequires, an array of strings of locations of JavaScript-defined modules

We're using 2 separate arrays for requirements to make it easier to resolve the filenames.

In src/runtime/module.js:

class Module {
  constructor(name, module, requires, nativeRequires) {
    this.name = name;
    this.module = module;
    this.requires = requires;
    this.nativeRequires = nativeRequires;
  }

  toString() {
    return `Module ${this.name}`;
  }
}

export const makeModule = (name, module, requires = [], nativeRequires = []) =>
  new Module(name, module, requires, nativeRequires);

We're going to define modules in the lib directory (not src/lib). We'll keep JavaScript-defined modules in the js subfolder of lib. The core library will go in lib/js/core.js.

Core Library Dependencies

The core library will require some dependencies, most of which have already been created. We're also going to install another NPM package to use in the core library though, so let's get that out of the way first.

We're going to add a package here called Fast Deep Equal.

I want Wanda to have a primitive check for structural equality, i.e. whether 2 objects have the same properties and values, not just the regular reference equality check you get with == in JavaScript, and this package is the simplest way to do that.

Add it with npm install fast-deep-equal.

Additional Util Functions

We're also going to diverge from JavaScript in another way: truthy and falsy values. In JavaScript, falsy values include false, null, undefined, the empty string, and 0.

Wanda is going to follow the example of several other Lisp-family languages and use only false and nil as primitive values.

To check for that, we'll need a utility function.

Add the file src/runtime/utils.js and add the following 3 functions:

export const isNil = (val) => val == null;
export const isFalsy = (val) => val === false || isNil(val);
export const isTruthy = (val) => !isFalsy(val);

Now we can go back to lib/js/core.js and import our dependencies:

import equal from "fast-deep-equal/es6/index.js";
import { makeModule } from "../../src/runtime/Module.js";
import { Cons, cons } from "../../src/shared/cons.js";
import { print } from "../../src/printer/print.js";
import { println } from "../../src/printer/println.js";
import { printString } from "../../src/printer/printString.js";
import { isTruthy } from "../../src/runtime/utils.js";

Note that since we're importing from a subfolder of the Fast Deep Equal package we have to use index.js to get Node to properly resolve the import.

Now the skeleton of the core module, which is exported from the file:

export const theModule = makeModule("Core", (rt, ns) => {});

Everything in the module will go in the body of the function being passed to makeModule. Note the parameters rt and ns, which stand for "runtime" and "namespace" respectively.

The ns parameter isn't used in the core module, but it will be used in other modules and will be passed as arguments to module constructors when we're using module constructors to build up programs in the future.

The rt parameter is the Runtime object that contains functions necessary for things like converting JavaScript values to Wanda values and other stuff that makes the language work when we translate it into JavaScript.

First, we're going to have a function in the runtime that converts JavaScript functions to Wanda functions. It's just a placeholder for now, but eventually we'll use it to add metadata to Wanda functions. We'll also add the util functions we defined above.

In src/runtime/makeFunction.js:

export const makeFunction = (func) => func;

Now let's create the Runtime object in src/runtime/makeRuntime.js:

import { makeFunction } from "./makeFunction.js";
import { isNil, isTruthy, isFalsy } from "./utils.js";

export const makeRuntime = () => {
  return {
    isFalsy,
    isNil,
    isTruthy,
    makeFunction,
  };
};

Note that we'll be using makeRuntime to inject the runtime object into a module's constructor, so there's no need to import it into lib/js/core.js.

We're going to need a function for getting the length of a list, since Cons doesn't have a useful length property (any time you try to access the length property on a Cons you get 2 because it's a 2-element array). We'll make this function recursive, traversing the list as long as the cdr is a Cons then returning true if the last cdr is null. This function goes at the top of the body of the module constructor:

  const isList = (obj) => {
    if (!rt.isNil(obj) && !(obj instanceof Cons)) {
      return false;
    } else if (rt.isNil(obj)) {
      return true;
    }

    // only option left is cons
    return isList(obj.cdr);
  };

The only other statement in the constructor is the return statement, which returns an object of names and functions that make up the core module. Remember, we're wrapping all the functions in rt.makeFunction so we can use it later to add metadata to our JavaScript-defined functions.

The property names on the object are the names by which the functions are accessed from within Wanda:

  return {
    print: rt.makeFunction(print),
    println: rt.makeFunction(println),
    cons,
    car: rt.makeFunction((pair) => pair.car),
    cdr: rt.makeFunction((pair) => pair.cdr),
    string: rt.makeFunction(printString),
    number: rt.makeFunction(Number),
    boolean: rt.makeFunction((val) => isTruthy(val)),
    symbol: rt.makeFunction((str) => Symbol.for(str)),
    keyword: rt.makeFunction((str) => Symbol.for(":" + str)),
    "+": rt.makeFunction((a, b, ...nums) =>
      nums.reduce((sum, n) => sum + n, a + b)
    ),
    "-": rt.makeFunction((a, b, ...nums) =>
      nums.reduce((diff, n) => diff - n, a - b)
    ),
    "*": rt.makeFunction((a, b, ...nums) =>
      nums.reduce((prod, n) => prod * n, a * b)
    ),
    "/": rt.makeFunction((a, b, ...nums) =>
      nums.reduce((quot, n) => quot / n, a / b)
    ),
    "%": rt.makeFunction((a, b, ...nums) =>
      nums.reduce((quot, n) => quot % n, a % b)
    ),
    "=": rt.makeFunction((a, b) => a === b),
    ">": rt.makeFunction((a, b) => a > b),
    ">=": rt.makeFunction((a, b) => a >= b),
    "<": rt.makeFunction((a, b) => a < b),
    "<=": rt.makeFunction((a, b) => a <= b),
    not: rt.makeFunction((x) => !x),
    list: rt.makeFunction((...args) => Cons.of(...args)),
    length: rt.makeFunction((obj) => {
      if (obj instanceof Cons) {
        let i = 0;
        for (let _ of obj) {
          i++;
        }
        return i;
      }

      return obj.length;
    }),
    get: rt.makeFunction((n, lst) => lst.get(n)),
    "list?": rt.makeFunction(isList),
    "pair?": rt.makeFunction((obj) => obj instanceof Cons),
    "number?": rt.makeFunction((obj) => typeof obj === "number"),
    "string?": rt.makeFunction((obj) => typeof obj === "string"),
    "boolean?": rt.makeFunction((obj) => typeof obj === "boolean"),
    "nil?": rt.makeFunction((obj) => obj == null),
    "keyword?": rt.makeFunction(
      (obj) => typeof obj === "symbol" && obj.description.startsWith(":")
    ),
    "equal?": rt.makeFunction((a, b) => equal(a, b)),
    "is?": rt.makeFunction((a, b) => Object.is(a, b)),
    append: rt.makeFunction((obj1, obj2) => {
      if (typeof obj1 === "string" && typeof obj2 === "string") {
        return obj1 + obj2;
      }
      return obj1.append(obj2);
    }),
  };

Populating Our Global Namespace

Now we need to make sure our global namespace is available to the compiler at compile time and to JavaScript at runtime.

Let's create 2 functions in src/runtime/makeGlobals.js:

import { theModule as Core } from "../../lib/js/core.js";
import { Namespace } from "./Namespace.js";
import { makeSymbol } from "./makeSymbol.js";
import { makeRuntime } from "./makeRuntime.js";

export const makeGlobal = () => {
  const globalNS = Namespace.new();
  const mod = Core.module(makeRuntime());

  for (let [k, v] of Object.entries(mod)) {
    globalNS.set(makeSymbol(k), v);
  }

  return globalNS;
};

export const makeGlobalNameMap = () => {
  const globalNS = Namespace.new();
  const mod = Core.module(makeRuntime());

  for (let [k] of Object.entries(mod)) {
    globalNS.set(k, makeSymbol(k));
  }

  return globalNS;
};

Now we need to actually emit code for the global functions so they can be used at runtime.

First, let's export a constant from the compiler's root directory that enables compiled code to import functions from within the compiler code (including libraries) using absolute paths. In root.js (not src/root.js):

import path from "path";
import { fileURLToPath } from "url";

export const ROOT_URL = import.meta.url;
export const ROOT_PATH = fileURLToPath(path.dirname(ROOT_URL));

Now that we can easily resolve an absolute path for any file in the compiler from within compiled code, let's create a function to emit the code that will place the names from the Core module into the compiled output as global names. In src/emitter/emitGlobalEnv.js:

import path from "path";
import { ROOT_PATH } from "../../root.js";
import { makeGlobal } from "../runtime/makeGlobals.js";

export const emitGlobalEnv = () => {
  const globalEnv = makeGlobal();
  let code = `import { makeGlobal } from "${path.join(
    ROOT_PATH,
    "./src/runtime/makeGlobals.js"
  )}";

const globalEnv = makeGlobal();
`;

  for (let [k] of globalEnv) {
    code += `${k} = globalEnv.get("${k}");\n`;
  }

  return code;
};

Note that in the for...of loop where we create the global variables there is no var, let, or const keyword.

Since there's no "use strict" directive in the IIFE constructed when we build our program (see below), we can use sloppy mode to our advantage and make sure the interpreter registers those variables as globals even though they're going to be emitted inside a function body.

You never, ever, ever want to do that when writing your own JavaScript by hand, but sometimes the needs of compiling force us to bend the rules a bit.

Now we're almost ready to compile and run code using our core library functions, but there's one issue we still need to resolve.

The Need for a Build Step

Actually, if we were just compiling the code and writing it to files that were then executed as ES2015 modules, we'd be ready to write the files and be done (as long as we were only running the code on the same machine used to compile it).

We have an evaluator to handle, though, and unfortunately it doesn't play nice with ES2015 modules.

Plus it might be nice to be able to run our code somewhere other than on the machine used to compile it.

To handle those scenarios, we need to implement a build step for the compiled output.

Bundling a Program with ESBuild

We're going to use a bundler called ESBuild to solve this for us.

A bundler is a program that takes JavaScript source files as input, detects the order used for imports across the whole program, and then "bundles" all the code from the various imports into a single file (or multiple files, known as "chunks") which is then executed as the main program.

We can configure ESBuild to wrap the whole bundle in an IIFE that will be executed at runtime with no need for imports or resolving paths.

The path resolving happens in advance when the bundle is built, so the whole program is all wrapped up together in a single file.

The short version of how it works is it starts with the main input file, finds any import or require statements, then resolves the source of the import/require and loads the file.

It then finds imports/requires in that file, and so on until it's gotten all of them from start to finish.

This process is called building the dependency graph, and we'll implement our own version of it when we add modules to Wanda. For now, though, there's no sense in reinventing the wheel when there are perfectly good tools like ESBuild that are battle-tested and ready for use.

Then, once the dependency graph is built, it traverses the graph and snags all the code from each dependency, outputting it into a bundle file.

The bundle file is what you execute as your program.

Installing and Using ESBuild

First we need to install ESBuild: npm install esbuild.

ESBuild is a natively compiled program with a JavaScript API, so it's way faster than it would be if it were written in JavaScript.

We'll implement the build step in src/cli/build.js. We'll need to pass data back and forth between temp files and ESBuild, so we'll use the Node.js fs and path modules. We'll also need the ROOT_PATH constant to resolve file paths. Here are the imports:

import fs from "fs";
import { join } from "path";
import * as esbuild from "esbuild";
import { ROOT_PATH } from "../../root.js";

We're going to use the synchronous APIs of both ESBuild and fs for simplicity.

We'll pass a string of code (JavaScript) into the build function, along with an optional out file name and module name. We'll create a temporary directory and add an output directory to it.

You'll want to add temp/ to your .gitignore file to go with node_modules/ (you are ignoring node_modules/, right?).

Then we'll use the ESBuild transformSync method to transpile the code, which makes some minor changes to make it easier for browsers and/or Node.js to use.

Then we'll output that into a temp file that we'll then use as the entrypoint for the ESBuild buildSync method to bundle the whole program.

Finally, we'll read the code in the file that method outputs, clean up our temporary files and directories, and return the built code as a string.

Here's the build function:

export const build = (code, outName = "main.js", moduleName = "main") => {
  const tmpPath = join(ROOT_PATH, "./tmp");
  const outPath = join(tmpPath, "./out");

  if (!fs.existsSync(tmpPath)) {
    fs.mkdirSync(tmpPath);
  }

  if (!fs.existsSync(outPath)) {
    fs.mkdirSync(outPath);
  }

  const transformed = esbuild.transformSync(code);
  const outFile = join(tmpPath, outName);

  fs.writeFileSync(outFile, transformed.code);

  esbuild.buildSync({
    entryPoints: [outFile],
    outdir: outPath,
    bundle: true,
    footer: { js: `${moduleName}.result` },
    format: "iife",
    banner: { js: `var ${moduleName} = {};\n` },
  });

  const builtCode = fs.readFileSync(join(outPath, outName), {
    encoding: "utf-8",
  });

  fs.rmSync(join(outPath, outName));
  fs.rmSync(join(tmpPath, outName));
  fs.rmdirSync(outPath);
  fs.rmdirSync(tmpPath);

  return builtCode;
};

I read somewhere that ESBuild has a virtual filesystem option that allows you to input/output without actually reading from/writing to files, but I couldn't find documentation or tutorials on using it. If you read this article and you know how, I would love it if you could leave a comment with information about it so we could avoid all this reading to/writing from temp files.

Changes to the CLI

We're almost done now! All that's left is to make some changes in the CLI to use our environment, library code, and build step with the REPL.

First, we need to slightly edit our compile function to use the compile-time environment, in src/cli/compile.js:

import { tokenize } from "../lexer/tokenize.js";
import { read } from "../reader/read.js";
import { expand } from "../expander/expand.js";
import { parse } from "../parser/parse.js";
import { desugar } from "../desugarer/desugar.js";
import { emit } from "../emitter/emit.js";

export const compile = (input, file = "stdin", ns = undefined) =>
  emit(desugar(parse(expand(read(tokenize(input, file))))), ns);

ns defaults to undefined so if it's not given the default namespace parameter in the emitter's constructor will be used.

You'll also want to make sure EVAL has the compile-time environment, in src/cli/eval.js:

import vm from "vm";
import { compile } from "./compile.js";

export const EVAL = (input, file = "global", ns = undefined) =>
  vm.runInThisContext(compile(input, file, ns));

Now we need a function to combine the compile and build steps. Currently I'm actually only using this in tests, but that could change later so I'm going to walk you through that function as well. In src/cli/compileAndBuild.js:

import { makeGlobalNameMap } from "../runtime/makeGlobals.js";
import { emitGlobalEnv } from "../emitter/emitGlobalEnv.js";
import { build } from "./build.js";
import { compile } from "./compile.js";

export const compileAndBuild = (
  wandaCode,
  {
    fileName = "stdin",
    contextCreator = emitGlobalEnv,
    nsCreator = makeGlobalNameMap,
    outName = "main.js",
    moduleName = "main",
  } = {}
) => {
  const contextCode = contextCreator();
  const compiledCode = `${contextCode}
  ${moduleName}.result = ${compile(wandaCode, fileName, nsCreator())}`;

  return build(compiledCode, outName, moduleName);
};

Assigning the result of the Program node's compiled IIFE to the result property on the module object allows us to return the value from the bundle created by ESBuild, which doesn't actually have a way to return a value from the IIFE it creates. It's a bit of a hack, but sometimes you have to use hacks to get the results you want.

Finally, we're going to rewrite the REPL both to use the bundler to access the core library and to allow the programmer to set options to print the AST from within the REPL.

In the repl function we'll create the compile-time environment and also emit and build the code for the global core library.

We'll use the Node.js vm module to run the built code, which will set all the globals as global variables in the runtime.

Then we'll compile the code entered at the REPL prompt and evaluate it using the vm module.

vm saves the context as long as the script is running, so since we set the globals earlier the variable names are still there to access as long as we properly map from Wanda symbols to JavaScript identifiers using the compile-time environment.

We'll also allow the programmer to enter :print-ast and :print-desugared keywords to both print the AST and evaluate the code at the same prompt.

Here's the new REPL in src/cli/repl.js:

import vm from "vm";
import readlineSync from "readline-sync";
import { pprintAST, pprintDesugaredAST } from "./pprint.js";
import { println } from "../printer/println.js";
import { makeGlobalNameMap } from "../runtime/makeGlobals.js";
import { emitGlobalEnv } from "../emitter/emitGlobalEnv.js";
import { build } from "./build.js";
import { compile } from "./compile.js";

const read = (prompt) => readlineSync.question(prompt);

export const repl = (mode = "repl") => {
  // Create global compile environment
  const compileEnv = makeGlobalNameMap();

  // Build global module and instantiate in REPL context
  // This should make all compiled global symbols available
  const globalCode = build(emitGlobalEnv());
  vm.runInThisContext(globalCode);

  let prompt = "wanda> ";

  while (true) {
    try {
      const input = read(prompt);

      switch (input) {
        // If it's a command, execute it
        case ":quit":
          process.exit(0);
        case ":print-ast":
          mode = "printAST";
          break;
        case ":print-desugared":
          mode = "printDesugared";
          break;
        // If it's code, compile and run it
        default:
          let compiled = compile(input, "stdin", compileEnv);
          let result = vm.runInThisContext(compiled);

          if (mode === "printAST") {
            console.log(pprintAST(input));
          } else if (mode === "printDesugared") {
            console.log(pprintDesugaredAST(input));
          }

          println(result);
      }
    } catch (e) {
      console.error(e.message);
    }
  }
};

Now we can use our REPL with all the functions we defined in our core library!

Conclusion

Whew... that was a ride. I promised y'all that we'd have a working compiler at the end of every article, and this time that meant having to do a lot of things to make sure the thing we just did would work.

Reading forms into linked lists required changing the parser, and having functions to call required both adding compile-time environments and changing the emitter. Having functions to call required adding a build step. One thing led to another and we ended up with over 8,100 words (including source code).

If you've made it this far, I commend you; you're clearly very interested in this stuff, and I appreciate that.

As always, You can see the current state of the code as of this article on GitHub using the tag for the article.

There are tests in the main repo you can run using npm run test if you're interested.

Plus you can see more about how the compiler internals work by looking at the tests if you're so inclined.

The repo code also has JSDoc block comments for most functions and types in the compiler.

I welcome pull requests for bug fixes and improvements. If you have an idea you think will improve Wanda I definitely want to see it.

In the next article we'll add variables to Wanda, which will require us to implement a semantic checking phase between parsing and code emitting.

I hope you're as excited as I am!

Create Your Own Programming Language 2: Primitives

Jason Barr — Fri, 02 Jun 2023 23:11:50 +0000

In the last post we added compiling and evaluating numbers to the Wanda Programming Language. Now we're going to add the other primitive types: strings, booleans, keywords, and nil.

If you haven't seen the previous post, where we scaffold the compiler and add numbers to the language, you can find it here.

As a reminder, strings are delimited with double quotes, and keywords are valid identifiers that start with a colon:

"hello"
:this-is-a-keyword
true
nil

We're also going to make it so you can pretty-print the AST in the REPL instead of evaluating the expression. This can help with debugging later. If you're not getting the results you expect from your Wanda program, you can print the AST and make sure that what the compiler's getting is the same thing you expect it to get.

High Level Overview of The Changes

Most of the work in this article is going to be done in the lexer. Tokenizing strings turns out to be a lot of work, for one thing.

We'll also make some changes to the reader, the parser, the emitter, and the printer as well as adding a function to print the AST. We'll also make some changes to the CLI so you can start the REPL in AST printing mode if you want.

We'll also add a dedicated file for running the CLI so you can add a wanda command to your terminal if you wish.

It may not seem like it takes as much work to get other primitives up and running like it did for numbers, but that's because there was a lot of scaffolding we had to do for the compiler as a whole just to make numbers work. Some of the articles will be like that: they'll feel like we didn't do a whole lot, but now we have at least one new language feature to show for it. Others will take a lot of work and not feel like much has changed at the end of the day. That's just how it goes sometimes!

The Lexer

The first thing we'll add to the lexer is new token types for the new primitives, in src/lexer/TokenTypes.js:

export const TokenTypes = {
  Number: "Number", // still there!
  String: "String",
  Boolean: "Boolean",
  Keyword: "Keyword",
  Nil: "Nil",
};

We're also going to add new character matchers for double-quotes, colons, valid identifier starting characters, and valid identifier characters. Plus string matchers for booleans and nil. In src/lexer/utils.js:

// after isPlus...
export const isDoubleQuote = (ch) => /"/.test(ch);
export const isColon = (ch) => /:/.test(ch);
export const isSymbolStart = (ch) => /[=<>%:|?\\/*\p{L}_$!+-]/u.test(ch);
export const isSymbolChar = (ch) =>
  /[:=@~<>%:&|?\\/^*&#'\p{L}\p{N}_$!+-]/u.test(ch);

// after isNumber...
export const isBoolean = (str) => /true|false/.test(str);
export const isNil = (str) => /nil/.test(str);

Now in the Lexer class we'll add new if/else clauses to the main lexer loop for the new primitives in src/lexer/Lexer.js:

// after the case for else if (isDigit(ch)) {
// ...
      } else if (isDoubleQuote(ch)) {
        tokens.push(this.readString());
      } else if (isColon(ch)) {
        tokens.push(this.readKeyword());
      } else if (isSymbolStart(ch)) {
        tokens.push(this.readSymbol());
      } else {
        // continue with error for unrecognized syntax

This means we'll need methods for readString, readKeyword, and readSymbol. If you want to have the same as what I've got, I'm putting the methods in alphabetical order.

First, readString since it's going to end up being the most complicated one:

  readString() {
    let { pos, line, col, file } = this.input;
    const srcloc = SrcLoc.new(pos, line, col, file);
    let str = this.input.next(); // collect opening double-quote

    str += this.readEscaped();
    return Token.new(TokenTypes.String, str, srcloc);
  }

No, it doesn't look too complicated... yet... but now we need to create a method called readEscaped.

readEscaped will take the string in, one character at a time. If it encounters a backslash, it recognizes it as the start of an escape sequence so it looks ahead and uses the readEscapeSequence method to read in the escaped character. If it hits an unexpected \n in a string literal it throws an error, since Wanda strings can only be single line. Also, if it encounters a backtick as part of the string it escapes it for the emitter, since we're going to be using backtick strings in the compiled output. This eliminates some issues we might otherwise have had with newline characters in double-quoted string literals in the output JavaScript.

Note that reading the escape characters in the lexer is a design decision. We could just as well have done them in the reader or parser. Since handling them now is the simplest way to do it, we'll do that.

Here's the readEscaped method:

  readEscaped() {
    let str = "";
    let escaped = false;
    let ended = false;

    while (!this.input.eof()) {
      let ch = this.input.next();

      if (escaped) {
        str += this.readEscapeSequence(ch);
        escaped = false;
      } else if (ch === "\\") {
        escaped = true;
      } else if (isDoubleQuote(ch)) {
        ended = true;
        str += ch;
        break;
      } else if (ch === "\n") {
        throw new Exception(
          "Unexpected newline in nonterminated single-line string literal"
        );
      } else if (ch === "`") {
        str += "\\`";
      } else {
        str += ch;
      }
    }

    if (!ended && this.input.eof()) {
      throw new Exception(
        "Expected double quote to close string literal; got EOF"
      );
    }

    return str;
  }

and readEscapeSequence:

  readEscapeSequence(c) {
    let str = "";
    let seq = "";

    if (c === "n") {
      str += "\n";
    } else if (c === "b") {
      str += "\b";
    } else if (c === "f") {
      str += "\f";
    } else if (c === "r") {
      str += "\r";
    } else if (c === "t") {
      str += "\t";
    } else if (c === "v") {
      str += "\v";
    } else if (c === "0") {
      str += "\0";
    } else if (c === "'") {
      str += "'";
    } else if (c === '"') {
      str += '"';
    } else if (c === "\\") {
      str += "\\";
    } else if (c === "u" || c === "U") {
      // is Unicode escape sequence
      seq += this.input.readWhile(isHexDigit);
      str += String.fromCodePoint(parseInt(seq, 16));
    }

    return str;
  }

Note that this will read Unicode escape characters in the format \uHHHH with any number of hex numbers necessary to complete a code point.

That's lexing strings. Keywords are considerably simpler:

// between readEscapeSequence and readNumber
  readKeyword() {
    let { pos, line, col, file } = this.input;
    const srcloc = SrcLoc.new(pos, line, col, file);
    const kw = this.input.next() + this.input.readWhile(isSymbolChar);

    return Token.new(TokenTypes.Keyword, kw, srcloc);
  }

We don't need to distinguish here between valid identifier starting characters and valid identifier body characters, since every valid identifier starting character is also a valid identifier body character. We'll use the same trick in the readSymbol method:

// below readString
  readSymbol() {
    let { pos, line, col, file } = this.input;
    const srcloc = SrcLoc.new(pos, line, col, file);
    const sym = this.input.readWhile(isSymbolChar);

    if (isBoolean(sym)) {
      return Token.new(TokenTypes.Boolean, sym, srcloc);
    } else if (isNil(sym)) {
      return Token.new(TokenTypes.Nil, sym, srcloc);
    }

    // Throw for now, since we haven't implemented symbols yet
    throw new SyntaxException(sym, srcloc);
  }

Note that we're just using the readSymbol method to read primitive literals for booleans and nil for now. Soon we'll use it to read identifiers as well, at which point we'll remove the exception at the end of the method.

The Reader

The changes to the reader are pretty simple: we just add cases to the readAtom function to handle the new token types. Just like with numbers, the reader handles the other primitive tokens themselves as the forms for those values.

In src/reader/read.js, here's the new readAtom function:

const readAtom = (reader) => {
  const tok = reader.peek();

  switch (tok.type) {
    case TokenTypes.Number:
      reader.skip();
      return tok;
    case TokenTypes.String:
      reader.skip();
      return tok;
    case TokenTypes.Boolean:
      reader.skip();
      return tok;
    case TokenTypes.Keyword:
      reader.skip();
      return tok;
    case TokenTypes.Nil:
      reader.skip();
      return tok;
    default:
      throw new SyntaxException(tok.value, tok.srcloc);
  }
};

The AST and Parser

First, we add new node types to the ASTTypes enum in src/parser/ast.js:

export const ASTTypes = {
  Program: "Program",
  NumberLiteral: "NumberLiteral",
  StringLiteral: "StringLiteral",
  BooleanLiteral: "BooleanLiteral",
  KeywordLiteral: "KeywordLiteral",
  NilLiteral: "NilLiteral",
};

Next, we add new constructor methods to the AST object for the new primitives:

export const AST = {
  // Program and NumberLiteral constructors
StringLiteral(token) {
    return {
      type: ASTTypes.StringLiteral,
      value: token.value,
      srcloc: token.srcloc,
    };
  },
  BooleanLiteral(token) {
    return {
      type: ASTTypes.BooleanLiteral,
      value: token.value,
      srcloc: token.srcloc,
    };
  },
  KeywordLiteral(token) {
    return {
      type: ASTTypes.KeywordLiteral,
      value: token.value,
      srcloc: token.srcloc,
    };
  },
  NilLiteral(token) {
    return {
      type: ASTTypes.NilLiteral,
      value: token.value,
      srcloc: token.srcloc,
    };
  },
}

Now in src/parser/parse.js we make changes to parsePrimitive that are very similar to what we did in the reader:

const parsePrimitive = (reader) => {
  const token = reader.peek();

  switch (token.type) {
    case TokenTypes.Number:
      reader.skip();
      return AST.NumberLiteral(token);
    case TokenTypes.String:
      reader.skip();
      return AST.StringLiteral(token);
    case TokenTypes.Boolean:
      reader.skip();
      return AST.BooleanLiteral(token);
    case TokenTypes.Keyword:
      reader.skip();
      return AST.KeywordLiteral(token);
    case TokenTypes.Nil:
      reader.skip();
      return AST.NilLiteral(token);
    default:
      throw new SyntaxException(token.value, token.srcloc);
  }
};

The Emitter

Now we need to emit code for the new primitive forms. In src/emitter/Emitter.js we'll need to add emitString, emitBoolean, emitKeyword, and emitNil methods to the Emitter class. We'll also add cases for each primitive node to the switch in the emit method.

Here's the new emit method:

  emit(node = this.program) {
    switch (node.type) {
      case ASTTypes.Program:
        return this.emitProgram(node);
      case ASTTypes.NumberLiteral:
        return this.emitNumber(node);
      case ASTTypes.StringLiteral:
        return this.emitString(node);
      case ASTTypes.BooleanLiteral:
        return this.emitBoolean(node);
      case ASTTypes.KeywordLiteral:
        return this.emitKeyword(node);
      case ASTTypes.NilLiteral:
        return this.emitNil(node);
      default:
        throw new SyntaxException(node.type, node.srcloc);
    }
  }

The emitBoolean method is about as simple as it can get:

  emitBoolean(node) {
    return node.value;
  }

Since keywords are JavaScript symbols under the hood, we'll need to use the Symbol.for constructor in emitting keywords (Note that we're emitting a string containing the "Symbol.for" call, not the constructor call itself):

  emitKeyword(node) {
    return `Symbol.for("${node.value}")`;
  }

The emitNil method just returns the string "null":

  emitNil(node) {
    return "null";
  }

Finally, emitString slices off the opening and closing double quotes and wraps the value in backticks to keep certain escape sequences from causing weird problems when evaluating the result:

  emitString(node) {
    return "`" + node.value.slice(1, -1) + "`";
  }

The Printer

Now that we have some new values besides numbers that will be output by the EVAL function in the CLI, we'll need to handle them in the printer. Here's the new printString function in src/printer/printString.js:

export const printString = (value, withQuotes) => {
  switch (typeof value) {
    case "number":
      return String(value);
    case "string":
      return withQuotes ? `"${value}"` : value;
    case "symbol":
      return value.description;
    case "boolean":
      return String(value);
    case "undefined":
      return "nil";
    case "object":
      if (value === null) {
        return "nil";
      }
    default:
      throw new Exception(`Invalid print value ${value}`);
  }
};

Note that since both null and undefined are represented as nil in Wanda we have to handle both cases in the printer. Also, currently null is the only "object" type value that will be produced by EVAL for now.

We're also going to implement an AST printer so we can use it for debugging purposes if needed. Create a new file printAST.js in src/printer and import the ASTTypes enum:

import { ASTTypes } from "../parser/ast.js";

We're going to do the AST printer as a class that implements the Visitor pattern then export a functional interface to it, just like the code emitter. First, the constructors:

class ASTPrinter {
  constructor(program) {
    this.program = program;
  }

  static new(program) {
    return new ASTPrinter(program);
  }
}

We'll need a print method to dispatch on the AST node type, and individual methods to handle those nodes. Since all our nodes except Program are primitives, we can actually handle all of those with a single method, printPrimitive:

  printPrimitive(node, indent) {
    return `${" ".repeat(indent)}${node.type}: ${
      node.type === "NilLiteral" ? "nil" : node.value
    }`;
  }

Note the indent property, which we'll eventually use to show how nodes are nested inside other nodes once we have compound forms that aren't just primitives.

We also need a method to handle the Program node:

  printProgram(node, indent) {
    let pStr = "";

    for (let n of node.body) {
      pStr += this.print(n, indent);
      +"\n";
    }

    return pStr;
  }

Finally, the print method to dispatch based on the node type:

  print(node = this.program, indent = 0) {
    switch (node.type) {
      case ASTTypes.Program:
        return this.printProgram(node, indent);
      case ASTTypes.NumberLiteral:
      case ASTTypes.StringLiteral:
      case ASTTypes.BooleanLiteral:
      case ASTTypes.KeywordLiteral:
      case ASTTypes.NilLiteral:
        return this.printPrimitive(node, indent);
    }
  }

Finally, at the bottom of printAST.js, let's export a function to use the ASTPrinter:

export const printAST = (ast) => ASTPrinter.new(ast).print();

Improving the CLI

We're going to add 2 AST pretty print functions to the CLI. We're also going to add the ability to invoke the CLI with a print command and option so the REPL will print the AST given to it as a program. This will allow us to debug things later when our programs get more complicated.

First, here's src/cli/pprint.js:

import { tokenize } from "../lexer/tokenize.js";
import { read } from "../reader/read.js";
import { expand } from "../expander/expand.js";
import { parse } from "../parser/parse.js";
import { desugar } from "../desugarer/desugar.js";
import { printAST } from "../printer/printAST.js";

export const pprintDesugaredAST = (input, file = "stdin") =>
  printAST(desugar(parse(expand(read(tokenize(input, file))))));

export const pprintAST = (input, file = "stdin") =>
  printAST(parse(expand(read(tokenize(input, file)))));

We're going to add a helper function that will let us throw an Exception from an expression position in the compiler's code, in src/shared/fail.js:

import { Exception } from "./exceptions.js";

export const fail = (msg, exn = Exception) => {
  throw new exn(msg);
};

Now we'll make some changes in src/cli/repl.js:

import readlineSync from "readline-sync";
import { EVAL } from "./eval.js";
import { pprintAST, pprintDesugaredAST } from "./pprint.js";
import { print } from "../printer/print.js";
import { fail } from "../shared/fail.js";

const read = (prompt) => readlineSync.question(prompt);

export const repl = (mode) => {
  const proc =
    mode === "repl"
      ? EVAL
      : mode === "printDesugared"
      ? pprintDesugaredAST
      : mode === "printAST"
      ? pprintAST
      : fail("Invalid REPL mode specified");
  let prompt = "wanda> ";

  while (true) {
    try {
      const input = read(prompt);

      if (input === ":quit") {
        process.exit(0);
      }

      let result = proc(input);

      print(result, mode === "repl");
    } catch (e) {
      console.error(e.message);
    }
  }
};

Note that now if the user enters the keyword :quit at the prompt it will exit the REPL.

Now in src/cli/run.js we're going to create an actual run function that will set the mode for the REPL instead of just importing and calling the repl function:

import { Exception } from "../shared/exceptions.js";
import { repl } from "./repl.js";

export const run = () => {
  let mode = "";
  switch (process.argv[2]) {
    case "print":
      if (process.argv[3] === "-d") {
        mode = "printDesugared";
        break;
      } else if (process.argv[3] === "-a") {
        mode = "printAST";
        break;
      }
    case undefined:
    case "-i":
    case "repl":
      mode = "repl";
      break;
    default:
      throw new Exception("Invalid command specified");
  }
  repl(mode);
};

Now we have a print command we can use when we execute the program, which is further used with either -a or -d options to print either the base AST or its desugared version. Right now there's no difference between the two, but that will change!

Creating The `wanda` Command

Now we need a way to call the run function. If you follow these instructions you'll be able to start the Wanda REPL just using the command wanda.

First, in the project root, open the package.json file. Somewhere in the toplevel properties (I've got it just below the "name" property), add:

{
// ...
  "bin": "bin/wanda.js",
// ...
}

Now in the project root create the bin directory (not src/bin) and add the file wanda.js to it. Here are its contents:

#!/usr/bin/env node

import { run } from "../src/cli/run.js";

run();

If the first line looks strange to you, it's called a shebang. It's a convention from UNIX shell scripts. It lets the system know to use Node.js to process the file, just like if you were writing a Bash script it would let the system know to use Bash to process the file.

After saving bin/wanda.js, make sure you're in the project root in your terminal and enter the command npm link. This will tell NPM to link your wanda command as a toplevel system command. Now whatever code is executed in bin/wanda.js will run when you enter the command wanda.

You can try it yourself: enter wanda into your terminal and hit ENTER.

Now you can enter values of any of the primitive types, including numbers, strings, booleans, keywords, and nil.

Want to see the AST printer in action? Run the command wanda print -a and see it work!

Conclusion

That wraps it up for today! We've now implemented all the primitive types in Wanda. Next time we'll add call expressions and change the data structure output by the reader to be more like that of a conventional Lisp language: using cons cells (linked lists) to represent the forms. This won't seem like a revolutionary change at first, but as it turns out this will enable us to do some pretty neat things in the future.

To see an exact snapshot of the state of the Wanda compiler and CLI as of the end of this article, check out the tag on GitHub. Note that the repo also includes JSDoc comments for the compiler as well as tests.

I'll see you next time!

Create Your Own Programming Language 1: Numbers

Jason Barr — Wed, 24 May 2023 00:07:00 +0000

In today's post we're going to scaffold out the compiler for Wanda and add numbers to the language.

Numbers will be represented by the JavaScript number type, which is a 64-bit floating point number.

Floating Point Numbers vs. Decimals

If you're not sure what that means, here's the tl;dr: the convention in regular use is to use decimal numbers, i.e. base 10. Computers can only deal with binary numbers, i.e. base 2. That means the numeral 10 in decimal means 10, whereas the numeral 10 in binary means 2.

Floating point numbers are a standard form used for representing decimal numbers in a computer's memory that deal with the issues that come with trying to convert base 10 numbers to base 2 numbers.

That means floating point numbers don't always work the exact same way as the decimals you're used to. For example, in decimal numbers 0.1 * 3 always equals 0.3. With floating point numbers, 0.1 * 3 is almost but not quite equal to 0.3 due to differences between rounding decimal numbers vs. rounding binary numbers.

To be exact, in floating point numbers 0.1 * 3 is 0.30000000000000004.

Some languages abstract these differences away by using BigDecimal numbers, but that would take away from our basic goal which is just to implement a language so we're not going to do it with Wanda. We'll stick to regular, old JavaScript numbers.

I am working on another language that does use BigDecimals, and I'll update you when there's something to show with that.

The Stages of Our Compiler

To recap the intro article, the compiler has several parts. Our compiler will have the following stages as it transforms a string of Wanda source code into its JavaScript output:

Lexing
Reading (producing an s-expression tree)
Expanding
Parsing (producing an Abstract Syntax Tree (AST))
Desugaring
Emitting

Note that later in the series we'll also add a stage for semantic analysis after the parsing stage.

In addition to compiling numbers, we're also going to start building out a CLI interface to the compiler with a REPL (Read-Eval-Print Loop) that will allow us to compile and evaluate code in a single step.

Most of the heavy lifting today will be done in the lexing phase, reading numbers from the text input into tokens. In fact, the expanding and desugaring phases won't do anything at all; they'll just take their input data structure and return it straight away. I'm just including them now so we don't have to add them to the pipeline later.

Project Setup

I use Prettier for code formatting, with the most basic .prettierrc config possible:

{}

You can run npm init if you want to initialize the project as a package. I have mine scoped to my name, @jasonsbarr/wanda. Note that I'm using Node v20.0.0, but you shouldn't need a brand new version of Node for the code to work.

Make sure that in your package.json file you have the directive "type": "module" so ES2015 modules work as specified.

All code is in the src directory unless specified otherwise.

Lexing

The lexer takes an input string of source code and transforms it into tokens that represent lexemes, which are valid bits of text that can be transformed into syntax objects as part of a tree. It takes in a string and outputs an array of token objects.

A token holds 3 bits of information: the token type, its text value, and an object that contains source location (srcloc) info: file, line, column, and position.

Source Location Info

Let's start by creating the SrcLoc class for the srcloc info. Create a new directory lexer in the src directory and put this in SrcLoc.js:

export class SrcLoc {
  constructor(pos, line, col, file) {
    this.pos = pos;
    this.line = line;
    this.col = col;
    this.file = file;
  }

  static new(pos, line, col, file) {
    return new SrcLoc(pos, line, col, file);
  }
}

Note that I add a static new method to all my classes as a personal preference so creating new objects is just a method call instead of a new expression. That's purely my preference; you can do the same with your classes if you like or leave it off. It's up to you.

The Token Type

In the lexer folder let's add TokenTypes.js for an enum-like object to represent token types. "Number" is the only token type for now:

export const TokenTypes = {
  Number: "Number",
};

Now let's add the Token class in src/lexer/Token.js:

export class Token {
  constructor(type, value, srcloc) {
    this.type = type;
    this.value = value;
    this.srcloc = srcloc;
  }

  static new(type, value, srcloc) {
    return new Token(type, value, srcloc);
  }
}

Helper Functions

Before we get into the lexer proper, we need 2 more things for support:

A set of helper functions to detect certain characters
An object to manage the input string as we traverse it and chunk it into tokens

First, the helper functions (in src/lexer/utils.js). We need to detect dots, digits, whitespace, semicolons (comments begin with a semicolon and continue to the end of the line), newlines, and dashes (minus signs), plus a function for checking if a numeric string is a well-formed number:

// Character matchers
export const isDot = (ch) => /\./.test(ch);
export const isDigit = (ch) => /\d/.test(ch);
export const isWhitespace = (ch) => /\s/.test(ch);
export const isSemicolon = (ch) => /;/.test(ch);
export const isNewline = (ch) => /\n/.test(ch);
export const isDash = (ch) => /\-/.test(ch);

// String matchers
export const isNumber = (str) => /^[+-]?\d+(\.\d+)?$/.test(str);

The InputStream Class

Now a class for the InputStream. This class manages the internal state of the input as the lexer traverses it and breaks it up into tokens.

The InputStream class has 5 methods: eof, lookahead, next, peek, and readWhile.

eof checks to see if the current position is at or past the end of the input string
lookahead gets the character at the position n spots ahead of the current position
next gets the character at the current position and advances the input by 1
peek gets the character at the current position, but does not advance the input
readWhile consumes every character in the input while a condition is true

src/lexer/InputStream.js:

import { isNewline } from "./utils.js";

export class InputStream {
  constructor(input, file) {
    this.input = input;
    this.file = file;
    this.pos = 0;
    this.line = 1;
    this.col = 1;
  }

  static new(input, file) {
    return new InputStream(input, file);
  }

  get length() {
    return this.input.length;
  }

  eof() {
    return this.pos >= this.length;
  }

  lookahead(n = 1) {
    return this.input[this.pos + n];
  }

  next() {
    const ch = this.input[this.pos++];

    if (isNewline(ch)) {
      this.line++;
      this.col = 1;
    } else {
      this.col++;
    }

    return ch;
  }

  peek() {
    return this.input[this.pos];
  }

  readWhile(pred) {
    let str = "";
    while (pred(this.peek())) {
      str += this.next();
    }

    return str;
  }
}

In-Language Exceptions

Let's add a SyntaxException class that inherits from a base Exception class in the language, which just inherits from the built-in JavaScript Error class. In src/shared/exceptions.js:

export class Exception extends Error {
  constructor(msg) {
    super(msg);
  }
}

export class SyntaxException extends Exception {
  constructor(value, srcloc) {
    super(
      `Syntax Exception: invalid syntax ${value} found at ${srcloc.file} (${srcloc.line}:${srcloc.col})`
    );
  }
}

Now we have the machinery set up so we can create the lexer itself.

The Lexer

We'll create a Lexer class that holds an InputStream object and has methods for tokenizing the input.

The main tokenize method creates an array for tokens, then has a loop that continues as long as we're not at the end of the input. Inside the loop we peek at the current character then perform an action based on it.

We'll skip whitespace and comments, and the only token we're reading right now is a number token. We'll do this with the readNumber method.

First, we import these functions and types into src/lexer/Lexer.js:

import { SyntaxException } from "../shared/exceptions.js";
import { SrcLoc } from "./SrcLoc.js";
import { Token } from "./Token.js";
import { TokenTypes } from "./TokenTypes.js";
import {
  isDash,
  isDigit,
  isDot,
  isNewline,
  isNumber,
  isSemicolon,
  isWhitespace,
} from "./utils.js";

The Lexer class constructor is simple:

export class Lexer {
    constructor(input) {
    this.input = input;
  }

  static new(input) {
    return new Lexer(input);
  }
}

For the readNumber method we first get the srcpos info off the input, then check to see if there's a minus sign. If there's a minus sign, add it to the number token. Then read any ensuing numbers and decimal points into the token value. We need to be able to tell if it's a valid or invalid number token, so we're just going to take all the numbers and dots in until there are no more left.

Then, if it's not a valid number, throw a SyntaxException. Otherwise, we create the number token.

In the Lexer class, beneath the new method:

  readNumber() {
    let { pos, line, col, file } = this.input;
    const srcloc = SrcLoc.new(pos, line, col, file);
    let num = "";

    if (isDash(this.input.peek())) {
      num += this.input.next();
    }

    num += this.input.readWhile((ch) => isDigit(ch) || isDot(ch));

    if (!isNumber(num)) {
      throw new SyntaxException(num, srcloc);
    }

    return Token.new(TokenTypes.Number, num, srcloc);
  }

The tokenize method runs a loop and dispatches on the current character. Right now all we're doing is skipping whitespace and comments, and reading number tokens. Anything else will cause a SyntaxException.

In the Lexer class below the readNumber method:

  tokenize() {
    let tokens = [];

    while (!this.input.eof()) {
      let ch = this.input.peek();
      if (isWhitespace(ch)) {
        this.input.readWhile(isWhitespace);
      } else if (isSemicolon(ch)) {
        this.input.readWhile((ch) => !isNewline(ch) && !this.input.eof());
      } else if (isDash(ch) && isDigit(this.input.lookahead(1))) {
        tokens.push(this.readNumber());
      } else if (isDigit(ch)) {
        tokens.push(this.readNumber());
      } else {
        const { pos, line, col, file } = this.input;
        throw new SyntaxException(ch, SrcLoc.new(pos, line, col, file));
      }
    }

    return tokens;
  }

That's all for the Lexer class for now! Now we'll abstract it away into a tokenize function in src/lexer/tokenize.js:

import { InputStream } from "./InputStream.js";
import { Lexer } from "./Lexer.js";

export const tokenize = (input, file) =>
  Lexer.new(InputStream.new(input, file)).tokenize();

Next we'll move onto the reader.

The Reader

"Reader" is the Lisp equivalent of a parser. A conventional reader dispatches based on the current character, similar to how our lexer works, and reads the input stream into a tree of s-expression data structures that are homoiconic with the source code. That means the code and data can essentially be substituted for one another in the programmer's imagination. The underlying data is the same as the outward representation in text.

Our compiler has both a reader and a parser for pedagogical reasons. We'll read the input into s-expressions so we can have all the power and elegance of Lisp macros, but we'll then parse that tree into a fuller AST so I can show you some additional compiler techniques that will help us make the language more interesting.

Technically we could have done all this with the s-expression tree produced by the reader and expander, but it's a little easier to do with a fuller AST and we're prioritizing understandability over efficiency here.

First we'll create a Reader class that manages the token stream in similar fashion to how the InputStream class manages the input string.

The Reader class has 4 methods:

eof, which checks to see if we're at the end of the token stream
next, which returns the current token and advances the stream by 1
peek, which returns the token at the current position
skip, which skips the current position without returning a token

In src/reader/Reader.js:

export class Reader {
  constructor(tokens) {
    this.tokens = tokens;
    this.pos = 0;
  }

  static new(tokens) {
    return new Reader(tokens);
  }

  get length() {
    return this.tokens.length;
  }

  eof() {
    return this.pos >= this.length;
  }

  next() {
    return this.tokens[this.pos++];
  }

  peek() {
    return this.tokens[this.pos];
  }

  skip() {
    this.pos++;
  }
}

The read functions take the Reader as a parameter and read syntactic forms from the token stream. Since we're only reading numbers right now, the read functions are very similar.

The main read function currently returns an array of forms. The only form we're reading right now is a number token. In the future, we'll change the data structure created by read from an array to an s-expression made up of linked lists using cons cells. If you don't know what that means, don't worry about it—I'll show you exactly what that means in a future post. We'll also add additional syntactic forms besides plain tokens and lists.

The read function dispatches to the readForm function, which reads forms from the current position in the token stream. Right now our only form is the number token, which is an atomic form, so readForm just passes the Reader through to readAtom. This will change soon.

In src/reader/read.js, import these types:

import { TokenTypes } from "../lexer/TokenTypes.js";
import { SyntaxException } from "../shared/exceptions.js";
import { Reader } from "./Reader.js";

Here's the readAtom function in its current state:

const readAtom = (reader) => {
  const tok = reader.peek();

  switch (tok.type) {
    case TokenTypes.Number:
      reader.skip();
      return tok;
    default:
      throw new SyntaxException(tok.value, tok.srcloc);
  }
};

As mentioned above, readForm currently just delegates to readAtom:

const readForm = (reader) => {
  return readAtom(reader);
};

And read processes the whole token stream:

export const read = (tokens) => {
  const reader = Reader.new(tokens);
  let parseTree = [];

  while (!reader.eof()) {
    parseTree.push(readForm(reader));
  }

  return parseTree;
};

That's the current iteration of the reader! Soon we'll add additional forms for it to read, including some reader macros that will transform the token stream as it reads it.

The Expander

The expander is simple, since there's nothing yet to expand. It merely takes the output of the reader and returns it. In src/expander/expand.js:

export const expand = (parseTree) => parseTree;

The Parser

Unlike most Lisp languages, this one has a separate parsing stage that converts the s-expression tree into a JSON-compatible AST made up of object nodes.

Since we're only compiling numbers for now, there are only 2 AST nodes: Program and NumberLiteral. I'm writing the AST module as an object of node constructor methods, rather than as a set of classes.

Every node has the properties type and srcloc that contain the node type and srcloc info, respectively. Different node types have different properties based on the data needed to represent the language form they signify.

The Program node has a body property that is an array of nodes that make up the program body. NumberLiteral has a value property that is the text value from the number token.

In src/parser/ast.js:

export const ASTTypes = {
  Program: "Program",
  NumberLiteral: "NumberLiteral",
};

export const AST = {
  Program(exprs) {
    return {
      type: ASTTypes.Program,
      body: exprs,
      srcloc: exprs[0]?.srcloc ?? SrcLoc.new(0, 0, 0, "none"),
    };
  },
  NumberLiteral(token) {
    return {
      type: ASTTypes.NumberLiteral,
      value: token.value,
      srcloc: token.srcloc,
    };
  },
};

The parser also uses the Reader class from the reader to manage its input state. This will change when we edit the reader to output a cons data structure instead of a JavaScript array, but it's good enough for now.

In src/parser/parse.js, import the following:

import { TokenTypes } from "../lexer/TokenTypes.js";
import { SyntaxException } from "../shared/exceptions.js";
import { AST } from "./ast.js";
import { Reader } from "../reader/Reader.js";

Like with the reader, we have a main parse function that dispatches to the parseExpr function. Since our only expression type right now is numbers, all parseExpr currently does is dispatch to parsePrimitive.

The parser will change quite a bit when we start returning a linked list from the reader instead of an array, but this will do for now:

const parsePrimitive = (reader) => {
  const token = reader.peek();

  switch (token.type) {
    case TokenTypes.Number:
      reader.skip();
      return AST.NumberLiteral(token);
    default:
      throw new SyntaxException(token.value, token.srcloc);
  }
};

const parseExpr = (reader) => {
  return parsePrimitive(reader);
};

export const parse = (readTree) => {
  let body = [];
  const reader = Reader.new(readTree);

  while (!reader.eof()) {
    body.push(parseExpr(reader));
  }

  return AST.Program(body);
};

That's it for the parser. In languages with more complicated syntax, parsing is a huge and complicated undertaking. Here, it's pretty simple. Even as we add more forms to the language that require parsing, it will still be much simpler for us than it would be in a language with more complicated syntax.

If there's interest, I may write a tutorial for a more complex parser in the future. Let me know in the comments if that's something you'd like to see!

The Desugarer

The desugarer is the first stage in the back end of the compiler. It takes higher, more complex forms in the language and converts them into simpler core forms. Sometimes you'll see this process called "lowering."

Like the expander above, the desugarer has nothing to do right now since we only have a single form in our language. That will change in the near future. For now, just put this in src/desugarer/desugar.js:

export const desugar = (ast) => ast;

And with that, we're ready to build the code emitter!

The Emitter

The emitter for our language is going to be fairly simple as well. It simply walks the desugared AST and emits code based on the nodes.

It uses a design pattern called the Visitor Pattern to "visit" each node of the tree and perform an action based on the node type. In this case, the action is emitting the code.

The emitter uses postorder traversal to visit the nodes, meaning it acts on the children at the bottom of the hierarchy before it acts on their parent nodes.

The Dispatcher Method

The emit method dispatches node-specific emitter methods depending on the node type. It uses a switch statement on the node.type property.

Since the only node types we have right now are Program and NumberLiteral, those are the only nodes we have emitters for.

The default case in emit throws a SyntaxException, though this is only for completeness to make sure we remember to implement emitter methods for every core node type.

In src/emitter/Emitter.js:

import { ASTTypes } from "../parser/ast.js";
import { SyntaxException } from "../shared/exceptions.js";

export class Emitter {
  constructor(program) {
    this.program = program;
  }

  static new(program) {
    return new Emitter(program);
  }

  emit(node = this.program) {
    switch (node.type) {
      case ASTTypes.Program:
        return this.emitProgram(node);
      case ASTTypes.NumberLiteral:
        return this.emitNumber(node);
      default:
        throw new SyntaxException(node.type, node.srcloc);
    }
  }

  emitNumber(node) {
    return node.value;
  }

  emitProgram(node) {
    let code = "";

    for (let n of node.body) {
      code += this.emit(n);
    }

    return code;
  }
}

Then in src/emitter/emit.js:

import { Emitter } from "./Emitter.js";

export const emit = (ast) => Emitter.new(ast).emit();

With that, we can create the whole compilation pipeline from Wanda input to JavaScript output.

The CLI

We're not going to build out a whole big CLI application just yet. I'll dedicate a whole article to that in the future. It would be nice to have some streamlined functions to let us compile and run the code though, as well as a REPL so we can run test programs.

Let's create a simple compilation pipeline in src/cli/compile.js:

import { tokenize } from "../lexer/tokenize.js";
import { read } from "../reader/read.js";
import { expand } from "../expander/expand.js";
import { parse } from "../parser/parse.js";
import { desugar } from "../desugarer/desugar.js";
import { emit } from "../emitter/emit.js";

export const compile = (input, file = "stdin") =>
  emit(desugar(parse(expand(read(tokenize(input, file))))));

Then an evaluator function using the Node.js vm module in src/cli/eval.js:

import vm from "vm";
import { compile } from "./compile.js";

export const EVAL = (input) => vm.runInThisContext(compile(input));

Note that EVAL is in all caps because eval is a reserved word in JavaScript.

For our simple REPL we'll need a way to get input from the console. The simplest way to do that is just to install the readline-sync package from NPM:

npm install readline-sync

And here's the simplest of REPLs in src/cli/repl.js:

import readlineSync from "readline-sync";
import { EVAL } from "./eval.js";
import { print } from "../printer/print.js";

const read = (prompt) => readlineSync.question(prompt);

export const repl = () => {
  let prompt = "wanda> ";

  while (true) {
    try {
      let result = EVAL(read(prompt)) ?? "";

      if (result === "") {
        process.exit(0);
      }

      print(result);
    } catch (e) {
      console.error(e.message);
    }
  }
};

Finally, we run the CLI by simply calling the repl function in src/cli/run.js:

import { repl } from "./repl.js";

repl();

Then run the REPL with node src/cli/run.js and enter numbers to your heart's content!

Note that any input that is not a valid number (integer or floating point, exponential notation not supported) will cause a SyntaxException. We'll start fixing that in the next article in the series.

Conclusion

Whew! That was a lot. But you've come a long way – you now have a fully functional compiler and evaluator with REPL! Granted, you can only run the simplest of programs, numeric literals, but we're going to make things much more interesting as we progress through the series.

I hope you understand better now how programming languages and compilers work, and I'm looking forward to adding additional features to Wanda with you in the days ahead.

You can see the code from this article in the Wanda GitHub repo. I'll conveniently tag the code from each article so you can see how things evolve as we move along. The repo code also has JSDoc comments for most of the functions and types, though I can't promise I've remembered to document every single one. There are also tests. You can run the test suite with the npm test command.

See you next time!

How To Create Your Own Programming Language

Jason Barr — Tue, 23 May 2023 18:59:00 +0000

Programming languages fascinate me.

They are the means by which we communicate human desires into machine-usable instructions that can create amazing art, produce scientific breakthroughs, and carry out just about every type of business on the planet.

We break down these human-readable words and phrases into electrical impulses that turn tiny switches on and off, and the entire modern world is based on the movement of those little switches.

To program is an act of creation, and programming languages are the means by which we programmers create our universes.

In this series of posts, we will build our own programming language together.

Why Create Your Own Language

There are hundreds of programming languages already. Why create your own?

Well, maybe there's something you want to do that existing languages just don't do well enough for your liking.

Maybe your passion is data visualization, and you want a language geared specifically for that task.

Languages are also fantastic for experiments. Some of the most beloved languages today have their roots in language experiments. For example, the ML family of languages (which includes OCaml and F#) started life as a theoretical type system. Lisp originally came from a paper on theories related to recursive functions. Ideas from these 2 language families have made their way into several modern languages that are in widespread use, including Rust and JavaScript.

Maybe you just enjoy a good programming challenge. Some of the hardest programming problems I've ever solved have come about while building interpreters and compilers. Creating a programming language brings together skills from a wide range of programming and computer science topics, including data structures and algorithms, computability theory, and more.

As for me? I just think creating languages is fun. I enjoy it, and I hope that maybe some of the people who read this series will come to enjoy it as well. I'm a better programmer because of the projects I've built while learning and creating new languages, and I'm sure you'll learn a lot from it too.

I'm far from an expert on language design and implementation. I just have a lot of fun doing it, and I'm here to document what I'm learning in case it's helpful for anyone else.

What Is A Programming Language?

Is a language the symbols (syntax) you use to tell the computer what to do? Is it the semantics (the meaning of those symbols)? Is it something else? A compiler or interpreter? All of the above?

I'm going to say it's a little bit of all of them. Obviously the semantics are the most important part. You can use the same syntax to implement vastly different semantic models. See the "Metacircular Evaluator" in chapter 4 of Structure and Interpretation of Computer Programs for an example of this. Using the same s-expression based syntax, the authors implement 4 radically different evaluator semantics.

But syntax matters as well. A good language probably feels good to write code in, and syntax obviously plays a part in that. Of course, what "feels good" might be very different for different programmers. Some people absolutely despise the minimal syntax of Lisp family languages with all its parentheses, but others find the symmetry between code and data structures that syntax allows to be the most beautiful thing in the world.

In the absence of a formal specification, the compiler and/or interpreter may serve as the source of truth for a language's behavior and semantics. Such will be the case for the language we build in this series. We're going to build a compiler that translates our source language into JavaScript. You might prefer to call that a transpiler instead of a compiler, since it doesn't emit assembly code or binary files. That's ok, you can call it whatever you want – it doesn't change the functionality.

How A Compiler Works

A compiler takes a string of code in a source language and converts it into another language that can be executed by a machine. Some compilers output binaries directly that are interpreted directly by the computer's hardware. We're going to compile our language to JavaScript, which can be executed either in a browser or by a server-side runtime like Node.js.

A compiler has several stages through which it transforms source code into its output, which can broadly be thought of as its front end and back end.

In the front end of the compiler, code gets read and transformed into a data structure used internally by the compiler. Usually a tree of some kind. This is usually done by lexical and syntactic analysis of the source code, or lexing and parsing.

During lexing, the compiler transforms the input text into a series of tokens that can be further analyzed by the parser.

The parser then takes the stream of tokens and converts them into a tree. This may be a parse tree, sometimes also known as a Concrete Syntax Tree, which contains all the data from the parsing stage, or it may produce an Abstract Syntax Tree (AST) which contains only the information necessary for further processing.

The compiler front end then moves into the semantic analysis phase which performs various checks on the AST to make sure the input program is well-formed.

Semantic analysis includes things like variable resolution, to make sure no uninitialized variables are used in the program, as well as type checking and inference (if it's a statically-typed language). Some languages perform even more semantic checks.

During the front end phases of the compiler, linguistic complexity tends to increase. We go from flat text to a data structure, and add annotations to the data structure during the process of semantic checking.

Then during the back end phases of the compiler the data structure created by the front end tends to be lowered and linguistic complexity decreases.

Higher level forms that are syntactic sugar get desugared into their core language forms.

Trees get converted into intermediate representations that allow for optimizations.

Finally, the compiler produces the end product and either outputs it into a file or feeds it into an evaluator for execution.

Our New Programming Language: Wanda

The language we're going to build throughout this series is called Wanda. Wanda is a fairly simple expression-based language with a few core forms, but it should be fully-featured enough to write interesting programs with.

Wanda's syntax is s-expression based, so it looks like a Lisp. Wanda code compiles to JavaScript values, and so like JavaScript it has first-class functions (i.e. functions that can be used like any other value, including as arguments to and return values from other functions).

Unlike JavaScript, though, we're going to restrict object creation to a class-based system. Under the hood the classes are built on top of JavaScript's prototypal delegation, but in Wanda itself you'll need to define a class to construct objects.

Wanda has a few primitive types:

; numbers are simply JavaScript numbers, i.e. 64 bit floats
15
; strings are JavaScript strings
"hello"
; booleans are JavaScript booleans
true
; nil combines JavaScript null and undefined
nil
; keywords are symbols with a colon at the beginning followed by a valid Wanda identifier
:keyword
; identifiers can have several characters not allowed in JavaScript
this-is+a*valid&identifier

Use the def keyword to define a constant or function:

(def name "Jason")
(def inc (x)
  (+ x 1))

Or var to define a variable and set! to change its value:

(var x 5)
(set! x 10)

You can define vector and map literals:

[1, 2, 3] ; note that commas are treated as whitespace
{:a "hi" :b 14}

Wanda has a for form that is an expression:

; for expression through the loop
; produces a list [0, 1, 2, 3, 4]
(for map ((i (range 5)))
  (println i)
  (inc i))

; produces the number 55
(for fold ((sum 0) (n (range 1 11)))
  (+ sum n))

Conditionals are expressions, so you can use them in variable assignment:

(def status (if (equal? state "Loading")
                "Loading"
                "Finished"))

Unlike with other Lisp-style languages, in Wanda you can access object properties with dot notation:

(def me (new Person "Jason" 42))
(println me.name) ; prints "Jason"

Wanda is gradually typed, including generics, so if you add type annotations to variables or functions it turns the type checker on:

(type Person { name: string, age: number })
(var (me: Person) { name: "Jason", age: 42 })
(def id (x: @a) -> @a
  x)

We'll cover additional language features, including classes and objects, pattern matching, and macros as we implement them. I'm especially excited to get to macros with you – implementing them is a real brain bender but they're pretty amazing once you get them working.

Like other Lisp-style languages, the s-expression syntax maps directly onto an in-language data structure in the reader. Wanda will then have an additional parsing step where the fully expanded tree produced by the reader (including expansion of any macros) is transformed into an object-based AST we can use for semantic analysis and additional compiler passes in the back end. We could just as easily output code based on the reader tree, but I'm using a full AST for learning purposes and because it's easier to do certain operations on an AST.

The Implementation Language

JavaScript is the implementation language for Wanda. I've chosen to write the compiler in JavaScript largely because it's an easy language to learn and most people who come across this series probably already know at least a little JavaScript. It's fairly easy to hack out simple projects in JavaScript, and I wanted to make this series as accessible as possible to as many people as I could.

An Incremental Approach

I've loosely based this series on Abdulaziz Ghuloum's seminal paper "An Incremental Approach to Compiler Construction" in that each article will end with a fully-functional compiler. We'll start out by compiling and evaluating numbers, and end up with a fully functional language that should hopefully be capable of supporting a wide range of interesting programs. It won't exactly be production grade in all aspects, but my goal is to make it interesting and useful for writing real programs and applications so you can see what it's like to implement a complete programming language that is more than just a toy.

I'll also cover some interesting and useful algorithms and data structures throughout the series that you may find helpful even if you never write another compiler.

Upcoming Articles

Right now the plan is to write articles implementing several features into the language, including:

Numbers
Additional primitive types (e.g. strings, booleans)
Cons pairs and lists
Variables
Vectors and Maps
Conditional expressions
Functions
Iteration
Objects and classes
Async
Exceptions and exception handling
Variants and pattern matching
Macros
Modules
Data flow analysis
AST transformations and optimization
A fully-featured CLI
Gradual typing (optional static typing)

What You Won't Find in this Series

You won't find a lot of programming language theory in this series, so if you're looking to learn about things like the role of deterministic finite automata in regular languages, context-free grammars, parsing algorithms, formal semantics, and so on, you'll need to consult a good book to learn about those.

There's nothing wrong with learning about that stuff.

Indeed, I'd say it's important to do so if you have an abiding interest in programming languages, but it's not the focus of this particular series.

I'm going to focus on implementation of a compiler with occasional notes about semantics and language design.

The goal is to produce something more in the spirit of Jack Crenshaw's classic Let's Build A Compiler series, though I wouldn't dream of putting my work on the same level as his.

I'll also cover some computer science topics along the way as they're relevant to what we're building, but as a means to help you better understand what we're building.

Inspiration

I've been blessed with the opportunity to use a number of excellent resources along my programming language journey, including a number of excellent books and courses, but Dmitry Soshnikov's series of programming language courses deserves special mention. He has courses on both theory and implementation, including building a parser, building interpreters and virtual machines, and even compiling a language to LLVM intermediate code. I wouldn't be where I am today without his materials and I highly recommend you check them out if you have further interest in the topic.

Conclusion

I hope you're as excited about this foray into language implementation as I am! The next entry in the series will present our first compiler, supporting numbers. It will also build up a lot of the boilerplate we'll need to compile more interesting expressions. Stay tuned!

A free (or mostly free) computer science education

Jason Barr — Thu, 01 Jul 2021 01:13:37 +0000

Update: Some of the materials I had originally listed are no longer available, so I've taken the dead links out and added new ones for the same courses, if possible, or if not, alternative options that should be just as good.

===
We programmers are lucky. A world-class education in the subject we practice is literally just a few searches away. There are myriad free and low-cost online courses available, and if you're not interested in the extras that come with taking actual courses you can just crib the notes from literally hundreds of course websites from university computer science courses all over the world.

In fact, if you've ever tried to put together your own self-study computer science/programming program you might find you have the opposite problem from what you'd expect: there's simply so much information out there that it's hard to know what to use.

Here's a list of online courses and a few other resources I've put together to study for myself in lieu of getting an actual degree.

Note that I'm already employed as a programmer, so a degree isn't necessary for me to work in the field. Your experience may vary, and you may decide an actual degree is the way to go. That's fine! There's nothing wrong with that. Sometimes I think we get stuck in this hacker mentality where we feel like we have to go our own way instead of following an established program. If a degree or other formal academic program is right for you, that's totally cool. Do what works best for you to help you meet your own goals.

If it's not, here's one way you might go about educating yourself.

The subjects

I wanted a list of resources that would cover most of the foundational subjects you would study in a top university CS program. These include:

Programming
Object-oriented programming and design
Data structures and algorithms
Low-level programming
Computer architecture and organization
Operating systems
Software engineering
Databases
Networks and networking
Programming languages

I also picked some "electives" based on my own interests:

More advanced algorithms
Parallel and distributed programming
Data science and machine learning
Functional programming
Interpreters and compilers

I also added some relevant subjects in math, since as an arts major my collegiate math background was inadequate for a CS education:

Mathematical thinking
Calculus
Discrete mathematics
Linear algebra
Statistics and probability

I've also added an optional physics course on electricity and magnetism, as it's a real-world prerequisite for the MIT Computation Structures course. There's also an optional course on theoretical computer science, which you should only work on if you have a good understanding of discrete mathematics (see the mathematics section).

How I chose my resources

I decided to prefer existing online courses that have certificate options, although these usually require payment of some kind. I find that having an external reward like a certificate helps keep me motivated and keeps me coming back to study when I'm tired after work and would rather just veg out and watch Netflix.

Even though in many cases I've linked to certificate programs that list a cost, you can access the content for the vast majority of courses I've picked for free, although there may be extra functionalities (like grading) that are unlocked by paying money.

This is simply my preference, and if you decide to choose other resources that are completely free that's totally fine; do what works for you.

I couldn't find adequate courses for all the relevant subjects, so in some cases I've listed books, video lectures, and materials available from course websites to fill the gaps.

I'm considering each top-level bullet point to be roughly the equivalent of a 3 credit, semester-long university course (except as noted), but YMMV. Using that as a guide, it should be pretty easy to pick enough from all the major topical areas to equal about 60 credits from a university. That's about average for major requirements in an undergraduate CS program here in the United States.

The program

Introduction to Computer Science and Programming

Choose 1 of the following 2 options:
- Option 1
  - HarvardX, CS50 Introduction to Computer Science
  - MITx, Computational Thinking using Python
    - Introduction to Computer Science and Programming Using Python
    - Introduction to Computational Thinking and Data Science
  - Optionally, you can take any of the following after CS50 to get the listed certificate (must pay for certificate):
- Option 2
  - Rice, Fundamentals of Computing Specialization
  - Princeton, Computer Science: Programming with a Purpose
  - Princeton, Computer Science: Algorithms, Theory, and Machines
    - These last 2 courses use Java, so you could potentially skip the Object-Oriented Programming section and add an optional course on a topic you find interesting
Optional:
- University of Illinois, Accelerated Computer Science Fundamentals Specialization

Object-Oriented Programming

Georgia TechX, Introduction to Object-Oriented Programming using Java
- Foundations and Syntax Basics
- Object-Oriented Programming and Algorithms
- Exceptions, Data Structures, Recursion, and GUIs

Software Design and Engineering

UBCx, Software Development (equal to 6 credits)
- How to Code: Simple Data
- How to Code: Complex Data
- Optional: finish the parts of How to Design Programs not covered by these courses
- Software Construction: Data Abstraction
- Software Construction: Object-Oriented Design
- Software Engineering: Introduction
- Software Development Capstone Project
MIT 6.005, Software Construction (Open Learning Library)
- Software Construction in Java
- Advanced Software Construction in Java
Additional Resources
- Software Debugging and Testing

Data Structures and Algorithms

Choose at least 1 of the following 3 options:
- Georgia TechX, Data Structures and Algorithms
  - ArrayLists, LinkedLists, Stacks and Queues
  - Binary Trees, Heaps, SkipLists and HashMaps
  - AVL and 2-4 Trees, Divide and Conquer Algorithms
  - Pattern Matching, Dijkstra’s, MST, and Dynamic Programming Algorithms
- StanfordX, Design and Analysis of Algorithms
  - Algorithms: Design and Analysis, Part 1
  - Algorithms: Design and Analysis, Part 2
- Princeton, Algorithms
For more advanced algorithm material, check out:
- Stanford 261: A Second Course in Algorithms (course website)
- MIT 6.046J: Design and Analysis of Algorithms (Open CourseWare)
- Harvard CS 224: Advanced Algorithms
  - Lecture playlist
  - Course website with assignments
- MIT 6.854J: Advanced Algorithms (Open CourseWare)
- MIT 6.851: Advanced Data Structures (Open CourseWare)

Low-Level Programming

DartmouthX, C Programming with Linux
- Getting Started
- Language Foundations
- Modular Programming and Memory Management
- Pointers and Memory Management
- Advanced Data Types
- Linux Basics: The Command Line Interface
- C Programming: Using Linux Tools and Libraries

Optional: Electricity and Magnetism

MITx, Introductory Electricity and Magnetism
- Electrostatics
- Magnetic Fields and Forces
- Maxwell's Equations
- This course is a required prerequisite for actual EE/CS students at MIT for the 6.004 course listed below
Optional:
- MIT 8.02 lecture videos
- All About Circuits

Computer Architecture and Organization

Hebrew University of Jerusalem, Build a Modern Computer from First Principles: From Nand to Tetris
- Part 1
- Part 2
Berkeley CS61C, Great Ideas in Computer Architecture
Optional:
- Princeton, Computer Architecture
- The following are 2 halves of the same course at Harvey Mudd, and require purchasing a hardware kit to build actual, physical systems
  - HarveyMuddX, Digital Design
  - HarveyMuddX, Computer Architecture
- MIT 6.004, Computation Structures (course website)

Parallel, Concurrent, and Functional Programming

Rice, Parallel, Concurrent, and Distributed Programming in Java Specialization
- Parallel Programming in Java
- Concurrent Programming in Java
- Distributed Programming in Java
École Polytechnique Fédérale de Lausanne, Functional Programming in Scala Specialization
- Functional Programming Principles in Scala
- Functional Program Design in Scala
- Parallel programming
- Big Data Analysis with Scala and Spark
Alternative FP resource:
- Cornell CS 3110, Data Structures and Functional Programming (2021 course website)
Optional:
- University of Illinois, Cloud Computing Specialization
- Advanced distributed systems:
  - MIT 6.824
    - Video lectures
    - Course website

Optional: Theoretical Computer Science

MIT 6.840J: Theory of Computation (Open Course Ware)
- Lecture videos
- You should have a solid grasp on discrete mathematics before attempting to work through this material
Building Blocks for Theoretical Computer Science (the textbook for CS 173: Discrete Structures at the University of Illinois)

Data Science and Machine Learning

Optional introductory course:
- University of Michigan, Applied Data Science with Python Specialization
MITx, Statistics and Data Science
- Probability - The Science of Uncertainty and Data
- Fundamentals of Statistics
- Machine Learning with Python: from Linear Models to Deep Learning
- Capstone Exam in Statistics and Data Science
- Data Analysis: Statistical Modeling and Computation in Applications
Stanford, Machine Learning Specialization
Optional:
- UC San Diego, Big Data Specialization
- IBM, Professional Certificate in Data Science
- IBM, Advanced Data Science Specialization
- Johns Hopkins, Data Science
  - Uses R instead of Python
- Johns Hopkins, Data Visualization & Dashboarding with R Specialization
- University of Illinois, Data Mining
- NYU, Information Visualization
  - Uses the D3.js JavaScript library

Math

Stanford, Introduction to Mathematical Thinking
UC San Diego, Mathematical Thinking in Computer Science
Stanford, Introduction to Mathematical Logic
- This one isn't strictly required, but it would be extremely helpful in preparation for Mathematics for Computer Science
MITx, Single Variable Calculus
- 1A: Differentiation
- 1B: Integration
- 1C: Coordinate Systems and Infinite Series
Discrete Mathematics introductory lecture notes
MIT, Mathematics for Computer Science
Georgia TechX, Linear Algebra
- Introductory Linear Algebra
  - Linear Equations
  - Matrix Algebra
- Applications of Linear Algebra
  - Determinants and Eigenvalues
  - Orthogonality & Symmetric Matrices and the SVD
Georgia TechX, Probability and Statistics
- Probability/Random Variables
  - A Gentle Introduction to Probability
  - Random Variables – Great Expectations to Bell Curves
- Statistics, Confidence Intervals and Hypothesis Tests
  - A Gentle Introduction to Statistics
  - Confidence Intervals and Hypothesis Tests
Optional:
- AdelaideX, MathTrackX
  - Polynomials, Functions, and Graphs
  - Special Functions
  - Differential Calculus
  - Integral Calculus
  - Probability
  - Statistics
- MIT 18.06, Linear Algebra (Open CourseWare)
- MITx, Multivariable Calculus I: Vectors and Derivatives
- MITx, Multivariable Calculus II: Integrals
- MITx, 18.03x: Differential Equations
  - Introduction to Differential Equations
  - Differential Equations: 2x2 Systems
  - Differential Equations: Linear Algebra and NxN Systems of Differential Equations
  - Differential Equations: Fourier Series and Partial Differential Equations
  - Transfer Functions and the Laplace Transform

For further study

The above list will give you a solid foundation in computer science and programming on par with what you would get from a BS degree. These are obviously just suggestions, though; feel free to tailor your own study to your interests and the things that motivate you.

If data science isn't your thing, maybe a course on iOS or Android app development will scratch your itch. You could also look into cloud-based development with programs from AWS or IBM. Chances are good that, if you've got an interest, there's a course out there for you.

I hope this list is helpful for you as you seek your own path to a well-rounded education as a developer.

DEV Community: Jason Barr

Create Your Own Programming Language 10: Optimization

A Normal Form

Tail Call Optimization

Fixing a Bug in The Parser

Preparing for A Normal Form

Transforming to A Normal Form

The Dispatcher Function

Transforming The Program

Transforming Call Expressions

Transforming Lambda Expressions

Transforming Variable Declarations

Transforming Vector Destructuring

Transforming Record Destructuring

Transforming Set Expressions

Transforming Vector Literals

Transforming Record Literals

Transforming Member Expressions

Transforming Do Expressions

Transforming As Expressions

Transforming If Expressions

Transforming When Expressions

Transforming Logical Expression

Tail Call Optimization

Changes to The Runtime

Changes to The Emitter

Changes to The CLI

Trying It Out

Conclusion

Create Your Own Programming Language 9: Iteration

Iteration in Wanda

New CLI Features

An Easy Iterator

Changes to The Lexer and Reader

Changes to The Parser

Changes to The Type Checker

Changes to The Visitor

Changes to The Desugarer

Changes to The Emitter

Changes to The Printer

Changes to The CLI

The New Readline

Getting The Version and Help Info

Commands

Changes to the REPL

Changes in Running The Wanda CLI

Changes in Running The Compiler

Trying It Out

Conclusion

Create Your Own Programming Language 8: Conditionals

Changes to The Lexer and Reader

Changes to The Core Library

Changes to The Parser

Changes to The Type Checker

Adding The Negation Type

Narrowing

Inference

Checking Against If and Cond Expressions

Checking with The Type Checker

Changes to The Visitor

Changes to The Desugarer

Changes to The Emitter

Changes to The Printer

Changes to The CLI

Changes to The REPL

Changes to The run Function

Compiling and Building Files

!/usr/bin/env node

Example Files

Conclusion

Create Your Own Programming Language 7: More Types

What All We're Doing Today

Singleton Types

Tuple Types

Union Types

Intersection Types

Type Ascriptions

Changes to Shared Types

Changes to The Lexer

Changes to The Reader

Changes to The `run` Function

Updated `fromTypeAnnotation`