DEV Community: Charper Bonaroo

User signup & login without dependencies

Toby Hinloopen — Mon, 27 Jan 2025 10:33:19 +0000

User Signup & Login

This blog is a part 3 of my bluebird blog. See part 1 and part 2

I don't think there's an easy way to send email without dependencies. I suppose if I really wanted email, I can use some email-as-a-service API using NodeJS' HTTP client. However, the email-as-a-service could be considered a dependency, so I won't.

Traditionally, if the user forgets their password, they can get a password reset link via email. Without email, this obviously isn't possible. For bluebird, my twitter-clone, forgetting your password results in losing access to your account.

Let's create a signup and login process. Before I start writing HTML, I want to implement the authentication (signup + login) process in a module for easy testing.

The authentication process will be handled by a class named Auth and will feature at least 2 methods: login and signup.

Signup

The signup method needs to create a user if it doesn't already exists. The created user should be returned. If the user does exist, some kind of error is returned.

For a method to return either an error or a user, I'm going to make the function method return an object with either an error property for failure, or a user property for success.

Following the Red-Green-Refactor rule, I'm writing a failing test first, then write a minimal implementation to make it pass.

// test/auth.test.mjs
import { describe, it, beforeEach } from "node:test";
import { Auth } from "../src/auth.mjs";
import assert from "node:assert";

describe("Auth", () => {
  describe("signup", () => {
    it("returns created user", async () => {
      const auth = new Auth();
      const { user } = await auth.signup("foo", "bar");
      assert(user);
    });
  });
});

// src/auth.mjs
export class Auth {
  async signup(name, pass) {
    return { user: { name } };
  }
}

Now to check for existing users. Note how I use JSDoc for typing. JSDoc is completely ignored by NodeJS, but it does help in some editors when the type of a variable isn't obvious.

// test/auth.test.mjs (partial)
describe("signup", () => {
  /** @type {Auth} */
  let auth;

  beforeEach(() => {
    auth = new Auth();
  });

  it("returns created user", async () => {
    const { user } = await auth.signup("foo", "bar");
    assert(user);
  });

  it("returns user-exists error if user already exists", async () => {
    const { user } = await auth.signup("foo", "bar");
    assert(user);

    const { error } = await auth.signup("foo", "bar");
    assert.equal(error, Auth.USER_EXISTS);
  });
});

// src/auth.mjs
/**
 * @typedef {Object} User
 * @property {string} name
 */

export class Auth {
  static USER_EXISTS = Symbol("USER_EXISTS");

  /** @type {Map<string, User>} */
  #users = new Map();

  /**
   *
   * @param {string} name
   * @param {string} pass
   * @returns {Promise<{ user: User }|{ error: symbol }>}
   */
  async signup(name, pass) {
    if (this.#users.has(name))
      return { error: Auth.USER_EXISTS };

    const user = { name };
    this.#users.set(name, user);
    return { user };
  }
}

Using the current implementation, I'm storing the created users in a variable. Later, I'll find a way to persist them in some kind of database.

Also note how I haven't implemented any password hashing. There is no way to test whether a password is stored or hashed properly without writing an additional method to verify the password, so that's what I'm going to do now using the login-method.

Login

I continue writing tests.

// test/auth.test.mjs (partial)
describe("login", () => {
  it("for unknown user, return USER_NOT_FOUND", async () => {
    const { error } = await auth.login("foo", "bar");
    assert.equal(error, Auth.USER_NOT_FOUND);
  });
});

// src/auth.mjs (partial)
class Auth {
  /**
   *
   * @param {string} name
   * @param {string} pass
   * @returns {Promise<{ user: User }|{ error: symbol }>}
   */
  async login(name, pass) {
    if (!this.#users.has(name))
      return { error: Auth.USER_NOT_FOUND };
  }
}

OWASP considers it a security risk to let the user know whether a username exists or not. They suggest to respond with a generic error message regardless of wether the password was incorrect, the account doesn't exist or the account is disabled.

However, in many implementations, there's still other ways for any user to find whether a username exists or not. Most notably, the signup screen generally shows you a user with that username already exists, and password recovery often also returns some kind of error indicating whether the email address is known or not.

OWASP suggests the signup page and account recovery to not report whether a user exists or not and instead send an email to the user. For this project, I don't have email available, so that's not going to work.

Additionally, I don't think the security issue is worth the hit on usability. I'm perfectly comfortable with rendering useful errors to users at the cost of this potential security risk. Let's continue!

// test/auth.test.mjs (partial)
it("for known user but invalid password, return INVALID_PASSWORD", async () => {
  await auth.signup("foo", "bar");
  const { error } = await auth.login("foo", "invalid");
  assert.equal(error, Auth.INVALID_PASSWORD);
});

// src/auth.mjs (partial)
class User {
  async login(name, pass) {
    if (!this.#users.has(name))
      return { error: Auth.USER_NOT_FOUND };
    return { error: Auth.INVALID_PASSWORD };
  }
}

// test/auth.test.mjs (partial)
it("for known user and valid password, return user", async () => {
  await auth.signup("foo", "bar");
  const { user } = await auth.login("foo", "bar");
  assert(user);
});

What's the minimal implementation to check a password? Yep: plain text!

class User {
    async signup(name, pass) {
    if (this.#users.has(name))
      return { error: Auth.USER_EXISTS };

    const user = { name, pass }; // <-- Don't do this!
    this.#users.set(name, user);
    return { user };
  }

  async login(name, pass) {
    const user = this.#users.get(name);
    if (!user)
      return { error: Auth.USER_NOT_FOUND };
    if (user.pass != pass) // <-- Don't do this!
      return { error: Auth.INVALID_PASSWORD };
    return { user };
  }
}

All tests pass. Now to find a way to make it not pass. First of all, I never want the password exposed. There's no perfect way to test whether the password is exposed by the returned user object, but a very minimal, naive approach would be to check none of the returned properties has the password as a value.

I'm going to reuse the last test I wrote for that purpose, iterating all properties of the returned object and checking for the password.

it("for known user and valid password, return user", async () => {
  await auth.signup("foo", "secret");
  const { user } = await auth.login("foo", "secret");
  assert(user);
  for (const k in user)
    assert.notEqual(user[k], "secret");
});

To hide the password, I'm going to hash the password in some way. The last time I hashed passwords myself, I used a single round of md5. This was a long time ago. After that, I only used bcrypt.

But that's a dependency. NodeJS seems to offer scrypt which is basically the same as bcrypt and only differs in one letter.

Unlike the bcrypt implementations I'm used to, scrypt requires you to bring your own salt. Scrypt seems to accept all kinds of parameters and options, so I'm going to wrap scrypt in my own hashing module which basically has a method for hashing a password and comparing a hash with a password.

I'm going to create a verify and hash method in the Pass class and their behaviour is pretty simple.

test/pass.test.mjs

import { describe, it, beforeEach } from "node:test";
import { Pass } from "../src/pass.mjs";
import assert from "node:assert";

describe("Pass", () => {
  /** @type {Pass} */
  let pass;

  beforeEach(() => {
    pass = new Pass({ cost: 2 });
  });

  it("hash creates a password hash", async () => {
    const hash = await pass.hash("my-password");
    assert.equal(typeof hash, "string");
  });

  it("Pass.verify returns true for the correct password", async () => {
    const hash = await pass.hash("my-password");
    assert.equal(await Pass.verify(hash, "my-password"), true);
  });

  it("Pass.verify returns false for any other password", async () => {
    const hash = await pass.hash("my-password");
    assert.equal(await Pass.verify(hash, "wrong-password"), false);
  });
});

src/pass.mjs

import crypto from "node:crypto";
import { promisify } from "node:util";

const scrypt = promisify(crypto.scrypt);
const randomBytes = promisify(crypto.randomBytes);

const pattern = /^\$s0\$(?<cost>\d+)\$(?<keylen>\d+)\$(?<salt>[^$]+)\$(?<key>[^$]+)$/

export class Pass {
  #keylen = 60;
  #cost = 16384;

  constructor({ keylen, cost } = {}) {
    this.#keylen = keylen ?? 60;
    this.#cost = cost ?? 16384;
  }

  /**
   *
   * @param {string} password
   * @returns {Promise<string>}
   */
  async hash(password) {
    return Pass.hash(password, await randomBytes(24), this.#keylen, this.#cost);
  }

  /**
   *
   * @param {string} hash
   * @param {string} password
   * @returns {Promise<boolean>}
   *
   */
  static async verify(hash, password) {
    const result = hash.match(pattern);
    if (!result) throw new Error(`Pass.verify was given an invalid string as a hash`);
    const cost = parseInt(result.groups.cost, 10);
    const keylen = parseInt(result.groups.keylen, 10);
    const salt = Buffer.from(result.groups.salt, "base64");
    const expectedHash = await Pass.hash(password, salt, keylen, cost);
    return expectedHash === hash;
  }

  /**
   *
   * @param {string} password
   * @param {Buffer} salt
   * @param {number} keylen
   * @param {number} cost
   */
  static async hash(password, salt, keylen, cost) {
    const key = await scrypt(password, salt, keylen, { cost });
    return `$s0$${cost}$${keylen}$${salt.toString("base64")}$${key.toString("base64")}`;
  }
}

I intentionally made Pass.verify a static method to make sure it doesn't rely on the config of a Pass-instance. That way, the tests automatically prove that a generated hash can be verified regardless the settings used at creation,
since there is no way to access instance variables from a static method.

I put all required parameters inside the generated password hash string, like bcrypt does. That way, I can safely change the parameters like cost, salt size or keylen in future releases without breaking existing password hashes.

Why $s0$ ? I don't know, I liked it. Bcrypt uses a similar prefix for versioning, so I thought I use it as well: s for scrypt and 0 for version 0.

For testing, I'm going to set the cost very low, but for real applications I would put the cost on a value to ensure password hashing takes about a second.

Back to the Auth-class. It's going to need an instance of Pass for hashing passwords.

// test/auth.test.mjs (partial)
beforeEach(() => {
  auth = new Auth({
    pass: new Pass({ cost: 2 })
  });
});

// src/auth.mjs
import { Pass } from "./pass.mjs";

/**
 * @typedef {Object} User
 * @property {string} name
 * @property {string} passHash
 */

export class Auth {
  static INVALID_PASSWORD = Symbol("INVALID_PASSWORD");
  static USER_EXISTS = Symbol("USER_EXISTS");
  static USER_NOT_FOUND = Symbol("USER_NOT_FOUND");

  /** @type {Map<string, User>} */
  #users = new Map();

  /** @type {Pass} */
  #pass;

  /** @param {{ pass: Pass }} opts */
  constructor({ pass }) {
    this.#pass = pass;
  }

  async signup(name, pass) {
    if (this.#users.has(name))
      return { error: Auth.USER_EXISTS };

    const user = { name, passHash: await this.#pass.hash(pass) };
    this.#users.set(name, user);
    return { user };
  }

  async login(name, pass) {
    const user = this.#users.get(name);
    if (!user)
      return { error: Auth.USER_NOT_FOUND };
    if (!await Pass.verify(user.passHash, pass))
      return { error: Auth.INVALID_PASSWORD };
    return { user };
  }
}

All tests now pass and the modules are ready for signup and login. Let's create the login and signup routes in App.

App: Controllers & Views

Our app already includes routes for login, specifically:

// src/app.mjs (partial)
export class App {
  async "GET /login"() {
    return render("login.ejs", { errorCode: null, params: { username: "" } });
  }

  async "POST /login"({ body }) {
    const { username, password } = body;
    const { user, error } = await this.#auth.login(username, password);

    if (user)
      return redirect("/profile", { cookies: { username: user.name } });

    return render("login.ejs", {
      errorCode: error.description,
      params: { username }
    });
  }
}

These routes were part of my previous blog.

The signup routes are just as simple. The GET /signup will just render a template with default params, and the POST /signup will delegate the params to the model (the Auth-module)

But before I write those, I'm going to write some failing tests first!

// app.test.mjs (partial)
describe("App", () => {
  describe("GET /signup", () => {
    it("renders signup template", async () => {
      await GET("/signup");
      assertRender("signup.ejs",
        { params: { username: "" }, errorCode: null });
    });
  });

  describe("POST /signup", () => {
    it("for existing user, return a USER_EXISTS error", async () => {
      await POST("/signup", { username: "foo", password: "bar" });

      assertRender("signup.ejs", {
        params: { username: "foo" },
        errorCode: "USER_EXISTS",
      });
    });

    it("with valid params, sets cookie and redirects to profile", async () => {
      await POST("/signup", { username: "foo", password: "bar" });
      assertRedirect("/profile", { cookies: { username: "foo" } });
    });
  });
});

These tests are very similar to the tests for login. The implementation is also very similar. So similar, I might as well copy/paste them.

// src/app.mjs (partial)
export class App {
  async "GET /signup"() {
    return render("signup.ejs", { errorCode: null, params: { username: "" } });
  }

  async "POST /signup"({ body }) {
    const { username, password } = body;
    const { user, error } = await this.#auth.signup(username, password);

    if (user)
      return redirect("/profile", { cookies: { username: user.name } });

    return render("signup.ejs", {
      errorCode: error.description,
      params: { username }
    });
  }
}

The template is just as interesting:

<!-- signup.ejs -->
<h1>Signup!</h1>

<form method="POST" action="/signup">
  <% if (errorCode) { %>
    <p><%= {
      "USER_EXISTS": "User exists"
    }[errorCode] ?? errorCode %></p>
  <% } %>

  <label>
    <strong>Username</strong><br>
    <input type="text" name="username" value="<%= params.username %>">
  </label><br>

  <label>
    <strong>Password</strong><br>
    <input type="password" name="password">
  </label><br>

  <button type="submit">Signup</button>
</form>

This completes a very basic signup feature. However, the /profile page does not yet exist. To complete this blog, I'm going to implement that page!

The Profile Page

I intent to make cookies readable like this:

// src/app.mjs (partial)
export class App {
  "GET /profile"({ cookies }) {
    return render("profile.ejs", { name: cookies.username });
  }
}

<!-- templates/profile.ejs -->
<h1>Hello, <%= name %>!</h1>

This route and template serve as a placeholder while I implement the handling of cookies. Let's first implement that any cookie gets passed to the request handler.

Insecure cookies

Let's first update the AppRequest type definition.

// src/app.mjs (partial)
/**
 * @typedef {Object} AppRequest
 * @property {AppRequestMethod} method
 * @property {string} path
 * @property {AppRequestParams} body
 * @property {Record<string, string|null>} cookies
 */

This has no use in NodeJS whatsoever, but it helps documenting the code and adds type hints to the editor. Now we just have to parse cookies from the cookie header:

// src/main.mjs (partial)
import { parseCookies } from "./cookies.mjs";

// inside http.createServer's callback:
let { status, json, headers, template, variables, cookies, location } =
  await app.handleRequest({
    method: req.method,
    path: url.pathname,
    cookies: parseCookies(req.headers.cookie),
    body,
  });

I was too lazy to implement parseCookies myself so I asked ChatGPT:

// cookies.mjs
export function parseCookies(cookieHeader) {
  if (!cookieHeader) return {};

  return cookieHeader.split(';').reduce((cookies, cookie) => {
    const [key, value] = cookie.split('=').map(part => part.trim());
    if (key) cookies[key] = decodeURIComponent(value || '');
    return cookies;
  }, {});
}

ChatGPT's implementation seems totally reasonable so I'll just go with it.

I have no intention to create a test for src/main.mjs at the moment, so I'll just manually test it (by starting the server and signing up via the browser). After a quick check, it seems to work perfectly fine.

I do intent to test /profile, so let's just break TDD's rules and write a test after writing the implementation.

First, I need to modify my GET and POST methods to handle cookies. I add an overrides param which can be used to include cookies. I use this new param in my new tests:

// app.test.mjs (partial)
describe("GET /profile", () => {
  it("with cookie, returns user", async () => {
    await GET("/profile", { cookies: { username: "foo" } });
    assertRender("profile.ejs", { name: "foo" });
  });

  it("without cookie, redirects to login", async () => {
    await GET("/profile");
    assertRedirect("/login");
  });
});

async function POST(path, body, overrides = {}) {
  res = await app.handleRequest({ method: "POST", path, body, cookies: {}, ...overrides });
}

async function GET(path, overrides = {}) {
  res = await app.handleRequest({ method: "GET", path, cookies: {}, ...overrides });
}

I need to handle the unauthenticated case in my GET /profile method:

// app.mjs
"GET /profile"({ cookies, path }) {
  if (!cookies.username)
    return redirect("/login")
  return render("profile.ejs", { name: cookies.username });
}

There is a huge flaw with this approach: Cookies are basically user input. Any visitor can alter the contents of the cookie, so anyone can now login as any user by just submitting a cookie with the user's username.

To reject tainted cookies, or cookies that are otherwise manipulated, I'll complete this blog by signing the cookies with a secret.

First, I'll generate a secret and save it in my .envrc:

echo "export SECRET='$(openssl rand -base64 48)'" >> .envrc
direnv allow

Now to sign the cookies with this secret using HMAC. I'm going to create two helper functions for this. I'll also have all cookies signed by default, and verify all cookies in every request.

// src/cookies.mjs (partial)
import crypto from "node:crypto";

/**.
 * @param {string} value
 * @param {string} [secret=process.env.SECRET]
 * @returns {string}
 */
function signCookie(value, secret = process.env.SECRET) {
  const hmac = crypto.createHmac("sha256", secret);
  hmac.update(value);
  const signature = hmac.digest("base64url");
  return `${value}.${signature}`;
}

/**
 * @param {string} signedValue
 * @param {string} [secret=process.env.SECRET]
 * @returns {string|null}
 */
function verifyCookie(signedValue, secret = process.env.SECRET) {
  const [value, signature] = signedValue.split(".");
  if (!value || !signature) return null;

  const expectedSignature = crypto
    .createHmac("sha256", secret)
    .update(value)
    .digest("base64url");

  if (crypto.timingSafeEqual(
    Buffer.from(signature), Buffer.from(expectedSignature))) {
    return value;
  }

  return null;
}

export function parseCookies(cookieHeader, secret = process.env.SECRET) {
  if (!cookieHeader) return {};

  return cookieHeader.split(';').reduce((cookies, cookie) => {
    const [key, value] = cookie.split('=').map(part => part.trim());
    if (key)
      cookies[key] = verifyCookie(decodeURIComponent(value || ''), secret);
    return cookies;
  }, {});
}

// src/main.mjs (partial)
if (cookies)
  for (const key in cookies)
    if (cookies.hasOwnProperty(key)) {
      const value = [
        `${key}=${signCookie(cookies[key]) ?? ""}`,
        `Path=/`,
        `HttpOnly`,
        `SameSite=Strict`,
        cookies[key] ? `Max-Age=${YEAR_SECONDS}` : `Max-Age=-1`
      ].join("; ");
      res.setHeader("set-cookie", value);
    }

Since the signing and verifying of cookies is handled in src/main.mjs, none of the request handlers need to change their implementation. The handlers receive verified cookies, and they can set cookies which are always signed automatically. I don't have to change any of my handlers or their tests.

Again, I tested the change manually by checking if the browser gets assigned a signed cookie, and altering the cookie breaks the login. It did, so it works! In a real production-ready application, I would very likely add some tests specifically testing the HTTP request & response handling.

Template language without dependencies

Toby Hinloopen — Mon, 27 Jan 2025 10:30:50 +0000

The Template Language

This blog is a part 2 of my bluebird blog. See part 1

For my twitter clone without dependencies, I've decided to design my route handlers in a way that they'll return variables and a template name. This allows for easy testing, as my tests can just do assertions on the template name and variables rather than inspecting a HTML document.

// request
{
  method: "GET",
  path: "/profile/1234",
  cookies: { "user-id": 54 },
}

// response
{
  status: 200,
  template: "public-profile-show",
  variables: {
    user: {
      id: 54,
      name: "John Doe",
    },
    posts: [
      { id: 55412, message: "Have you seen the new iThing?",
        createdAt: 1699788972 }
    ]
  }
}

In this blog, I'm going to implement this template language.

Designing the template language

The template language I need, needs to output HTML documents with only a set of variables as input. I want a template to be compiled to a JS function. For example, I want Hello <%= name %> to compile to something like this:

({ name }) => `Hello ${escapeHtml(name)}`;

I'll go with a classic <%= %> syntax, because it's incredibly common and well known. Most developers that see this syntax will intuitively know that you can just write regular code in there and the output of the code will be added to the output.

It must support variables and auto-escape HTML entities. Loops, if/else statements and including other templates must also be supported. It would be nice if we could invoke arbitrary functions and do some basic math;

So basically, I want it to be able to execute arbitrary javascript.

The implementation

I guess I just get started writing code and see where I end up. First, a test.

it("simple template", () => {
  const fn = Template.parse("Hello, <%= name %>");
  assert.equal(fn({ name: "world" }), "Hello, world");
});

The simplest implementation I can think of is to use a regular expression. All content outside <%= %> will just be added to the output, and the content between will be executed as JS.

The regular expression used is /(.*?)<%(.*?)%>/sg. This regular expression captures any text until the first <% it finds using (.*?)<%. Then it captures any text until %> using (.*?)%>. The s modifier allows the . (dot) to match newlines. The g modifier allows multiple matches.

Javascript's replace function on string allows executing code for every match while also returning a replacement value, "" in my code. Because every match is replaced by an empty string, only the text after the last %> is returned by the replace-function, which I call the tail.

I use JSON.stringify to create a string literal.

const Template = {
  parse(template) {
    let body = [
      "eval(`var { ${Object.keys(vars).join(', ')} } = vars;`);",
      `out = [];`
    ];
    const tail = template.replace(/(.*?)<%(.*?)%>/sg, (_, content, code) => {
      body.push(`out.push(${JSON.stringify(content)});`);

      if (code.startsWith("="))
        body.push(`out.push(${code.substr(1)});`);

      return "";
    });

    body.push(`out.push(${JSON.stringify(tail)});`);
    body.push(`return out.join("");`);

    return new Function('vars', body.join("\n"))
  }
};

For the template in the test, this function returns a function like this:

function(vars) {
  eval(`var { ${Object.keys(vars).join(', ')} } = vars;`);
  out = [];
  out.push("Hello, ");
  out.push(name);
  out.push("");
  return out.join("");
}

Another notable part of this code is the eval statement. To allow the template to refer to any variable in vars (name in this example), I need to make the properties in vars available as local variables in the function.

There is no easy way to determine the possible variables while compiling, so I generate them at runtime. The only way I know to assign arbitrary local variables at runtime, is to use eval.

var { foo } = { foo: 1 };
// foo = 1
eval('var { bar } = { bar: 2 }');
// bar = 2

Another method is to use the with-statement, which is discouraged. Let's try it anyway.

function(vars) {
  with (vars) {
    out = [];
    out.push("Hello, ");
    out.push(name);
    out.push("");
    return out.join("");
  }
}

The generated function works perfectly. Too bad the feature is discouraged, legacy or deprecated, depending on who you ask. So far my options are evil eval or deprecated with. Ideally, I want to determine the variables used at compile-time, but this requires compiling the Javascript code to determine the variables used.

There is no easy way to get an abstract syntax tree of some piece of Javascript using plain NodeJS.

Now to escape HTML entities, support if/else statements and add some minor fixes.

// Template.parse
let body = [
  "eval(`var { ${Object.keys(vars).join(', ')} } = vars;`);",
  `out = [];`
];
const tail = template.replace(/(.*?)<%(.*?)%>/sg, (_, content, code) => {
  if (content)
    body.push(`out.push(${JSON.stringify(content)});`);

  if (code.startsWith("="))
    body.push(`out.push(escapeHtml(${code.substr(1)}));`);
  else if (code.startsWith("-"))
    body.push(`out.push(${code.substr(1)});`);
  else
    body.push(code);

  return "";
});

if (tail.length > 0) body.push(`out.push(${JSON.stringify(tail)});`);

body.push(`return out.join("");`);
body = body.join("\n");

const fn = new Function('vars', body);
return (vars) => fn({ ...vars, ...Template.locals });

// Template.locals
locals: {
  escapeHtml: (str) => `${str}`.replace(/[<>&"']/g, s =>
    ({ "<": "&lt;", ">": "&gt;", "&": "&amp;", '"': "&quot;", "'": "&#39;" })[s])
}

I also added some more tests.

describe("Template.parse", () => {
  it("simple template", () => {
    const fn = Template.parse("Hello, <%= name %>");
    assert.equal(fn({ name: "world" }), "Hello, world");
  });

  it("math template", () => {
    const fn = Template.parse("1 + 1 = <%= 1 + 1 %>");
    assert.equal(fn({}), "1 + 1 = 2");
  });

  it("function template", () => {
    const fn = Template.parse("Hello <%= foo() %>");
    assert.equal(fn({ foo: () => "world" }), "Hello world");
  });

  it("if-else template", () => {
    const fn = Template.parse(`Answer: <% if (answer) { %>Yes<% } else { %>No<% } %>`);
    assert.deepEqual(
      [fn({ answer: true }), fn({ answer: false })],
      ["Answer: Yes", "Answer: No"]);
  });

  it("multiline template", () => {
    const fn = Template.parse(`
    Answer:
    <% if (answer) { %>
      Yes
    <% } else { %>
      No
    <% } %>
  `);
    assert.deepEqual(
      [delws(fn({ answer: true })), delws(fn({ answer: false }))],
      ["Answer: Yes", "Answer: No"]);
  });

  it("escape html", () => {
    const fn = Template.parse("Hello, <%= name %>");
    assert.equal(fn({ name: "<script> & \" '" }), "Hello, &lt;script&gt; &amp; &quot; &#39;");
  });
});

function delws(str) {
  return str.replace(/^\s+|\s+$/g, "").replace(/\s+/g, " ");
}

To allow includes, I'm going to add a function to parse all template files in a directory. This function will keep a dictionary with template names as keys and their parsed template functions as values.

src/template.mjs

export const Template = {
  /**
   *
   * @param {string} path Directory containing one or more template files
   * @returns {Promise<(template: string, vars: Record<string, any>) => string>}
   */
  async parseDirectory(path) {
    /** @type {Map<string, function>} */
    const templates = new Map();

    const include = (templateName, vars) => {
      const template = templates.get(templateName);
      if (!template) throw new Error(`Template ${path}/${templateName} does not exist, `
        + `templates found: ${Array.from(templates.keys()).join(", ")}`);
      return template({ ...vars, include });
    };

    const readDir = async (prefix) => {
      const innerPath = join(path, prefix);
      const fileNames = await promises.readdir(join(innerPath));

      for (const fileName of fileNames) {
        const templateName = join(prefix, fileName);
        const filePath = join(innerPath, fileName);
        if (fileName.endsWith(".ejs")) {
          const body = await promises.readFile(filePath, { encoding: "utf-8" });
          templates.set(templateName, Template.parse(body, { filePath }));
        } else if ((await promises.stat(filePath)).isDirectory) {
          await readDir(join(prefix, fileName));
        }
      }
    };

    await readDir("");

    return include;
  }
}

test/templates/**.ejs

<!-- layout/header.ejs -->
<h1>Header</h1>

<!-- bar.ejs -->
Hello <%= name %>

<!-- foo.ejs -->
<%- include("layout/header.ejs") %>
<%- include("bar.ejs", { name }) %>

test/template.test.mjs

  describe("parseDirectory", () => {
    it("parses a tree", async () => {
      const fn = await Template.parseDirectory("test/templates");
      assert.equal(delws(fn("foo.ejs", { name: "World!" })),
        "<h1>Header</h1> Hello World!");
    });
  });

Now to integrate this template engine in the main.mjs file to render the template using the .ejs templates.

templates/home.ejs

<h1>Hello, <%= name %>!</h1>

src/main.mjs

const render = await Template.parseDirectory("templates");

const server = http.createServer(async (req, res) => {
  let { status, json, headers, template, variables } = await app.handleRequest();

  // (hidden)

  if (template) {
    res.setHeader("content-type", "text/html");
    res.end(render(template, variables));
    return;
  }
});

curl http://localhost:8000/
# <h1>Hello, World!</h1>

Now we're ready to start writing our application, which will continue in the next blog

Building NodeJS applications without dependencies

Toby Hinloopen — Mon, 01 Jan 2024 12:41:00 +0000

Video to accompany this blog post

https://www.youtube.com/watch?v=QZyX1ZjZqJs

Introduction

I'm going to build a twitter clone using NodeJS, but without any dependencies.

I've written many Node applications using dependencies like express, jest, lodash, webpack, typescript, typeorm and many more. My applications can have tens of these dependencies, and some of these dependencies will also have many dependencies, and I end up with literally 1000s of dependencies.

Updating these projects can be a major pain as dependencies get outdated and need updating, and I got tired of it. I decided to challenge myself to write an application with not a single dependency.

In this blog, I'll start with a simple HTTP server and some tests.

Normally, I use express as a webserver, and I use jest for testing. Additionally, for testing HTTP requests, I like to use supertest and jsdom for checking the HTML responses.

Naturally, without dependencies, I can't use any of these things.

Before I start working on an actual HTTP server, let's consider how to test the application. I can start an HTTP server for testing and use Node's fetch or http client to query the server, but this makes testing slow and dependent on the network stack.

So instead, I'm going to abstract away any HTTP details. I'm going to define a boundary that accepts plain objects representing HTTP requests and returns plain objects representing HTTP responses.

// request
{
  method: "GET",
  path: "/profile/1234",
  cookies: { "user-id": 54 },
}

// response
{
  status: 200,
  headers: { "Content-Type": "text/html" },
  content: "<!DOCTYPE ..."
}

That way I can convert a http request as implemented by NodeJS to the plain object representing the request, pass it to the implementation behind the boundary, and then write the response to the NodeJS http response.

Meanwhile, in the tests, I can just mock the requests by creating object literals representing the requests and do assertions on the response object.

Assertions on content in the HTML document

Checking HTTP responses where an application would return a HTML response could be a bit of a pain. For example, let's assert the page returns an H1 containing "hello world":

expect(response.content).toContain("<h1>Hello world</h1>");

For the following responses, the assertion would fail:

<h1>
  Hello world
</h1>

<h1 class="my-header">Hello world</h1>

<h2 class="h1">Hello world</h2>

Meanwhile, I usually don't care the exact formatting or semantics of the HTML document. I just want to know if some content is or isn't rendered. Easy, you might think:

expect(response.content).toContain("Hello world");

This causes all the tests to pass. Great. But what if the page doesn't render a 'Hello world'? Well...

FAILED: "Hello world" not found in

<main id="content" class="mw-body" role="main"> <header class="mw-body-header vector-page-titlebar"> <nav role="navigation" aria-label="Contents" class="vector-toc-landmark"> <div id="vector-page-titlebar-toc" class="vector-dropdown vector-page-titlebar-toc vector-button-flush-left"> <input type="checkbox" id="vector-page-titlebar-toc-checkbox" role="button" aria-haspopup="true" data-event-name="ui.dropdown-vector-page-titlebar-toc" class="vector-dropdown-checkbox " aria-label="Toggle the table of contents"> <label id="vector-page-titlebar-toc-label" for="vector-page-titlebar-toc-checkbox" class="vector-dropdown-label cdx-button cdx-button--fake-button cdx-button--fake-button--enabled cdx-button--weight-quiet cdx-button--icon-only" aria-hidden="true"><span class="vector-icon mw-ui-icon-listBullet mw-ui-icon-wikimedia-listBullet"></span> <span class="vector-dropdown-label-text">Toggle the table of contents</span> </label> <div class="vector-dropdown-content"> <div id="vector-page-titlebar-toc-unpinned-container" class="vector-unpinned-container"> <div id="vector-toc" class="vector-toc vector-pinnable-element"> <div class="vector-pinnable-header vector-toc-pinnable-header vector-pinnable-header-pinned" data-feature-name="toc-pinned" data-pinnable-element-id="vector-toc"> <h2 class="vector-pinnable-header-label">Contents</h2> <button class="vector-pinnable-header-toggle-button vector-pinnable-header-pin-button" data-event-name="pinnable-header.vector-toc.pin">move to sidebar</button> <button class="vector-pinnable-header-toggle-button vector-pinnable-header-unpin-button" data-event-name="pinnable-header.vector-toc.unpin">hide</button> </div> <ul class="vector-toc-contents" id="mw-panel-toc-list"> <li id="toc-mw-content-text" class="vector-toc-list-item vector-toc-level-1 vector-toc-level-1-active vector-toc-list-item-active"> <a href="#" class="vector-toc-link"> <div class="vector-toc-text">(Top)</div> </a> </li> <li id="toc-History" class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded"> <a class="vector-toc-link" href="#History"> <div class="vector-toc-text"> <span class="vector-toc-numb">1</span>History</div> </a> <button aria-controls="toc-History-sublist" class="cdx-button cdx-button--weight-quiet cdx-button--icon-only vector-toc-toggle" aria-expanded="true"> <span class="vector-icon vector-icon--x-small mw-ui-icon-wikimedia-expand"></span> <span>Toggle History subsection</span> </button> <ul id="toc-History-sublist" class="vector-toc-list"> <li id="toc-Origin" class="vector-toc-list-item vector-toc-level-2"> <a class="vector-toc-link" href="#Origin"> <div class="vector-toc-text"> <span class="vector-toc-numb">1.1</span>Origin</div> </a> <ul id="toc-Origin-sublist" class="vector-toc-list"> <li id="toc-Simultaneous_references" class="vector-toc-list-item vector-toc-level-3"> <a class="vector-toc-link" href="#Simultaneous_references"> <div class="vector-toc-text"> <span class="vector-toc-numb">1.1.1</span>Simultaneous references</div> </a> <ul id="toc-Simultaneous_references-sublist" class="vector-toc-list"> </ul> </li> </ul> </li> <li id="toc-Growth_in_2008" class="vector-toc-list-item vector-toc-level-2"> <a class="vector-toc-link" href="#Growth_in_2008"> <div class="vector-toc-text"> <span class="vector-toc-numb">1.2</span>Growth in 2008</div> </a> <ul id="toc-Growth_in_2008-sublist" class="vector-toc-list"> </ul> </li> <li id="toc-Later_usage" class="vector-toc-list-item vector-toc-level-2"> <a class="vector-toc-link" href="#Later_usage"> <div class="vector-toc-text"> <span class="vector-toc-numb">1.3</span>Later usage</div> </a> <ul id="toc-Later_usage-sublist" class="vector-toc-list"> </ul> </li> </ul> </li> <li id="toc-Reaction" class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded"> <a class="vector-toc-link" href="#Reaction"> <div class="vector-toc-text"> <span class="vector-toc-numb">2</span>Reaction</div> </a> <ul id="toc-Reaction-sublist" class="vector-toc-list"> </ul> </li> <li id="toc-See_also" class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded"> <a class="vector-toc-link" href="#See_also"> <div class="vector-toc-text"> <span class="vector-toc-numb">3</span>See also</div> </a> <ul id="toc-See_also-sublist" class="vector-toc-list"> </ul> </li> <li id="toc-References" class="vector-toc-list-item vector-toc-level-1 vector-toc-list-item-expanded"> <a class="vector-toc-link" href="#References"> <div class="vector-toc-text"> <span class="vector-toc-numb">4</span>References</div> </a>

There's no fun in debugging that response.

There are other options.

I can make the server only respond with JSON and handle the rendering on the client using a React app. That way, I can just test the JSON responses, which are much easier to debug and reason about.

However, I don't want to introduce a whole new app just to make testing the server easier. Single Page Applications can quickly become very complex and are prone to errors. In my experience, Server-Side Rendering applications are easier to build and maintain, less prone to errors and can be just as responsive.

In server-side generated webapplications, you usually use templates to convert data to html. So what if I make the boundary just return a template name and all data required to render the page using that template?

// request
{
  method: "GET",
  path: "/profile/1234",
  cookies: { "user-id": 54 },
}

// response
{
  status: 200,
  template: "public-profile-show",
  variables: {
    user: {
      id: 54,
      name: "John Doe",
    },
    posts: [
      { id: 55412, message: "Have you seen the new iThing?",
        createdAt: 1699788972 }
    ]
  }
}

Then in my test, I can just assert the template name and data, which is much easier to reason about and results in much more accurate failure messages.

As long as I don't too much crazy stuff in the templates, I can generally assume that if the data and template returned are correct and the template renders without error, the page should pass the test.

This method of testing isn't perfect. It's possible for a template to just render nothing at all, and the tests would still pass. In my experience, there's just no way to test whether a HTML document actually renders some content. Even if the content would be present in the HTML document, it could be rendered incorrectly or not at all using CSS.

expect(response.content).toContain("Hello world");

<!-- PASS! -->
<h1 style="display: none;">Hello world!</h1>

<!-- LGTM! -->
<h1 style="position: absolute; top: -1000;">Hello world!</h1>

<!-- SURE! -->
<h1 style="color: transparent;">Hello world!</h1>

<!-- PERFECT! -->
<h1 style="font-size: 0px;">Hello world!</h1>

<!-- YEAH! -->
<h1 id="foo">Hello world!</h1>
<script>document.querySelector("#foo").innerHTML = "";</script>

Therefore, in my opinion, templates can only be tested by a pair of human eyes, rendering them using a real browser. Ideally, this manual testing can be done in multiple browsers very quickly.

To test the templates, I'll could render the page in every test and save the output. If there's a fatal error rendering the page, the test will fail. If the rendering succeeds, I can manually check the HTML document by opening it in my browser. Maybe I'll create an overview page with all HTML documents in iframes to quickly review multiple pages.

Implementation

After implementing some example pages and building an .ejs-like template language, the code and tests look like this:

app.mjs

class App {
  async handleRequest(req) {
    const fn = this[`${req.method} ${req.path}`];
    if (!fn)
      return render("not-found.ejs", req, { status: 404 });
    return fn.call(this, req);
  };

  async "GET /login"() {
    return render("login.ejs", { errorCode: null, params: { username: "" } });
  }

  async "POST /login"({ body }) {
    const { username, password } = body;
    const { user, error } = await this.#auth.login(username, password);

    if (user)
      return redirect("/profile", { cookies: { username: user.name } });

    return render("login.ejs", {
      errorCode: error.description,
      params: { username }
    });
  }
}

The render and redirect functions just return response objects.

function render(template, variables = {}, overrides = {}) {
  return { template, variables, status: 200, ...overrides };
}

function redirect(location, overrides = {}) {
  return { location, status: 302, ...overrides };
}

They're very simple one-liners meant to improve readability and set some defaults.

app.test.mjs

describe("GET /login", () => {
  it("renders login template", async () => {
    await GET("/login");
    assertRender("login.ejs", { params: { username: "" }, errorCode: null });
  });
});

describe("POST /login", () => {
  it("with unknown username param, returns user not found error", async () => {
    await POST("/login", { username: "foo", password: "bar" });

    assertRender("login.ejs", {
      params: { username: "foo", },
      errorCode: "USER_NOT_FOUND",
    });
  });

  it("with valid params, sets cookie and redirects to profile", async () => {
    await auth.signup("foo", "bar");
    await POST("/login", { username: "foo", password: "bar" });
    assertRedirect("/profile", { cookies: { username: "foo" } });
  });
});

Testing utilities

In a beforeEach function, the app is initialized. In my testing file, I have POST and GET functions available. Even though they're oneliners, they improve readability by reducing noise.

async function POST(path, body) {
  res = await app.handleRequest({ method: "POST", path, body });
}

async function GET(path) {
  res = await app.handleRequest({ method: "GET", path });
}

assertRender and assertRedirect check the response in res, assigned by GET or POST. assertRender also renders the template, to check for errors in the template.

function assertRender(template, variables = {}, overrides = {}) {
  assert.deepStrictEqual(res,
    { status: 200, template, variables, ...overrides });
  if (res.template)
    renderTemplate(res.template, res.variables);
}

function assertRedirect(location, overrides = {}) {
  assert.deepStrictEqual(res, { status: 302, location, ...overrides });
}

renderTemplate is created by my template engine. It is a function that accepts a template name and variables and returns HTML as a string.

This template engine renders an EJS-like template like this:

<h1>Login!</h1>

<form method="POST" action="/login">
  <% if (errorCode) { %>
    <p><%= {
      "USER_NOT_FOUND": "User not found",
      "INVALID_PASSWORD": "Invalid password"
    }[errorCode] ?? errorCode %></p>
  <% } %>

  <label>
    <strong>Username</strong><br>
    <input type="text" name="username" value="<%= params.username %>">
  </label><br>

  <label>
    <strong>Password</strong><br>
    <input type="password" name="password">
  </label><br>

  <button type="submit">Login</button>
</form>

To some nice HTML.

In the next blog, we'll do a deep dive into this template language.

As a bonus, here is the code responsible for actually processing HTTP requests and generating responses.

const server = http.createServer(async (req, res) => {
  try {
    const url = new URL(req.url, `http://${req.headers.host}`);

    const body = await new Promise((resolve) => {
      if (req.method != "POST") {
        resolve(null);
        return;
      }

      const bodyData = [];

      req.on("data", (chunk) => bodyData.push(chunk));
      req.on("end", () => { resolve(querystring.parse(bodyData.join(""))); });
    });

    let { status, json, headers, template, variables, cookies, location } = await app.handleRequest({
      method: req.method,
      path: url.pathname,
      body,
    });

    if (status) res.statusCode = status;

    if (headers)
      for (const key in headers)
        if (headers.hasOwnProperty(key))
          res.setHeader(key, value);

    if (cookies)
      for (const key in cookies)
        if (cookies.hasOwnProperty(key)) {
          const value = [
            `${key}=${cookies[key] ?? ""}`,
            `Path=/`,
            `HttpOnly`,
            `SameSite=Strict`,
            cookies[key] ? `Max-Age=${YEAR_SECONDS}` : `Max-Age=-1`
          ].join("; ");
          res.setHeader("set-cookie", value);
        }

    if (location) {
      res.setHeader("Location", location);
      res.end();
      return;
    } else if (json) {
      res.setHeader("content-type", "application/json");
      res.end(JSON.stringify(json));
      return;
    } else if (template) {
      res.setHeader("content-type", "text/html");
      res.end(render(template, variables));
      return;
    }

    throw new Error(`Request handler had no response`);
  } catch (error) {
    res.statusCode = 500;
    res.setHeader("content-type", "text/plain");
    res.end(`SOMETHING WENT WRONG\n${error.stack}`);
  }
});

Original: https://www.bonaroo.nl/2023/12/12/building-node-applications-without-dependencies.html

Circuit Simulator: Compiling a bitmap

Toby Hinloopen — Thu, 02 Nov 2023 12:55:41 +0000

Introduction

I want to build a game where you can draw a circuit and have it be running while you're drawing. The circuit should control some thing in a virtual environment, like a tiny car or traffic lights.

The circuit "language", as designed by realhet, features only 3 building blocks: The wire (any pixel where the R, G or B is >=244), a wire crossing and a not gate.

My first goal is to build the bitmap compiler, where "the game" can only load an image of this CPU made by realhet (which I've slightly modified). It will compile the image and run it as fast as possible.

In this blog, I'll share the development proces of building the bitmap compiler by sharing snippets of code.

About me

I usually write web-applications using languages like Javascript, Ruby and Elixir. I'm not a C programmer. This C code may be unconventional or otherwise wrong.

This blog includes incomplete code samples, including a mix of code from header files (.h) and source files (.c).

Reading an image

To start with, I need to read the image from disk to memory. I'll be using libpng to read the bitmap file to a custom image_data_t-struct.

typedef struct {
  uint8_t r;
  uint8_t g;
  uint8_t b;
} rgb_t;

typedef struct {
  uint32_t width;
  uint32_t height;
  rgb_t *data;
} image_data_t;

For those not familiar with C, a struct is simply an object with predefined fields and no methods. uint8_t and uint32_t are unsigned (u) integers (int) of 8-bits or 32-bits in size. These integer types are defined in stdint.h

With the help of Google and ChatGPT, I've scrambled together an obsene amount of code for reading an image. Every time I encounter code like this, it's a reminder how much is actually happening under the hood for a simple task like reading an image. Because this blog isn't about reading PNG files, I've shared the code here.

I then write a test using a very tiny testing header found here.

image_data_t image_data;
image_data_initialize(&image_data);

image_data_png_read(&image_data, "./assets/fixed-cpu.png");

mu_assert("error, image_data.width != 1024", image_data.width == 1024);
mu_assert("error, image_data.height != 1024", image_data.height == 1024);

Success!

Rules of the bitmap language

Any pixel that has an R, G or B value of >=224 is considered a conductive pixel. Any other pixel is considered non-conductive. Conductive and non-conductive pixels can have any color, it only matters whether they have an R, G or B value of >=224.

For the circuit simulation, a conductive pixel can be powered or unpowered, and it's powered-state can change at most once per tick. Conductive pixels that touch sides share their "powered"-state. This allows the player to draw "wires". Powering the wire at any point powers all the pixels in the whole line at once. A line is powered if it has one or more sources of power.

The wire crossing allows crossing 2 independent wires without connecting them. A 3x3 grid containing 4 conductive pixels in a +-shape is considered a wire crossing. The lines connecting to the upper and lower pixels share state, and the lines connecting to the left and right pixels share state, but otherwise their state is kept separate, because they don't touch.

A NOT-gate acts as a power source, powering the output wire if the input wire is unpowered. Like the wire crossing, the NOT-gate is a 3x3 grid, but with a C-shaped input and a single pixel for the output.

The state of the input is independent from the output, but the output will become powered if the input is unpowered. Once the input wire is powered, the NOT-gate will no longer act as a power source and the output wire might become unpowered if there's no other power source for that wire.

For example, two not-gates with an output connected to a single wire will power the output wire if the input of any not-gate is unpowered. Only if both not-gates are powered, the output wire will be unpowered, basically making a NAND gate. You can connect the outwire wire to another not-gate to make an AND-gate, where the output of that new gate is powered when both input not-gates are powered.

Compiling the image: Finding wires, NOT-gates and crossings

The first step of compiling the image will be finding wires, not-gates and crossings. I'll store these in a struct named nodemap_t. This struct will be like image_data_t, but instead of storing a color per pixel, it will store whether a pixel is "nothing", a conductive wire, the center of a crossing, or the center of some NOT-gate.

For faster processing, I also keep a list of pixel indeces for wire crossings, and pixel indeces for the inputs and outputs for not gates.

typedef struct {
  uint32_t size;
  uint32_t capacity;
  uint32_t *data;
} uint32_list_t;

void uint32_list_add(uint32_list_t *uint32_list, uint32_t value);

typedef struct {
  uint32_t width;
  uint32_t height;

  // for every not-gate, there will be 2 indeces here, one for the in-index and
  // one for the out-index.
  uint32_list_t not_gate_index_pairs;

  // pixel-data with 1 element (byte) per pixel. Value is one of
  // the NODE_* constants.
  uint8_t *data;
} nodemap_t;

const uint8_t NODE_NONE = 0;
const uint8_t NODE_WIRE = 1;
const uint8_t NODE_CROSSING = 2;
const uint8_t NODE_NOT_EAST = 3;
const uint8_t NODE_NOT_WEST = 4;
const uint8_t NODE_NOT_NORTH = 5;
const uint8_t NODE_NOT_SOUTH = 6;

// Resize nodemap to have a certain width and height, and reallocate to fit
// that amount of data.
void nodemap_resize(nodemap_t *nodemap, uint32_t width, uint32_t height);

void nodemap_put(nodemap_t *nodemap, uint32_t x, uint32_t y, uint8_t value);

// Parse wire-nodes and assign not-gates and crossings, populating
// not_gate_index_pairs.
void nodemap_parse(nodemap_t *nodemap);

bool rgb_is_high(rgb_t rgb);

To start compiling the image, I first scan the image data for conductive pixels and write the result to nodemap_t.data using nodemap_put.

const uint8_t RGB_HIGH = 224;

inline bool rgb_is_high(rgb_t rgb) {
  return rgb.r >= RGB_HIGH || rgb.g >= RGB_HIGH || rgb.b >= RGB_HIGH;
}

// Get a pointer to a specific pixel in the image data
rgb_t *image_data_at(image_data_t *image_data, uint32_t x, uint32_t y);

void image_data_to_nodemap(image_data_t *image_data, nodemap_t *nodemap) {
  nodemap_resize(nodemap, image_data->width, image_data->height);
  for (uint32_t y = 0; y < image_data->height; y++) {
    for (uint32_t x = 0; x < image_data->width; x++) {
      nodemap_put(nodemap, x, y, rgb_is_high(*image_data_at(image_data, x, y)));
    }
  }
  nodemap_parse(nodemap);
}

I scan the nodemap wire-data to find not-gates and crossings. A NOT-gate will have the same 4 conductive pixels around the center pixel as the wire crossing, so I can start looking at any location that has 4 conductive pixels around a non-conductive center pixel.

// For each pixel (except all pixels at the edge of the image). EG:
//    for (y = 1; y < height - 1; y++)
//      for (x = 1; x < width - 1; x++)
uint8_t *self = nodemap->data + index;
if (is_wire(self)) continue;

uint8_t *yn = self - width;
uint8_t *yp = self + width;

if (!is_wire(yn) || !is_wire(yp) || !is_wire(self - 1) ||
    !is_wire(self + 1))
  continue;

// these 4 variables are named based on whether they're relative to the
// YX of self or index: n = -1, p = +1. So pn is a y+1, x-1.
bool nn = is_wire(yn - 1);
bool np = is_wire(yn + 1);
bool pn = is_wire(yp - 1);
bool pp = is_wire(yp + 1);
if (nn && np && !pn && !pp) {
  // Write nodemap->data[index] with the node-type (not-gate pointing south)
  *self = NODE_NOT_SOUTH;

  // Add not-gate input and output index to not_gate_index_pairs
  uint32_list_add2(not_gate_index_pairs, index - width, index + width);
} else if (pn && pp && !nn && !np) {
  *self = NODE_NOT_NORTH;
  uint32_list_add2(not_gate_index_pairs, index + width, index - width);
} else if (nn && pn && !np && !pp) {
  *self = NODE_NOT_EAST;
  uint32_list_add2(not_gate_index_pairs, index - 1, index + 1);
} else if (np && pp && !nn && !pn) {
  *self = NODE_NOT_WEST;
  uint32_list_add2(not_gate_index_pairs, index + 1, index - 1);
} else if (!nn && !np && !pn && !pp) {
  *self = NODE_CROSSING;
}

Next comes the hard part: Determining which pixels belong to the same wire.

Mapping pixels to wire IDs.

Next, I need to combine conductive pixels that share the same wire somehow. The plan is to assign each wire a unique ID, and give all pixels making a wire that same ID.

Like before, I'm storing the result in an image-like struct.

typedef struct {
  uint32_t width;
  uint32_t height;
  uint32_t max_id;
  uint32_t *data;
} wiremap_t;

In this struct, every pixel stores the ID of the wire. If the ID is 0, it is part of no wire. max_id will store the highest wire-id. I'm going to loop through every pixel, and for every unassigned conductive pixel I find, I'm going to assign it a new ID. Then, all adjecent conductive pixels will be given the same ID.

I'm going to use a flood fill algorithm to find all adjecent pixels. I'm going to write a very simple algorithm where I basically walk through every adjecent pixel to see if the pixel is conductive. If I encounter a crossing, I jump it by advancing an extra pixel (2 pixels)

For testing, I'm going to use this slightly more complicated test image. For clarification, the expected result is shown as a colored image where each color represents a wire id.

Let's write the flood fill!

static void _fill(const nodemap_t *nodemap, wiremap_t *wiremap,
                  uint32_list_t *stack, uint32_t x, uint32_t y,
                  uint32_t next_id) {
  uint32_t width = nodemap->width;
  uint32_t height = nodemap->height;
  uint32_t width_minus_one = width - 1;
  uint32_t height_minus_one = height - 1;

  uint32_list_clear(stack);
  uint32_list_add2(stack, x, y);
  while (stack->size > 0) {
    y = uint32_list_pop(stack);
    x = uint32_list_pop(stack);

    uint32_t index = y * width + x;
    if (nodemap->data[index] != NODE_WIRE || wiremap->data[index]) continue;

    wiremap->data[index] = next_id;

    if (x < width_minus_one) {
      uint8_t node = nodemap->data[index + 1];
      if (node == NODE_WIRE)
        uint32_list_add2(stack, x + 1, y);
      else if (node == NODE_CROSSING)
        uint32_list_add2(stack, x + 2, y);
    }

    if (x > 0) {
      uint8_t node = nodemap->data[index - 1];
      if (node == NODE_WIRE)
        uint32_list_add2(stack, x - 1, y);
      else if (node == NODE_CROSSING)
        uint32_list_add2(stack, x - 2, y);
    }

    if (y < height_minus_one) {
      uint8_t node = nodemap->data[index + width];
      if (node == NODE_WIRE)
        uint32_list_add2(stack, x, y + 1);
      else if (node == NODE_CROSSING)
        uint32_list_add2(stack, x, y + 2);
    }

    if (y > 0) {
      uint8_t node = nodemap->data[index - width];
      if (node == NODE_WIRE)
        uint32_list_add2(stack, x, y - 1);
      else if (node == NODE_CROSSING)
        uint32_list_add2(stack, x, y - 2);
    }
  }
}

_fill is called for every XY coordinate that has a conductive pixel and is not yet assigned an ID. next_id is an unused ID to be assigned to the current pixel, and any adjecent conductive pixel.

stack is just an array where I keep track of pixels that need to be visited. I pass it as a param to reuse any allocated memory. Clearing the stack using uint32_list_clear doesn't free memory, it just sets the size to 0.

I'm sure it isn't the most efficient implementation, but it works!

Now the wires have assigned IDs, I can combine the wire IDs with the NOT-gates.

Creating NOT-gates list

For the simulation, the state of a wire is determined by NOT-gates it is connected to. A wire is powered if there is any NOT-gate with an unpowered input.

Therefore, the only thing the simulation needs to be aware of, is the list of NOT-gates attached to a wire, and the powered/unpowered state of a wire.

The powered-state of a wire can simply be an array of booleans. To advance the simulation, I need to iterate through all NOT-gates and change the output of each NOT-gate based on the powered-state of the input wire.

To start with implementing this, I need a list of NOT-gates with the ID of their input and output wires. I'm also going to store the position of each NOT-gate output and input pixel as the index.

typedef struct {
  uint32_t in_id;
  uint32_t out_id;
  uint32_t in_index;
  uint32_t out_index;
} not_gate_t;

typedef struct {
  uint32_t size;
  uint32_t capacity;
  not_gate_t *not_gates;
} not_gate_list_t;

void not_gate_list_initialize(not_gate_list_t *not_gate_list);
void not_gate_list_clear(not_gate_list_t *not_gate_list);
void not_gate_list_ensure_capacity(not_gate_list_t *not_gate_list,
                                   uint32_t required_capacity);
void not_gate_list_add(not_gate_list_t *not_gate_list, not_gate_t not_gate);
void not_gate_list_free(not_gate_list_t *not_gate_list);

To populate the not_gate_list, I need the list of NOT-gates and the IDs of wires per pixel. The list of NOT-gates is stored in nodemap's not_gate_index_pairs. The IDs of wires is stored in the wiremap.

I'm going to loop through not_gate_index_pairs and create a not_gate_t for each pair.

void nodemap_wiremap_to_not_gate_list(const nodemap_t *nodemap,
                                      const wiremap_t *wiremap,
                                      not_gate_list_t *not_gate_list) {
  const uint32_t size = nodemap->not_gate_index_pairs.size / 2;
  not_gate_list_clear(not_gate_list);
  not_gate_list_ensure_capacity(not_gate_list, size);
  const uint32_t *pair_data = nodemap->not_gate_index_pairs.data;
  for (uint32_t i = 0; i < size; i++) {
    not_gate_t not_gate;
    not_gate.in_index = *(pair_data++);
    not_gate.in_id = wiremap->data[not_gate.in_index];
    not_gate.out_index = *(pair_data++);
    not_gate.out_id = wiremap->data[not_gate.out_index];
    not_gate_list_add(not_gate_list, not_gate);
  }
}

This list is the last thing I need to finish the compile proces. Now I can use the not_gate_list to create a program.

Creating & Running a program

Loading the program involves clearing any existing program, loading the NOT-gates, randomizing the order of NOT-gates, and making sure the state is initialized (all zeroed)

For efficiency, the not-gates input IDs and output IDs are copied to their own arrays.

typedef uint32_t gate_io_t;
typedef uint8_t gate_state_t;
typedef enum fixed_state {
  FIXED_NONE = 0,
  FIXED_LOW = 1,
  FIXED_HIGH = 2
} fixed_state_t;

typedef struct program_state {
  uint32_t id_state_size;
  uint32_t id_state_capacity;
  bool *id_state;

  // The fixed-state status per ID. A fixed state forces the value to either
  // FALSE or TRUE of a state at the end of every iteration. If the fixed-state
  // is 0, no state is enforced.
  fixed_state_t *fixed_state_map;

  // Number of high src_gates per ID. If >0, the id_state should be true.
  // fixed_state can override the final value.
  uint32_t *high_count;

  // NotGate sized arrays
  uint32_t not_gate_size;
  uint32_t not_gate_capacity;

  not_gate_t *not_gates;
  gate_state_t *gate_state;
  float *slow_state;
  gate_io_t *in_ids;
  gate_io_t *out_ids;

  // A set of all state-ids that are affected by fixed_state. Basically, this
  // set includes all non-zero IDs in fixed_state_map.
  uint32_hash_set_t fixed_state_ids;

  // Seed for random. This value can change any time as random values are
  // created.
  unsigned int random_seed;
} program_state_t;

void program_state_next(program_state_t *program_state) {
  const uint32_t not_gate_size = program_state->not_gate_size;
  for (uint32_t i = 0; i < not_gate_size; i++)
    _update_gate_state(program_state, i, !_get_gate_input(program_state, i));
  _force_fixed_state(program_state);
}

Running the program involves checking the state of the input wire of each gate and changing the output if the input is changed. The input is checked using _get_gate_input, and the output is updated using _update_gate_state.

static gate_state_t _get_gate_input(const program_state_t *program_state,
                                    uint32_t index) {
  uint32_t in_id = program_state->in_ids[index];
  uint8_t fixed_state = program_state->fixed_state_map[in_id];
  if (fixed_state > 0) return fixed_state == FIXED_HIGH;
  return program_state->high_count[in_id] > 0;
}

_get_gate_input is a function that should return the current state of the input wire for a gate.

If the state of the input wire is fixed by fixed_state, that state is used for the input wire. Otherwise, high_count is used to determine the number of gates with "high" (powered) outputs per wire. If at least one gate powers the output wire, the wire is powered and get_gate_input returns true.

The target state of a gate will be the opposite of the input; If the gate input was false, the target_state will be true, and vice-versa. _update_gate_state is called with the inverse of _get_gate_input:

static void _update_gate_state(program_state_t *program_state, uint32_t index,
                               gate_state_t target_state) {
  gate_state_t *gate_state = program_state->gate_state;
  float *slow_state = program_state->slow_state;

  if (target_state) {
    if (gate_state[index]) return;
    if (slow_state[index] >= 0.5f) {
      _set_gate_state(program_state, index, true);
      return;
    }
    slow_state[index] +=
        random_range_f(&program_state->random_seed, 0.5f, 1.0f);
    if (slow_state[index] >= 1.0f) _set_gate_state(program_state, index, true);
  } else {
    if (!gate_state[index]) return;
    if (slow_state[index] <= 0.5f) {
      _set_gate_state(program_state, index, false);
      return;
    }
    slow_state[index] -=
        random_range_f(&program_state->random_seed, 0.5f, 1.0f);
    if (slow_state[index] <= 0.0f) _set_gate_state(program_state, index, false);
  }
}

_update_gate_state checks if the target state is different from the current state, and will change slow_state by some random amount. If the slow_state exceeds the 0..1 range, the state-change will be confirmed using _set_gate_state.

The randomness allows the system to end up in a stable state, as the gates will not update simultaneously.

You can play around with (an outdated copy) of the simulation here: https://charperbonaroo.github.io/bls/#0

Originally published at: https://www.bonaroo.nl/2023/10/30/microengineer-compiler.html

Hello, Developer;

Toby Hinloopen — Sat, 29 Aug 2020 18:00:31 +0000

Welcome. If you're reading this, you might be hired by Charper Bonaroo. You're here to help us help our clients. To help you help us help our clients, here's a short article to describe some common instructions regarding common problems.

Before you start: Accessing our Slack

Communication within Charper Bonaroo is done within our slack. We don't have fixed locations - some work from home, some work at our office. To keep our communication independent from location, we use slack for communications.

[ ] Ensure you have access to our Slack
[ ] You're happy about the terms and agreements on the job

About work times, locations & schedules

We don't work on fixed schedules, but we're usually working on weekdays somewhere between 9:00 to 18:00. (Amsterdam Time, GMT +01:00 winter or +02:00 in summer). We usually don't mind if you work other hours.

We're mostly interested in getting things done. We don't mind if you're doing it at night, weekends or any other time. We do like to know what your usual schedule is, so we know when we can expect questions and when we need to ensure we've assigned some issues to you.

Getting started: Running the project

Usually you're here to work on a specific project. Ensure you've got access to the project's source, and see if you can get the project running. You should be assigned some issues soon. Before you start working on specific issues, see if you can get the project running.

The README.md on the project should always be sufficient to get started, but mistakes do slip in. If the readme wasn't sufficient to get started or you encounter any other issue, please let us know.

[ ] Get access to the project
[ ] Check the readme
[ ] Ensure you're able to locally start the project
[ ] Ensure the tests are all running & green

If you need any help, feel free to ask questions.

Understanding the project

Our projects aren't usually that well documented. We like to ensure the intent for a project is clear from code and context as far as reasonabily possible. Sometimes we just didn't have the time to properly document our project (note: we usually request help when we're low on time, not when we've time to spare).

To keep communication as low as possible, we'll assume you'll take some time to attempt to understand the application based on code, GUIs and readme, and to ask questions whenever your understanding is missing.

It might also be useful to understand who the client is. The software is here to help our client solve some problem. Often the client isn't the user of the project, but their role is still important for the project.

[ ] Get a rough idea what the users of the project are
[ ] Get a rough idea who the client is

You don't have any issue to work on

Please let us know.

You've been assigned an issue

Great. See if you understand the issue. Issues might be greatly detailed, or extremely short. Maybe the issue is just a dump with a quote from the client. The issue might be incomplete or unclear.

Sometimes the issue just describe a problem that needs to be fixed without a proposed solution. Sometimes the issue proposes a solution without describing the problem. Idealy the issue does both.

[ ] Ensure you understand the problem that needs to be solved
[ ] Ensure you see a potential solution

Please let the author now if you have any questions. If the issue is missing a suggested solution, it would be nice if you could comment your approach to solve the issue before you start working on it.

Document your progress

The issue is the central authority of information about that problem. If you found any required information about the issue outside the issue, please add the information to the issue. If you asked a question and got an answer, copy paste it into the issue.

For small issues, it's also fine to not document anything but just to create a solution.

Tracking time

Please track time using the method discussed with you.

Ensure that for each block of time, we can determine:

[ ] What project you've worked on
[ ] What issue you've worked on (EG #55 is sufficient)
[ ] A small summary of the work done (EG creating a users index)

If you've not worked on a specific issue, it's especially important to mention what you've done otherwise.

We use time tracking to keep track of pay-per-hour billing, but also to keep track of progress and time spent on specific issues.

At the end of each block of time, we want to see relevant code commits, if any.

Repository Guidelines

We're using common guidelines like this one:

https://www.gnu.org/software/gnuastro/manual/html_node/Commit-guidelines.html

Unless otherwise specified.

Code starts at an issue. From an issue, a potential solution is created. This potential solution is presented in a branch, forked from the latest master branch.

After your first commit, create a Pull Request for your branch. Your branch name should include the issue ID and a short summary, eg 55-create-user-index. Your git commit should have a short message that finishes the sentence "This commit will...", eg git commit -m "add user index".

Mark your new PR as a work in progress if your solution isn't done yet. In this PR, we can keep track of progress. Keep the WIP until you're ready to propose your solution.

We'll then review the PR. If it passes review, it gets merged. If it doesn't, please fix PR based on the discussions/comments.

Feel free to work on other issues if you're waiting for a code review.

Github

Create a pull request after your first commit, and add "WIP" to the title. Remove "WIP" when you're done. Some github repos use other conventions, like a WIP label or WIP-bot. We'll let you know.

Gitlab

If you're working on gitlab, use the "Create Merge Request" feature to create both a branch and merge request. Use "WIP: " in the title to mark a PR (called a Merge Request in Gitlab) as a work in progress. When you're done, remove "WIP: " from the title.

Contributing to open source project JS-DOM

Toby Hinloopen — Mon, 06 Jul 2020 18:27:43 +0000

Contributing to open source project JS-DOM

We use JSDOM for testing clientside applications in NodeJS. JSDOM lowers the complexity of writing tests for clientside code by omitting the browser and replacing it with a fake one: JSDOM.

However, there's one JSDOM dependency that concerned me a bit: request, with request-promise-native. Request has been deprecated, and request-promise-native does nasty things using stealthy-require. I'm not sure why anyone would use stealthy-require, but I trust there's a good reason for to use it.

request has already been a discussed to be replaced with something else in an issue #2792: Replace request with something better. Since there were no pull requests for the issue, I decided to see if I can help out and fix it myself. In this blog post, I'll describe my process.

Contributing to foreign projects

Changing code inside a foreign project is commonly quite the challenge. There is usually a lot of code and a lot of things to consider, many you just don't know about. This is why tests are really important.

For a complex project like JSDOM, without a comprehensive suite of tests, there is no way to be sure your changes might break something. Even with perfect code coverage, there is still no guarantee your changes don't break something, but you can still be pretty sure your code at least runs in the cases presented by the tests.

Fork & Clone.

I forked & cloned the repository, and created a new branch to start my experimental replacement.

git clone git@github.com:tobyhinloopen/jsdom.git
cd jsdom
git checkout -b 2792-replace-request-with-node-fetch

Now let's see if there are some tests I can run.

$ npm i
npm ERR! code EUNSUPPORTEDPROTOCOL
npm ERR! Unsupported URL Type "link:": link:./scripts/eslint-plugin

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/hinloopen/.npm/_logs/2020-05-10T15_02_02_981Z-debug.log

Uh... ok. Let's consult the README first. There is a README.md and Contributing.md. Both might be relevant.

In Contributing.md, it's already mentioned they're using yarn. Eager to start, I ignore the rest and use yarn install to install the dependencies.

Let's run some tests without consulting the readme or contributing guidelines and see if they run.

$ yarn test
# ...
1) "before all" hook: $mochaNoSugar in "{root}"
2) "after all" hook: $mochaNoSugar in "{root}"

0 passing (16ms)
2 failing

1) "before all" hook: $mochaNoSugar in "{root}":
    Error: Host entries not present for web platform tests. See https://github.com/web-platform-tests/wpt#running-the-tests
    at /Users/hinloopen/Projects/Github/jsdom/test/web-platform-tests/start-wpt-server.js:62:13
    at async /Users/hinloopen/Projects/Github/jsdom/test/web-platform-tests/run-tuwpts.js:25:32
# ...

Looks like the tests require more setup. Let's consult the readme again. The readme refers to The web-platform-tests Project. It looks like this project allows you to run a test suite (that you have to provide yourself in some way) inside a set of browsers. You have to clone the repo and run the code.

I'll just assume this web-platform-tests project starts some kind of server and you have to open a page in a real browser. Since we're testing a fake browser (JSDOM), I also assume JSDOM somehow registers to WPT as a real browser, so it can the same tests in JSDOM, as if JSDOM was a browser. Let's try it out.

$ git clone https://github.com/web-platform-tests/wpt.git
# ...
$ cd wpt
$ ./wpt serve
# ...
CRITICAL:web-platform-tests:Failed to start HTTP server on port 59514; is something already using that port?
CRITICAL:web-platform-tests:Please ensure all the necessary WPT subdomains are mapped to a loopback device in /etc/hosts.

Right. RTFM. I added the setup instructions to .envrc in the WPT project folder.

$ nano .envrc
python -m ensurepip --user
export PATH="$PATH:$HOME/Library/Python/2.7/bin"
pip install --user virtualenv

Additionally:

To get the tests running, you need to set up the test domains in your hosts file. (source)

Let's do that.

./wpt make-hosts-file | sudo tee -a /etc/hosts
# ...

I think that command fails when a password is asked. I used sudo ls to make my system ask for a password so I can run another sudo command without asking for password. I'm sure there is a better way, but hey, it works.

After that, let's retry serve:

$ ./wpt serve
# ...
INFO:web-platform-tests:Starting http server on web-platform.test:8000
INFO:web-platform-tests:Starting http server on web-platform.test:59632
INFO:web-platform-tests:Starting https server on web-platform.test:8443

Hey, it works! Let's open it with a browser!

Well that's not very interesting at all. Am I done now? Let's get back to JSDOM and run the tests.

yarn test
# ...

Cool! It's running tests. Thousands of them. While the tests are running and are heating my macbook, let's take a peak at our goal: Removing request. Let's see where it is used.

Finding usages of request

The first, most naive way to find usages of request is to look for require("request") and require("request-promise-native"):

lib/jsdom/living/helpers/wrap-cookie-jar-for-request.js

"use strict";
const request = require("request");

module.exports = cookieJar => {
  const jarWrapper = request.jar();
  jarWrapper._jar = cookieJar;
  return jarWrapper;
};

lib/jsdom/living/xhr/xhr-utils.js

// ...
const request = require("request");
// ...
const wrapCookieJarForRequest = require("../helpers/wrap-cookie-jar-for-request");
// ...
  function doRequest() {
    try {
      const client = request(options);

      if (hasBody && flag.formData) {
        const form = client.form();
        for (const entry of body) {
          form.append(entry.name, entry.value, entry.options);
        }
      }

      return client;
    } catch (e) {
      const client = new EventEmitter();
      process.nextTick(() => client.emit("error", e));
      return client;
    }
  }
/// ...

test/util.js

// ...
const request = require("request");
// ...
/**
 * Reads a static fixture file as utf8.
 * If running tests from node, the file will be read from the file system
 * If running tests using karma, a http request will be performed to retrieve the file using karma's server.
 * @param {string} relativePath Relative path within the test directory. For example "jsdom/files/test.html"
 */
exports.readTestFixture = relativePath => {
  const useRequest = exports.inBrowserContext();

  return exports.nodeResolverPromise(nodeResolver => {
    if (useRequest) {
      request.get(exports.getTestFixtureUrl(relativePath), { timeout: 5000 }, nodeResolver);
    } else {
      fs.readFile(path.resolve(__dirname, relativePath), { encoding: "utf8" }, nodeResolver);
    }
  })
  // request passes (error, response, content) to the callback
  // we are only interested in the `content`
    .then(result => useRequest ? result[1] : result);
};

lib/jsdom/browser/resources/resource-loader.js

// ...
const request = require("request-promise-native");
const wrapCookieJarForRequest = require("../../living/helpers/wrap-cookie-jar-for-request");
// ...
  fetch(urlString, options = {}) {
    const url = parseURL(urlString);
    // ...
    switch (url.scheme) {
      // ...
      case "http":
      case "https": {
        const requestOptions = this._getRequestOptions(options);
        return request(urlString, requestOptions);
      }
      // ...
    }
  }

test/web-platform-tests/start-wpt-server.js

// ...
const requestHead = require("request-promise-native").head;
// ...
function pollForServer(url) {
  return requestHead(url, { strictSSL: false })
    .then(() => {
  // ...
}

Looks good! Looking for require('request') yields no results, so I'll assume there is either a strict merge policy or some kind of linter ensuring double quoted strings are used everywhere.

There might be other ways request or request-promise-native is required. One could have aliased the require to something else. Maybe someone used require("re" + "quest") to mess with me. Maybe someone's using import somewhere.

Instead of hunting other possible dependencies, let's try to fix the found dependencies first and re-run the tests.

Narrowing down the tests

Running all tests takes ages. However, I'm not sure how to narrow down the number of tests. While trying to figure out how to narrow down the number of tests, the test runner finally finished after 11 minutes.

Reading the contributing guidelines, it is mentioned you can run only JSDOM api tests, or even a set of tests for one specific function. Since the JSDOM API includes a fromUrl function, I'll assume fromUrl fetches the document using request.

There is a test suite specifically for fromUrl and based on the contributing guidelines, I can run it using yarn test-mocha test/api/from-url.js. Let's try that.

$ yarn test-mocha test/api/from-url.js
yarn run v1.22.4
$ mocha test/api/from-url.js


  API: JSDOM.fromURL()
    ✓ should return a rejected promise for a bad URL
    ✓ should return a rejected promise for a 404
    ✓ should return a rejected promise for a 500
    ✓ should use the body of 200 responses (54ms)
    ✓ should use the body of 301 responses
    ✓ should be able to handle gzipped bodies
    ✓ should send a HTML-preferring Accept header
    ✓ should send an Accept-Language: en header
    user agent
      ✓ should use the default user agent as the User-Agent header when none is given
    referrer
      ✓ should reject when passing an invalid absolute URL for referrer
      ✓ should not send a Referer header when no referrer option is given
      ✓ should use the supplied referrer option as a Referer header
      ✓ should canonicalize referrer URLs before using them as a Referer header
      ✓ should use the redirect source URL as the referrer, overriding a provided one
    inferring options from the response
      url
        ✓ should use the URL fetched for a 200
        ✓ should preserve full request URL
        ✓ should use the ultimate response URL after a redirect
        ✓ should preserve fragments when processing redirects
        ✓ should disallow passing a URL manually
      contentType
        ✓ should use the content type fetched for a 200
        ✓ should use the ultimate response content type after a redirect
        ✓ should disallow passing a content type manually
    cookie jar integration
      ✓ should send applicable cookies in a supplied cookie jar
      ✓ should store cookies set by the server in a supplied cookie jar
      ✓ should store cookies set by the server in a newly-created cookie jar


  25 passing (234ms)

✨  Done in 1.09s.

Phew. That's better. One second. Let's first try to break these tests by changing the code that requires request. I'm hoping these tests touches the request-requires at some point.

The test messages also mention cookie jar. I'm hoping this is somehow related to lib/jsdom/living/helpers/wrap-cookie-jar-for-request.js so we can test our changes in that file using this test.

Removing request from test/util.js

Before we can drop request, we need a replacement. I'll be using node-fetch. node-fetch is a NodeJS implementation for the browser's Fetch API. I like the idea of using a library that implements an existing standard because even if you don't longer like or want to use the library, you can just replace the fetch library with some other fetch implementation.

Since JSDOM also runs in the browser, you can use the browser's Fetch implementation in the browser. Isn't that great?

npm install nod-- oh right, we're using YARN now.

$ yarn install node-fetch
error `install` has been replaced with `add` to add new dependencies. Run "yarn add node-fetch" instead.
$ yarn add node-fetch
# ...
✨  Done in 7.80s.

Ok. Now, let's naively replace request with fetch somewhere. Let's start with test/util.js, since I'll assume it's only used from tests. It is most likely the easiest one to replace.

test/util.js

// ...
const fetch = require("node-fetch");
// ...
exports.readTestFixture = relativePath => {
  const useRequest = exports.inBrowserContext();

  if (useRequest) {
    const url = exports.getTestFixtureUrl(relativePath);
    // timeout is a node-fetch specific extention.
    fetch(url, { timeout: 5000 }).then((response) => {
      if (!response.ok) {
        throw new Error(`Unexpected status ${response.status} fetching ${url}`);
      }
      return response.text();
    });
  } else {
    return exports.nodeResolverPromise(nodeResolver => {
      fs.readFile(path.resolve(__dirname, relativePath), { encoding: "utf8" }, nodeResolver);
    });
  }
};

Looks fine, I suppose. Let's run the tests.

$ yarn test-mocha test/api/from-url.js
yarn run v1.22.4
$ mocha test/api/from-url.js
# ...
  25 passing (234ms)
✨  Done in 1.02s.

All tests are passing, but I don't know if the tests even touch my changes. Let's just throw inside the method.

test/util.js

exports.readTestFixture = relativePath => {
  const useRequest = exports.inBrowserContext();
  if (useRequest) {
    throw new Error("???");
// ...

$ yarn test-mocha test/api/from-url.js
yarn run v1.22.4
$ mocha test/api/from-url.js
# ...
  25 passing (234ms)
✨  Done in 1.02s.

No thrown errors or failing tests, so it's still not touching my changes. Let's run all API tests for good measure. Otherwise, I'll have to run all tests.

yarn test-api
# ...
  419 passing (4s)

✨  Done in 4.56s.

Still no error. Let's run all tests until something goes bad. While the tests are running forever, let's CMD+F for readTestFixture.

It looks like all occurences are in test/to-port-to-wpts. CMD+F for to-port-to-wpts yields to this result in the readme:

Ideally we would only use Mocha for testing the JSDOM API itself, from the outside. Unfortunately, a lot of web platform features are still tested using Mocha, instead of web-platform-tests. These are located in the to-port-to-wpts subdirectory.

So maybe running all mocha tests will trigger my intentional failure. While the main test suite is running, I run the mocha tests using yarn test-mocha, hoping it will run faster.

After a minute, I cancelled the mocha runner since it there seems to be no obvious speed improvement by invoking mocha this way.

What about yarn test-mocha test/to-port-to-wpts/*.js?

$ yarn test-mocha test/to-port-to-wpts/*.js

  379 passing (6s)
  1 pending

✨  Done in 9.78s.

That runs the tests, but the tests aren't failing. Confused, I read the jsdoc comment above the function:

test/util.js

/**
 * Reads a static fixture file as utf8.
 * If running tests from node, the file will be read from the file system
 * If running tests using karma, a http request will be performed to retrieve the file using karma's server.
 * @param {string} relativePath Relative path within the test directory. For example "jsdom/files/test.html"
 */
exports.readTestFixture = relativePath => {

So my error will only be thrown when running from inside a browser. Well, I don't need node-fetch inside a browser, do I? I can just use window.fetch, but I won't get the timeout, since the timeout option isn't supported on window.fetch.

How did request implement the timeout? I suppose it uses XMLHttpRequest in the background and aborts after a certain amount of time. Let's ignore that for now and see if we can run the tests inside a browser. The jsdoc mentions karma. Let's CMD+F karma in the readmes.

Contributing.md

The mocha test cases are executed in Chrome using karma. Currently, web platform tests are not executed in the browser yet.

To run all browser tests: yarn test-browser

To run the karma tests in an iframe: yarn test-browser-iframe

To run the karma tests in a web worker: yarn test-browser-worker

Sure. Let's try that.

$ yarn test-browser
yarn run v1.22.4
$ yarn test-browser-iframe && yarn test-browser-worker
$ karma start test/karma.conf.js
[...]
HeadlessChrome 81.0.4044 (Mac OS X 10.15.4) ERROR
  Uncaught Error: ???
  at /var/folders/bf/29ljwt3s4dscb7tdd2z5zz0h0000gn/T/test/util.js:162:1 <- /var/folders/bf/29ljwt3s4dscb7tdd2z5zz0h0000gn/T/91efe4665a6210ee2f5edcae3a8f463c.browserify.js:293540:5

  Error: ???
      at exports.readTestFixture (/var/folders/bf/29ljwt3s4dscb7tdd2z5zz0h0000gn/T/test/util.js:162:1 <- /var/folders/bf/29ljwt3s4dscb7tdd2z5zz0h0000gn/T/91efe4665a6210ee2f5edcae3a8f463c.browserify.js:293540:11)
      [...]

My ??? error is thrown! Now, let's retry without the intentional failure.

$ yarn test-browser
[...]
HeadlessChrome 81.0.4044 (Mac OS X 10.15.4) jsdom/namespaces should set namespaces in HTML documents created by jsdom.env() FAILED
    TypeError: Cannot read property 'then' of undefined
        [...]
HeadlessChrome 81.0.4044 (Mac OS X 10.15.4) jsdom/namespaces should set namespace-related properties in HTML documents created by innerHTML FAILED
    TypeError: Cannot read property 'then' of undefined
        [...]
HeadlessChrome 81.0.4044 (Mac OS X 10.15.4) jsdom/namespaces should set namespace-related properties in HTML-SVG documents created by jsdom.env() FAILED
    TypeError: Cannot read property 'then' of undefined
        [...]
HeadlessChrome 81.0.4044 (Mac OS X 10.15.4) jsdom/namespaces should set namespace-related properties in HTML-SVG documents created by innerHTML FAILED
    TypeError: Cannot read property 'then' of undefined
        [...]
HeadlessChrome 81.0.4044 (Mac OS X 10.15.4) jsdom/parsing real-world page with < inside a text node (GH-800) FAILED
    TypeError: Cannot read property 'then' of undefined
        [...]
HeadlessChrome 81.0.4044 (Mac OS X 10.15.4) jsdom/xml should ignore self-closing of tags in html docs FAILED
    TypeError: Cannot read property 'then' of undefined
        [...]
HeadlessChrome 81.0.4044 (Mac OS X 10.15.4) jsdom/xml should handle self-closing tags properly in xml docs FAILED
    TypeError: Cannot read property 'then' of undefined
        [...]
HeadlessChrome 81.0.4044 (Mac OS X 10.15.4): Executed 1209 of 2460 (7 FAILED) (skipped 1251) (7.437 secs / 6.708 secs)
TOTAL: 7 FAILED, 1202 SUCCESS
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Failures! TypeError: Cannot read property 'then' of undefined? Oh... i forgot to return. Oops.

test/util.js

  if (useRequest) {
    const url = exports.getTestFixtureUrl(relativePath);
    // timeout is a node-fetch specific extension
    return fetch(url, { timeout: 5000 }).then((response) => {
      if (!response.ok) {
        throw new Error(`Unexpected status ${response.status} fetching ${url}`);
      }
      return response.text();
    });
  }

$ yarn test-browser
[...]
HeadlessChrome 81.0.4044 (Mac OS X 10.15.4): Executed 1209 of 2460 (skipped 1251) SUCCESS (7.497 secs / 6.723 secs)
TOTAL: 1209 SUCCESS

That's great! Now, since it's run inside a browser, let's drop the node-fetch requirement and use the browser's fetch.

test/util.js

  if (exports.inBrowserContext()) {
    return fetch(exports.getTestFixtureUrl(relativePath)).then((response) => {
      if (!response.ok) {
        throw new Error(`Unexpected status ${response.status} fetching ${response.location}`);
      }
      return response.text();
    });
  }

$ yarn test-browser
[...]
HeadlessChrome 81.0.4044 (Mac OS X 10.15.4): Executed 1209 of 2460 (skipped 1251) SUCCESS (7.561 secs / 6.812 secs)
TOTAL: 1209 SUCCESS

Great. The best dependency is the one not being used, am I right?

Removing request from test/web-platform-tests/start-wpt-server.js

The second request usage by tests is inside test/web-platform-tests/start-wpt-server.js.

test/web-platform-tests/start-wpt-server.js

// ...
const requestHead = require("request-promise-native").head;
// ...
function pollForServer(url) {
  return requestHead(url, { strictSSL: false })
    .then(() => {
      console.log(`WPT server at ${url} is up!`);
      return url;
    })
    .catch(err => {
      console.log(`WPT server at ${url} is not up yet (${err.message}); trying again`);
      return new Promise(resolve => {
        setTimeout(() => resolve(pollForServer(url)), 500);
      });
    });
}

Based on the name of the file and some of the error messages, it looks like this code is used to check whether WPT is running. This code is used at the start of the test runner. That should be easy enough to test. Let's replace request with node-fetch.

The strictSSL option is no part of the Fetch standard, but stack overflow tells me I can use rejectUnauthorized: false instead.

const fetch = require("node-fetch");
const https = require("https");
// ...

const httpsAgent = new https.Agent({
  rejectUnauthorized: false,
});

function pollForServer(url) {
  const agent = url.startsWith("https")
    ? new https.Agent({ rejectUnauthorized: false })
    : null;
  return fetch(url, { method: "HEAD", agent })
    .then(({ ok, status }) => {
      if (!ok) {
        throw new Error(`Unexpected status=${status}`);
      }
      console.log(`WPT server at ${url} is up!`);
      return url;
    })
    .catch(err => {
      console.log(`WPT server at ${url} is not up yet (${err.message}); trying again`);
      return new Promise(resolve => {
        setTimeout(() => resolve(pollForServer(url)), 500);
      });
    });
}

I've added throw new Error("Foo") (not shown above) to intentionally fail at first. Let's run the tests and see if they fail. I'll assume they fail early, so I'll run all tests.

$ yarn test
[...]
  1) "before all" hook: $mochaNoSugar in "{root}"
  2) "after all" hook: $mochaNoSugar in "{root}"

  0 passing (22ms)
  2 failing

  1) "before all" hook: $mochaNoSugar in "{root}":
     Error: foo

I was right. Let's kill it and retry without the intentional failure.

$ yarn test
[...]

The tests are running again. I let them run, but I assume my change is fine.

Removing request from lib/jsdom/browser/resources/resource-loader.js

Now that the test utilities are fixed, let's get our hands dirty on the lib code. There are only 2 files where request is actually invoked. The 3rd is only a helper:

lib/jsdom/living/helpers/wrap-cookie-jar-for-request.js

"use strict";
const request = require("request");

module.exports = cookieJar => {
  const jarWrapper = request.jar();
  jarWrapper._jar = cookieJar;
  return jarWrapper;
};

Since this helper is a dependency of the other 2 files, I'll look at the helper last. Let's first look at resource-loader.

lib/jsdom/browser/resources/resource-loader.js

// ...
const request = require("request-promise-native");
const wrapCookieJarForRequest = require("../../living/helpers/wrap-cookie-jar-for-request");
// ...
  _getRequestOptions({ cookieJar, referrer, accept = "*/*" }) {
    const requestOptions = {
      encoding: null,
      gzip: true,
      jar: wrapCookieJarForRequest(cookieJar),
      strictSSL: this._strictSSL,
      proxy: this._proxy,
      forever: true,
      headers: {
        "User-Agent": this._userAgent,
        "Accept-Language": "en",
        Accept: accept
      }
    };

    if (referrer && !IS_BROWSER) {
      requestOptions.headers.referer = referrer;
    }

    return requestOptions;
  }
// ...
  fetch(urlString, options = {}) {
    const url = parseURL(urlString);
    // ...
    switch (url.scheme) {
      // ...
      case "http":
      case "https": {
        const requestOptions = this._getRequestOptions(options);
        return request(urlString, requestOptions);
      }
      // ...
    }
  }

Seems easy enough. Let's convert the request options to fetch options.

encoding: null: This causes request to return a buffer. With node-fetch, we might be able to use response.arrayBuffer() for that.
jar: wrapCookieJarForRequest(cookieJar): Somehow cookies are reused this way. The cookieJar variable is converted to a request-compatible cookie jar to allow keeping track of cookies. I don't know if fetch has features like this. I suppose we can just manually read/write the cookies.
strictSSL: this._strictSSL: Just like before, use the HTTPS agent with rejectUnauthorized.
proxy: this._proxy: Enables proxy. There is no obvious way to implement this in node-fetch. I also don't know what's in this._proxy. I might need to use https-proxy-agent for this.
forever: true: Sets keepAlive on the HTTPS agent. Since we're replacing the agent anyway, we might as well set keepAlive: true for both http and https agents.

Let's make a first attempt to implement resource-loader's fetch function using fetch instead of request. Because I don't know how to implement the proxy or cookies, I'll ignore those for now.

lib/jsdom/browser/resources/resource-loader.js

_getFetchOptions({ cookieJar, referrer, accept = "*/*" }) {
  /** @type RequestInit */
  const fetchOptions = {};

  // I don't know what these variables hold exactly - let's log them!
  console.log("cookieJar", cookieJar);
  console.log("this._proxy", this._proxy);

  fetchOptions.headers = {
    "User-Agent": this._userAgent,
    "Accept-Language": "en",
    Accept: accept,
  };

  if (!IS_BROWSER) {
    const httpAgent = new http.Agent({ keepAlive: true });
    const httpsAgent = new https.Agent({ keepAlive: true, rejectUnauthorized: this._strictSSL });

    fetchOptions.headers.referrer = referrer;
    fetchOptions.agent = (url) => url.protocol == 'http:' ? httpAgent : httpsAgent;
  }

  return fetchOptions;
}

// ...
case "http":
case "https": {
  return fetch(urlString, this._getFetchOptions(options))
    .then((response) => {
      if (!response.ok) {
        throw new Error(`Unexpected status=${response.status} for ${urlString}`);
      }
      return response.arrayBuffer();
    })
}

Let's run the tests and see the mess I've created. I get a lot of failures from the tests, as expected. Some are related to cookies. The console.logs look like this:

cookieJar CookieJar { enableLooseMode: true, store: { idx: {} } }
this._proxy undefined

cookieJar CookieJar { enableLooseMode: true, store: { idx: {} } }
this._proxy http://127.0.0.1:51388

So the proxy is just a URL. I'm not sure how to implement the proxy from fetch, if it is even possible. I suppose I can use a proxy agent on the server, but I don't know an alternative for the browser.

The cookie jar is still a mystery. Since package.json mentions tough-cookie, I'll just assume the cookie jar is from that library. I'm just going to assume this is also only used server-side, since the browser's fetch handles cookies automatically.

To add tough-cookie's cookie-jar to node-fetch, I'm going to use a library called fetch-cookie. fetch-cookie has no other dependencies except for tough-cookie so it can be used independently from Fetch implementations. fetch-cookie is also pretty small: about 50 lines of code.

yarn add fetch-cookie

lib/jsdom/browser/resources/resource-loader.js

_getFetchOptions({ cookieJar, referrer, accept = "*/*" }) {
  /** @type RequestInit */
  const fetchOptions = {};

  // I don't know what these variables hold exactly - let's log them!
  console.log("cookieJar", cookieJar);
  console.log("this._proxy", this._proxy);

  fetchOptions.headers = {
    "User-Agent": this._userAgent,
    "Accept-Language": "en",
    "Accept-Encoding": "gzip",
    Accept: accept,
  };

  if (!IS_BROWSER) {
    const httpAgent = new http.Agent({ keepAlive: true });
    const httpsAgent = new https.Agent({ keepAlive: true, rejectUnauthorized: this._strictSSL });

    fetchOptions.headers.referrer = referrer;
    fetchOptions.agent = (url) => url.protocol == 'http:' ? httpAgent : httpsAgent;
  }

  return fetchOptions;
}

// ...
case "http":
case "https": {
  const cookieJar = options.cookieJar;
  cookieJar.__setCookie = cookieJar.setCookie;
  cookieJar.setCookie = (...args) => {
    if (args.length === 3) {
      args.splice(2, 0, {});
    }
    if (args.length === 4) {
      args[2].ignoreError = true;
    }
    return cookieJar.__setCookie(...args);
  }
  const targetFetch = fetchCookie(fetch, cookieJar);
  const fetchOptions = this._getFetchOptions(options);
  return targetFetch(urlString, fetchOptions)
    .then((response) => {
      if (!response.ok) {
        throw new Error(`Unexpected status=${response.status} for ${urlString}`);
      }
      return response.arrayBuffer();
    });
}

I got a lot of errors when handling the cookies. Turns out, when adding cookies, the request library sets ignoreError on true by default (like a browser would do), but fetch-cookie doesn't allow you to change the options when setting cookies.

To "fix" this, I hijacked the setCookie function to silence the errors, only to get different errors. I'll find a proper fix later.

1) Cookie processing
      document.cookie
        reflects back cookies set from the server while requesting the page:
    TypeError: Cannot read property 'headers' of undefined
    at /Users/hinloopen/Projects/Github/jsdom/lib/api.js:138:28
    at processTicksAndRejections (internal/process/task_queues.js:93:5)

Let's see what's inside lib/api.js:

lib/api.js

const req = resourceLoaderForInitialRequest.fetch(url, {
  accept: "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
  cookieJar: options.cookieJar,
  referrer: options.referrer
});

return req.then(body => {
  const res = req.response;

  options = Object.assign(options, {
    url: req.href + originalHash,
    contentType: res.headers["content-type"],
    referrer: req.getHeader("referer")
  });

  return new JSDOM(body, options);
});

So that's interesting. Apparently, the promise returned by request-promise not only has a .then method, it also has a .response property containing the response. I didn't know that, and I don't see it documented anywhere on the request-promise readme. I would just have used resolveWithFullResponse but whatever.

Let's see if we can replicate this behavior.

We need to return a promise-like object that has a .then and a .catch (like a promise), but it also needs to have a .response getter, .href getter, and a .getHeader function.

Again, quick and dirty, let's make it work the easiest way possible.

lib/jsdom/browser/resources/resource-loader.js

const cookieJar = options.cookieJar;
cookieJar.__setCookie = cookieJar.setCookie;
cookieJar.setCookie = (...args) => { /* ... */ }
const targetFetch = fetchCookie(fetch, cookieJar);
const fetchOptions = this._getFetchOptions(options);
const fetchResult = targetFetch(urlString, fetchOptions);

let result;
result = {
  response: null,
  href: urlString,
  then: fetchResult.then((response) => {
    if (!response.ok) {
      throw new Error(`Unexpected status=${response.status} for ${urlString}`);
    }
    result.response = response;
    return response.arrayBuffer();
  }).then.bind(fetchResult),
  catch: fetchResult.catch.bind(fetchResult),
  getHeader(name) {
    return fetchOptions.headers[name];
  }
};

return result;

The previously failing test now succeeds, but many others still fail. Let's fix the next one:

  1) Cookie processing
       should share cookies when a cookie jar is shared:
     TypeError: Cannot read property 'innerHTML' of null
      at /Users/hinloopen/Projects/Github/jsdom/test/api/cookies.js:288:75
      at processTicksAndRejections (internal/process/task_queues.js:93:5)

test/api/cookies.js

it("should share cookies when a cookie jar is shared", () => {
  const cookieJar = new CookieJar();

  return JSDOM.fromURL(testHost + "/TestPath/set-cookie-from-server", { cookieJar }).then(() => {
    return JSDOM.fromURL(testHost + "/TestPath/html-get-cookie-header", { cookieJar });
  }).then(({ window }) => {
    const cookieHeader = window.document.querySelector(".cookie-header").innerHTML;

    assertCookies(cookieHeader, [
      "Test1=Basic",
      "Test2=PathMatch",
      "Test6=HttpOnly",
      "Test9=Duplicate",
      "Test10={\"prop1\":5,\"prop2\":\"value\"}",
      "Malformed"
    ]);

    assertCookies(window.document.cookie, [
      "Test1=Basic",
      "Test2=PathMatch",
      "Test9=Duplicate",
      "Test10={\"prop1\":5,\"prop2\":\"value\"}",
      "Malformed"
    ]);
  });
});

So the .cookie-header element couldn't be found in the /html-get-cookie-header page. Maybe there is a hint somewhere in the document's HTML. Let's log window.document.body.innerHTML using console.log({ html: window.document.body.innerHTML });

{ html: '[object Response]' }

I strongly suspect somewhere inside my new fetch implementation, the HTML body's toString returns "[object Response]". Let's check our implementation again.

lib/jsdom/browser/resources/resource-loader.js

const fetchOptions = this._getFetchOptions(options);
const fetchPromise = targetFetch(urlString, fetchOptions);

let result;
const then = function(onfulfilled, onrejected) {
  return fetchPromise.then((response) => {
    if (!response.ok) {
      throw new Error(`Unexpected status=${response.status} for ${urlString}`);
    }
    result.response = response;
    return response.arrayBuffer();
  }).then(onfulfilled, onrejected);
};

result = {
  response: null,
  href: urlString,
  then,
  catch: function(onrejected) { return then(undefined, onrejected) },
  getHeader(name) {
    return fetchOptions.headers[name];
  }
};

return result;

Now we get, yet again, different errors. One includes The "buf" argument must be one of type Buffer, TypedArray, or DataView. Received type object. I suspect this has to do with the ArrayBuffer returned by node-fetch: This is NOT the same as a NodeJS Buffer. Let's make it a Buffer for NodeJS only:

lib/jsdom/browser/resources/resource-loader.js

const then = function(onfulfilled, onrejected) {
  return fetchPromise.then((response) => {
    if (!response.ok) {
      throw new Error(`Unexpected status=${response.status} for ${urlString}`);
    }
    result.response = response;
    return response.arrayBuffer();
  })
  .then((arrayBuffer) => {
    if (typeof Buffer === "undefined") {
      return arrayBuffer;
    } else {
      return Buffer.from(arrayBuffer);
    }
  })
  .then(onfulfilled, onrejected);
};

The next error I encounter is this one:

  1) API: resource loading configuration
       set to "usable"
         canceling requests
           should abort a script request (with no events) when stopping the window:
     TypeError: openedRequest.abort is not a function
      at RequestManager.close (lib/jsdom/browser/resources/request-manager.js:25:21)
      at Window.stop (lib/jsdom/browser/Window.js:608:15)
      at /Users/hinloopen/Projects/Github/jsdom/test/api/resources.js:559:20
      at processTicksAndRejections (internal/process/task_queues.js:93:5)

.abort is not a function. Is openedRequest our fetch result?

lib/jsdom/browser/resources/request-manager.js

/**
 * Manage all the request and it is able to abort
 * all pending request.
 */
module.exports = class RequestManager {
  // ...
  close() {
    for (const openedRequest of this.openedRequests) {
      openedRequest.abort();
    }
    this.openedRequests = [];
  }
  // ...
};

Let's implement .abort, make it do nothing, and see if the error changes.

lib/jsdom/browser/resources/resource-loader.js

result = {
  response: null,
  abort: () => { console.log("TODO ABORT"); },
  href: urlString,
  then,
  catch: function(onrejected) { return then(undefined, onrejected) },
  getHeader(name) {
    return fetchOptions.headers[name];
  }
};

TODO ABORT
Error: Could not load script: "http://127.0.0.1:58978/"
  1) API: resource loading configuration
       set to "usable"
         canceling requests
           should abort a script request (with no events) when stopping the window:

      The error event must not fire
      + expected - actual

      -true
      +false

      at /Users/hinloopen/Projects/Github/jsdom/test/api/resources.js:920:12
      at async Promise.all (index 0)
      at async /Users/hinloopen/Projects/Github/jsdom/test/api/resources.js:561:9

Right, time to properly implement .abort. Can we even implement .abort using the browser's Fetch API? According to MDN, it is experimental technology. Browser support might be incomplete, but I suspect it's only used in NodeJS anyway.

node-fetch also supports aborting requests, and it is implemented in the same way! It requires an AbortController implementation - abort-controller is suggested.

sh

yarn add abort-controller

lib/jsdom/browser/resources/resource-loader.js

const AbortController = require("abort-controller");

// ...
const targetFetch = fetchCookie(fetch, cookieJar);
const fetchOptions = this._getFetchOptions(options);
const abortController = new AbortController();
fetchOptions.signal = abortController.signal;
const fetchPromise = targetFetch(urlString, fetchOptions);

let result;
const then = function(onfulfilled, onrejected) {
  return fetchPromise.then((response) => {
    if (!response.ok) {
      throw new Error(`Unexpected status=${response.status} for ${urlString}`);
    }
    result.response = response;
    return response.arrayBuffer();
  })
  .then((arrayBuffer) => typeof Buffer === "undefined" ? arrayBuffer : Buffer.from(arrayBuffer))
  .then(onfulfilled, onrejected);
};

result = {
  response: null,
  abort: () => { abortController.abort(); },
  href: urlString,
  then,
  catch: function(onrejected) { return then(undefined, onrejected) },
  getHeader(name) {
    return fetchOptions.headers[name];
  }
};

Using abort still throws an error, causing the test to fail:

Error: Could not load script: "http://127.0.0.1:61567/"

# ...

  type: 'aborted',
  message: 'The user aborted a request.'

# ...

  1) API: resource loading configuration
       set to "usable"
         canceling requests
           should abort a script request (with no events) when stopping the window:

      The error event must not fire
      + expected - actual

      -true
      +false

I'm not sure how request would have handled the abort, but based on this failure, it wasn't by throwing an error. I can't find any documentation about it. The source seems to just cancel the request and destroy the response without throwing an error. Maybe the promise just never resolves?

Let's implement it that way, see if it works.

lib/jsdom/browser/resources/resource-loader.js

let aborted = false;
let result;
const then = function(onfulfilled, onrejected) {
  return fetchPromise.then((response) => {
    if (!response.ok) {
      throw new Error(`Unexpected status=${response.status} for ${urlString}`);
    }
    result.response = response;
    return response.arrayBuffer();
  })
  .then((arrayBuffer) => typeof Buffer === "undefined" ? arrayBuffer : Buffer.from(arrayBuffer))
  .then((result) => { if (!aborted) return onfulfilled(result); })
  .catch((error) => { if (!aborted) return onrejected(error); });
};

result = {
  response: null,
  abort: function() {
    aborted = true;
    abortController.abort();
  },
  href: urlString,
  then,
  catch: function(onrejected) {
    return then(undefined, onrejected)
  },
  getHeader(name) {
    return fetchOptions.headers[name];
  }
};

A lot of green tests this round! Looking good. Still, there are tens of failing tests, some mentioning the proxy. Others mentioning the Referer header.

It looks like I assigned the referrer to a header named Referrer instead of Referer. Let's fix that and look at the next error.

lib/jsdom/browser/resources/resource-loader.js

// inside _getFetchOptions
if (!IS_BROWSER) {
  const httpAgent = new http.Agent({ keepAlive: true });
  const httpsAgent = new https.Agent({ keepAlive: true, rejectUnauthorized: this._strictSSL });

  if (referrer) {
    fetchOptions.headers.referer = referrer;
  }
  fetchOptions.agent = (url) => url.protocol == 'http:' ? httpAgent : httpsAgent;
}

The other two errors are going to be a problem, and are related to redirects:

  1) Cookie processing
       sent with requests
         should gather cookies from redirects (GH-1089):

      AssertionError: expected [ 'Test3=Redirect3' ] to deeply equal [ Array(3) ]
      + expected - actual

       [
      +  "Test1=Redirect1"
      +  "Test2=Redirect2"
         "Test3=Redirect3"
       ]

      at assertCookies (test/api/cookies.js:383:10)
      at /Users/hinloopen/Projects/Github/jsdom/test/api/cookies.js:247:9
      at processTicksAndRejections (internal/process/task_queues.js:93:5)

  2) API: JSDOM.fromURL()
       referrer
         should use the redirect source URL as the referrer, overriding a provided one:

      AssertionError: expected 'http://example.com/' to equal 'http://127.0.0.1:55863/1'
      + expected - actual

      -http://example.com/
      +http://127.0.0.1:55863/1

      at /Users/hinloopen/Projects/Github/jsdom/test/api/from-url.js:135:14
      at processTicksAndRejections (internal/process/task_queues.js:93:5)

fetch uses transparent redirects, and it appears that fetch-cookie doesn't store cookies around redirects. Reading the documentation, there is actually a fix for that. Let's apply that fix.

It looks like it is as easy as changing the require to const fetchCookie = require('fetch-cookie/node-fetch');. Let's do that, and re-run the tests.

  1) API: JSDOM.fromURL()
       referrer
         should use the redirect source URL as the referrer, overriding a provided one:

      AssertionError: expected 'http://example.com/' to equal 'http://127.0.0.1:56188/1'
      + expected - actual

      -http://example.com/
      +http://127.0.0.1:56188/1

The other error is gone. Now let's see how we fix this one. I can make an educated guess what's being tested here, but let's look at the source.

it("should use the redirect source URL as the referrer, overriding a provided one", async () => {
  const [requestURL] = await redirectServer("<p>Hello</p>", { "Content-Type": "text/html" });

  const dom = await JSDOM.fromURL(requestURL, { referrer: "http://example.com/" });
  assert.strictEqual(dom.window.document.referrer, requestURL);
});

So... it's checking document.referrer. I've no idea where this is assigned and I don't want to find out. Instead, since this test is testing JSDOM.fromURL specifically, let's see if JSDOM.fromURL assigns the referrer somewhere.

lib/api.js

static fromURL(url, options = {}) {
  return Promise.resolve().then(() => {
    // Remove the hash while sending this through the research loader fetch().
    // It gets added back a few lines down when constructing the JSDOM object.
    const parsedURL = new URL(url);
    const originalHash = parsedURL.hash;
    parsedURL.hash = "";
    url = parsedURL.href;

    options = normalizeFromURLOptions(options);

    const resourceLoader = resourcesToResourceLoader(options.resources);
    const resourceLoaderForInitialRequest = resourceLoader.constructor === NoOpResourceLoader ?
      new ResourceLoader() :
      resourceLoader;

    const req = resourceLoaderForInitialRequest.fetch(url, {
      accept: "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
      cookieJar: options.cookieJar,
      referrer: options.referrer
    });

    return req.then(body => {
      const res = req.response;

      options = Object.assign(options, {
        url: req.href + originalHash,
        contentType: res.headers["content-type"],
        referrer: req.getHeader("referer")
      });

      return new JSDOM(body, options);
    });
  });
}

Interesting - it uses this req.getHeader("referer"). req is the object I'm returning, so it actually calls my getHeader function. This function returns the header of the first request.

This is a problem: Because the request was redirected, a new request was started. However, my getHeader fetches the header of the first request, not the last request in the redirect chain.

This is also an issue for req.href, which returns the first request URL, not the last, but I haven't confirmed a failing test for this problem.

Let's see if we can peek into the redirect requests. Since fetch-cookie also fixed this problem for assigning cookies, I bet their fix shows how you can peek into redirect requests. Let's take a look at fetch-cookie/node-fetch

fetch-cookie's node-fetch.js

module.exports = function nodeFetchCookieDecorator (nodeFetch, jar) {
  const fetchCookie = require('./')(nodeFetch, jar)

  return function nodeFetchCookie (url, userOptions = {}) {
    const opts = Object.assign({}, userOptions, { redirect: 'manual' })

    // Forward identical options to wrapped node-fetch but tell to not handle redirection.
    return fetchCookie(url, opts)
      .then(res => {
        const isRedirect = (res.status === 303 || res.status === 301 || res.status === 302 || res.status === 307)

        // Interpret the proprietary "redirect" option in the same way that node-fetch does.
        if (isRedirect && userOptions.redirect !== 'manual' && userOptions.follow !== 0) {
          const statusOpts = {
            // Since the "follow" flag is not relevant for node-fetch in this case,
            // we'll hijack it for our internal bookkeeping.
            follow: userOptions.follow !== undefined ? userOptions.follow - 1 : undefined
          }

          if (res.status !== 307) {
            statusOpts.method = 'GET'
            statusOpts.body = null
          }

          const redirectOpts = Object.assign({}, userOptions, statusOpts)

          return nodeFetchCookie(res.headers.get('location'), redirectOpts)
        } else {
          return res
        }
      })
  }
}

So basically, their fix is to set the redirect-mode to manual and just call fetch again for every redirect. Because it calls fetch for every redirect, the cookies can be assigned and extracted every request by fetch-cookie.

The easiest way to keep track of all the redirect requests without also interfering with fetch-cookie's fix is by wrapping the node-fetch instance, keeping track of the last request.

Let's try that.

lib/jsdom/browser/resources/resource-loader.js

_getFetchOptions({ accept = "*/*" }) {
  /** @type RequestInit */
  const fetchOptions = {};

  fetchOptions.headers = {
    "User-Agent": this._userAgent,
    "Accept-Language": "en",
    "Accept-Encoding": "gzip",
    Accept: accept,
  };

  if (!IS_BROWSER) {
    const httpAgent = new http.Agent({ keepAlive: true });
    const httpsAgent = new https.Agent({ keepAlive: true, rejectUnauthorized: this._strictSSL });
    fetchOptions.agent = (url) => url.protocol == 'http:' ? httpAgent : httpsAgent;
  }

  return fetchOptions;
}

// inside fetch(urlString, options = {})
let lastUrl = options.referrer;
let lastOpts = null;

const myFetch = (url, opts) => {
  if (lastUrl && !IS_BROWSER) {
    opts.headers.referer = lastUrl;
  }
  lastUrl = url;
  lastOpts = opts;
  return fetch(url, opts);
};

const targetFetch = fetchCookie(myFetch, cookieJar);
const fetchOptions = this._getFetchOptions(options);
const abortController = new AbortController();
fetchOptions.signal = abortController.signal;
const fetchPromise = targetFetch(urlString, fetchOptions);

let aborted = false;
let result;
const then = function(onfulfilled, onrejected) {
  return fetchPromise.then((response) => {
    if (!response.ok) {
      throw new Error(`Unexpected status=${response.status} for ${urlString}`);
    }
    result.response = response;
    result.href = lastUrl;
    return response.arrayBuffer();
  })
  .then((arrayBuffer) => typeof Buffer === "undefined" ? arrayBuffer : Buffer.from(arrayBuffer))
  .then((result) => { if (!aborted) return onfulfilled(result); })
  .catch((error) => {
    if (!aborted) {
      if (onrejected) {
        return onrejected(error);
      } else {
        throw error;
      }
    }
  });
};

result = {
  response: null,
  abort: function() {
    aborted = true;
    abortController.abort();
  },
  href: null,
  then,
  catch: function(onrejected) {
    return then(undefined, onrejected)
  },
  getHeader(name) {
    return lastOpts.headers[name];
  }
};

return result;

So we now have fetch, myFetch and targetFetch. Bad variable names aside, the redirect-related failures seem solved. Let's run the tests and look at the next errors.

# ...
      with a Content-Type header specifying csiso88598e
        1) should sniff no-bom-charset-http-equiv-no-quotes.html as ISO-8859-8
        2) should sniff no-bom-charset-http-equiv-tis-620.html as ISO-8859-8
        3) should sniff no-bom-charset-koi8.html as ISO-8859-8
        4) should sniff no-bom-charset-utf-16.html as ISO-8859-8
        5) should sniff no-bom-charset-utf-16be.html as ISO-8859-8
        6) should sniff no-bom-charset-utf-16le.html as ISO-8859-8
        7) should sniff no-bom-no-charset.html as ISO-8859-8
# ...
  2) API: encoding detection
       fromURL
         with a Content-Type header specifying csiso88598e
           should sniff no-bom-charset-http-equiv-tis-620.html as ISO-8859-8:

      AssertionError: expected 'windows-874' to equal 'ISO-8859-8'
      + expected - actual

      -windows-874
      +ISO-8859-8
# ...

I have questions. Maybe the test provides some details.

test/api/encoding.js

describe("fromURL", { skipIfBrowser: true }, () => {
  let server;
  let host;
  before(() => {
    return createServer((req, res) => {
      const [, fixture, query] = /^\/([^?]+)(\?.*)?$/.exec(req.url);

      const headers = { "Content-Type": "text/html" };
      if (query === "?charset=csiso88598e") {
        headers["Content-Type"] = "text/html;charset=csiso88598e";
      }

      res.writeHead(200, headers);
      fs.createReadStream(fixturePath(fixture)).pipe(res);
    }).then(s => {
      server = s;
      host = `http://127.0.0.1:${s.address().port}`;
    });
  });

  after(() => server.destroy());

  describe("with no Content-Type header given", () => {
    for (const encodingFixture of Object.keys(encodingFixtures)) {
      const { name, body } = encodingFixtures[encodingFixture];

      it(`should sniff ${encodingFixture} as ${name}`, () => {
        return JSDOM.fromURL(`${host}/${encodingFixture}`).then(dom => {
          assert.strictEqual(dom.window.document.characterSet, name);
          assert.strictEqual(dom.window.document.body.textContent, body);
        });
      });
    }
  });

  describe("with a Content-Type header specifying csiso88598e", () => {
    for (const encodingFixture of Object.keys(encodingFixtures)) {
      const { nameWhenOverridden, bodyWhenOverridden } = encodingFixtures[encodingFixture];

      it(`should sniff ${encodingFixture} as ${nameWhenOverridden}`, () => {
        return JSDOM.fromURL(`${host}/${encodingFixture}?charset=csiso88598e`).then(dom => {
          assert.strictEqual(dom.window.document.characterSet, nameWhenOverridden);
          assert.strictEqual(dom.window.document.contentType, "text/html"); // encoding should be stripped

          if (bodyWhenOverridden) {
            assert.strictEqual(dom.window.document.body.textContent, bodyWhenOverridden);
          }
        });
      });
    }
  });
});

Looking at other tests, this csiso88598e content-type is also tested when invoking the constructir directly, and the expectations are similar, and these tests are not failing:

constructor, given binary data
  with a contentType option specifying csiso88598e
    Buffer
      ✓ should sniff no-bom-charset-http-equiv-no-quotes.html as ISO-8859-8
      ✓ should sniff no-bom-charset-http-equiv-tis-620.html as ISO-8859-8
      ✓ should sniff no-bom-charset-koi8.html as ISO-8859-8
      ✓ should sniff no-bom-charset-utf-16.html as ISO-8859-8
      ✓ should sniff no-bom-charset-utf-16be.html as ISO-8859-8
      ✓ should sniff no-bom-charset-utf-16le.html as ISO-8859-8
      ✓ should sniff no-bom-no-charset.html as ISO-8859-8
      ✓ should sniff utf-8-bom.html as UTF-8
      ✓ should sniff utf-16be-bom.html as UTF-16BE
      ✓ should sniff utf-16le-bom.html as UTF-16LE

fromURL
  with no Content-Type header given
    ✓ should sniff no-bom-charset-http-equiv-no-quotes.html as ISO-8859-5 (48ms)
    ✓ should sniff no-bom-charset-http-equiv-tis-620.html as windows-874
    ✓ should sniff no-bom-charset-koi8.html as KOI8-R
    ✓ should sniff no-bom-charset-utf-16.html as UTF-8
    ✓ should sniff no-bom-charset-utf-16be.html as UTF-8
    ✓ should sniff no-bom-charset-utf-16le.html as UTF-8
    ✓ should sniff no-bom-no-charset.html as windows-1252
    ✓ should sniff utf-8-bom.html as UTF-8
    ✓ should sniff utf-16be-bom.html as UTF-16BE
    ✓ should sniff utf-16le-bom.html as UTF-16LE
  with a Content-Type header specifying csiso88598e
    1) should sniff no-bom-charset-http-equiv-no-quotes.html as ISO-8859-8
    2) should sniff no-bom-charset-http-equiv-tis-620.html as ISO-8859-8
    3) should sniff no-bom-charset-koi8.html as ISO-8859-8
    4) should sniff no-bom-charset-utf-16.html as ISO-8859-8
    5) should sniff no-bom-charset-utf-16be.html as ISO-8859-8
    6) should sniff no-bom-charset-utf-16le.html as ISO-8859-8
    7) should sniff no-bom-no-charset.html as ISO-8859-8

Correctly handling this csiso88598e content-type should be done by the constructor. Looking at the source and the tests, the constructor accepts a contentType option that, when equal to csiso88598e, parses the response as ISO-8859-8.

Additionally, the test-server returns a Content-Type: text/html;charset=csiso88598e header. This content-type should be passed to the JSDOM constructor from fromURL:

lib/api.js

static fromURL(url, options = {}) {
  return Promise.resolve().then(() => {
    return req.then(body => {
      const res = req.response;

      options = Object.assign(options, {
        url: req.href + originalHash,
        contentType: res.headers["content-type"],
        referrer: req.getHeader("referer")
      });

      return new JSDOM(body, options);
    });
  });
}

Let's take a look at res.headers inside one of the failing tests using console.log(res.headers, res.headers["content-type"]);:

Headers {
  [Symbol(map)]: [Object: null prototype] {
    'content-type': [ 'text/html;charset=csiso88598e' ],
    date: [ 'Mon, 29 Jun 2020 20:44:07 GMT' ],
    connection: [ 'keep-alive' ],
    'transfer-encoding': [ 'chunked' ]
  }
} undefined

So the content-type is there, but res.headers["content-type"] is undefined. That's because res.headers is not a regular object, but instead is a Headers object. Apperently, you cannot use the [] operator to access the Header's properties. Instead, you should use .get.

For backwards compatibility, let's change response to have a headers property that's just a plain JS object.

lib/jsdom/browser/resources/resource-loader.js

// inside `then`
const { ok, status } = response;
if (!ok) {
  throw new Error(`Unexpected status=${status} for ${urlString}`);
}
const headers = {};
for (const [ key, value ] of response.headers) {
  headers[key] = value;
}

result.response = {
  status,
  headers,
};
result.href = lastUrl;
return response.arrayBuffer();

All encoding-related tests are now green. Let's see what's next. There a lot less failures now, so waiting for a failing test now takes minutes.

There are some interesting failures. A common one is a maximum call stack size exceeded error in setCookie:

RangeError: Maximum call stack size exceeded
    at Array.values (<anonymous>)
    at CookieJar.cookieJar.setCookie [as __setCookie] (/Users/hinloopen/Projects/Github/jsdom/lib/jsdom/browser/resources/resource-loader.js:148:28)
    at CookieJar.cookieJar.setCookie [as __setCookie] (/Users/hinloopen/Projects/Github/jsdom/lib/jsdom/browser/resources/resource-loader.js:148:28)
    at CookieJar.cookieJar.setCookie [as __setCookie] (/Users/hinloopen/Projects/Github/jsdom/lib/jsdom/browser/resources/resource-loader.js:148:28)
    at CookieJar.cookieJar.setCookie [as __setCookie] (/Users/hinloopen/Projects/Github/jsdom/lib/jsdom/browser/resources/resource-loader.js:148:28)
    at CookieJar.cookieJar.setCookie [as __setCookie] (/Users/hinloopen/Projects/Github/jsdom/lib/jsdom/browser/resources/resource-loader.js:148:28)
    at CookieJar.cookieJar.setCookie [as __setCookie] (/Users/hinloopen/Projects/Github/jsdom/lib/jsdom/browser/resources/resource-loader.js:148:28)
    at CookieJar.cookieJar.setCookie [as __setCookie] (/Users/hinloopen/Projects/Github/jsdom/lib/jsdom/browser/resources/resource-loader.js:148:28)
    at CookieJar.cookieJar.setCookie [as __setCookie] (/Users/hinloopen/Projects/Github/jsdom/lib/jsdom/browser/resources/resource-loader.js:148:28)
    at CookieJar.cookieJar.setCookie [as __setCookie] (/Users/hinloopen/Projects/Github/jsdom/lib/jsdom/browser/resources/resou

Another one is mentioning the proxy, which I have not yet implemented:

  1) API: resource loading configuration
       With a custom resource loader
         should be able to customize the proxy option:

      AssertionError: expected 1 to equal 3
      + expected - actual

      -1
      +3

A timeout:

  2) web-platform-tests
       cors
         credentials-flag.htm:
     Error: Error: test harness should not timeout: cors/credentials-flag.htm

And cookies being sent for preflight requests:

  31) web-platform-tests
       xhr
         access-control-preflight-request-must-not-contain-cookie.htm:
     Failed in "Preflight request must not contain any cookie header":
assert_unreached: Unexpected error. Reached unreachable code

There might also be some other errors in between, but the logs are full with the setCookie stacktraces, so let's first fix that one.

It seems that the cookieJar keeps being patched over and over again, which was not my intention. Fixing this should fix the stack-level-too-deep error, and it might also fix the timeout error.

Let's add a check to ensure the cookieJar is only patched once:

lib/jsdom/browser/resources/resource-loader.js

// inside `fetch(urlString, options = {})`
const cookieJar = options.cookieJar;
if (!cookieJar.__setCookie) {
  cookieJar.__setCookie = cookieJar.setCookie;
  cookieJar.setCookie = (...args) => {
    if (args.length === 3) {
      args.splice(2, 0, {});
    }
    if (args.length === 4) {
      args[2].ignoreError = true;
    }
    return cookieJar.__setCookie(...args);
  }
}

4917 passing (11m)
563 pending
1 failing

1) API: resource loading configuration
      With a custom resource loader
        should be able to customize the proxy option:

    AssertionError: expected 1 to equal 3
    + expected - actual

    -1
    +3

    at /Users/hinloopen/Projects/Github/jsdom/test/api/resources.js:666:16
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)

4917 passing tests, 1 failing. Only the proxy implementation remains.

Implementing proxy

It seems that one can replace the node-fetch HTTP(s) agents with a proxy agent using https-proxy-agent as mentioned by jimliang.

Looking at the dependencies of https-proxy-agent, it seems there are two: agent-base and debug.

I feel like this debug dependency should have been optional, but who am I to judge. The agent-base dependency seems sensible.

I also noticed there is a http-proxy-agent variant, without the https. I'm not sure if we need both. I'm hoping the https one just supports both HTTP and HTTPS so I don't have to install both.

Let's try https-proxy-agent.

yarn add https-proxy-agent

lib/jsdom/browser/resources/resource-loader.js

const HttpsProxyAgent = require("https-proxy-agent");

// _getFetchOptions({ accept = "*/*" }) {
if (!IS_BROWSER) {
  const proxyAgent = this._proxy ? new HttpsProxyAgent(this._proxy) : null;
  const httpAgent = new http.Agent({ keepAlive: true });
  const httpsAgent = new https.Agent({ keepAlive: true, rejectUnauthorized: this._strictSSL });
  fetchOptions.agent = (url) => proxyAgent ? proxyAgent : (url.protocol == 'http:' ? httpAgent : httpsAgent);
}

Let's run the tests, see if this works.

# (with .only on "should be able to customize the proxy option")
0 passing (6s)
1 failing

1) API: resource loading configuration
      With a custom resource loader
        should be able to customize the proxy option:
    Error: Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/Users/hinloopen/Projects/Github/jsdom/test/index.js)
    at listOnTimeout (internal/timers.js:531:17)
    at processTimers (internal/timers.js:475:7)

Timeout? That's not helpful at all. Since the proxy is HTTP, let's blindly try http-proxy-agent. Additionally, the keepAlive and rejectUnauthorized options are not passed to the proxy agent. Let's add them. Both proxy agents accept either a URL or an object post, hostname, protocol: The output of url.parse. I'm assuming the remaining options are passed to http(s).Agent.

Let's combine all my assumptions and see if we get anything other than a timeout. Let's also increase the timeout, just in case something is just being slow.

yarn add http-proxy-agent

lib/jsdom/browser/resources/resource-loader.js

const url = require("url");
const HttpProxyAgent = require("http-proxy-agent");
const HttpsProxyAgent = require("https-proxy-agent");

// _getFetchOptions({ accept = "*/*" }) {
if (!IS_BROWSER) {
  const agentOpts = { keepAlive: true, rejectUnauthorized: this._strictSSL };
  const proxyOpts = { ...agentOpts, ...(this._proxy ? url.parse(this._proxy) : {}) };
  const httpAgent = this._proxy ? new HttpProxyAgent(proxyOpts) : new http.Agent(agentOpts);
  const httpsAgent = this._proxy ? new HttpsProxyAgent(proxyOpts) : new https.Agent(agentOpts);
  fetchOptions.agent = (url) => url.protocol == 'http:' ? httpAgent : httpsAgent;
}

# npm t -- --timeout 9999
# (with .only on "should be able to customize the proxy option")
this._proxy http://127.0.0.1:63767
this._proxy http://127.0.0.1:63767
      ✓ should be able to customize the proxy option (80ms)


  1 passing (4s)

Success!

Let's do a minor cleanup to create agents on-demand, and re-run all tests to make sure everything still works.

lib/jsdom/browser/resources/resource-loader.js

/**
 *
 * @param {string} protocol "http:" or "https:"
 */
_getAgent(protocol) {
  const isHttps = protocol === "https:";
  const agentOpts = { keepAlive: true, rejectUnauthorized: this._strictSSL };
  if (this._proxy) {
    agentOpts.rejectUnauthorized = this._strictSSL;
    const proxyOpts = { ...url.parse(this._proxy), ...agentOpts };
    return isHttps ? new HttpsProxyAgent(proxyOpts) : new HttpProxyAgent(proxyOpts);
  } else {
    return isHttps ? new https.Agent(agentOpts) : new http.Agent(agentOpts);
  }
}

// inside _getFetchOptions({ accept = "*/*" }) {
if (!IS_BROWSER) {
  fetchOptions.agent = (url) => this._getAgent(url.protocol);
}

All tests are gean. Great. This is the final result. I intent to clean it up after the remaining request dependencies are removed.

lib/jsdom/browser/resources/resource-loader.js

/**
 *
 * @param {string} protocol "http:" or "https:"
 */
_getAgent(protocol) {
  const isHttps = protocol === "https:";
  const agentOpts = { keepAlive: true, rejectUnauthorized: this._strictSSL };
  if (this._proxy) {
    agentOpts.rejectUnauthorized = this._strictSSL;
    const proxyOpts = { ...url.parse(this._proxy), ...agentOpts };
    return isHttps ? new HttpsProxyAgent(proxyOpts) : new HttpProxyAgent(proxyOpts);
  } else {
    return isHttps ? new https.Agent(agentOpts) : new http.Agent(agentOpts);
  }
}

// inside _getFetchOptions({ accept = "*/*" }) {
case "http":
case "https": {
  const cookieJar = options.cookieJar;
  if (!cookieJar.__setCookie) {
    cookieJar.__setCookie = cookieJar.setCookie;
    cookieJar.setCookie = (...args) => {
      if (args.length === 3) {
        args.splice(2, 0, {});
      }
      if (args.length === 4) {
        args[2].ignoreError = true;
      }
      return cookieJar.__setCookie(...args);
    }
  }

  let lastUrl = options.referrer;
  let lastOpts = null;

  const myFetch = (url, opts) => {
    if (lastUrl && !IS_BROWSER) {
      opts.headers.referer = lastUrl;
    }
    lastUrl = url;
    lastOpts = opts;
    return fetch(url, opts);
  };

  const targetFetch = fetchCookie(myFetch, cookieJar);
  const fetchOptions = this._getFetchOptions(options);
  const abortController = new AbortController();
  fetchOptions.signal = abortController.signal;
  const fetchPromise = targetFetch(urlString, fetchOptions);

  let aborted = false;
  let result;
  const then = function(onfulfilled, onrejected) {
    return fetchPromise.then((response) => {
      const { ok, status } = response;
      if (!ok) {
        throw new Error(`Unexpected status=${status} for ${urlString}`);
      }
      const headers = {};
      for (const [ key, value ] of response.headers) {
        headers[key] = value;
      }

      result.response = {
        status,
        headers,
      };
      result.href = lastUrl;
      return response.arrayBuffer();
    })
    .then((arrayBuffer) => typeof Buffer === "undefined" ? arrayBuffer : Buffer.from(arrayBuffer))
    .then((result) => { if (!aborted) return onfulfilled(result); })
    .catch((error) => {
      if (!aborted) {
        if (onrejected) {
          return onrejected(error);
        } else {
          throw error;
        }
      }
    });
  };

  result = {
    response: null,
    abort: function() {
      aborted = true;
      abortController.abort();
    },
    href: null,
    then,
    catch: function(onrejected) {
      return then(undefined, onrejected)
    },
    getHeader(name) {
      return lastOpts.headers[name];
    }
  };

  return result;
}

Because this post has become fairly large, I'll continue this post in a part 2. To be continued...

Full Rich Text Search with Slate and PostgreSQL 12

matheus-b — Wed, 12 Feb 2020 10:14:47 +0000

The following approach will work for all JSON content but was used here for full text search on content created with Slate. In the format Slate produces by default all plain text content is conveniently placed in fields with key text. Slate can emit plain text but here all content will be stored in a PostgreSQL jsonb column and we will be doing full texts search using a functional GIN index directly.

Postgres can produce tsvectors (the format postgres uses to store text search data) on actual json data with the json(b)_to_tsvector function, but this will index all values in the json document, so we need to do some filtering first. Postgres 12 introduced the SQL/JSON Path Language machinery which exactly allows for this.

Let's consider the following content stored in our documents table:

[
    {
        "data": {
        },
        "type": "introduction",
        "nodes": [
            {
                "text": "This is a test document",
                "marks": [
                ],
                "object": "text"
            }
        ],
        "object": "block"
    },
    {
        "data": {
        },
        "type": "paragraph",
        "nodes": [
            {
                "text": "Just to show what content format we might have.",
                "marks": [
                ],
                "object": "text"
            }
        ],
        "object": "block"
    },
    {
        "data": {
        },
        "type": "conclusion",
        "nodes": [
            {
                "text": "I hope you get the general idea.",
                "marks": [
                ],
                "object": "text"
            }
        ],
        "object": "block"
    }
]

Jumping right to the good bit, jsonb_path_query_array(content, 'strict $.**.text') will produce jsonb:

["This is a test document", "Just to show what content format we might have.", "I hope you get the general idea."].

Note the strict keyword here, leaving it out will produce:

["This is a test document", "This is a test document", "Just to show what content format we might have.", "Just to show what content format we might have.", "I hope you get the general idea.", "I hope you get the general idea."].

According to the documentation strict mode should throw actual errors when a non-existing path is queried (which does not happen because of the explicit ** expansion (otherwise one might use 'strict $.** ? (exists(@.text)).text')), but it seems to me lax mode should not return double results. Anyway, whatever works.

Note that

select jsonb_path_query(content, 'strict $.** ? (exists(@.text))')->>'text' from documents;

produces nice rows of plain text content (as opposed to jsonb-text).

We can now do a full text search of our documents:

select * from documents where
    jsonb_to_tsvector('english',
                      jsonb_path_query_array(content, 'strict $.**.text'),
                      '["string"]')
    @@ to_tsquery('belgian & beer');

Note that json(b)_to_tsvector actually generates a tsvector for every element in our json array and then concatenates these, this might not be what we want. Results do look a bit different, with our original content:

to_tsvector('english', jsonb_path_query_array(content, 'strict $.**.text')::text)

gives

'content':10 'document':5 'format':11 'general':20 'get':18 'hope':16 'idea':21 'might':13 'show':8 'test':4

and

jsonb_to_tsvector('english', jsonb_path_query_array(content, 'strict $.**.text'), '["string"]')

gives

'content':11 'document':5 'format':12 'general':22 'get':20 'hope':18 'idea':23 'might':14 'show':9 'test':4

A quick check does not show any difference in performance.

To speed things up and allow for split second full rich text search on millions of records create a functional index:

CREATE INDEX ON paper USING gin(jsonb_to_tsvector('english', jsonb_path_query_array(content, 'strict $.**.text'), '["string"]'));

Note that you need to exactly reproduce the expression used in your index in your query to make use of this.

And that's that!

If you want to know more or have any other questions, please get in touch with us via info@bonaroo.nl

Undo typing

Toby Hinloopen — Thu, 06 Feb 2020 12:37:12 +0000

Did you drop your phone again while typing? You cannot undo dropping your phone, but at least you can undo the typing you’ve done on your phone before breaking the back glass.

But what actually happens in the DOM when you undo typing?

Test setup

Let's create a textarea and capture all events to see what events are actually triggered on undo. Creating a textarea is easy enough, but how do we capture all events? There isn't a addEventListener("*") for any event.

After some googling, some suggest iterating over all properties that start with on: onkeydown, onkeyup, etc. Let's try that!

const textarea = document.createElement("textarea");
document.body.appendChild(textarea);
const pre = document.createElement("pre");
document.body.appendChild(pre);
const eventNames = [];
for (const key in textarea) {
  if (key.substr(0, 2) === "on") {
    eventNames.push(key.substr(2));
  }
}
console.log(eventNames); /* ["copy", "cut", "paste", "abort", "blur", "cancel", "canplay",
  "canplaythrough", "change", "click", "close", "contextmenu", "cuechange",
  "dblclick", "drag", "dragend", "dragenter", "dragleave", "dragover", "dragstart",
  "drop", "durationchange", "emptied", "ended", "error", "focus", "input", "invalid",
  "keydown", "keypress", "keyup", "load", "loadeddata", "loadedmetadata", "loadstart",
  "mousedown", "mouseenter", "mouseleave", "mousemove", "mouseout", "mouseover", "mouseup",
  "mousewheel", "pause", "play", "playing", "progress", "ratechange", "reset", "resize",
  "scroll", "seeked", "seeking", "select", "stalled", "submit", "suspend", "timeupdate",
  "toggle", "volumechange", "waiting", "wheel", "auxclick", "gotpointercapture",
  "lostpointercapture", "pointerdown", "pointermove", "pointerup", "pointercancel",
  "pointerover", "pointerout", "pointerenter", "pointerleave", "selectstart", "selectionchange",
  "animationend", "animationiteration", "animationstart", "transitionend", "formdata",
  "pointerrawupdate", "beforecopy", "beforecut", "beforepaste", "search", "fullscreenchange",
  "fullscreenerror", "webkitfullscreenchange", "webkitfullscreenerror"] */

const eventCounts = {};
for (const eventName of eventNames) {
  textarea.addEventListener(eventName, (event) => {
    if (!eventCounts[eventName]) {
      eventCounts[eventName] = 0;
    }
    console.log(eventName, event);
    eventCounts[eventName]++;
    pre.textContent = JSON.stringify(eventCounts, null, 2);
  });
}

That's a LOT of events. I never actually realised there were so many!

Testing this in my desktop browser, there seems to be an input event I didn't know about. The input event is triggered (together with keyboard events) when using Ctrl + Z. Even when using the Cut feature from the Edit toolbar of the browser (I mean, who does that?), an input event is triggered, together with a cut event.

Now let's try the undo feature on my phone: I enter some text, shake the phone, and press undo. Only an input event is triggered!

So what's this InputEvent? Let's see what Mozzila has to say:

Looks like there is all kinds of interesting data on the InputEvent interface.

Most notably: event.inputType is "historyUndo". Is this even a documented standard somewhere? Looking at InputEvent's inputType specification at w3c, there is an editor's draft containing a list of inputType values:

https://w3c.github.io/input-events/#overview

Both "historyUndo" and "historyRedo" are there. Cool.

So there's that: When you undo typing on your browser, a special input event is triggered, with inputType = "historyUndo". Now you know.

Thank you for reading! And if you want to know more or have any other questions, please get in touch with us via info@bonaroo.nl

Creating & testing a TOTP plugin (commonly used for 2-Factor Authentication) for express: Creating the package

Toby Hinloopen — Fri, 24 Jan 2020 18:06:27 +0000

This blog post focuses on the creation of a express.js middleware package, using the Creating a TS-written NPM package for usage in Node-JS or Browser: The Long Guide as a basis. I'll use that guide up to Configuring GIT to generate a basic Typescript project. From that point, we'll continue below.

Goal of the package

The package we'll be writing will be providing a middleware to easily add TOTP (Time-based One-Time Password, the 6-digit codes generated by your Authenticator app), commonly used for Two-Factor Authentication, to any express project.

The exact package structure is yet to be determined, but I'd imagine it will look like the snippet below. In this example, all options are included. In our final package, all options are optional: The library should function without any configuration.

import * as express from "express";
import { initTotp, ensureTotpVerified } from "bonaroo-totp-express";

const app = express();

// Configure our library, storing the settings in the express app.
// Additionally, will check every request whether TOTP was verified.
app.use(initTotp({
  storage: "signed-cookie", // or "session"
  property: "totpVerified",
  setup: {
    // path of setup page
    path: "/user/totp-setup",
    // view to render setup page
    view: "/user/totp-setup",
    // assigns to `res.locals` when rendering the setup page
    locals: {
      totpSecret: "totpSecret",
      totpPeriod: "totpPeriod",
      totpQrCodeSvg: "totpQrCodeSvg",
      totpErrorCode: "totpErrorCode",
      totpFormTarget: "totpFormTarget",
      totpFormMethod: "totpFormMethod",
      totpCodeInputName: "totpCodeInputName",
    },
  },
  verify: {
    // path of the verify page
    path: "/user/totp-verify",
    // view to render verify page
    view: "/user/totp-verify",
    // assigns to `res.locals` when rendering the verify page
    locals: {
      totpFormTarget: "totpFormTarget",
      totpFormMethod: "totpFormMethod",
      totpCodeInputName: "totpCodeInputName",
      totpErrorCode: "totpErrorCode",
    },
  },
  getIssuer = (req) => "Sample Issuer",
  getUserAccountName = (req) => req.user ? req.user.name : null,
  getUserTotpSecret = (req) => req.user ? req.user.totpSecret : null,
  async setUserTotpSecret(req: express.Request, totpSecret: string) {
    req.user.totpSecret = totpSecret;
    await req.user.save();
  },
}));

app.get("/sign-in", /* */);
app.get("/sign-out", /* */);

// For all routes below, require totp verification.
app.use(ensureTotpVerified());

app.get("/user", /* */);
app.get("/admin", /* */);

Goal of this post

I'll be writing the progress of this library to demonstrate the flow of writing an actual working NodeJS package, including comprehensive tests and configuration options for package users.

This post will also include trouble-shooting and mistakes. Therefore, this post will be incredibly long.

About TOTP: Time-based One Time Password

HTOP: (HMAC-based One-Time Password) is an algorithm where a stream of passwords is generated using some kind of hashing method, a secret, and a counter.

This secret is shared between the authenticator and the authenticated, allowing the authenticated to generate passwords derived from the secret to confirm their identity with the authenticator. The authenticator checks the password against a configurable window of passwords generated from the same counter.

This does require both the authenticator and the authenticated to synchronise their counters somehow. One could, for example, increment the authenticated counter by 1 everytime a password is generated by the authenticated, and increment the authenticator counter by 1 when a password is used used at the authenticator.

To automate the counter synchronisation process, one could use time as a counter: Time-based One-Time Password is born. Both parties increment their counter every n seconds (usually 30), starting at some point in time. The authenticated generates a password, and the authenticator checks the password against a small window, usually about up to one minute earlier and later.

Because the passwords are 6 digits, and are valid for 30 seconds up to a few minutes, brute-forcing the passwords is a real danger. Therefore, persistent throttling is required for a secure implementation. The authenticator should lock-out a user with an increasing timeout after receiving invalid codes.

Step 1: Creating and testing a middleware that always assigns `totpVerified=false`.

Because I use TDD to write my code, I start writing a test first. Because we're writing a middleware that tightly integrates with express.js, we'll need a way to test express applications.

I'll be using a library called Supertest to test our express application.

$ npm i -D supertest @types/supertest
npm WARN deprecated superagent@3.8.3: Please note that v5.0.1+ of superagent removes User-Agent header by default, therefore you may need to add it yourself (e.g. GitHub blocks requests without a User-Agent header).  This notice will go away with v5.0.2+ once it is released.
npm WARN bonaroo-totp-express@1.0.0 No description

+ @types/supertest@2.0.8
+ supertest@4.0.2
added 10 packages from 62 contributors and audited 876631 packages in 4.896s
found 0 vulnerabilities

Now, let's create an express app in our tests and test it using supertest.

test/index.spec.ts

import { hello } from "../src";
import * as express from "express";
import supertest = require("supertest");

test("hello", () => {
  expect(hello("foo")).toEqual("Hello foo");
});

test("supertest", () => {
  const app = express();

  app.get("/", (req, res) => res.type("txt").send("Hi"));

  return supertest(app)
    .get("/")
    .expect(200, "Hi")
    .expect("Content-Type", /text/);
});

$ npm t

> bonaroo-totp-express@1.0.0 test /Users/hinloopen/Projects/Github/bonaroo-totp-express
> jest

 PASS  test/index.spec.ts
  ✓ hello (2ms)
  ✓ supertest (15ms)

Test Suites: 1 passed, 1 total
Tests:       2 passed, 2 total
Snapshots:   0 total
Time:        0.626s, estimated 1s
Ran all test suites.

That works. Now let's write a test that expects req.totpVerified to equal false.

Writing a test to test the middleware of an express app is a bit tricky, since the user of the middleware uses it inside request handlers. Therefore, you're interested in testing the behavour of the middleware inside a request handler of an express app.

To allow testing our middleware, we want to run some assertions inside a request handler while a request is being handled. Therefore, we must do the following:

create an express server
create and add our to-be-tested middleware to the server
add a request handler to actually check the behavior of the middleware
invoke the server to execute the middleware and
check the result of the assertions inside the request handler

Let's start with writing a bare middleware that can be added to an express app, so we can implement the middleware later.

src/totpInit.ts

export function totpInit(options: any) {
  return (req: Request, res: Response, next: NextFunction) => {
    next();
  }
}

With our new middleware, let's setup a (failing) test that can assert the state of a request after the middleware is executed.

test/totpInit.spec.ts

import { totpInit } from "../src/totpInit";
import * as express from "express";
import supertest = require("supertest");

test("totpInit assigns req.totpVerified to false", (done) => {
  const app = express();

  // add our middleware
  app.use(totpInit({}));

  // test our middleware
  app.get("/", (req, res) => {
    expect((req as any).totpVerified).toEqual(false);
    res.type("txt").send("Hi");
  });

  // catch errors & invoke
  app.use((error, req, res, next) => { done(error); next(error); });
  supertest(app).get("/").expect(200).then(() => done(), done);
});

This test explained: What are these done's?

This test is asynchronous, using callbacks. When the function passed to test() returns, the test is not yet completed. To let Jest know this is an async test, we can either return a promise or accept a callback function in the test callback, here called done.

This done callback needs to be called at least once, with either no arguments (indicating a success), or one argument (usually an error) indicating a failure.
app.use((error, req, res, next) => { done(error); next(error); });
This line registers an express error handler: Any unhandled error thrown in any express request handler or middleware is captured by this error handler.
When our expect fails, an error is thrown. This error is caught with this express error handler. We call Jest's done with the error to let Jest know our test finished with error. After that, we let express also know about the error so it can render a 500 response.
supertest(app).get("/").expect(200)
This line starts and calls the express server and expects a 200 response. A Promise is returned. The promise either resolves to the request's response, or any error.
.then(() => done(), done);
.then accepts up to 2 callbacks: The first is the resolve callback, invoked with a successful response. The second is the error callback, invoked with an error.

When the response was successful, we want to let Jest know the test finished without error. We invoke () => done() to call done without any argument (because passing any argument causes Jest to consider it a failure).

When the response was not successful, we want to let Jest know the test failed with error, and pass the error to jest. You could use (err) => done(err) as an error callback, but since done accepts an error as first argument, and the callback is also invoked with an error as first argument, we can just use done as an error callback.

$ npm t

> bonaroo-totp-express@1.0.0 test /Users/hinloopen/Projects/Github/bonaroo-totp-express
> jest

 PASS  test/index.spec.ts
 FAIL  test/totpInit.spec.ts
  ● totpInit assigns req.totpVerified to false

    expect(received).toEqual(expected) // deep equality

    Expected: false
    Received: undefined

       9 |
      10 |   app.get("/", (req, res) => {
    > 11 |     expect((req as any).totpVerified).toEqual(false);
         |                                       ^
      12 |     res.type("txt").send("Hi");
      13 |   });
      14 |

      at test/totpInit.spec.ts:11:39
      at Layer.handle [as handle_request] (node_modules/express/lib/router/layer.js:95:5)
      at next (node_modules/express/lib/router/route.js:137:13)
      at Route.dispatch (node_modules/express/lib/router/route.js:112:3)
      at Layer.handle [as handle_request] (node_modules/express/lib/router/layer.js:95:5)
      at node_modules/express/lib/router/index.js:281:22
      at Function.process_params (node_modules/express/lib/router/index.js:335:12)
      at next (node_modules/express/lib/router/index.js:275:10)
      at src/totpInit.ts:5:5
      at Layer.handle [as handle_request] (node_modules/express/lib/router/layer.js:95:5)

Test Suites: 1 failed, 1 passed, 2 total
Tests:       1 failed, 2 passed, 3 total
Snapshots:   0 total
Time:        0.9s, estimated 2s
Ran all test suites.
npm ERR! Test failed.  See above for more details.

Yep. That's a failing test. Now let's make it succeed.

src/totpInit.ts

import { Request, Response, NextFunction } from "express";

export function totpInit(options: any) {
  return (req: Request, res: Response, next: NextFunction) => {
    req.totpVerified = false;
    next();
  }
}

O-oh: Typescript doesn't allow req.totpVerified to be assigned since it doesn't exist on express's Request type. Let's fix that:

src/types.d.ts

declare namespace Express {
  export interface Request {
    totpVerified?: boolean;
  }
}

Now we can remove our as any hack from the test as well:

test/totpInit.ts (partial)

  app.get("/", (req, res) => {
    expect(req.totpVerified).toEqual(false);
    res.type("txt").send("Hi");
  });

Step 2: Creating the Setup page

The TOTP setup page includes creating a secret, sharing this secret with the user, and letting the user confirm the secret is stored by letting the user generate a password using their app.

This page consists of 2 routes: One GET for the setup form, and one POST to submit the setup, either rendering a failure or redirecting after success.

To render the setup page, we'll need a TOTP secret, and a handy QR code containing a special URL containing the secret.

For our library, we want to provide a default view, but we also want to allow the user to only generate the secret and QR code in a middleware before rendering their own view.

Let's first create this middleware. We can expose this middleware to our library user, and use it ourself in our library's default view.

This middleware will assign the secret and QR code in locals named totpSecret and totpQrCodeUrl respectively.

Let's start with the test:

test/totpSetup.spec.ts

import { totpSetup } from "../src/totpSetup";
import * as express from "express";
import supertest = require("supertest");
import url = require("url");

test("totpSetup assigns a secret and qr code", (done) => {
  const app = express();

  app.get("/", totpSetup(), (req, res) => {
    expect(typeof res.locals.totpSecret).toEqual("string");
    expect(typeof res.locals.totpQrCodeUrl).toEqual("string");
    res.type("txt").send("Hi");
  });

  app.use((error, req, res, next) => { done(error); next(error); });
  supertest(app).get("/").expect(200).then(() => done(), done);
});

Since the secret is a random value, you cannot really test the value. You can, however, check whether it's a string.

Next we check the QR Code URL. There is no real easy way to confirm the QR code's content is valid. Let's just assert it's a string again.

The test obviously fails with compile errors: totpSetup doesn't exist.

Let's fix the compile errors first by creating a bare middleware named totpSetup.

src/totpSetup.ts

import { Request, Response, NextFunction } from "express";

export function totpSetup() {
  return (req: Request, res: Response, next: NextFunction) => {
    next();
  }
}

 FAIL  test/totpSetup.spec.ts
  ✕ totpSetup assigns a secret and qr code (14ms)

  ● totpSetup assigns a secret and qr code

    expect(received).toEqual(expected) // deep equality

    Expected: "string"
    Received: "undefined"

       8 |
       9 |   app.get("/", totpSetup(), (req, res) => {
    > 10 |     expect(typeof res.locals.totpSecret).toEqual("string");
         |                                          ^
      11 |     expect(typeof res.locals.totpQrCodeUrl).toEqual("string");
      12 |     res.type("txt").send("Hi");
      13 |   });

totpSetup now exists, removing the comile errors. Now we get expectation failed errors, because totpSetup doesn't actually do anything.

To implement totpSetup, let's create the TOTP secret & QR code. There are many packages that can help us generate & verify TOTP secrets. Let's find some by searching for TOTP in the npm search.

Sorting by popularity, totp and speakeasy are on top. Totp looks like a poorly documented port with little weekly downloads - I suspect it's only at top because it's an exact match.

The next thing is a library called speakeasy. I never used speakeasy, but given the 39K weekly downloads I'll assume it's alright. Let's see if it is actually something we're looking for.

After a quick peek at the readme, it looks like generating secrets & verifying tokens is present in the package, and it looks easy to use. It's well documented and even has examples for creating a QR code.

Let's install it, and use it. The docs mention using qrcode to generate QR codes, so let's grab that one as well.

npm i speakeasy qrcode && npm i -D @types/speakeasy @types/qrcode

src/totpSetup.ts

import { Request, Response, NextFunction } from "express";
import { generateSecret } from "speakeasy";
import { toDataURL } from "qrcode";

export function totpSetup() {
  return (req: Request, res: Response, next: NextFunction) => {
    const secret = generateSecret();
    res.locals.totpSecret = secret.base32;
    toDataURL(secret.otpauth_url, (err, url) => {
      res.locals.totpQrCodeUrl = url;
      next();
    });
  }
}

The test succeeds, but that does not mean our code actually works. The test only checks whether the secret and URL are strings.

We're not sure whether the QR code is actually correct, or whether it's even an URL. I'm also not sure how we could test that.

We could somehow scan the image using a QR code parser, parsing the URL, and use the URL to generate a code. Maybe we could write an integration test for this later.

I'm also not sure whether the secret is actually a real secret. To verify the secret, we somehow need to verify every request a unique, random secret is generated. Testing whether something returns a random thing is pretty hard.

Let's first check whether the QR code actually resolves to an URL, and whether that URL is a valid OTP url.

Testing the QR code

To parse the QR code, we're installing a QR code reader package in development. The QR code reader package I picked at random requires jimp to read the image data. Fine.

npm i -D qrcode-reader jimp

It seems jimp doesn't play nice with Typescript without changes. Jimp requires allowSyntheticDefaultImports to be set to true according to their documentation. However, even with this setting enabled, I still either get runtime errors (cannot call read on Jimp) or compiler errors (property read does not exists on Jimp). I'm not interested to find out why, so let's convert its type to any before using it.

Jimp allows reading files or buffers, but it cannot read base64 data URLs. Therefore, I'm asserting the URL is a data url, and I'm converting the base64 data to a buffer.

Because reading the image dataurl and reading the QR code is a messy bit of code with type-hacks and mixing callbacks and promises, I'm hiding it in a function somewhere, never to be seen again.

test/support/expectAndParseQrCode.ts

import * as Jimp from "jimp";
import * as QrCode from "qrcode-reader";

export async function expectAndParseQrCodeDataUrl(maybeDataUrl: any): Promise<string> {
  expect(typeof maybeDataUrl).toEqual("string");
  const dataUrl = maybeDataUrl as string;
  expect(dataUrl).toMatch(/^data:image\/png;base64,/);
  const buffer = Buffer.from(dataUrl.substr("data:image/png;base64,".length), "base64");
  const image = await (Jimp as any).read(buffer);

  return new Promise((resolve, reject) => {
    const qr = new QrCode();
    qr.callback = (error, value) => error ? reject(error) : resolve(value.result);
    qr.decode(image.bitmap);
  });
}

We'll be creating more of these support functions. Let's export them all from a single file to make importing them easier.

test/support/index.ts

export * from "./expectAndParseQrCode";

Now that we can parse our QR code, we want to assert the parsed result is a valid URL. After a quick Google search, I found a small article about the structure of the URL.

It looks like the scheme must be otpauth:// and the secret is stored in a secret querystring param. The secret value must be base32 encoded.

The URL host must must be either hotp or totp (totp in our case), followed by the label or name of the account. Example URL:

otpauth://totp/ACME%20Co:john.doe@email.com?secret=HXDMVJECJJWSRB3HWIZR4IFUGFTMXBOZ&issuer=ACME%20Co&algorithm=SHA1&digits=6&period=30

Let's finish our test to check whether the URL is looks like an otpauth url as described by Google's docs.

test/totpSetup.spec.ts (additions)

import { expectAndParseQrCodeDataUrl } from "./support/expectAndParseQrCode";
import { parse } from "url";

test("totpSetup's totpQrCodeUrl is a QR code containing an OTP auth url", (done) => {
  const app = express();

  app.get("/", totpSetup(), async (req, res, next) => {
    try {
      const value = await expectAndParseQrCodeDataUrl(res.locals.totpQrCodeUrl);
      const url = parse(value, true);
      expect(url).toMatchObject({
        protocol: "otpauth:",
        slashes: true,
        host: "totp",
        query: { secret: res.locals.totpSecret },
      });
      res.type("txt").send("Hi");
    } catch (error) {
      next(error);
    }
  });

  app.use((error, req, res, next) => { done(error); next(error); });
  supertest(app).get("/").expect(200).then(() => done(), done);
});

Great. That works.

Refactoring our tests

I really don't like that const app = express(); app.use; supertest(app) pattern in every test. Let's attempt to extract that inside a support function.

test/support/testMiddleware.ts

import * as express from "express";
import { Request, Response, NextFunction } from "express";
import supertest = require("supertest");

export function testMiddleware(
  middleware: (req: Request, res: Response, next: NextFunction) => void,
  test: (req: Request, res: Response) => Promise<void>,
): Promise<supertest.Response> {
  const app = express();
  app.get("/", middleware, async (req, res, next) => {
    try {
      await test(req, res);
      res.type("txt").send("Hi");
    } catch (error) {
      next(error);
    }
  });
  return new Promise((resolve, reject) => {
    app.use((error, req, res, next) => { reject(error); next(error); });
    supertest(app).get("/").expect(200).then(resolve, reject);
  });
}

Now I can use this new function in my tests that use a similar pattern, rewriting all tests for totpSetup and totpInit.

test/totpSetup.spec.ts

import { totpSetup } from "../src/totpSetup";
import { expectAndParseQrCodeDataUrl } from "./support/expectAndParseQrCode";
import { parse } from "url";
import { testMiddleware } from "./support";

test("totpSetup assigns a secret and qr code",
() => testMiddleware(totpSetup(), async (req, res) => {
  expect(typeof res.locals.totpSecret).toEqual("string");
  expect(typeof res.locals.totpQrCodeUrl).toEqual("string");
}));

test("totpSetup's totpQrCodeUrl is a QR code containing an OTP auth url",
() => testMiddleware(totpSetup(), async (req, res) => {
  const value = await expectAndParseQrCodeDataUrl(res.locals.totpQrCodeUrl);
  const url = parse(value, true);
  expect(url).toMatchObject({
    protocol: "otpauth:",
    slashes: true,
    host: "totp",
    query: { secret: res.locals.totpSecret, },
  });
}));

test/totpInit.spec.ts

import { totpInit } from "../src/totpInit";
import { testMiddleware } from "./support";

test("totpInit assigns req.totpVerified to false",
() => testMiddleware(totpInit({}), async (req, res) => {
  expect((req as any).totpVerified).toEqual(false);
}));

I'm not perfectly sure whether this is an improvement or not. Readability is still pretty poor, but at least the boilerplate is gone.

Handling the setup POST

So far, totpSetup only included locals for rendering the form. Next, we need to handle submissions. The form submission should include:

The generated secret included in the QR code
An example code to verify the user setup their app correctly

Let's call these totpSecret and totpToken. We expect these values to be submitted via POST. I want to create a separate middleware for handling the setup POST request, and I'll call it totpSetupSubmit. I also want a middleware that can be used to handle both the GET (to render the form) and POST (to submit the form). Finally, I want to an isolated middleware for only rendering the form.

By creating 2 middlewares, we keep the responsibilities isolated and the tests simple. By creating a combined middleware, we allow the user to use our library with less code.

First, let's rename our totpSetup middleware to totpSetupForm, since it's only responsibility is to populate the form. Using my editor's Replace All function, I replace totpSetup with totpSetupForm, and I finish my remame by renaming the files of the test and source.

Next, let's create the tests for totpSetupSubmit before implementing it. I like to write empty tests with descriptions to define the features of the thing I'll be testing. For totpSetupSubmit, I want to test the following:

When submitted with secret and valid token, store the secret using totpInit's setUserTotpSecret.
When submitted without secret, add SECRET_REQUIRED to totpErrorCodes.
When submitted without token, add TOKEN_REQUIRED to totpErrorCodes.
When submitted with secret but random token, add TOKEN_VERIFY_FAILURE to totpErrorCodes.
When submitted with secret but old token, add TOKEN_VERIFY_FAILURE to totpErrorCodes.

Let's convert these to tests. I'll with the easy ones: The REQUIRED-errors. Because we cannot use testMiddleware in it's current state, let's first write our test "the old" way.

test/totpSetupSubmit.spec.ts

import * as express from "express";
import supertest = require("supertest");
import { totpInit } from "../src/totpInit";
import { totpSetupSubmit } from "../src/totpSetupSubmit";

test("totpSetupSubmit() without secret, add SECRET_REQUIRED to totpErrorCodes", async () => {
  const app = express();
  app.post("/", totpInit({}), totpSetupSubmit(), async (req, res, next) => {
    try {
      expect(res.locals).toMatchObject({
        totpErrorCodes: ["SECRET_REQUIRED"],
      });
      res.type("txt").send("Hi");
    } catch (error) {
      next(error);
    }
  });
  return new Promise((resolve, reject) => {
    app.use((error, req, res, next) => { reject(error); next(error); });
    supertest(app)
      .post("/")
      .send({ token: "123456", secret: "" })
      .expect(200)
      .then(resolve, reject);
  });
});

Now let's make the test succeed in the laziest way possible, so we can focus on improving our test code.

src/totpSetupSubmit.ts

import { NextFunction, Request, Response } from "express";

export function totpSetupSubmit() {
  return (req: Request, res: Response, next: NextFunction) => {
    res.locals.totpErrorCodes = ["SECRET_REQUIRED"];
    next();
  };
}

Great. Now we can figure out how to change testMiddleware to make it compatible with our POST request. Maybe add an options hash? We also need to be able to include multiple middleware functions, so let's make it accept an array.

To read the POST request body, we'll need body-parser as well. Let's install it.

npm i -D body-parser @types/body-parser

test/support/testMiddleware.ts

import * as express from "express";
import { Request, Response, NextFunction } from "express";
import supertest = require("supertest");
import bodyParser = require("body-parser");

/**
 * Test an express middleware by creating an express app & sending a request to
 * the app. The `test`-callback is invoked inside a request handler handling the
 * request.
 * @param middleware The middleware to add to the request app
 * @param test A function to test the request and/or response with inside a
 * request handler.
 * @param options
 */
export function testMiddleware(
  middleware: testMiddleware.Middleware|testMiddleware.Middleware[],
  test: (req: Request, res: Response) => Promise<void>,
  options: Partial<testMiddleware.Options> = {},
): Promise<supertest.Response> {

  const { path, method, data } = { ...testMiddleware.DEFAULT_OPTIONS, ...options };
  const app = express();

  app.use(bodyParser.json());
  app.use(bodyParser.urlencoded({ extended: true }));
  app.use(...(Array.isArray(middleware) ? middleware : [middleware]));

  // Create a request handler to test the middleware with.
  // If the test didn't fail, render a 200 OK response.
  app[method](path, async (req, res, next) => {
    try {
      await test(req, res);
      res.type("txt").send("Hi");
    } catch (error) {
      next(error);
    }
  });

  return new Promise((resolve, reject) => {
    // Register an error handler to capture errors in the request handler.
    // If an error is caught, reject the promise to indicate a test failure.
    // To ensure the express request will finish, we'll also pass the same error
    // back to express.
    app.use((error, req, res, next) => { reject(error); next(error); });

    // Invoke the request handler by sending a request to the express app.
    supertest(app)[method](path).send(data).expect(200).then(resolve, reject);
  });
}

export namespace testMiddleware {
  export type Middleware = (req: Request, res: Response, next: NextFunction) => void;

  export const DEFAULT_OPTIONS: Options = {
    path: "/",
    method: "get",
    data: {},
  }

  export interface Options {
    path: string;
    method: "get" | "post";
    data: string | object;
  }
}

Because readability of this function is pretty bad, I did attempt to improve it with some comments describing what is going on.

test/totpSetupSubmit.spec.ts

import { totpInit } from "../src/totpInit";
import { totpSetupSubmit } from "../src/totpSetupSubmit";
import { testMiddleware } from "./support/testMiddleware";

test("totpSetupSubmit() without secret, add SECRET_REQUIRED to totpErrorCodes", async () => {
  await testMiddleware([totpInit({}), totpSetupSubmit()], async (req, res) => {
    expect(res.locals).toMatchObject({ totpErrorCodes: ["SECRET_REQUIRED"] })
  }, { method: "post", data: { token: "123456", secret: "" } });
});

While not the most readable, it is shorter and we don't repeat ourselves in future tests. Let's add the second test. With our second test, we test without code but with secret. I generated a secret to use in my test using the speakeasy library.

test/totpSetupSubmit.spec.ts (addition)

test("totpSetupSubmit() without token, add TOKEN_REQUIRED to totpErrorCodes", async () => {
  await testMiddleware([totpInit({}), totpSetupSubmit()], async (req, res) => {
    expect(res.locals).toMatchObject({ totpErrorCodes: ["TOKEN_REQUIRED"] })
  }, { method: "post", data: { token: "", secret: "ORRSC22KLVQUWW3HKRKEU6BDPVITSLBSIMSE4S3FPJZFCXRDJJTA" } });
});

To make this test succeed, we're going to need some actual logic in our totpSetupSubmit middleware. Let's check for the params.

src/totpSetupSubmit.ts

import { NextFunction, Request, Response } from "express";

export function totpSetupSubmit() {
  return (req: Request, res: Response, next: NextFunction) => {
    const totpErrorCodes: string[] = [];
    if (!req.body.token) {
      totpErrorCodes.push("TOKEN_REQUIRED");
    }
    if (!req.body.secret) {
      totpErrorCodes.push("SECRET_REQUIRED");
    }
    res.locals.totpErrorCodes = totpErrorCodes;
    next();
  };
}

Now let's actually verify the code. First, I'll write the tests for a success case, and 2 failure cases (invalid code and old token).

test/totpSetupSubmit.spec.ts (addition)

const SECRET = speakeasy.generateSecret().base32;
const TOKEN = speakeasy.totp({ encoding: "base32", secret: SECRET });
const OLD_TOKEN = speakeasy.totp({ encoding: "base32", secret: SECRET, time: Date.parse("2000-01-01") / 1000 });
const INVALID_TOKEN = "123456";

test("totpSetupSubmit() with secret but random token, add TOKEN_VERIFY_FAILURE to totpErrorCodes", async () => {
  await testMiddleware([totpInit({}), totpSetupSubmit()], async (req, res) => {
    expect(res.locals).toMatchObject({ totpErrorCodes: ["TOKEN_VERIFY_FAILURE"] })
  }, { method: "post", data: { token: INVALID_TOKEN, secret: SECRET } });
});

test("totpSetupSubmit() with secret but old token, add TOKEN_VERIFY_FAILURE to totpErrorCodes", async () => {
  await testMiddleware([totpInit({}), totpSetupSubmit()], async (req, res) => {
    expect(res.locals).toMatchObject({ totpErrorCodes: ["TOKEN_VERIFY_FAILURE"] })
  }, { method: "post", data: { token: OLD_TOKEN, secret: SECRET } });
});

test("totpSetupSubmit() with secret and valid token, no error is assigned", async () => {
  await testMiddleware([totpInit({}), totpSetupSubmit()], async (req, res) => {
    expect(res.locals).toMatchObject({ totpErrorCodes: [] })
  }, { method: "post", data: { token: TOKEN, secret: SECRET } });
});

And the implementation...

src/totpSetupSubmit.ts

import { NextFunction, Request, Response } from "express";
import * as speakeasy from "speakeasy";

export function totpSetupSubmit() {
  return (req: Request, res: Response, next: NextFunction) => {

    const totpErrorCodes: string[] = [];
    res.locals.totpErrorCodes = totpErrorCodes;

    const { token, secret } = req.body;
    if (!token) {
      totpErrorCodes.push("TOKEN_REQUIRED");
    }
    if (!secret) {
      totpErrorCodes.push("SECRET_REQUIRED");
    }

    if (totpErrorCodes.length > 0) {
      next();
      return;
    }

    const isVerifiedToken = speakeasy.totp.verify({
      secret: secret,
      encoding: "base32",
      token,
      window: 2
    });
    if (!isVerifiedToken) {
      totpErrorCodes.push("TOKEN_VERIFY_FAILURE");
    }

    next();
  };
}

Next, on success, I want to provide the user the chance to store the secret. There are multiple ways to implement this. I could assign something like res.locals.totpSetupSuccess=true, and let the user store the secret in a second middleware, or I could add a callback in the totpSetupSubmit function that is invoked on success.

Because storing the secret is essential, I think I'll stick with the callback implementation. I'll be adding an options-argument to totpSetupSubmit containing a callback that is invoked with the secret on any successful setup.

This callback is very likely to perform some I/O to store the secret, so I'll be sure to await the callback function. I'll call this callback setUserTotpSecret.

To start, I'll add 2 tests: One testing to ensure setUserTotpSecret is invoked on success, and one to ensure setUserTotpSecret is NOT invoked on failure.

test/totpSetupSubmit.spec.ts (partial)

test("totpSetupSubmit() invokes setUserTotpSecret on success", async () => {
  const setUserTotpSecret = jest.fn();
  let request: Request;
  await testMiddleware(
    [
      totpInit({}),
      totpSetupSubmit({ setUserTotpSecret }),
    ],
    async (req, res) => { request = req; },
    { method: "post", data: { token: TOKEN, secret: SECRET } },
  );
  expect(setUserTotpSecret).toHaveBeenCalledWith(request, SECRET);
});

test("totpSetupSubmit() does not invoke setUserTotpSecret on failure", async () => {
  const setUserTotpSecret = jest.fn();
  await testMiddleware(
    [
      totpInit({}),
      totpSetupSubmit({ setUserTotpSecret }),
    ],
    async (req, res) => {},
    { method: "post", data: { token: INVALID_TOKEN, secret: SECRET } },
  );
  expect(setUserTotpSecret).not.toHaveBeenCalled();
});

I'm still struggling to make the tests more readable. I've added some whitespace, maybe that helps?

src/totpSetupSubmit.ts (partial)

export function totpSetupSubmit(options: Partial<totpSetupSubmit.Options> = {}) {
  return async (req: Request, res: Response, next: NextFunction) => {
    try {
      const { setUserTotpSecret } = { ...totpSetupSubmit.DEFAULT_OPTIONS, ...options };

      // ...

      if (!isVerifiedToken) {
        totpErrorCodes.push("TOKEN_VERIFY_FAILURE");
        next();
        return;
      }

      await setUserTotpSecret(req, secret);

      next();
    } catch (error) {
      next(error);
    }
  };
}

export namespace totpSetupSubmit {
  export interface Options {
    setUserTotpSecret(req: Request, secret: string): Promise<void>|void;
  }

  export const DEFAULT_OPTIONS: Options = {
    setUserTotpSecret: () => {},
  };
}

If you want to use await inside a request handler, you'll have to capture the error manually: express ignores the returned promise. I've added try { } catch (error) { next(error); } to catch any async error and pass it through the next middleware.

To allow package users to either render a success or failure response, we'll also need to assign the result of the setup in either the request or response. My plan is to set the following properties:

totpSetupSuccess: Was the token verified and was the secret stored?
totpSecret: The BASE32 secret
totpVerified: Will be set to true on totpSetupSuccess.

I'll assign these both to req (with type-definition, for usage in other middleware) and res.locals (without type-definition, for usage in views).

Test & implementation below.

test/totpSetupSubmit.spec.ts (partial)

test("totpSetupSubmit() on success assigns res.locals. & req. totpVerified & totpSetupSuccess=true", () => testMiddleware(
  totpSetupSubmit({}),
  async (req, res) => {
    expect(res.locals.totpSetupSuccess).toEqual(true);
    expect(req.totpSetupSuccess).toEqual(true);
    expect(res.locals.totpVerified).toEqual(true);
    expect(req.totpVerified).toEqual(true);
  },
  { method: "post", data: { token: TOKEN, secret: SECRET } },
));

test("totpSetupSubmit() assigns res.locals. & req. totpVerified & totpSetupSuccess=false", () => testMiddleware(
  totpSetupSubmit({}),
  async (req, res) => {
    expect(res.locals.totpSetupSuccess).toEqual(false);
    expect(req.totpSetupSuccess).toEqual(false);
    expect(res.locals.totpVerified).toEqual(false);
    expect(req.totpVerified).toEqual(false);
  },
  { method: "post", data: { token: INVALID_TOKEN, secret: SECRET } },
));

test("totpSetupSubmit() assigns res.locals. & req.totpSecret", () => testMiddleware(
  totpSetupSubmit({}),
  async (req, res) => {
    expect(res.locals.totpSecret).toEqual(SECRET);
    expect(req.totpSecret).toEqual(SECRET);
  },
  { method: "post", data: { token: TOKEN, secret: SECRET } },
));

src/totpSetupSubmit.ts (partial)

export function totpSetupSubmit(options: Partial<totpSetupSubmit.Options> = {}) {
  return async (req: Request, res: Response, next: NextFunction) => {
    try {
      // ...

      const totpErrorCodes: string[] = [];
      res.locals.totpErrorCodes = totpErrorCodes;
      res.locals.totpVerified = req.totpVerified = false;
      res.locals.totpSetupSuccess = req.totpSetupSuccess = false;

      // ...

      if (!isVerifiedToken) {
        totpErrorCodes.push("TOKEN_VERIFY_FAILURE");
        next();
        return;
      }

      await setUserTotpSecret(req, secret);

      res.locals.totpVerified = req.totpVerified = true;
      res.locals.totpSetupSuccess = req.totpSetupSuccess = true;
      next();
    } catch(error) {
      // ...
    }
  };
}

The middleware is now usable. I'll add some nice-to-haves later.

Step 3: Verifying the token

Now let's focus on creating a middleware to verify the token. For setup, we had 2 middlewares: One for rendering the form, and one for submitting the form. For verification, we don't need any data to render a form. We only need to be able to submit it, hence we're only creating a middleware for submitting the form.

I'll call this middleware totpVerifySubmit.

To verify the token being submitted, we'll be needing a previously stored secret and a token from the request body. We'll ask the package user to provide the current user's secret using a getUserTotpSecret callback in an options hash. The token will be extracted from a param named token.

Let's start off with tests.

test/totpVerifySubmit.spec.ts

import { testMiddleware } from "./support/testMiddleware";
import * as speakeasy from "speakeasy";
import { totpVerifySubmit } from "../src/totpVerifySubmit";

const SECRET = speakeasy.generateSecret().base32;
const TOKEN = speakeasy.totp({ encoding: "base32", secret: SECRET });
const OLD_TOKEN = speakeasy.totp({ encoding: "base32", secret: SECRET, time: Date.parse("2000-01-01") / 1000 });
const INVALID_TOKEN = "123456";
const getUserTotpSecret = () => Promise.resolve(SECRET);

it("totpVerifySubmit() without token assigns TOKEN_REQUIRED to totpErrorCodes", () => testMiddleware(
  totpVerifySubmit({ getUserTotpSecret }),
  async (req, res) => {
    expect(res.locals).toMatchObject({ totpErrorCodes: ["TOKEN_REQUIRED"] });
    expect(req.totpVerified).toEqual(false);
  },
  { method: "post", data: { token: "" } }
));

it("totpVerifySubmit() with null-secret from getUserTotpSecret assigns SECRET_REQUIRED to totpErrorCodes", () => testMiddleware(
  totpVerifySubmit({ getUserTotpSecret: () => null }),
  async (req, res) => {
    expect(res.locals).toMatchObject({ totpErrorCodes: ["SECRET_REQUIRED"] });
    expect(req.totpVerified).toEqual(false);
  },
  { method: "post", data: { token: TOKEN } }
));

it("totpVerifySubmit() with blank-secret from getUserTotpSecretassigns SECRET_REQUIRED to totpErrorCodes", () => testMiddleware(
  totpVerifySubmit({ getUserTotpSecret: () => "" }),
  async (req, res) => {
    expect(res.locals).toMatchObject({ totpErrorCodes: ["SECRET_REQUIRED"] });
    expect(req.totpVerified).toEqual(false);
  },
  { method: "post", data: { token: TOKEN } }
));

it("totpVerifySubmit() with valid secret and token assigns totpVerified=true", () => testMiddleware(
  totpVerifySubmit({ getUserTotpSecret }),
  async (req, res) => {
    expect(res.locals).toMatchObject({ totpErrorCodes: [] });
    expect(req.totpVerified).toEqual(true);
  },
  { method: "post", data: { token: TOKEN } }
));

it("totpVerifySubmit() with random token token assigns TOKEN_VERIFY_FAILURE to totpErrorCodes", () => testMiddleware(
  totpVerifySubmit({ getUserTotpSecret }),
  async (req, res) => {
    expect(res.locals).toMatchObject({ totpErrorCodes: ["TOKEN_VERIFY_FAILURE"] });
    expect(req.totpVerified).toEqual(false);
  },
  { method: "post", data: { token: INVALID_TOKEN } }
));

it("totpVerifySubmit() with old token token assigns TOKEN_VERIFY_FAILURE to totpErrorCodes", () => testMiddleware(
  totpVerifySubmit({ getUserTotpSecret }),
  async (req, res) => {
    expect(res.locals).toMatchObject({ totpErrorCodes: ["TOKEN_VERIFY_FAILURE"] });
    expect(req.totpVerified).toEqual(false);
  },
  { method: "post", data: { token: OLD_TOKEN } }
));

Now let's implement totpVerifySubmit and make the tests succeed.

src/totpVerifySubmit.ts

import { NextFunction, Request, Response } from "express";
import * as speakeasy from "speakeasy";

export function totpVerifySubmit(options: Partial<totpVerifySubmit.Options> = {}) {
  return async (req: Request, res: Response, next: NextFunction) => {
    try {
      const { getUserTotpSecret } = { ...totpVerifySubmit.DEFAULT_OPTIONS, ...options };

      const totpErrorCodes: string[] = [];
      res.locals.totpErrorCodes = totpErrorCodes;
      res.locals.totpVerified = req.totpVerified = false;

      const { token } = req.body;
      const secret = await getUserTotpSecret(req);

      if (!token) {
        totpErrorCodes.push("TOKEN_REQUIRED");
      }
      if (!secret) {
        totpErrorCodes.push("SECRET_REQUIRED");
      }

      if (totpErrorCodes.length > 0) {
        next();
        return;
      }

      const isVerifiedToken = speakeasy.totp.verify({
        secret,
        encoding: "base32",
        token,
        window: 2,
      });

      if (!isVerifiedToken) {
        totpErrorCodes.push("TOKEN_VERIFY_FAILURE");
        next();
        return;
      }

      res.locals.totpVerified = req.totpVerified = true;

      next();
    } catch (error) {
      next(error);
    }
  };
}

export namespace totpVerifySubmit {
  export interface Options {
    getUserTotpSecret(req: Request): Promise<string>|string;
  }

  export const DEFAULT_OPTIONS: Options = {
    getUserTotpSecret: () => { throw new Error(`getUserTotpSecret not implemented`); },
  };
}

Now both the tests and the implementations for totpSetupSubmit and totpVerifySubmit are pretty similar. Let's see if we can extract some duplicate code.

Both middlewares assign totpVerified and totpErrorCodes, and both middlewares verify a token with a secret. Both extract options and have that try { } catch(error) { } pattern.

Because I'm writing a package with express middlewares, maybe I can extract the token verification part to its own middleware, but without the specifics of extracting the token or secret. This new middleware should be able to be used by both totpSetupSubmit and totpVerifySubmit.

To achieve this separation, both middlewares must first extract their secret and token in a way our new middleware can access it: Let's store it in req!

src/types.d.ts (partial)

declare namespace Express {
  export interface Request {
    // ...
    totpToken?: string;
    totpErrorCodes?: string[];
  }
}

Now let's move some stuff around: Let's create our new middleware. Because we're only refactoring our code without changing functionality, writing new tests isn't necessary. Instead, let's rely on the tests to verify our refactoring is done correctly.

While refactoring, I'll keep watch of npm t -- --watchAll, running all tests on every change.

Also, to make it easier to track our middleware's state, instead of using a list of variables all prefixed totp, I think it's better to use a single variable (both on req and res.locals) called totp, and store our package request-state in there.

src/TotpState.ts

export interface TotpState {
  verified?: boolean;
  secret?: string;
  setupSuccess?: boolean;
  token?: string;
  errorCodes?: string[];
}

Because sometimes our changes to Request weren't picked up, I've added them to both "express".Request and Express.Request.

src/types.d.ts

import { TotpState } from "./TotpState";

declare namespace Express {
  export interface Request {
    totp?: TotpState;
  }
}

declare module "express" {
  export interface Request {
    totp?: TotpState;
  }
}

Now refactor our assignments from req.totp* to req.totp.*, practially rewriting all tests and implementations.

To avoid having to ensure totp is set on req and res.locals, I'll ensure it's done once in totpInit. The changes from req.totpVerified to req.totp.verified (and all other assignments) are not included in this post, because they involve almost all files while the change is trivial.

src/totpInit.ts

import { NextFunction, Request, Response } from "express";

export function totpInit(options: any) {
  return (req: Request, res: Response, next: NextFunction) => {
    if (!req.totp) {
      res.locals.totp = req.totp = {};
    }
    req.totp.verified = false;
    next();
  };
}

With this change in totpInit, for any other middleware, I'll assume req.totp is always equal to res.locals.totp, and they are always set. To ensure totpInit is always invoked before my other middleware, I'll always add totpInit in all middleware tests.

test/totpSetupSubmit.spec.ts (partial)

test("totpSetupSubmit() without secret, add SECRET_REQUIRED to totpErrorCodes", () => testMiddleware(
  [totpInit({}), totpSetupSubmit()],
  async (req, res) => expect(req.totp).toMatchObject({ errorCodes: ["SECRET_REQUIRED"] }),
  { method: "post", data: { token: TOKEN, secret: "" } })
);

Now let's implement totpVerify and change totpVerifySubmit and totpSetupSubmit to be used with totpVerify.

src/totpVerify.ts

import { Request, Handler } from "express";
import * as speakeasy from "speakeasy";
import "./types";

export function totpVerify(): Handler {
  return (req: Request, res, next) => {
    const errorCodes = req.totp.errorCodes = [] as string[];
    const { token, secret } = req.totp;

    if (!token) {
      errorCodes.push("TOKEN_REQUIRED");
    }
    if (!secret) {
      errorCodes.push("SECRET_REQUIRED");
    }

    if (errorCodes.length > 0) {
      next();
      return;
    }

    req.totp.verified = speakeasy.totp.verify({
      secret,
      encoding: "base32",
      token,
      window: 2,
    });

    if (!req.totp.verified) {
      errorCodes.push("TOKEN_VERIFY_FAILURE");
    }

    next();
  }
}

Now I want to change totpVerifySubmit to only extract the secret and token, and use totpVerify to verify the token. For this, I want to change totpVerifySubmit to combine 2 middlewares: A middleware for extracting secret (from getUserTotpSecret) and token (from body), and the new totpVerify middleware.

Express can be used to combine multiple middlewares into a single middleware.

src/totpVerifySubmit.ts (partial)

import * as express from "express";
import { NextFunction, Request, Response } from "express";
import { totpVerify } from "./totpVerify";
import "./types";

export function totpVerifySubmit(options: Partial<totpVerifySubmit.Options> = {}) {
  const app = express();

  app.use(totpVerifySubmit.assignSecretAndTokenForVerification(options));
  app.use(totpVerify());

  return app;
}

export namespace totpVerifySubmit {
  export function assignSecretAndTokenForVerification(options: Partial<Options> = {}) {
    const { getUserTotpSecret } = { ...DEFAULT_OPTIONS, ...options };
    return async (req: Request, res: Response, next: NextFunction) => {
      try {
        const { token } = req.body;
        const secret = await getUserTotpSecret(req);

        req.totp.secret = secret;
        req.totp.token = token;

        next();
      } catch (error) {
        next(error);
      }
    }
  }

  // ...
}

src/totpSetupSubmit.ts (partial)

import * as express from "express";
import { NextFunction, Request, Response } from "express";
import "./types";
import { totpVerify } from "./totpVerify";

export function totpSetupSubmit(options: Partial<totpSetupSubmit.Options> = {}) {
  const app = express();

  app.use(totpSetupSubmit.extractTokenAndSecretForSubmission);
  app.use(totpVerify());
  app.use(totpSetupSubmit.completeSubmitAfterVerification(options));

  return app;
}

export namespace totpSetupSubmit {
  export function extractTokenAndSecretForSubmission(req: Request, res: Response, next: NextFunction) {
    const { token, secret } = req.body;
    req.totp.secret = secret;
    req.totp.token = token;
    next();
  }

  export function completeSubmitAfterVerification(options: Partial<Options> = {}) {
    const { setUserTotpSecret } = { ...DEFAULT_OPTIONS, ...options };
    return async (req: Request, res: Response, next: NextFunction) => {
      try {
        req.totp.setupSuccess = req.totp.verified;
        if (req.totp.setupSuccess) {
          await setUserTotpSecret(req, req.totp.secret!);
        }
        next();
      } catch (error) {
        next(error);
      }
    }
  }

  // ...
}

Both verify and setup still work, but the code is a bit cleaner in my opinion.

Now let's clean index.ts and export our middlewares.

src/index.ts

export * from "./ITotpState";
export * from "./totpInit";
export * from "./totpSetupForm";
export * from "./totpSetupSubmit";
export * from "./totpVerify";
export * from "./totpVerifySubmit";

Now, let's also test whether all middlewares are actually exported. That way, we can be sure our package users can actually use our middlewares.

test/index.spec.ts

import * as src from "../src";

for (const name of ["totpInit", "totpSetupForm", "totpSetupSubmit", "totpVerify", "totpVerifySubmit"]) {
  test(`middleware ${name}() is exported and a function`, () => {
    expect(src).toHaveProperty(name);
    expect(typeof src[name]).toEqual("function");
  });
}

Recap

To recap, we now have the following exports:

import { Handler } from "express";

/**
 * An interface representing the current TOTP state, present on every request.
 */
interface ITotpState {
  verified?: boolean;
  secret?: string;
  setupSuccess?: boolean;
  token?: string;
  qrCodeUrl?: string;
  errorCodes?: string[];
}

/**
 * Assign a blank ITotpState on `req.totp` and `res.locals.totp`.
 */
function totpInit(options?: any): Handler;

/**
 * Generate a new secret and QR code and assign them to `totp.secret` and
 * `totp.qrCodeUrl`.
 */
function totpSetupForm(): Handler;

/**
 * Extract the secret and token from the request body, verify the token against
 * the secret, and store the secret when verification was successful using
 * `setUserTotpSecret` in `options`.
 */
function totpSetupSubmit(options: Partial<totpSetupSubmit.IOptions>): Handler;

/**
 * Verify `totp.token` against `totp.secret` and assign `totp.verified` with
 * `true` on success, or `totp.errorCodes` with error codes on failure.
 */
function totpVerify(): Handler;

/**
 * Extract totp token from request body and verify it against a secret fetched
 * from `getUserTotpSecret` in `options`.
 *
 * Assign `totp.verified` with `true` on success, or `totp.errorCodes` with
 * error codes on failure.
 */
function totpVerifySubmit(): Handler;

Let's deploy it to NPM and test our package. I already wrote about publishing packages to NPM here, so I'll keep it short.

Deploying to NPM

tsconfig.json (partial)

{
  "compilerOptions": {
    // ...
    "declaration": true
  },
}

package.json (partial)

{
  "main": "dist/index.js",
  "files": ["dist"],
  "scripts": {
    "prepublishOnly": "npm run ci",
    // ...
  }
  // ...
}

$ npm publish --dry-run

> bonaroo-totp-express@1.0.0 prepublishOnly .
> npm run ci


> bonaroo-totp-express@1.0.0 ci /home/toby/bonaroo-totp-express
> npm run lint & npm run build & npm t & wait


> bonaroo-totp-express@1.0.0 lint /home/toby/bonaroo-totp-express
> tslint -c tslint.json -p tsconfig.json


> bonaroo-totp-express@1.0.0 build /home/toby/bonaroo-totp-express
> tsc


> bonaroo-totp-express@1.0.0 test /home/toby/bonaroo-totp-express
> jest

 PASS  test/index.spec.ts
 PASS  test/totpVerifySubmit.spec.ts
 PASS  test/totpSetupSubmit.spec.ts
 PASS  test/totpInit.spec.ts
 PASS  test/totpSetupForm.spec.ts

Test Suites: 5 passed, 5 total
Tests:       24 passed, 24 total
Snapshots:   0 total
Time:        2.056s
Ran all test suites.
npm notice
npm notice 📦  bonaroo-totp-express@1.0.0
npm notice === Tarball Contents ===
npm notice 358B  dist/index.js
npm notice 79B   dist/ITotpState.js
npm notice 388B  dist/totpInit.js
npm notice 497B  dist/totpSetup.js
npm notice 604B  dist/totpSetupForm.js
npm notice 5.4kB dist/totpSetupSubmit.js
npm notice 1.0kB dist/totpVerify.js
npm notice 5.1kB dist/totpVerifySubmit.js
npm notice 1.3kB package.json
npm notice 170B  dist/index.js.map
npm notice 112B  dist/ITotpState.js.map
npm notice 396B  dist/totpInit.js.map
npm notice 468B  dist/totpSetup.js.map
npm notice 487B  dist/totpSetupForm.js.map
npm notice 1.3kB dist/totpSetupSubmit.js.map
npm notice 842B  dist/totpVerify.js.map
npm notice 1.1kB dist/totpVerifySubmit.js.map
npm notice 23B   README.md
npm notice 192B  dist/index.d.ts
npm notice 897B  dist/ITotpState.d.ts
npm notice 167B  dist/totpInit.d.ts
npm notice 188B  dist/totpSetupForm.d.ts
npm notice 761B  dist/totpSetupSubmit.d.ts
npm notice 257B  dist/totpVerify.d.ts
npm notice 663B  dist/totpVerifySubmit.d.ts
npm notice === Tarball Details ===
npm notice name:          bonaroo-totp-express
npm notice version:       1.0.0
npm notice package size:  5.8 kB
npm notice unpacked size: 22.9 kB
npm notice shasum:        ea5fd4c4b0a1f77b514a2cd1bcf0fe912fd42df1
npm notice integrity:     sha512-o/I6Tfd19PP4B[...]FtW7UUZ3EGODQ==
npm notice total files:   25
npm notice
+ bonaroo-totp-express@1.0.0

LGTM.

npm publish

Thank you for reading! And if you want to know more or have any other questions, please get in touch with us via info@bonaroo.nl.

In my next blog post, I'll write an example project using the middlewares from my new package. After the example project, I'll add some default views and routes in the package to make it as easy as possible to integrate the package in express applications.

Creating a TS-written NPM package for use in Node-JS or Browser.

Toby Hinloopen — Wed, 27 Nov 2019 10:48:37 +0000

Creating a TS-written NPM package for use in Node-JS or Browser: The Long Guide

In this guide, I'll explain how to create an NPM package for NodeJS or the browser using Typescript without leaving built artifacts in your repository. At the end, my example library will be able to be included in any Javascript or Typescript project, including:

Imported as a script in a <script> tag, using either direct download or a free CDN service.
Installed in a client-side application using npm and a bundler like webpack.
Installed in a server-side NodeJS application using npm.

Furthermore, the whole build and publish process will be automated as much as possible, while keeping the repository free from builds.

For this post, I'll be using a tiny library I wrote as an example. The library itself is meaningless and not very useful, which makes it a fine distraction-free example for this guide.

The example library

The example library will be called bonaroo-able, only exporting a namespace called Able.

Able contains a small set of functions for managing a list of strings that act as abilities (permissions) for some user. This example library is written in Typescript and it has no browser- or NodeJS specific dependencies (EG it doesn't rely on the DOM or filesystem). More about this library later. For now, let's start with creating some config files.

The NPM package

First, we need a package.json. The package.json file contains details about your Javascript package, including the name, author and dependencies. You can read about package.json files in the npm docs.

To create a package.json file, we use npm. In your library folder, run npm init and follow the instructions. For this guide, I'll be using jest to test my library. We can just use jest as a test command: We'll be installing this dependency later.

The entry point is the file that will be included when our package is included in another project. To allow our package to be used in non-Typescript projects, this entry point must be a regular Javascript file.

This Javascript file must include all of our library. I like to have an index.js file that requires all our of library. Because this is a Typescript project, we will have separate Typescript and Javascript files. We keep these in src (written source) and dist (distributed files) folders.

We'll be writing a src/index.ts file importing all of our library, and use the Typescript compiler to generate a Javascript variant in dist/index.js. This dist/index.js will be our package's entry point. We'll configure the Typescript compiler later.

$ npm init
This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sensible defaults.

See `npm help json` for definitive documentation on these fields
and exactly what they do.

Use `npm install <pkg>` afterwards to install a package and
save it as a dependency in the package.json file.

Press ^C at any time to quit.
package name: (bonaroo-able)
version: (1.0.0)
description: A tiny library handling abilities
entry point: (index.js) dist/index.js
test command: jest
git repository: https://github.com/tobyhinloopen/bonaroo-able
keywords: Abilities, Permissions
author: Charper Bonaroo BV
license: (ISC) UNLICENSED
About to write to /home/toby/bonaroo-able//package.json:

{
  "name": "bonaroo-able",
  "version": "1.0.0",
  "description": "A tiny library handling abilities",
  "main": "dist/index.js",
  "scripts": {
    "test": "jest"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/tobyhinloopen/bonaroo-able.git"
  },
  "keywords": [
    "Abilities",
    "Permissions"
  ],
  "author": "Charper Bonaroo BV",
  "license": "UNLICENSED",
  "bugs": {
    "url": "https://github.com/tobyhinloopen/bonaroo-able/issues"
  },
  "homepage": "https://github.com/tobyhinloopen/bonaroo-able#readme"
}


Is this OK? (yes)
$

Next, we'll be needing some dependencies. Obviously you'll be needing Typescript. We'll also be installing jest, ts-jest and @types/jest.

npm i -D typescript jest ts-jest @types/jest

Configuring Typescript

Next, we need to configure Typescript. Let's create a minimal tsconfig.json file.

tsconfig.json

{
  "compilerOptions": {
    "outDir": "dist",
    "lib": ["es2016"],
    "sourceMap": true
  },
  "include": [
    "src/**/*.ts"
  ]
}

About these options

compilerOptions.outDir: Generate JS files in this directory.

compilerOptions.lib: Tell the Typescript compiler the es2016 standard functions are available, like Array.prototype.includes.

include: Ensure only the Typescript in src is compiled & ignore all other source files (like the tests)

Since you cannot invoke Node binaries directly in all environments, I like to add all my commonly used commands to npm scripts. Add "build": "tsc" to the scripts section in your package.json

package.json (partial)

  "scripts": {
    "build": "tsc",
    "test": "jest"
  },

To test whether everything is setup correctly, I like to create an entry point with a dummy function.

src/index.ts

export function hello(name: string): string {
  return `Hello ${name}`;
}

Let's attempt to built this:

$ npm run build

> bonaroo-able@1.0.0 build /home/toby/bonaroo-able
> tsc

$

No errors. That's great. Also, note that Typescript has created some Javascript files for us! If you take a look at dist/index.js, you'll see a Javascript variant of our Typescript file. My generated file looks like this:

dist/index.js (generated)

"use strict";
exports.__esModule = true;
function hello(name) {
    return "Hello " + name;
}
exports.hello = hello;

Note that all type information has been stripped, and the file has been changed to be compatible with older Javascript runtimes by changing the template string to a regular string with concat operator: "Hello " + name.

Writing a test

Now test our "library": Let's write a test!

I like to create tests in a test directory, with a filenames matching the src files. For example, to test src/Foo.ts, I put my tests in test/Foo.spec.ts.

test/index.spec.ts

import { hello } from "../src";

test("hello", () => {
  expect(hello("foo")).toEqual("Hello foo");
});

To be able to write our tests in Typescript, we need to configure jest first. We can generate a config file with ts-jest config:init.

$ node_modules/.bin/ts-jest config:init

Jest configuration written to "/home/toby/bonaroo-able/jest.config.js".
$

Now we're ready to confirm our test suite is working:

$ npm t

> bonaroo-able@1.0.0 test /home/toby/bonaroo-able
> jest

 PASS  test/index.spec.ts
  ✓ hello (2ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        1.267s, estimated 2s
Ran all test suites.
$

Configuring GIT

Before we continue, let's configure source control to persist our working setup.

To keep our git repository clean, we omit node_modules and dist from the git repository.

.gitignore

dist/
node_modules/

Now let's create a git repository. Replace the remote with your git repo.

git init
git add --all
git commit -m "Initial commit"
git remote add origin git@github.com:tobyhinloopen/bonaroo-able.git
git push -u origin master

Writing our Library

Now let's write the code for our library. Writing code is outside the scope of this guide. Here's an overview of my Able library. The filename points to the current version of the complete file on github.

src/Able.ts (overview, no function bodies)

export namespace Able {
  export type AbilitySet = string[];
  export interface GroupDefinition { [key: string]: AbilitySet; }
  export interface ValueMap { [key: string]: string|string[]; }

  export function flatten(definition: GroupDefinition, abilities: AbilitySet): AbilitySet;
  export function extractValues(abilities: AbilitySet): [ValueMap, AbilitySet];
  export function applyValues(abilities: AbilitySet, values: ValueMap): AbilitySet;
  export function resolve(definition: GroupDefinition, abilities: AbilitySet): AbilitySet;
  export function getMissingAbilities(abilities: AbilitySet, requiredAbilities: AbilitySet): AbilitySet;
  export function canAccess(appliedAbilities: AbilitySet, requiredAbilities: AbilitySet): boolean;
}

src/index.ts

import { Able } from "./Able";
export default Able;
Object.assign(module.exports, Able);

test/index.spec.ts (snippet, remaining tests removed)

import { Able } from "../src/Able";

describe("Able", () => {
  it("flatten() includes own name", () => {
    expect(Able.flatten({}, ["foo"])).toContain("foo");
  });

  // ...remaining tests...
});

test/Able.spec.ts

import Able from "../src";

test("Able is exported", () => {
  expect(Able).toBeInstanceOf(Object);
});

Testing our build

In some cases, our tests might succeed while our build fails, or the build is
somehow invalid. To ensure the build is working, I like to add a very crude test to confirm the build is working and the exports are in-place.

This test will build the code, and run a simple JS file using the build to confirm the build is working.

In this build test, we copy one of our test suite's tests. I think it is safe to assume that if one test actually using the library succeeds, the library is built and exported correctly.

test-build.js

const assert = require("assert");
const Able = require("./dist");

const definition = { foo: ["bar"] };
const abilities = ["foo", "bam"];
const result = Able.flatten(definition, abilities).sort();
assert.deepStrictEqual(result, ["foo", "bar", "bam"].sort());

Note that we're importing ./dist here: We are explicitly importing dist/index.js that way. We need to build our code before we can import dist/index.js.

To build the code & run test-build.js, we'll add a script to package.json, called test-build.

package.json (partial)

  "scripts": {
    "build": "tsc",
    "test": "jest",
    "test-build": "npm run build && node test-build.js"
  },

I like to run all automated checks, currently npm t and npm run test-build, from a single script called ci. This script will run all automated checks and only pass when all automated checks passed.

Let's add ci to the scripts as well:

package.json (partial)

  "scripts": {
    "build": "tsc",
    "ci": "npm run test-build & npm t & wait",
    "test": "jest",
    "test-build": "npm run build && node test-build.js"
  },

This ci script will be used to verify our build every release. Let's try it!

$ npm run ci

> bonaroo-able@1.0.0 ci /home/toby/bonaroo-able/
> npm run test-build & npm t & wait


> bonaroo-able@1.0.0 test-build /home/toby/bonaroo-able/
> npm run build && node test-build.js


> bonaroo-able@1.0.0 test /home/toby/bonaroo-able/
> jest


> bonaroo-able@1.0.0 build /home/toby/bonaroo-able/
> tsc

 PASS  test/Able.spec.ts
 PASS  test/index.spec.ts

Test Suites: 2 passed, 2 total
Tests:       11 passed, 11 total
Snapshots:   0 total
Time:        1.816s
Ran all test suites.

Later we'll make sure to only accept changes in the master branch that have passed this npm run ci call. That way, we'll make sure the master branch always features a valid build.

Let's commit all of our changes to git and start deploying our library.

NPM release

The first and most useful release is the npm release. This allows our library users to npm i our library in most projects.

Both server-side projects and client-side projects that use a bundler like webpack can use an npm release without any changes.

Let's prepare our library for publication to NPM.

Preparing our package for release

Let's first define what files we actually want to include in our package. You can peek the contents of your package-to-be using npm publish --dry-run:

$ npm publish --dry-run
npm notice
npm notice 📦  bonaroo-able@1.0.0
npm notice === Tarball Contents ===
npm notice 862B  package.json
npm notice 56B   .git
npm notice 69B   jest.config.js
npm notice 284B  test-build.js
npm notice 114B  tsconfig.json
npm notice 3.9kB dist/Able.d.ts
npm notice 6.1kB dist/Able.js
npm notice 3.4kB dist/Able.js.map
npm notice 52B   dist/index.d.ts
npm notice 184B  dist/index.js
npm notice 198B  dist/index.js.map
npm notice 6.0kB src/Able.ts
npm notice 24B   src/index.ts
npm notice 3.4kB test/Able.spec.ts
npm notice 108B  test/index.spec.ts
npm notice === Tarball Details ===
...
+ bonaroo-able@1.0.0

This built includes all kinds of things the user wouldn't care about. With package.json's files property you can whitelist the files you want to include.

Only the built files are required to use our library: Let's add only the dist folder to the package:

package.json (partial)

{
  "main": "dist/index.js",
  "files": ["dist"],
  // ...
}

Now let's peek at our package's contents again:

$ npm publish --dry-run
npm notice
npm notice 📦  bonaroo-able@1.0.0
npm notice === Tarball Contents ===
npm notice 1.3kB  package.json
npm notice 3.9kB  dist/Able.d.ts
npm notice 6.1kB  dist/Able.js
npm notice 3.4kB  dist/Able.js.map
npm notice 52B    dist/index.d.ts
npm notice 184B   dist/index.js
npm notice 198B   dist/index.js.map
npm notice === Tarball Details ===
npm notice name:          bonaroo-able
...
+ bonaroo-able@1.0.0

That seems about right to me. Let's publish it!

Publishing to NPM

Either sign-in npm login or sign-up npm adduser. After that, we're ready to publish our package.

npm publish

$ npm publish
npm notice
npm notice 📦  bonaroo-able@1.0.0
npm notice === Tarball Contents ===
npm notice 883B   package.json
npm notice 3.9kB  dist/Able.d.ts
npm notice 6.1kB  dist/Able.js
npm notice 3.4kB  dist/Able.js.map
npm notice 52B    dist/index.d.ts
npm notice 184B   dist/index.js
npm notice 198B   dist/index.js.map
npm notice === Tarball Details ===
npm notice name:          bonaroo-able
npm notice version:       1.0.0
npm notice package size:  2.3 kB
npm notice unpacked size: 7.1 kB
npm notice shasum:        4b25f5d01b4ef46259d947d0c0ce1455b92b8433
npm notice integrity:     sha512-mX7RA0CS8hprb[...]lFsx3AGk5XIeA==
npm notice total files:   7
npm notice
+ bonaroo-able@1.0.0

Nice!

Testing our release in Node

Now we can use our package in Node projects! Let's create a temporary Node project to test our package.

mkdir /tmp/node-test
cd $_
npm i bonaroo-able
node

> const Able = require("bonaroo-able");
undefined
> const definition = { foo: ["bar"] };
undefined
> const abilities = ["foo", "bam"];
undefined
> result = Able.flatten(definition, abilities).sort();
[ 'bam', 'bar', 'foo' ]

Testing our release in a webpack project

To use our package in the brower, the package user might be using webpack. Let's try our package in webpack!

mkdir /tmp/webpack-test
cd $_
npm init -y
npm i bonaroo-able
npm i -D webpack webpack-cli html-webpack-plugin webpack-dev-server clean-webpack-plugin

webpack.config.js

const path = require('path');
const HtmlWebpackPlugin = require('html-webpack-plugin');
const { CleanWebpackPlugin } = require('clean-webpack-plugin');

module.exports = {
  entry: {
    app: './src/index.js',
  },
  plugins: [
    new CleanWebpackPlugin(),
    new HtmlWebpackPlugin({ title: "Titled Document" }),
  ],
  output: {
    filename: '[name].bundle.js',
    path: path.resolve(__dirname, 'dist'),
  },
};

src/index.js

const Able = require("bonaroo-able");

document.addEventListener("DOMContentLoaded", () => {
  const definition = { foo: ["bar"] };
  const abilities = ["foo", "bam"];
  const result = Able.flatten(definition, abilities);

  const code = document.createElement("code");
  code.textContent = result.join(", ");
  document.body.appendChild(code);
});

package.json (partial)

  "scripts": {
    "build": "webpack",
    "start": "webpack-dev-server --open"
  },

Let's start the webpack dev server:

npm start

We are greeted with foo, bam, bar in our browser: Webpack build works!

Building our library for usage in browser

One cannot use the dist/* files in the browser directly - we must combine these files somehow to create a single bundle for the browser.

Bundling libraries for use in the browser is a hairy subject. There are many solutions, none of them are perfect. In this guide, I'll cover only one solution: We'll be creating something called an IIFE build using rollup.js.

An IIFE build looks something like this:

var Able = (function() {
  var Able = {};
  var otherVars = 1;

  Able.flatten = /* ... */

  return Able;
})();

Because the library is defined inside a function expression that is invoked immediately using (function() {})(), all definitions inside the function are hidden, and only the return value is exposed to the global scoped.

Since the Function Expression is Immediately Invoked, it is called an IIFE.

Let's install rollup, add a build command to our package.json, and add a config file for rollup. Also, let's also add a reference to our browser bundle in the package.json's browser property.

npm i -D rollup rollup-plugin-commonjs rollup-plugin-node-resolve rollup-plugin-babel-minify

package.json (partial)

{
  "browser": "dist/bonaroo-able.min.js",
  "scripts": {
    // ...
    "rollup": "rollup -c"
    // ...
  }
}

rollup.config.js

import resolve from 'rollup-plugin-node-resolve';
import commonjs from 'rollup-plugin-commonjs';
import minify from 'rollup-plugin-babel-minify';
import pkg from './package.json';

export default [{
  input: 'dist/index.js',
  output: {
    name: "Able",
    file: pkg.browser,
    format: 'iife',
    sourcemap: true,
  },
  plugins: [
    resolve(),
    commonjs(),
    minify({ comments: false }),
  ],
}];

Let's test or browser build:

example.html

<!DOCTYPE html>
<title>bonaroo-able test</title>
<script src="./dist/bonaroo-able.min.js"></script>
<script>
document.addEventListener("DOMContentLoaded", function() {
  const definition = { foo: ["bar"] };
  const abilities = ["foo", "bam"];
  const result = Able.flatten(definition, abilities);

  const code = document.createElement("code");
  code.textContent = result.join(", ");
  document.body.appendChild(code);
});
</script>

You should see foo, bam, bar again in your browser when opening example.html.

Build before publish

You can configure NPM to build automatically before publishing by adding a prepublish script to your package.json. Because npm publish publishes the built files, we want to make sure the files are built and tested before every publish.

We already have npm run ci to both build & test our build. Let's add rollup to ci, and add npm run ci to prepublishOnly:

package.json (partial)

  "scripts": {
    // ...
    "ci": "(npm run test-build && npm run rollup) & npm t & wait",
    "prepublishOnly": "npm run ci && npm run rollup",
    // ...
  }

Let's publish our new build. NPM uses semantic versioning. Every release, you must update your version number. Since we introduced a new feature (browser build) without breaking changes, we can release a new minor version. You can increment your build number with npm version minor, push our new version to git with git push, and finish with npm publish to publish our new version.

npm version minor
git push
npm publish

Including our library in a browser directly from a CDN

unpkg is a fast, global content delivery network for everything on npm. Use it to quickly and easily load any file from any package using a URL like:
unpkg.com/:package@:version/:file

Thanks unpkg - I couldn't have explained it better myself. Let's try this!

package: Our package name, bonaroo-able.
version: We just minor-bumped our version to 1.1.0.
file: The browser file: dist/bonaroo-able.min.js.

That makes https://unpkg.com/bonaroo-able@1.1.0/dist/bonaroo-able.min.js. Let's grab our example.html again, and change the script source to this URL:

example.html

<!DOCTYPE html>
<title>bonaroo-able test</title>
<script src="https://unpkg.com/bonaroo-able@1.1.0/dist/bonaroo-able.min.js"></script>
<script>
document.addEventListener("DOMContentLoaded", function() {
  const definition = { foo: ["bar"] };
  const abilities = ["foo", "bam"];
  const result = Able.flatten(definition, abilities);

  const code = document.createElement("code");
  code.textContent = result.join(", ");
  document.body.appendChild(code);
});
</script>

Great. Works for me. Now let's write a readme.

Writing a readme

A readme is the entry point of the documentation of our library and should include a short summary of the following:

What is our library?
Why does it exist?
What can it be used for?
How to install it
How to use it
Requirements & dependencies

Writing a good readme is outside the scope of this guide. This guide will only cover installation instructions.

README.md (partial)

## Installation - NPM

```sh
npm i bonaroo-able
```

## Installation - Browser

```html
<script src="https://unpkg.com/bonaroo-able@1.1.1/dist/bonaroo-able.min.js"></script>
```

The script tag in the readme now includes the version number, which will not be updated automatically. Let's add a simple script that bumps the version in the readme everytime we update the NPM version.

When using npm version, npm will invoke multiple hooks automatically, two of which are called preversion (Run BEFORE bumping the package version) and version (Run AFTER bumping the package version, but BEFORE commit).

My approach is to dump the version before bumping the version, and after bumping the version to replace all occurances of the old version in the README.md with the new version.

preversion.sh

#!/usr/bin/env bash
node -e 'console.log(require("./package.json").version)' > .old-version

version.sh

#!/usr/bin/env bash
sed "s/$(cat .old-version)/$(node -e 'console.log(require("./package.json").version)')/g" < README.md > ~README.md
rm README.md .old-version
mv ~README.md README.md
git add README.md

package.json (partial)

  "scripts": {
    // ...
    "preversion": "./preversion.sh",
    // ...
    "version": "./version.sh",
    // ...
  },

sh

chmod +x preversion.sh version.sh

Now let's commit our changes & bump the library version.

sh

git add --all
git commit -am "Introduce README.md"
npm version patch
git push
npm publish

Our readme is now updated! Neat.

## Installation - NPM

```sh
npm i bonaroo-able
```

## Installation - Browser

```html
<script src="https://unpkg.com/bonaroo-able@1.1.2/dist/bonaroo-able.min.js"></script>
```

Final words

Now every time you change something about your library, commit the changes, update the version, push the version change & publish your new version:

git add --all
git commit -m "Describe your changes here."
npm version minor
git push
npm publish

If you're still here, thanks so much for reading! And if you want to know more or have any other questions, please get in touch with us via info@bonaroo.nl

Implementing projects with the JAMstack

Toby Hinloopen — Tue, 26 Nov 2019 13:59:57 +0000

Implementing projects with the JAMstack

We like to implement projects with the JAMstack. Read how we create modern, responsive and interactive cloud applications for web and mobile.

What is the JAMstack?

According to jamstack.org, it’s a new way of building websites and apps that delivers better performance, higher security, lower cost of scaling, and a better developer experience.

The frontend is written in Javascript, running entirely on the client. This can be done with any frontend framework or library, or even vanilla Javascript.

All serverside processing and database transactions are abstracted into APIs, accessed over HTTP with Javascript. These can be custom-built or leverage third-party services.

Finally, templated Markup (usually static websites like bonaroo.nl) should be prebuilt before deploying.

The Javascript Client

We start with a client-side application written in web technologies and is compiled to static HTML, JS en CSS files. Whenever the user visits the URL for this application, the application is downloaded and executed. The application does not require downloading new assets to load other views.

Since these files are static, they can be cached so that the user's next visit doesn't require downloading the application again.

Whenever the application actually requires user-data, it is loaded on-demand using APIs. The raw data provided by the API is transformed client-side for the user.

User-input is handled similarly: User-created data is transformed client-side to compact raw data and pushed to the API. The client is aware of common domain constraints and most validations can be performed even without a single API call.

For security, the same (and usually more) validations are also performed server-side, and any validation errors are returned. Validation errors from the API are transformed to user-friendly errors client-side.

Because rendering is done client-side without delay, the developer gets freedom to create highly interactive user-interfaces rarely seen in more conventional web applications.

Because this client-side application runs independent from all other components, it can be developed and tested efficiently and thoroughly. The client-side application scales perfectly.

The API Server

Since the client-side application is fully static, all dynamic content is fetched from the server-side application through APIs. The server-side application responsible for reading and writing all data generated by users and from external sources. The server-side application is not concerned with the user or the interfaces presented to the user. It only pushes raw data around.

The server-side application is also responsible for authentication, authorisation and access control. It determines who can and cannot access any resource, and handles access violations accordingly.

The server-side application is finally responsible for communicating with other external servers, and for sending e-mails and push-notifications.

Because this server-side application runs independent from the user has no graphical interface, it is developed against a suite of tests. These tests are compact and execute in-memory, so we can write a lot of tests in little time, and execute them fast.

The result is a thoroughly tested and efficient server-side application that scales exceptionally well and is usually only limited by its database.

The Markup

An application usually requires a dedicated website telling potential users about the application. Since the website is usually fully static, it can be generated at deployment. The result is a set of static files, just like the client-side application.

A static website can still be fully responsive, can still include analytics and can still include some dynamic elements. All dynamic elements are written in client-side Javascript, like the client-side application. Any server-side processing is done through APIs.

Load-times are exceptionally low, since no server-side processing is required. There is no page being generated on-the-fly. There is no server-side framework that can be abused or exploited. There is no processing required to render the pages, since all pages are already rendered at deployment.

Static websites scale perfectly, like the client-side application.

Hybrid mobile apps

Native apps usually behave like the client-side application describe earlier. They are completely self-contained and fetch user-data from the server using APIs.

We like to bundle the client-side Javascript application in a native app. When the app is opened, the app renders the bundled static files, rendering the client-side application like a native app. This way, we don't have to write dedicated native apps: This is called a Hybrid App.

While hybrid apps are written in web technologies, they have complete access to all native feature and component of any device. We only need to make small modifications in the client-side application to make them work on any mobile platform.

Well-written hybrid apps provide a near-native experience: It is very likely you have been using hybrid apps without realising it. Major examples include Yelp and Instagram.

Development speed

Writing JAMstack applications can be done with similar development speed of conventional server-rendered web application. Modern client-side frameworks are essential to create a similar development speed.

Any potential overhead caused by writing abstractions in multiple separate code repositories to communicate from the client to the server is compensated by ease of development and testability both client-side and server-side.

Used technologies

Here is a list of tools, frameworks and other keywords we use for our JAMstack:

Client-side: Javascript, Vue, Flow, SCSS.
Server-side: Phoenix, Elixir, Ruby, Rails, NodeJS
Markup and websites: Jekyll

We also use sentry.io for error reporting on any native app, hybrid app, client-side application and server-side application.

Potential overhead cause by implementing some logic both server-side and client-side be reduced by sharing code between the client and server by writing both the server-side and client-side code in the same language: Javascript. However, we like to use Phoenix and Elixir on the server since we find it to be more reliable.

Resulting application

The final result is a thoroughly tested and exceptionally fast application that has very few potential scaling bottlenecks. It also include iOS and Android apps without exceptional development overhead, and the apps behave the same as the client-side application in the browser.

Links

If you have any thoughts or questions, please get in touch with us via info@bonaroo.nl.