DEV Community: Ruairidh W McHardy

Partial Application and Functional Programming in JavaScript

Ruairidh W McHardy — Tue, 19 Jan 2021 00:00:00 +0000

How many times have you heard about making your code 'functional'? You know it makes things more composable, more efficient, and easy to reason with, but what's a practical example of this in action? Something you could use at work?

One technique I like to use in my daily coding is partial application. Wikipedia defines this as:

In computer science, partial application (or partial function application) refers to the process of fixing a >number of arguments to a function, producing another function of smaller arity.

...what?

Put simply, we take a function which could accept multiple inputs, but make it more flexible by splitting it into smaller functions to allow for better reusability. But, it's going to be easier to explain this with some code.

Imagine we have an API call we want to make. We could do something like this:

const getData = async (url: string) => {

  try {

  const response = await fetch(URL);



  const result = await response.json();

  } catch (error) {

  console.error(error);

  }

 };

So it works, but if we wanted to then amend our getData function to use another endpoint on the API, we'd need to pass in a whole new URL. It's also not very descriptive. We can do better!

This is where partial application comes in. We can split our function into its constituent parts:

const getAPIData = (baseUrl: string) => (endpoint: string) => async (callback) => {

 try {

  const response = await fetch(`${baseUrl}/${endpoint}`);

  const result = await response.json();

  callback(result);

  } catch (error) {
  console.error(error);
  }
 };

So what's different? Here we are using currying to allow us to reuse certain elements of the function, making it more efficient. Check out the usage:

const fetchInternalData = getAPIData("https://internal.myco.io");

// Now we want to fetch employees and clients
const fetchEmployees = fetchInternalData("employees");

const fetchClients = fetchInternalData("clients");

const callback = (data) => console.log(data); // This could do anything you want. It's just a callback.

// So putting it all together

fetchEmployees(callback);
fetchClients(callback);

// We can then use the same function for an external API

const fetchExternalData = getAPIData("https://api.github.com");

const fetchUsers = fetchExternalData("/users");

// We want to get the login names here

fetchUsers((data) => {
  console.log(data.map((user) => user.login));
});

// And the avatars here

fetchUsers((data) => {
  console.log(data.map((user) => user.avatar_url));
});

And, that's it! Just a simple way of splitting up a rigid function and making it more composable, saving you and your team from needing to reinvent the wheel. It also is easier to reason with which makes code reviews a more pleasant experience for everyone involved!

P.S. Want to sound clever when you talk about this stuff? You could mention that it reduces the arity of your functions. Put simply, arity is just a fancy way of talking about the number of parameters your function takes. If you've coded for a while, then you'll actually have used this already:

A function that takes one parameter is unary
A function which takes two parameters is binary
A function which takes three parameters is ternary

...and so on.

Thanks to Kyle Shevin who I learned this from on Egghead, and Mark Barry for a neat async refactoring which saved me some time when putting this quick post together.

An introduction to micro-frontends

Ruairidh W McHardy — Sat, 28 Nov 2020 00:00:00 +0000

Introduction

If you work on an application which involves multiple engineers, you'll have noticed that it's hard to approach working on a frontend simultaneously. This can lead to a lot of frustration and delay across teams, and the recent trend of splitting up monolithic frontends into smaller pieces has become popular.

This is known as a micro-frontend, and this article will look at how they work, why they're effective, and how you can leverage this architecture in your own team.

Additionally, we'll look at the benefits and costs so that you can establish whether you should be using a micro-frontend, rather than just chasing the latest craze.

By the end of this post, you should understand:

The pros of micro-frontends
The cons of micro-frontends
The differences in integration approaches of micro-frontends
How to implement a client-side integration of a micro-frontend

What is a micro-frontend?

A micro-frontend is an architecture where independent frontend codebases are combined into a larger application. You could create applications using different libraries such as React, or Vue, and also allow teams to work on applications independently, before bringing them together.

There are numerous advantages to this approach, namely that multiple teams can work on the frontend simultaneously without blocking one another, and you can easily version or customise components.

Integration of micro-frontends

If you can make a frontend application, congratulations! You can make a micro-frontend. There are a few approaches to implementing a micro-frontend though. The key difference lies in client-side integration, vs build-time integration.

Build-time integration

This integration strategy involves giving the container access to the dependency applications' source code before it is all loaded in the browser.

The steps involved are:

Work on application
Deploy it as an NPM package (in a private registry if you wish)
Install the package as a dependency in the container application
Output a bundle which contains all the code for our application

This has a few disadvantages as the container must be re-deployed every time a dependency application is updated, and it can be very tempting to tightly couple dependent applications which defeats the purpose of a micro-frontend architecture.

Client-side integration

This integration strategy involves loading the dependency application source code after the container application is loaded in the browser. We simply expose an entry point and render the output.

Work on application
Deploy application to a static JS file such as https://remote.foo.com/widget.js
User navigates to https://container.foo.co
Container fetches widget.js and executes it

This is advantageous as the widget.js dependency application can be deployed independently at any time, and different versions of it can be deployed, deferring to the container as to which version should be used.

Which should I use?

It's up to you! In this article we will discuss using a client-side integration. This a slightly trickier thing to achieve than simply combining the constituent applications at build time, but it allows us greater flexibility as you don't need to re-deploy every time you change a dependency application, and you avoid tightly coupling applications.

Client-side micro-frontends

We're using client-side integration via Webpack Module Federation. This is a flexible and performant solution which will give us a scalable architecture.

This works by:

Designating our container as the host application, and our dependency applications as remote.
In the remote applications, we decide which modules we want to make available to other projects.
We expose those files via module federation.
In the host we decide what we want to fetch from the remotes.
We load the host's entry point asynchronously.
We load whatever we want from the remotes.

As you can see in the example repo, we have added a Module Federation Plugin into our webpack config. There is a sample remote application added there called 'widget'. We can see this calls a localhost url. This works as follows:

Our remote application goes through its normal webpack bundling process, but additionally is processed by the Module Federation Plugin.

The remoteEntry.js file acts as a manifest and tells the container application how to load the files for the 'widget' application.

This means we can run the 'widget' application in isolation, but by adding the module federation plugin, we are able to include the 'widget' application in other applications.

On the container application side, we process our code with the Module Federation Plugin, and we generate the contents of 'index.js'. This way, Webpack knows that it needs to fetch something from the remote applications.

As a flow, this looks like:

main.js (generated by Webpack) is loaded and executed
Webpack sees we need to load and executes bootstrap.js
Bootstrap tells Webpack it needs a file from widgets'. It fetches remoteEntry.js to figure out what.
It fetches the dependencies.
Once fetched, bootstrap.js is executed

Sharing dependencies

As many remote applications may need the same dependencies, it doesn't make sense to install these multiple times. Instead, look at the shared property of the Module Federation Plugin. If you set this in common across both remote applications, the dependency will be shared between them.

For example:

shared: { react: {singleton: true}}

We load our dependencies asynchronously to prevent an eager consumption error. An example is contained in bootstrap.ts. This follows the same asynchronous loading pattern we see above in the container.

Versioning is handled automatically. If you specify a different version of a package in one application, than exists in the other, and they're attempting to share, then both versions will be loaded and the correct one is used in the respective applications.

The Module Federation Plugin looks at the shared dependencies, and reconciles them with what is contained in package.json . If the versions diverge, then both are loaded.

We can prevent this from occurring for packages such as react which rely on there being only one instance running, by passing a singleton property which ensure that only one copy is ever loaded.

Shared module selection can also be delegated by importing package.json and adding our dependencies. This is purely optional however.

You can see an example of this in the repo:

shared: packageJson.dependencies,

What about the execution?

We want to be able to run our code in isolation, and as part of a container. So how do we handle the assumption of where it will be rendered?

When developing in isolation we can assume that the element with id='foo' is present. But when running it as part of a container application, how do we know what the id of the element rendering our code will be?

The best pattern for handling this is to wrap our application in a 'mount' function which accepts an HTML element or React Element as an argument. This will then allow us to render the code in the correct place.

As you can see in the repo we achieve this in the bootstrap file:

const mount = (el: Element) => {
    ReactDOM.render(<App />, el);
};

Pattern for importing remote applications

If you look in App.tsx and components/WidgetApp.tsx then you will see an example of how to include remote applications in the container. We use the useRef hook to create an element which the app will be injected into, and a useEffect hook to ensure that we only load it in once:

import React, { useRef, useEffect } from "react";
import { mount } from "widget/WidgetApp";

export default () => {
 const ref = useRef(null);

// Pass in our ref and render it once.

useEffect(() => {
  mount(ref.current);
});

return <div ref={ref} />;
};

What about communicating between apps?

Ideally, you don't want your apps to be aware of each other, as that can create issues, but there will be times you need to pass data between them.

There are a few ways of achieving this, but my preferred pattern is to follow react's example and pass callbacks and state downwards from the container application to the remote applications.

This has the advantage of being explicit in how data flows through the application.

It's important to avoid your micro-frontend applications sharing state. This then couples them and makes maintenance extremely difficult. At that point, you may as well just have a monolithic frontend, which may be the more appropriate solution.

What are the downsides of micro-frontends?

It's important to understand when to use, and not to use a micro-frontend. They have tradeoffs and you shouldn't be tempted to use this pattern just because you can.

Bundle size

The first downside is obvious. We end up shipping more code to the client. It's very important to be sensitive to this, and I've tried to include best practices in my example repo.

As a quick guide, you should:

Share dependencies wherever possible
Lazy load components to prevent unecessary code download
Avoid bundling enormous packages such as moment

None of these things should come as a surprise. Try to keep your dependencies lean, and keep an eye on your bundle size.

Organisation

The next downside is organisational. Whilst it's great that you can split up code across teams and release autonomously, you can end up with a lack of communication about features, schedules, and code practices.

This can be avoided by good communication and documentation, but it's worth bearing in mind.

Complexity

Micro-services can appear intimidating if you're used to dealing exclusively with monolithic architectures. Questions such as how the applications communicate, where state lives, how to develop a good release pipeline, and test components are all common.

Before rushing to implement micro-frontends, you should take the time to fully understand how they work, and try to communicate this with your team. Once everyone is at a similar level of understanding, it's easier to move ahead.

Conclusion

Frontend software engineering has become vastly more complex over recent years, and that trend is likely to continue.

We are pushing more and more functionality to the client side, with incredibly sophisticated applications. Understanding how to separate your code into modules and splitting their development can deliver real benefits.

Hopefully by the end of this tutorial you now understand:

The pros of micro-frontends
The cons of micro-frontends
The differences in integration approaches of micro-frontends
How to implement a client-side integration of a micro-frontend

Found this useful? Let me know on Twitter

Build your own programming language

Ruairidh W McHardy — Sat, 16 May 2020 19:52:20 +0000

Introduction

I write JavaScript nearly every day just now and love it, but I don’t really know what happens to my code once I run it. Normally I run it through node, or the browser, and sometimes it works.

But I want to know what’s actually happening beneath the surface. How does my syntax become a working program? I’m no computer scientist, so I’m not going super deep into this, but I want to at least have a superficial understanding of my tools.

As part of this, I thought it would be a fun exercise to try and create a very, very basic programming language in JavaScript. I’m more or less following Steve Kinney’s path in this, so I’ve got a good example.

What does a programming language consist of?

So a programming language needs a way to understand the syntax it has been provided, and a way to interpret or compile these instructions into machine-readable code. Effectively we are turning our high level code into slightly lower level code.

I’m keeping this very simple and building a basic lexical analysis tool or lexer and a simple syntactic analysis tool, or AST (Abstract Syntax Tree).

This will accept my string of syntax, tokenize it, and then run the logic.

The current post will focus on building our lexer, and a subsequent post will handle our AST and related tooling.

Building a lexer in JavaScript

So a lexer basically takes a string of code and splits it into individual elements, or ‘tokens’. A token is just a small unit of the language. For example, look at this string in JavaScript:

sum(2, 1)

A lexer will split it into individual elements like this:

sum + ( + 2 + , + 1 + )

We effectively accept a string of code, iterate through each character, and check each character to see if it matches a pre-defined set of tokens. If so, we add it to our tokens collection and return them at the end to be interpreted.

Getting started

Now we have a rough idea of how a lexer works, let’s get started building one! Firstly, we’ll make some helper functions to determine character types:

const LETTER = /[a-zA-Z]/
const WHITESPACE = /\s+/
const NUMBER = /^[0-9]+$/
const OPERATORS = ["+", "-", "*", "/", "%"]

const isLetter = character => LETTER.test(character)
const isWhitespace = character => WHITESPACE.test(character)
const isNumber = character => NUMBER.test(character)
const isOpeneningParenthesis = character => character === "("
const isClosingParenthesis = character => character === ")"
const isParenthesis = character =>
  isOpeneningParenthesis(character) || isClosingParenthesis(character)
const isQuote = character => character === '"'
const isOperator = character => OPERATORS.includes(character)

const helpers = {
  isLetter,
  isWhitespace,
  isNumber,
  isOpeneningParenthesis,
  isClosingParenthesis,
  isParenthesis,
  isQuote,
  isOperator,
}

export default helpers

As we can see here, we have a number of methods which accept a character and run a simple RegEx (regular expression) on it to determine whether it matches a pre-determined type which we have created as a constant at the top of the file. In particular, we are looking for letters, whitespace, numbers, and operators.

Because the language we are building is inspired by Lisp, we will definitely need to know about parentheses, so we create specific helpers for these.

Building our token parser

Now we have some helpers to determine the characters we are working with, we want to put them to use! So let’s build a simple tokeniser:

import helpers from './helpers';

const tokenize = (input) => {
  const tokens = [];

  let cursor = 0;

  while (cursor < input.length) {
    const character = input[cursor];

    if (helpers.isParenthesis(character)) {
      tokens.push({
        type: 'Parenthesis',
        value: character,
      });
      cursor++;
      continue;
    }

      cursor++;
      continue;
    }

    throw new Error(`${character} is not valid.`);
  }
  return tokens;
};

export default tokenize;

Let’s walk through this. Firstly we define our tokenize function and accept an input.

Next, we create an empty array for our tokens which we’ll fill later. We also create a cursor variable which we’ll use to track our position in the input.

With that initial setup done, let’s walk through the input. We’re using a while loop here since it’s fast and allows us a good deal of control over our cursor position. We could also use something like reduce but we could work with some very big inputs in theory and this can give us performance issues along with making it harder to control exactly where the cursor is (but please get in touch if you have a cool way of doing this).

So we traverse the length of our input which is the code, and we assign the current position to our character variable for the sake of legibility.

Time to run our first check! We want to see if it is an opening or closing parenthesis. To do this, we use our isParenthesis helper and if so, we push an object to our tokens array providing the type and value. So we could express this in a test:

it("should tokenize a pair of parentheses", () => {
  const input = "()"

  const result = [
    { type: "Parenthesis", value: "(" },
    { type: "Parenthesis", value: ")" },
  ]

  expect(tokenize(input)).toEqual(result)
})

So now we’re capturing parentheses, we want to figure out the rest of our tokens:

    if (helpers.isWhitespace(character)) {
      cursor++;
      continue;
    }

    if (helpers.isNumber(character)) {
      let number = character;

      /**
       * We want to account for multi-digit numbers, so we
       * look ahead in our string to see if the next character
       * is a number. We assume white space is the end of a number.
       */
      while (helpers.isNumber(input[++cursor])) {
        number += input[cursor];
      }

      tokens.push({
        type: 'Number',
        value: parseInt(number, 10),
      });

      continue;
    }

    if (helpers.isLetter(character)) {
      let symbol = character;

      /**
       * We want to account for words, so we look ahead in our
       * string to see if the next character is a letter.
       *
       * We assume white space is the end of a word.
       */
      while (helpers.isLetter(input[++cursor])) {
        symbol += input[cursor];
      }

      tokens.push({
        type: 'Name',
        value: symbol,
      });

      continue;
    }

    if (helpers.isQuote(character)) {
      let string = '';

      while (!helpers.isQuote(input[++cursor])) {
        string += input[cursor];
      }
      tokens.push({
        type: 'String',
        value: string,
      });

      cursor++;
      continue;
    }
    ```

Some of these are simple such as a check for whitespace, but others are more complex, so we’ll dig into these.

Tokenizing Digits

Tokenizing a single digit is pretty straightforward, but it becomes more complex with multi-digit numbers. If we didn’t take this into account, we could have 101 as an input but it would be split into 1, 0, 1. This could be pretty disastrous for our tiny language!

So instead we need to look ahead of our current character and see if the next item is a number as well. If so, we can assume that it is a continuous number. So we introduce a while loop and increment our cursor to see that the next character is a number. If so, we append it to our current number variable, until we reach the end of the number.

As some example tests, we can do this:



it("should tokenize a single digit", () => {
  const input = "3"
  const result = [{ type: "Number", value: 3 }]

  expect(tokenize(input)).toEqual(result)
})

it("should tokenize a continuous number", () => {
  const input = "33"
  const result = [{ type: "Number", value: 33 }]

  expect(tokenize(input)).toEqual(result)
})

Tokenizing Words

The logic for tokenizing a word is more or less the same here so you can refer to the same logic, but for an example test:



it("should tokenize a continuous Name", () => {
  const input = "abc"
  const result = [{ type: "Name", value: "abc" }]

  expect(tokenize(input)).toEqual(result)
})

Tokenizing Quotes

Finally, we want to be able to handle strings inside quotes. There are a few gotchas here which haven’t been implemented, like parsing single and double quotes and escaping strings, but for our purposes it works fine.

In this case we don’t really care about the quotation marks other than the fact they operate as boundaries for the beginning and end of a quoted string. To account for this, we inverse the logic and for every item which is not a quote mark, we add it to our string variable. When we hit our closing quotation, the loop breaks and we continue to iterate the tokenizer.

As a simple test, we could run:



it("should handle a quoted string", () => {
  const input = '"hello"'

  const result = [{ type: "String", value: "hello" }]

  expect(tokenize(input)).toEqual(result)
})

Finished Result

All in, your code should look something like this:



import helpers from "./helpers"

const tokenize = input => {
  const tokens = []

  let cursor = 0

  while (cursor < input.length) {
    const character = input[cursor]

    if (helpers.isParenthesis(character)) {
      tokens.push({
        type: "Parenthesis",
        value: character,
      })
      cursor++
      continue
    }

    if (helpers.isWhitespace(character)) {
      cursor++
      continue
    }

    if (helpers.isNumber(character)) {
      let number = character

      /**
       * We want to account for multi-digit numbers, so we
       * look ahead in our string to see if the next character
       * is a number. We assume white space is the end of a number.
       */
      while (helpers.isNumber(input[++cursor])) {
        number += input[cursor]
      }

      tokens.push({
        type: "Number",
        value: parseInt(number, 10),
      })

      continue
    }

    if (helpers.isLetter(character)) {
      let symbol = character

      /**
       * We want to account for words, so we look ahead in our
       * string to see if the next character is a letter.
       *
       * We assume white space is the end of a word.
       */
      while (helpers.isLetter(input[++cursor])) {
        symbol += input[cursor]
      }

      tokens.push({
        type: "Name",
        value: symbol,
      })

      continue
    }

    if (helpers.isQuote(character)) {
      let string = ""

      while (!helpers.isQuote(input[++cursor])) {
        string += input[cursor]
      }
      tokens.push({
        type: "String",
        value: string,
      })

      cursor++
      continue
    }

    throw new Error(`${character} is not valid.`)
  }
  return tokens
}

export default tokenize

Conclusion

And…that’s pretty much it! Although a lexical analysis tool sounds pretty tricky, the basic version is actually quite simple to make.

Don’t be fooled though, to build an actually useful one would take a lot of time and effort. Yes, JavaScript was famously written in ten days, but that’s more a testament to Brendan Eich’s skill than the complexity of the task. This stuff really is hard!

With that said, we’ve done a good job today. Going from zero to a functional lexer is no mean feat and we got there!

Next step is to write an AST to break the code into a more meaningful structure so that we can see what our tokens want to achieve, and then transpile this into JavaScript, and we will do precisely that in another post.

Building a Jest Clone

Ruairidh W McHardy — Wed, 13 May 2020 14:52:20 +0000

Introduction

I use Jest nearly every day when working, and it’s a fantastic tool. It lets me ship my code with confidence, knowing that I have produced something which works as intended.

More often than not, when I write tests for my code, I end up catching something I hadn’t considered and go back to take that into account. It saves me from getting getting called in the evening to fix something in production. For me, that’s a big deal.

With that said, I didn’t really know how jest worked. I used it all the time, but had no real clue what was going on under the hood.

Recently, I bought Kent C Dodd’s excellent Testing JavaScript course which has been incredible for increasing my knowledge.

As part of that, he digs into how jest works under the hood, and this encouraged me to try building my own tiny version. So I want to pass on that knowledge to you!

This will be a small series of posts on each part, but you can skip ahead and see the full repo here.

Building our test runner

Today’s focus will be building the actual test runner. This simply accepts a test with a title, and a callback. We run the callback and if there are no errors, the test passes!

If an error is thrown, then we deem the test to have failed.

Take a look:

const test = async (title: string, callback: Function) => {
  try {
    await callback();

    console.log(chalk.green(`\u2713 ${title}`));
  } catch (error) {
    console.error(chalk.red(`✕ ${title}`));

    console.error(error);
  }

So as you can see, we have our test function which accepts a title which is a string, and a callback which is the function which we want to test.

We then run our callback function in a try/catch block, and assuming that nothing is caught, we say the test has passed.

This allows us to run something like this:

test("sum adds numbers", () => {
  const sum = (a, b) => a + b

  const result = sum(3, 7)

  const expected = 10

  expect(result).toBe(expected)
})

So as you can see, we pass a title, and a function where we sum two numbers: 3 and 7. But on our final line, we see something new: expect and toBe. These are our assertions and we have nothing to handle them yet.

Building our assertions

Fundamentally, an assertion is a very simple concept of whether two inputs match. Does sum(3,7) result in 10? We don’t want to have to write logic like this for everything:

if (sum(3, 7) !== 10) {
  throw new Error("Sums do not match")
}

Instead, we want to extract this out to a helper utility we will call expect, much like as you see in jest.

const expect = (actual: Function) => {
  return {
    toBe(expected: String | Object | Array) {
      if (actual !== expected) {
        throw new Error(`${actual} is not equal to ${expected}`)
      }
    },
  }
}

As you can see, we take in a function which is what we are running our assertion on. We then have a series of methods on a returned object allowing us to expand our test matchers later.

In this case we are wanting a simple toBe test for an exact match. So we accept a variable called expected on toBe and compare that to the result of our function that we passed in earlier.

Going back to our previous example where we add 3, and 7, we can do something like this:

expect(sum(3, 7)).toBe(10)

Pretty simple right? Although jest itself is a far more complex tool, this is more or less how it works. We can extend our matchers as well. For example, if we wanted to check for a falsy value, we could do something like this:

  toBeFalsy() {
      if (actual) {
        throw new Error(`${actual} is not falsy.`);
      }
    },

As you can see, our expect function can easily be extended to cater for whatever assertions you need to make.

Putting it together, you can access it within our test runner and the above code will return:

✅ sum adds numbers

And that’s it! You’ve just made a very, very simple test runner with some assertions. In the next post in this series, I’ll be showing you how to write your own test mocks which allow you to simulate function calls so that you don’t run them in reality. It’s a very useful technique.

If you want to see the full code, just take a look at the repo!

Create a COVID-19 tracker in React

Ruairidh W McHardy — Tue, 21 Apr 2020 14:52:20 +0000

Introduction

At the time of writing, there are 2,494,915 confirmed COVID-19 cases worldwide. Many more are expected, and a huge number of people are confined to their homes. It’s grim news, and it’ll take time to get better.

With that said, it’s not a bad time to try and improve your skills if you’re in the right headspace to do so. It’s completely understandable if not though, these are stressful times and there is no expectation that you should be doing anything than getting through this.

If you’d like to learn how to make a cool COVID-19 heatmap in React, read below! If you want to skip directly to the full code, click here

Project setup

To keep this simple, we’re going to use create-react-app to get started. Run the following commands:

npx create-react-app covid-tracker && cd covid-tracker

This will allow you to use create-react-app without having to install it to your system, create a project called covid-tracker and enter the directory.

You’ll see plenty of boilerplate code which you can ignore for now. Go to src/App.js and clear the code in the return () statement.

First let’s get hold of some data. To do this, we’ll use the free Corona API. I’m specifically using the /v2/countries endpoint. This shows the latitude and longitude of each country where a COVID-19 case is present, and some statistics.

To pull this data into our component, we want to load it on the first render so that we have access to the data in our map. To do so, we take advantage of the useEffect hook. This is pretty close to the lifecycle methods we previously had such as ComponentWillMount and ComponentDidMount.

Our effect will look like this:

const [getCases, setCases] = useState(undefined)
const [loading, setLoading] = useState(true)
/**
 * Get our data on the first render, and prevent from
 * fetching on subsequent renders. If our request fails
 * or takes too long, then clean up.
 */
useEffect(() => {
  let isCancelled = false
  let source = axios.CancelToken.source()
  function getFetchUrl() {
    return "https://corona.lmao.ninja/v2/countries"
  }
  async function fetchData() {
    let result
    if (!isCancelled) {
      result = await axios(getFetchUrl())
    }
    setCases(result.data)
    setLoading(false)
  }

  fetchData()

  return () => {
    isCancelled = true
    source.cancel("Cancelling in cleanup")
  }
}, [])

Let’s break this down. So firstly we declare that we’re using a hook, useEffect. Then we create a variable isCancelled. This will become useful down the line for cleaning up our operation.

If the operation isn’t cancelled, we then use axios (a popular data-fetching library) to asynchronously fetch our endpoint. We have to declare this as its own function inside the useEffect hook as asynchronous functions return a promise, which the hook doesn’t expect. Instead, the hook expects that either nothing is returned, or a function is returned.

In the future, using React Suspense will remove this issue but for now, this is the workaround.

Once the resource is fetched, we update our state with the data returned, and set loading to false.

We also have a function below this which acts as our cleanup. This effectively acts as ComponentWillUnmount and we use this to cancel our axios request mid-flight.

Finally, we pass an empty array as an optional argument to useEffect which prevents it from triggering each time the component renders.

Ok, so now we have some data. We now need to convert it to GeoJSON to be displayed in react-map-gl. To do this, we’re going to write a quick utility function to transform our current data into the appropriate format.

Create a folder called utils and add makeGeoJSON.js to it. The code is:

const makeGeoJSON = data => {
  return {
    type: "FeatureCollection",
    features: data.map(feature => {
      return {
        type: "Feature",
        properties: {
          id: feature.countryInfo?._id,
          value: feature.cases,
        },
        geometry: {
          type: "Point",
          coordinates: [feature.countryInfo.long, feature.countryInfo.lat],
        },
      }
    }),
  }
}

export default makeGeoJSON

This takes in our data as a variable, and we map over each item in the array to add its coordinates. We now have valid GeoJSON!

In our main script, we want to pass our data to our new utility function:

// Convert our JSON to GeoJSON
let data
if (!loading) {
  data = makeGeoJSON(getCases)
}

Finally, we want to add this to the map! First, add the following dependencies:

yarn add react-map-gl axios

Firstly, we need to set some default parameters for our map on its initialization:

// Set our initial map variables
const [viewport, setViewport] = useState({
  latitude: 55.8609825,
  longitude: -4.2488787,
  zoom: 4,
  width: "100vw",
  height: "100vh",
})

This simply sets the initial latitude and longitude to Glasgow, Scotland (where I live) but you can make it whatever you want. Then we set a zoom level (smaller being further our, larger being closer in).

Finally we set the default height and width which I’ve just made the whole page.

Now we have our map, we can render it like so:

  return (
  <div className="App">
    {loading && <h1>Loading</h1>}
    {!loading && (
      <ReactMapGL
        {...viewport}
        mapboxApiAccessToken={process.env.REACT_APP_MAPBOX_TOKEN}
        onViewportChange={(viewport) => setViewport(viewport)}
        mapStyle="mapbox://styles/mapbox/dark-v9"
      >
        <Source type="geojson" data={data}>
            <Layer {...heatMapLayer} />
        </Source>
      </ReactMapGL>
    )}
  </div>
);
}

This is fairly self-explanatory but as you can see:

We check the loading state, and display an appropriate holding message while we load our data.

If we are not loading the data, then we render the map with the default map variables, and pass it our token (which you can create for free on Mapbox).

We then add a method onViewportChange which is provided by react-map-gl and allows us to make the map interactive. It provides us with the viewport variable which contains the lat/lng/zoom etc and we simply update our state with that data.

Finally we add a mapStyle. There are many online but I just went with a simple dark theme from mapbox.

Once we have rendered the map, we then pass it a custom layer. This uses heatMapLayer which we will now create in our utils folder:

const MAX_ZOOM_LEVEL = 9

const heatMapLayer = {
  maxzoom: MAX_ZOOM_LEVEL,
  type: "heatmap",
  threshold: 0.03,
  radiusPixels: 30,
  paint: {
    // Increase the heatmap weight based on frequency and property magnitude
    "heatmap-weight": ["interpolate", ["linear"], ["get", "mag"], 0, 0, 6, 1],
    // Increase the heatmap color weight weight by zoom level
    // heatmap-intensity is a multiplier on top of heatmap-weight
    "heatmap-intensity": [
      "interpolate",
      ["linear"],
      ["zoom"],
      0,
      1,
      MAX_ZOOM_LEVEL,
      80,
    ],
    // Color ramp for heatmap. Domain is 0 (low) to 1 (high).
    // Begin color ramp at 0-stop with a 0-transparancy color
    // to create a blur-like effect.
    "heatmap-color": [
      "interpolate",
      ["linear"],
      ["heatmap-density"],
      0,
      "rgba(10,0,0,0)",
      0.2,
      "rgb(100,0,0)",
      0.4,
      "rgb(120,0,0)",
      0.6,
      "rgb(1300,0,0)",
      0.8,
      "rgb(140,0,0)",
      2.1,
      "rgb(255,0, 0)",
    ],
    // Adjust the heatmap radius by zoom level
    "heatmap-radius": [
      "interpolate",
      ["linear"],
      ["zoom"],
      0,
      2,
      MAX_ZOOM_LEVEL,
      30,
    ],
    // Transition from heatmap to circle layer by zoom level
    "heatmap-opacity": ["interpolate", ["linear"], ["zoom"], 7, 1, 9, 0],
  },
}

export default heatMapLayer

This is from Uber’s example. I just customised it a bit for sizing and to have a red colour. You can easily customise this to your needs.

Your full code in App.js should look like this:

import axios from "axios"
import React, { useEffect, useState } from "react"
import ReactMapGL, { Layer, Source } from "react-map-gl"
import { heatMapLayer, makeGeoJSON } from "./utils"

function App() {
  const [getCases, setCases] = useState(undefined)
  const [loading, setLoading] = useState(true)

  // Set our initial map variables
  const [viewport, setViewport] = useState({
    latitude: 55.8609825,
    longitude: -4.2488787,
    zoom: 4,
    width: "100vw",
    height: "100vh",
  })

  /**
   * Get our data on the first render, and prevent from
   * fetching on subsequent renders. If our request fails
   * or takes too long, then clean up.
   */
  useEffect(() => {
    let isCancelled = false
    let source = axios.CancelToken.source()
    function getFetchUrl() {
      return "https://corona.lmao.ninja/v2/countries"
    }
    async function fetchData() {
      let result
      if (!isCancelled) {
        result = await axios(getFetchUrl())
      }
      setCases(result.data)
      setLoading(false)
    }

    fetchData()

    return () => {
      isCancelled = true
      source.cancel("Cancelling in cleanup")
    }
  }, [])

  // Convert our JSON to GeoJSON
  let data
  if (!loading) {
    data = makeGeoJSON(getCases)
  }

  return (
    <div className="App">
      {loading && <h1>Loading</h1>}
      {!loading && (
        <ReactMapGL
          {...viewport}
          mapboxApiAccessToken={process.env.REACT_APP_MAPBOX_TOKEN}
          onViewportChange={viewport => setViewport(viewport)}
          mapStyle="mapbox://styles/mapbox/dark-v9"
        >
          <Source type="geojson" data={data}>
            <Layer {...heatMapLayer} />
          </Source>
        </ReactMapGL>
      )}
    </div>
  )
}

export default App

Now this is completed, you can run:

yarn start

And you will see something like this:

From here, you can easily add more context to the heatmap, further data such as US county data, or change the styling.

Conclusion

As you can see, it’s very easy to get up and running with react-map-gl and a basic dataset. There are so many excellent data sources, and being able to see them visually is a very powerful technique.

If you want a to see the full code, click here.

Made something cool by following this guide? Tweet me @ruairidhwm and let me know!

How to understand a new codebase

Ruairidh W McHardy — Sat, 18 Jan 2020 16:00:00 +0000

Introduction

We’ve all been there. You’ve started a new job, or been assigned to a new project. All of a sudden, you’re looking at an existing codebase which has been worked on by many people. Not all of them are still working at the company. There are thousands of files, hundreds of thousands of lines of code, and you understand precisely none of it.

Sound familiar?

This is a particularly daunting situation when you’re a junior software engineer, or you’re wanting to make a good first impression in a new job. You want to feel like you’re competent, and that you’re making a valuable contribution to your team. But where to start?

This post is a collection of tips I’ve used in various roles and projects. This is my personal approach to reviewing a new codebase, but I’d love to hear about anything you’d suggest though, so reach out via Twitter, or subscribe to my mailing list below!

Understanding a new codebase

Why are you writing this code?

First things first, understand what the code actually does. Not on the level of what a function returns, or how the data moves around the system, but actually what it is designed to do. Perhaps you’re working for an online marketplace, so you understand that the code is to make it easy for people to buy and sell something online. Or you’re building a notification system, so a message needs sent from one entity to another.When you understand why you’re making something, then it becomes a lot easier to figure out what the code is doing, and you know what comes next. This is essential domain knowledge.

A lot of companies are terrible at providing this to software engineers during on-boarding as the assumption is that you only ‘write code’. You need to know why you’re writing code, so don’t hesitate to ask about how the company works.

What is the busiest part of the code?

This might be a little controversial, but I think that you can learn a lot from a codebase by seeing which parts of it are the busiest. Some files don’t get touched much once created, like small utility functions, whereas areas that deal with business logic, data, and routing are far more likely to be altered on a frequent basis. These often form the ‘meat’ of the application and I like to review them first.

A handy way to find out which areas of the code change the most is through the git effort command in Git Extras.

This shows you the number of commits per file which shows where the most activity is. You quickly get a visual overview of the codebase’s activity.

What has come before?

Now that you’ve had a look around some of the busier parts of the codebase, it’s time to see what activity there has been in the repo. If your team is disciplined then you should have informative commit messages, pull requests, and a strong code review culture.

I start with the pull requests to see how features have taken shape, and the comments provide a lot of context in terms of why some code has come to be there. How often have you looked at some lines of code and wondered what drove an engineer to write it that way?

Before you criticise either internally or externally (something I have been guilty of in the past), take the time to try to understand why it ended up that way. Yeah, the author might have been incompetent, or, more likely, there was a particular reason for the code ending up like that.

I try to look at the last few months of activity to see what has been prioritised, and what hasn’t. Hopefully there’s discussion in the repo about the approach, tradeoffs, and what is left to do. Even better if your team uses an issue tracker.

Tests

This is a great place to see the logic in action. Depending on the size of your codebase, there may be thousands of tests, so I’m not saying check them all. It’s worthwhile however to see how the tests have been written and what the coverage looks like.

Also, tests are a brilliant way to get involved with the codebase in a non-blocking way. You can improve test coverage which forces you to start working within the application’s logic and ensure that you get it right in order to have your test pass.

By writing some tests, you expose yourself to many different parts of the codebase and get to follow the logic around through various files which increases familiarity.

Database structure

I find it helpful to have a look at how the database is structured in order to give more context to the way the application is written. Are we using a relational or non-relational structure? Are we using something like a graph database?

Understanding how data is structured and stored gives you more confidence when it comes to processing and transforming the data within your application. You can anticipate how it will be handled down the line, and allow you to write more performant code.

Talk to your colleagues

Now that you have a pretty reasonable overview of the codebase, no doubt you’ll have more specific questions. If so, now is the time to ask! If you can take a colleague out to lunch and let them know that you’d like to pick their brains on the code that’s great.

If you feel that you want a less direct approach, then I find a quick email / DM asking if they could make some time to run through something with you is great.

Finally, when submitting a pull request, that’s a great time to add clarifying questions or comments. It has the added benefit of preserving knowledge for the next person to come along and read the commits.

Write documentation

Be the change you want to see in the world. If you felt there was information missing when you started, add it for the next person! Nobody is going to complain about having improved documentation and it is a valuable contribution to your team. It also has the added benefit of reducing the Bus Factor.

Conclusion

There are just a small collection of tips I have for people getting used to a new codebase that I’ve used in the past. There are obviously so many ways of doing this, and different things work for different people.

If you have any suggestions then let me know! I’d love to hear what approaches other people take.

Stop using offset pagination

Ruairidh W McHardy — Thu, 05 Dec 2019 18:37:00 +0000

What is offset pagination?

If you’ve been working with software for any length of time, you’ll be very used to consuming results from APIs. Typically you’ll make a request, and end up with a list of information which is quite long. People are usually bad at reading very long lists, so as developers, it’s good practice to shorten them to something more manageable.

If you’ve used Google, you’ve seen this in action. You may have 9000 results, but you get 10 per page by default. People can handle 10. But 9000 in one go is just far too much.

We achieve this result via pagination. We split all our results into a series of pages. A common (and easy) way to do this is to say to the API - “Give me the first ten results”. Then when we click on our ‘next’ button, we say, “Skip the first ten results, and serve up the next ten”. In essence, we are offsetting our results by an amount of our choosing.

This is particularly easy when you’re working on the frontend, but it’s actually not very efficient at all.

What’s the problem with offset pagination?

When you are working with an offset, you make your request as detailed above and receive the expected results. But from the database’s perspective it looks a bit more like this:

Select every single row that has been requested.
Now order them.
Finally, drop the number of rows specified in the offset parameter.

Offset only cares about how many rows you want to drop. It doesn’t care about optimising a query, so it fetches everything, then drops what has been requested. This applies across both SQL and NoSQL databases, and creates a heavy workload for the database.

Ok, so we’ve seen that’s an issue, but it’s also problematic because you probably don’t work with one request at a time. What happens if a new row is created whilst you’re fetching? It turns out that offset tries to be clever and counter this by inserting duplicates in case something else was inserted during the fetch. This is a weird quirk and obviously frustrating to deal with.

So when you’re coding a pagination function, you’re unlikely to be giving the database enough context to efficiently and accurately provide you with the information that you want.

How to paginate data properly

Ok, that’s a bold title. This is a potential way to achieve your desired result though. It’s actually simple - you can use a WHERE clause to return the information that you haven’t retrieved. As shown in the incredible article by Use The Index Luke on which this is based, you can achieve your desired result with this:

SELECT someCol, anotherCol
  FROM someTable
 WHERE 'some_condition' > 1
   AND id < ?last_seen_id
 ORDER BY id DESC
 FETCH FIRST 10 ROWS ONLY

This is called keyset_pagination and it offers the advantages of being faster than offset and also prevents the strange duplication of resuls, or other frustrating anomalies.

There are some downsides however. You can’t go to an arbitrary page for example, so it’s a method better suited to things like infinite scrolling in a way similar to Instagram, than it is to clicking on page numbers. It’s a judgment call as to what will fit your use-case the best.

Conclusion

We’ve now understood the difference between offset and keyset pagination, and the advantages and disadvantages of both. I feel that while offset pagination is convenient, there are some tradeoffs when it comes to speed and predictability. I’d be very curious to know how Google achieves this.

Overall, my advice would be to use code similar to the snippet above, and avoid a simplistic offset where possible if you are concerned about performance in your application.

Want to know more about this kind of stuff? Sign up to my newsletter below!

React Suspense - A first look

Ruairidh W McHardy — Wed, 06 Nov 2019 18:15:13 +0000

Introduction

If you’ve been following the React community, then you’ll know about a hotly-anticipated feature called ‘Suspense’. This is still an experimental feature, but essentially it allows you to ‘wait’ for some code to load and specify a loading state such as a spinner whilst we’re waiting.

Here’s the example that the React documentation gives us:

const ProfilePage = React.lazy(() => import("./ProfilePage")) // Lazy-loaded

// Show a spinner while the profile is loading
;<Suspense fallback={<Spinner />}>
  <ProfilePage />
</Suspense>

In simple terms, this allows your components to wait for a condition to be true until they render. Most people imagine this in the context of fetching data, but it equally applies to things like images, scripts, or anything asynchronous.

What’s the point?

Currently, we fetch data in a number of ways - such as fetching on render. This is typically achieved through useEffect() and performing our fetch operation within the effect.

This means that our component renders, and the fetch operation doesn’t start till that render completes. If other components on your page depend on data being present, then this cascades down your application. The code sample that React provides is a great illustration:

function ProfilePage() {
  const [user, setUser] = useState(null)

  useEffect(() => {
    fetchUser().then(u => setUser(u))
  }, [])

  if (user === null) {
    return <p>Loading profile...</p>
  }
  return (
    <>
      <h1>{user.name}</h1>
      <ProfileTimeline />
    </>
  )
}

function ProfileTimeline() {
  const [posts, setPosts] = useState(null)

  useEffect(() => {
    fetchPosts().then(p => setPosts(p))
  }, [])

  if (posts === null) {
    return <h2>Loading posts...</h2>
  }
  return (
    <ul>
      {posts.map(post => (
        <li key={post.id}>{post.text}</li>
      ))}
    </ul>
  )
}

Here we are waiting for our user details to be fetched, then we start fetching the posts only after we have fetched the user details. Each of these takes time and the user is left waiting.

Or we could be clever and use something like GraphQL and fetch our data in a single step, but the user is still left waiting for this operation to complete. In the first example, we rendered, then fetched. With GraphQL, we start fetching, finish fetching, and then render.

With Suspense, we make a subtle change and instead we:

Start fetching
Start rendering
Finish fetching

This speeds everything up as the user gets a response before we’ve finished grabbing all of our data.

The effect is to stream the response as data becomes available and make content available sooner, rather than waiting for all data to become available before displaying it.

Data that hasn’t been fetched yet simply falls back to a loading indicator. This has the added advantage that we can remove if(loading) checks from our code.

How can I use React Suspense?

The feature is currently experimental and shouldn’t be used on production systems. If you want to play around with it however, you can easily install it via

yarn add react@experimental react-dom@experimental

This introduces Concurrent Mode which Suspense is a part of. There are some pretty big changes involved, including the way that you render your React application.

Instead of:

import ReactDOM from "react-dom"

ReactDOM.render(<App />, document.getElementById("root"))

you would now have this for your entry point:

import ReactDOM from "react-dom"

ReactDOM.createRoot(document.getElementById("root")).render(<App />)

The reason for this change is that Concurrent Mode is a pretty big change to how React works, and so you have to opt-in to it wholesale, rather than the ‘drop-in’ features of Fragments, or Context.

Conclusion

Concurrent Mode and React Suspense look amazing! It’s a big paradigm shift in how we think about data and our code, and I think that it will really change the way that we perceive the responsiveness of applications. From a DX perspective, this will lead to clearer code, and encourage us to load data closer to where it’s actually being used.

It’s going to be a little while until Concurrent Mode and Suspense are ready for production, but they’re already showing a lot of potential.

Building an AI-powered security camera

Ruairidh W McHardy — Tue, 08 Oct 2019 19:45:13 +0000

Building an AI-powered security camera

In my spare time, I have been experimenting with the potential of facial recognition using Python. Facial recognition and computer vision are becoming much simpler thanks to high level libraries such as Face Recognition.

My goal with this script was to see if I could make something which recognised that somebody was present, then would recognise whether it was me or not. If it was, then nothing happens - otherwise I use the computer’s text-to-speech utility to tell the intruder to ‘Go away!‘.

Here’s an example:

The code, and how it works

You can see the repo, otherwise the script is below:

import face_recognition
import cv2
import numpy as np
import os
import glob

# Get a reference to webcam #0 (the default one)
video_capture = cv2.VideoCapture(0)

# make array of sample pictures with encodings
known_face_encodings = []
known_face_names = []
dirname = os.path.dirname( __file__ )
path = os.path.join(dirname, 'people/')

# make an array of all the saved jpg files' paths
list_of_files = [f for f in glob.glob(path+'*.jpg')]
# find number of known faces
number_files = len(list_of_files)

names = list_of_files.copy()

for i in range(number_files):
    globals()['image_{}'.format(i)] = face_recognition.load_image_file(
        list_of_files[i])
    globals()['image_encoding_{}'.format(i)] = face_recognition.face_encodings(
        globals()['image_{}'.format(i)])[0]
    known_face_encodings.append(globals()['image_encoding_{}'.format(i)])

    # Create array of known names
    names[i] = names[i].replace("people/", "")
    known_face_names.append(names[i])

# Initialize some variables
face_locations = []
face_encodings = []
face_names = []
owner = 'Ruairidh'
process_this_frame = True

while True:
    # Grab a single frame of video
    ret, frame = video_capture.read()

    # Resize frame of video to 1/4 size for faster face recognition processing
    small_frame = cv2.resize(frame, (0, 0), fx=0.25, fy=0.25)

    # Convert the image from BGR color (which OpenCV uses) to RGB color (which face_recognition uses)
    rgb_small_frame = small_frame[:, :, ::-1]

    # Only process every other frame of video to save time
    if process_this_frame:
        # Find all the faces and face encodings in the current frame of video
        face_locations = face_recognition.face_locations(rgb_small_frame)
        face_encodings = face_recognition.face_encodings(
            rgb_small_frame, face_locations)

        face_names = []
        for face_encoding in face_encodings:
            # See if the face is a match for the known face(s)
            matches = face_recognition.compare_faces(
                known_face_encodings, face_encoding)
            name = 'Unknown'

            face_distances = face_recognition.face_distance(
                known_face_encodings, face_encoding)
            best_match_index = np.argmin(face_distances)
            if matches[best_match_index]:
                name = known_face_names[best_match_index]

            face_names.append(name.rsplit('.', 1)[0].capitalize())

        if owner not in face_names and name is 'Unknown':
            os.system('say Go away!')

    process_this_frame = not process_this_frame

    # Display the results
    for (top, right, bottom, left), name in zip(face_locations, face_names):
        # Scale back up face locations since the frame we detected in was scaled to 1/4 size
        top *= 4
        right *= 4
        bottom *= 4
        left *= 4

        # Draw a box around the face
        cv2.rectangle(frame, (left, top), (right, bottom), (0, 0, 255), 2)

        # Draw a label with a name below the face
        cv2.rectangle(frame, (left, bottom - 35),
                      (right, bottom), (0, 0, 255), cv2.FILLED)
        font = cv2.FONT_HERSHEY_DUPLEX
        cv2.putText(frame, name, (left + 6, bottom - 6),
                    font, 1.0, (255, 255, 255), 1)

    # Display the resulting image
    cv2.imshow('Video', frame)

    # Hit 'q' on the keyboard to quit!
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# Release handle to the webcam
video_capture.release()
cv2.destroyAllWindows()

As you can see, it’s a pretty short script! To achieve the desired goal however, it needs to do four things:

Establish whether there are any faces present.
Create a ‘map’ of the face and transform it to centre it as much as possible.
Encode the face through an embedding (more on this later).
Compare the embedding to that of a previously recognised face.

If you save a photo of yourself in ‘people’ and run this, the script will draw a boundary box over your face, labelled with your name. Otherwise, it will tell anyone else to go away!

How face detection works

Face detection is everywhere. It’s on your phone’s camera to help with focusing, it’s used by Snapchat for all those fun filters, and when Facebook suggests who is in a photo, it’s all face detection.

So how does it work? A typical process is to convert the image to black and white since we don’t need colours to recognise a face. Then we can look at every single pixel of the image and note the surrounding pixels. We figure out in which direction the image becomes darker, and then draw an arrow in that direction, this arrow is known as a gradient.

This allows us to make meaningful comparisons of faces, even with light differences. Since the direction of the gradient will remain the same in different conditions, we end up with the same representation of the face.

Ok, but isn’t that a bit computationally heavy? Instead we can split the image into segments and set a gradient for the direction in which we have the most incidences.

This is known as a Histogram of oriented gradients, or HOG for short.

How face mapping works

So we can now detect there is a face present somewhere. But how do we handle the fact that different photos contain a multitude of poses, and it’s not like a series of passport photos where you get the same predictable pose and expression.

Well, we can adjust the positioning of the face itself so that it’s roughly in the same place. It’s kind of creating that passport standard, but with any photo.

To achieve this, a technique called face landmark estimation is used which creates a series of data points, or ‘landmarks’ that exist on nearly every face. Things like the space between eyes, the end of your chin, the edges of your brow, etc. Then our machine learning algorithm will look for those datapoints in order to make comparisons between faces and make predictions.

Now we can find out where the features are, we edit the image to place them as close to the centre as possible. It’s important to note that no 3d transforms happen to the image as that would warp it and ruin our comparisons. Instead we use things like rotation and scaling.

How encoding a face works

Now that we have our face positioned nicely, we want to extract a few basic measurements so that we can then find the face with the closest measurements, and so be most likely to be our match. People do this intuitively since we have evolved to do so.

Computers are less able to do this, and so we rely on deep learning to allow the computer to come up with its own way of recognising a face. This approach was previously used by Google researchers. For those that don’t want to spend a ton of time doing this themselves, there’s a handy dataset called OpenFace.

How facial recognition works

Now that we have our detected and embedded photos, we want to check if there’s a match! This is one of the easiest parts as it’s a simple classification problem. We provide the embedding of our new face and see if we can categorise it with one of our previously known faces. The closest match is then returned.

In practice

If you’re really into this, then you could implement it yourself, otherwise you could use the excellent Face Recognition python library. The creator of the library has written an excellent article on the process which inspired this one to solidify my own learning.

Cleaner error handling in Javascript

Ruairidh W McHardy — Tue, 08 Oct 2019 17:00:00 +0000

Providing clearer error messages in Javascript

As developers, we come across errors every single day. We don’t want to see errors, and we definitely don’t want to introduce them - but sadly it’s an everyday part of the job. So why is it that so many errors lack proper context, and look something like this:

Uncaught Error: Cannot read property 'foo' in Bar at <anonymous>:1:7

Ugh. So now we have a vague idea that maybe we’re calling an undefined property on an object, but we’re relying on our developer to have provided sensible variable names, and using our own domain knowledge of the application to infer exactly where the error is coming from. It’s not ideal.

When we’re creating our own applications, a good developer is conscious of what information they return to the end user (which may be another developer) at all times. This is something done well by some of the best API providers out there, such as Stripe or Twilio. In the Stripe API documentation, you have a clear expectation of exactly what an error is going to look like, and have a good idea of how to fix it.

How to write better error messages

Firstly, consider exactly what happened, and where. Let’s say that you are trying to let a user reset their password, and want to check whether their token has expired yet. Your code might look like this:

if (isExpired(new Date(user.passwordTokenExpiry))) {
  throw new Error("invalid token")
}

There is no context as to what the token is. A JWT? Password reset token? Maybe something else. The developer (and by extension, the user) consuming this error is none the wiser. Separately, the developer who needs to look at this later, won’t know where the error is coming from. It provides a poor developer experience.

So how to fix this? Well, custom error classes to the rescue! As seen in the mdn documentation you can extend the Error object provided by javascript to create your own ones with far more context. For example:

class PasswordTokenError extends Error {
  constructor(message) {
    super(message)
    this.name = "PasswordTokenError"
    this.message = message
  }
  toJSON() {
    return {
      error: {
        name: this.name,
        message: this.message,
      },
    }
  }
}

When we call this in our code, we can now provide a great deal more context:

if (isExpired(new Date(user.passwordTokenExpiry))) {
  throw new PasswordTokenError("Password reset token has expired")
}

So an error is thrown as before, but a developer can now check what type of error has been thrown, and take action accordingly. The error returns JSON like the following:

{
  "error": {
    "name": "PasswordTokenError",
    "message": "Password reset token has expired"
  }
}

Conclusion

Although conceptually very simple, this is a really easy way to clean up your errors and make debugging a lot easier for you and other developers. It provides context and a uniform way to catch errors relating to your application.

I’ve just introduced this to my code at work and am confident that it will make life a lot easier when it comes to debugging!

With thanks to Iain Collins for his excellent article on this topic.

My Experience with GraphQL

Ruairidh W McHardy — Tue, 30 Jul 2019 18:42:13 +0000

What is GraphQL?

GraphQL is described as ‘a query language for your API’. It was created by Facebook back in 2012 for use in their mobile applications, but was made open source in 2015 and has grown in popularity ever since.

GraphQL is an alternative to REST architecture and allows users to specify the data they would like to receive, rather than a REST endpoint simply returning everything.

An endpoint returning unnecessary data is known as ‘overfetching’ and results in requests which are larger than they need to be, the frontend needing to parse data it won’t use, and potentially a client having to perform multiple network requests until it has all the information it needs.

For example, compare these two queries:

REST Query

GET /users/1
{
  name: "John Doe",
  email: "john@test.com"
  // ...more user information
}

Now if we also wanted to retrieve the comments of this user, we may need to perform another query before we can get all the information we need. Something like:

GET /users/1/comments
{
    {
        title: "I really don't like too many requests"
        body: "REST is great but sometimes makes me have to fetch a lot of data"
    },
    {
        title: "I wonder if there's another way of doing this?"
        body: "Surely this is a problem people have encountered before?"
    }
  // ...more user comments
}

GraphQL Query

In GraphQL, things are much simpler. We can just do:

query {
  User(id: "1") {
    name
    email
    comments {
      title
      body
    }
  }
}

The above would return a single object containing all the information in just one request. It’s far more efficient and we don’t have to receive all the fields that we would otherwise need to discard had we queried a REST endpoint.

Conversely, we don’t underfetch either. This is when an endpoint doesn’t give us enough information so we have to make n additional requests to get hold of everything that we need. For example, we may retrieve a list of 100,000 users but then need to separately request their comments.

This would be an expensive operation (and probably bad design regardless of whether you’re using REST or GraphQL).

GraphQL is a specification

Many people imagine GraphQL to be like an ORM or a replacement database. This is not the case. It’s merely a specification that can be implemented in any language and describes a way to relate to your data. The spec is easy to read and can be checked out here.

Pros of using GraphQL

No Overfetching

As described above, the client does not receive data that it won’t use. Instead we simply request what we need and GraphQL returns the appropriate data. This means that we have smaller payloads, more efficient use of resources, and more flexibility on the frontend.

Modern

This matters more to some people than others, but it’s undeniable that GraphQL is on the ‘cooler’ end of the developer spectrum. It is an interesting technology that has a lot of potential.

In the State of Javascript 2018 survey, GraphQL won the ‘Highest Interest’ category with a staggering 87.7% of developers who have heard about GraphQL wanting to learn it!

Strongly Typed

GraphQL uses a strong type system which maps the API. Everything exposed in the API is contained in the GraphQL schema using a schema definition language. Once the schema is created, there is a single souce of truth for both the backend and the frontend and no need for translation between the two.

This allows for fewer errors, validation at compile-time, and you can be certain of what you are passing to the API and receiving from it.

Flexible

By allowing the client to dictate the data it requires, your frontend can move a lot faster without tying up a great deal of backend resources. This is significantly better than a 1:1 matching of endpoint to request where the frontend needs to normalize a great deal of data in order to present the correct values to the end user.

Cons of GraphQL

It’s New

This isn’t necessarily a bad thing, but the ecosystem is - while growing - less mature than what is available should you choose to go with a standard REST setup.

For many companies choosing the ‘boring’ option makes far more sense. There’s nothing wrong with that! For others, then GraphQL is an exciting new way to do things.

Complex Queries

This is something of a ‘solved’ problem in GraphQL but it can trip people up. For example, what happens if a user (perhaps maliciously) makes a request like this:

query {
  User(id: "1") {
    name
    email
    about
    address
    city
    zip
    country
    comments {
      title
      body
    }
    pictures {
      url
      order
    }
    likes {
      website {
        title
        url
      }
    }
  }
}

This is a bit of a trivial example, but you get the idea. A query can rapidly bloat and consume a lot of server resources. In a REST API you could potentially tweak each endpoint to retrieve its specific data in the most specific way.

In GraphQL you can use tools like DataLoader to handle your requests via batching and caching to reduce the load on your server.

Additionally you can limit the complexity of queries through Query Cost Analysis. There are a few packages which make this easier such as GraphQL-Cost-Analysis which computes the cost of the query and optionally blocks it if over a certain threshold.

Learning Curve

As GraphQL is quite new, it may take you time to get up to speed with it and figure out if its a good fit for you. Also, there are a few ‘gotchas’ that you may be unfamiliar with compared to REST. There are many resources covering these however.

My experience with GraphQL

At BeatGig we’re using GraphQL to power our new API and my experience has been largely very positive. Although it presented a bit of a learning curve, it has allowed for very rapid development and a much quicker integration between the backend and the frontend.

It’s not a silver bullet but when it comes to needing a very flexible way to query and subscribe to our data, it works well for us. There were some initial pains in the setup and understanding exactly how to map our existing functionality to GraphQL but this has gone fairly smoothly so far.

Something that has benefitted us hugely however, is using GraphQL-Yoga, and Prisma which gives us a lot of the CRUD features out of the box, and has pre-configured an express server with Apollo in the case of GraphQL-Yoga.

This allowed us to get up and running very quickly and development has been a very enjoyable experience since!

Conclusion

GraphQL isn’t going to be for everyone, but it’s a very interesting technology that reduces some of the frustrations you may feel (particularly on the frontend) when you’re using REST endpoints.

It has an intial learning curve but allows for rapid development and flexible access to your underlying data. The ecosystem is growing very quickly and a large number of developers are interested in learning it, so that ecosystem should continue to grow.

Overall it’s a useful tool in any developer’s arsenal and definitely worth trying out!

The Javascript Event Loop

Ruairidh W McHardy — Wed, 27 Mar 2019 11:03:00 +0000

Understanding how the Event Loop Works

Introduction

Have you ever heard somebody saying that slow code is ‘blocking’? Or perhaps you’ve seen Call Stack Limit Exceeded errors when you’ve been coding? What does it all mean?

Well, it comes down to something called the Event Loop which is how your code is handled by the Javascript Runtime and its environment. It actually makes a lot of sense, and is useful to know - so I’ve tried to document my own learning here.

I recommend checking out Philip Roberts’ excellent talk if you’d like more detail. It provided the inspiration for this post.

How do things fit together?

If we look at the Javascript Runtime itself (V8 in Chrome for example), you see a heap (where memory allocation happens), and the call stack.

You also have Web APIs which the browser provides - things like the DOM, fetch, setTimeout etc. Then you also have the callback queue.

What is the stack?

Javascript has a single-threaded runtime which means it has a single call stack. That means technically it can only run one thing at a time.

For example:

function sayHello(x) {
  return `Hi ${x}, how are you today?`
}

function greet(x) {
  return sayHello(x)
}

function greetMe(x) {
  var greeting = greet(x)

  console.log(greeting)
}

greetMe("Ruairidh")

What does this code do? Well you can see multiple functions, which call each other.

Our stack looks something like this:

sayHello(x)
greet(x)
greetMe("Ruairidh")
main()

Blocking

What happens when things are slow? Things which are slow (image processing, network requests, etc) and are on the stack can be referred to as blocking.

If you had three synchronous requests in a row then you would need to wait for them to complete before anything else on the stack happens. This is a problem in browsers, as you can’t do anything else until those requests are complete. This would create a horrible user experience.

So what’s the solution? Well, asynchronous callbacks. They allow you to run code and give it a callback (or function) which we can execute later. But what happens to them? This is where the event loop comes in.

The Event Loop

Javascript as a runtime can only do one thing at a time, but we can do things concurrently thanks to APIs we can access in places such as the browser. Node does effectively the same thing.

When you call an asynchronous function, the relevant API picks it up and the function is popped off the stack. It’s being handled elsewhere, and the stack moves on to the next item.

When your asynchronous function is completed, it pushes your code onto the task queue. The event loop then checks the stack and the task queue. If the stack is empty, then it goes to the task queue and pushes the first item there onto the stack.

The Javascript Runtime then executes the code on the stack, and you receive the output.

This is why the hack of using setTimeout and setting the value to 0 existed. It was a way of deferring something until the stack is clear. You can now use Promises to achieve the same result but in a more efficient way.

Promises have higher priority in the event loop stack as they create a ‘micro task’ and a setTimeout creates a ‘macro task’ which also has a minimum delay of 4ms. Also, setTimeout doesn’t provide a guaranteed time of execution, it provides a minimum time to execution. There’s a great post here if you’d like to learn more.

If you’d like a way of visualising this process, check out Philip Roberts’ Loupe tool.

Conclusion

The Javascript Event Loop is actually one of the simplest parts of a more complex process. It allows you to write non-blocking code in a single-threaded language and defer slower tasks to the queue while the rest of your code runs.

This means that you can develop more responsive and more efficient code and not have to wait on larger operations.