DEV Community: Andrew Healey

Creating Randomness Without Math.random

Andrew Healey — Wed, 29 Jul 2020 19:53:17 +0000

In JavaScript, you can create random numbers using Math.random(). But what if we wanted to create our own random values in the browser without this function?

The ECMAScript Language Specification defines the requirements of Math.random():

Returns a Number value with positive sign, greater than or equal to 0 but less than 1, chosen randomly or pseudo randomly with approximately uniform distribution over that range, using an implementation-dependent algorithm or strategy. This function takes no arguments.

Each Math.random function created for distinct realms must produce a distinct sequence of values from successive calls.

Number Generation

Here's an example of a number generator. It uses a closure to maintain internal state and creates a sequence of numbers based off an initial seed value. Here the seed is fixed and is always initialized to 0.

Math.random = (function () {
  let seed = 0
  return function () {
    seed += 1
    return seed
  }
})()

// We can iterate through the sequence
Math.random() // 1
Math.random() // 2
Math.random() // 3

A pseudorandom number generator (PRNG) works in a similar manner. A PRNG maintains an internal state and applies math to that state every time a new random number is requested. The seed can be manual or automatic. In the Go programming language, you must seed math/rand yourself. In the browser, Math.random requests random data under the hood from the operating system (OS) to use as a seed.

PRNGs are deterministic. The same seed will always produce the same sequence of numbers. Often, a deterministic outcome is preferred. For example, to generate the same random events on all clients without them having to talk over a network. Or for reproducible performance benchmarks.

A hash function can be used to create a PRNG. In spinning-balls, one of Chrome's benchmarks, we can see an example of this:

// v8/benchmarks/spinning-balls/v.js

// To make the benchmark results predictable, we replace Math.random
// with a 100% deterministic alternative.
Math.random = (function () {
  var seed = 49734321
  return function () {
    // Robert Jenkins' 32 bit integer hash function.
    seed = seed & 0xffffffff
    seed = (seed + 0x7ed55d16 + (seed << 12)) & 0xffffffff
    seed = (seed ^ 0xc761c23c ^ (seed >>> 19)) & 0xffffffff
    seed = (seed + 0x165667b1 + (seed << 5)) & 0xffffffff
    seed = ((seed + 0xd3a2646c) ^ (seed << 9)) & 0xffffffff
    seed = (seed + 0xfd7046c5 + (seed << 3)) & 0xffffffff
    seed = (seed ^ 0xb55a4f09 ^ (seed >>> 16)) & 0xffffffff
    return (seed & 0xfffffff) / 0x10000000
  }
})()

Like our number generator, it alters its internal state while calculating the next random number. This state-change allows the next call to produce a different number.

More on Pseudorandom Number Generators

One of the oldest and most well known types of PRNG is the linear congruential generator (LCG). Which, despite its somewhat scary name, does not require many lines of code.

@bryc provides an example and a warning:

Commonly called a Linear congruential generator (LCG), but in this case, more correctly called a Multiplicative congruential generator (MCG) or Lehmer RNG. It has a state and period of 2^31-1. It's blazingly fast in JavaScript (likely the fastest), but its quality is quite poor.

function LCG(a) {
  return function () {
    a = Math.imul(48271, a) | 0 % 2147483647
    return (a & 2147483647) / 2147483648
  }
}

(This is the first time I've come across Math.imul() — which provides C-like 32-bit multiplication of the two parameters.)

What does @bryc's comment, "its quality is quite poor" mean in this context? Well, given certain even seeds, this algorithm has a pattern when the final step (the division) is removed.

// https://gist.github.com/blixt/f17b47c62508be59987b#gistcomment-2792771

// @bryc:
// "Looking at the output without the division, and in hexadecimal, the
// first bits are always the same. This shows a clear pattern in the
// first 8 bits of the output: 1000 000, and it happens each time,
// infinitely. This is mostly caused by using an even seed."
const LCG = (s) => (_) => (s = Math.imul(48271, s) >>> 0)
const nxt = LCG(3816034944)
for (let i = 0; i < 9; i++) {
  console.log(nxt().toString(16))
}

/* Outputs:
4b6c5580 <-- notice the last two digits
b04dc280 <--
9645a580
16717280
d974f580
5c9f2280
9a3a4580
f196d280
b5d59580 */

There are many ways to test the quality of randomness. Some of the methodology and results of these tests can be understood by a layperson. One of the Diehard battery of tests plays 200000 games of craps and looks at the distribution of wins and the number of throws each game.

There's also a test for LCGs called the spectral test which plots the sequence in two or more dimensions. In the example below, we can see the hyperplanes that the spectral test measures for.

![Hyperplanes of an LCG in three dimensions](https://dev-to-uploads.s3.amazonaws.com/i/rq9ftbb8fm6l7e3oy142.gif)

A PRNG eventually repeats its sequence. In this context, the period is the length of steps until the cycle repeats. Simpler PRNGs such as Mulberry32 have a period as low as ~4 billion whereas the Mersenne Twister has a period of 2^19,937 - 1. In 2015, the V8 team said that their implementation of Math.random() uses an algorithm called xorshift128+ which has a period of 2^128 - 1. Its introduction can been seen in this diff.

If a PRNG eventually repeats itself, you might wonder why we call it repeatedly. Why not use the first number and then reset the internal state with a new seed? The problem with this is that the seed needs to originate from somewhere. If we continue to ask the OS for more random data there is a chance that the call may block (as the OS waits for more randomness to be generated) and our program will stall.

Entropy Required

So you've settled on a PRNG and replaced window.Math.random. You've shipped it to your users and, at first, everyone seems to be happy.

But wait! You forgot about the seed. And now your users are complaining about the sequence of random numbers they get. It's the same every time their customers' page loads. All of their software is predictable. As a result, the web games they built are easy to beat.

Huzaifa Sidhpurwala reminds us:

Entropy is the measurement of uncertainty or disorder in a system. Good entropy comes from the surrounding environment which is unpredictable and chaotic.

When required, the generation of securely random numbers in the browser is performed by Crypto.getRandomValues() from the Web Cryptography API. Which is seeded by "a platform-specific random number function, the Unix /dev/urandom device, or other source of random or pseudorandom data."

The Linux source suggests where this pseudorandom data can come from:

Sources of randomness from the environment include inter-keyboard timings, inter-interrupt timings from some interrupts, and other events which are both (a) non-deterministic and (b) hard for an outside observer to measure.

There are also hardware devices that use quantum mechanical physical randomness.

You can find many prominent examples of random number generator attacks which occurred because the wrong type (or not enough) entropy was used. Cloudflare famously uses lava lamps as an entropy source. Since we are not attempting to create a secure algorithm, predictable sources of entropy like time are fine.

We can use Date.now() our seed state. This means that we will get a different random sequence for every millisecond. We could also use performance.now() which returns the length of time since the time origin.

Other possible ways of getting entropy in the browser:

crypto.getRandomValues, crypto key generation, or similar (feels like cheating)
Mouse/touch events, ambient light events, mic/webcam noise (hard to use on page load)
Geolocation API, Bluetooth API, or similar (need permission, doesn't work on page load)
WebGL/video performance shenanigans
Most APIs listed here

Here's our slower (because it's not native code) and unstable (because I haven't tested it) replacement for Math.random(). Also note that PRNGs have requirements for the seed state (e.g. prime numbers, 128-bit). Our algorithm doesn't comply with the seed recommendations for the Xoshiro family.

// https://github.com/bryc/code/blob/master/jshash/PRNGs.md
// xoshiro128+ (128-bit state generator in 32-bit)
Math.random = (function xoshiro128p() {
  // Using the same value for each seed is _screamingly_ wrong
  // but this is 'good enough' for a toy function.
  let a = Date.now(),
    b = Date.now(),
    c = Date.now(),
    d = Date.now()
  return function () {
    let t = b << 9,
      r = a + d
    c = c ^ a
    d = d ^ b
    b = b ^ c
    a = a ^ d
    c = c ^ t
    d = (d << 11) | (d >>> 21)
    return (r >>> 0) / 4294967296
  }
})()

Math.random() // 0.5351827056147158
Math.random() // 0.2675913528073579

So, Mission Accomplished?

Sadly it's impossible to create a fully ECMAScript compliant replacement for Math.random() since the specification requires "distinct realms [to] produce a distinct sequence of values from successive calls." A realm roughly means a different global environment (e.g. a different window, or a different WebWorker). Our version cannot reach outside its realm thus cannot make this guarantee.

However, there have been proposals for a Realms API. It's not inconceivable that such an API would provide access to something like an incrementing realm id. This would give our algorithm the loophole it needs — access to Realm-unique entropy!

Thanks to JN~commonswiki for the 3D GIF of the spectral test.

Join 300+ people signed up to my newsletter about code and how I write it!

I tweet about tech @healeycodes.

A Practical Introduction to Graphs (Network Diagrams)

Andrew Healey — Wed, 08 Jul 2020 06:42:28 +0000

A graph is a structure that represents the connections between objects. They can be used for social network analysis, working out someone's Erdős number, finding the shortest path through a maze, and many other applications.

We'll be calculating the path between two countries with the least border crossings. The method we'll use is Breadth-First Search (BFS).

What's in a Graph?

Graphs are made up of vertices and edges. Below is the representation of France and its land border crossings.

An example of a vertex is FR or DE — they are also known as points or nodes. Here they represent countries.

An example of an edge is {FR, IT} or {FR, BE} — they are also known as lines, links, or connections. Here they represent the land border crossing between two countries.

Let's build a slightly more complicated graph. If our goal is to find the shortest path between two countries then we need a structure containing all of the countries of the world. We'll actually need multiple graphs because some sets of countries aren't connected to other sets of countries (and islands are just lone vertices).

All the code for this post can be found on healeycodes/country-borders.

We can get all the data we need from borders.csv. This file is structured as country_code, country_name, country_border_code, country_border_name.

As a side-quest, we'll also store a relationship between country codes and their full names to make our output more user-friendly later on.

import csv

country_border_graph = {}
country_code_lookup = {}

with open('borders.csv') as csvfile:
    reader = csv.reader(csvfile, delimiter=',')
    next(reader, None)  # skip the headers
    for row in reader:
        country_code_lookup[row[0]] = row[1]
        country_a = row[0]
        country_b = row[2]

        if country_a in country_border_graph:
            country_border_graph[country_a].append(country_b)
        else:
            country_border_graph[country_a] = [country_b]

A graph is a mathematical structure and there is no singular way to represent them in code. Here, we have chosen a dictionary that maps a country to a list of its neighbors.

If we isolate France inside country_border_graph, it looks like this:

{
    'FR': ['AD', 'BE', 'DE', 'IT', 'LU', 'MC', 'ES', 'CH']
}

Displaying Graphs

So how do we print this to the screen? First we alter the structure of our data then we plug it into some libraries.

In order to build a DataFrame for the pandas library, we need to create two lists that describe all of our edges. We do that by iterating through the structure and adding a vertex to two lists (from_verts and to_verts) when we want to describe a connection.

from_verts = []
to_verts = []
for country in country_border_graph:
    for i in range(0, len(country_border_graph[country])):
        if country_border_graph[country][0] == '':
            # skip islands
            continue
        from_verts.append(country)
        to_verts.append(country_border_graph[country][i])

For France, the lists would look like this:

# from_verts
['FR', 'FR', 'FR', 'FR', 'FR', 'FR', 'FR', 'FR']

# to_verts
['AD', 'BE', 'DE', 'IT', 'LU', 'MC', 'ES', 'CH']

Using pandas, networkx, and matplotlib we can display our non-islands in a popout window.

import pandas as pd
import networkx as nx
import matplotlib.pyplot as plt


def show(from_verts, to_verts):
    df = pd.DataFrame(
        {'from': from_verts, 'to': to_verts})
    G = nx.from_pandas_edgelist(df, 'from', 'to')

    nx.draw(G,
            with_labels=True,
            node_size=1500,
            node_color="skyblue",
            node_shape="o",
            linewidths=4,
            font_size=25,
            font_color="grey",
            font_weight="bold",
            width=4,
            edge_color="grey",
            dpi=1000)
    plt.show()

The result is below. (I briefly attempted to visually unsquish the vertices from each other but didn't think it was worth the effort. If you want to display this data more aesthetically, start by checking out this StackOverflow answer.)

While this image, at first glance, looks similar to a world map the positions of the countries are not tied to their position on Earth.

Shortest paths

You can calculate the shortest path between some countries by hand. Take the path from Canada to Mexico as an example. At the lowest point of the largest set in the above image, we can see that CA is connected to US which is connected to MX. It's clear there is only one route between Canada and Mexico and therefore we have found the shortest path.

What if there are multiple paths? Let's take the path from France to China as an example.

Breadth-First Search (BFS) works by first looking at all of the nearby vertices and then working further and further away from the start point. By using FR as a starting point, our algorithm will first check ['AD', 'BE', 'DE', 'IT', 'LU', 'MC', 'ES', 'CH']. When it doesn't find China among those, it will then look at the neighbors of AD and then the neighbors of BE and so on.

In the diagram below, we can see the order in which BFS would visit each vertex of a tree.

The algorithm we use has been adapted from an outdated essay on Python.org. Eryk Kopczyński corrected the efficiency of the essay's pathing function ten years after its publication date. In linear time, it computes and stores the direct paths from the start point to every other point and then returns the path from the start point to the endpoint.

His original code returns the path in a strange format. FR to CN is [[[[['FR'], 'DE'], 'PL'], 'RU'], 'CN']. We can unfurl this nested list with flatten.

from collections import Iterable


def flatten(l):
    # flatten a nested list
    # https://stackoverflow.com/a/2158532

    for el in l:
        if isinstance(el, Iterable) and not isinstance(el, (str, bytes)):
            yield from flatten(el)
        else:
            yield el


def find_shortest_path(graph, start, end):
    # breadth first search (BFS)
    # adapted from https://www.python.org/doc/essays/graphs/

    dist = {start: [start]}
    q = [start]
    while len(q):
        at = q.pop(0)
        for country in graph[at]:
            if country not in dist:
                dist[country] = [dist[at], country]
                q.append(country)

    raw_path = dist.get(end)
    return list(flatten(dist.get(end)))

We can use this function on the original country_border_graph dictionary we built. With the country_code_lookup we made, we can print out user-friendly information.

france = country_border_graph["FR"]
france_names = [country_code_lookup[code] for code in france]

print(f'France is connected to {len(france_names)} countries.')
print(f'They are {", ".join(france_names)}.')

path = find_shortest_path(country_border_graph, 'FR', 'CN')
path_names = [country_code_lookup[code] for code in path]

print(f'The shortest path from France to China is: {" -> ".join(path_names)}.')

This prints:

France is connected to 8 countries.
They are Andorra, Belgium, Germany, Italy, Luxembourg, Monaco, Spain, Switzerland.
The shortest path from France to China is: France -> Germany -> Poland -> Russian Federation -> China.

To display this path, we can use the show function we defined earlier.

path = find_shortest_path(country_border_graph, 'FR', 'CN')

from_verts = []
to_verts = []
for i in range(1, len(path)):
    # we're drawing a line
    # so we describe each edge 
    from_verts.append(path[i-1])
    to_verts.append(path[i])

graph.show(from_verts, to_verts)

We can now visually show the shortest path between any two connected countries.

(If we attempted to use this code to calculate the path between two non-connected countries, e.g. GB and CN, the code would throw an error. Let's blame geography for this instead of the code's author.)

Thanks to Alexander Drichel for the Breadth-First Tree diagram.

Join 300+ people signed up to my newsletter about code and how I write it!

I tweet about tech @healeycodes.

Polyfilling the Fetch API for Old Browsers and Node.js

Andrew Healey — Sun, 28 Jun 2020 09:33:26 +0000

First some definitions.

A polyfill will try to emulate certain APIs, so can use them as if they were already implemented.

A transpiler on the other hand will transform your code and replace the respective code section by other code, which can already be executed.

(Thanks Sirko for those!)

Examples of polyfilling and transpiling

By polyfilling the Fetch API we make it usable in browsers where it isn't available by default. Another example would be making it available in its original functionality in Node.js.

By transpiling Spread syntax (an ES6 feature) into ES5 compatible JavaScript, we end up with source code that is easier for us to write and deployable code that works in older browsers!

Here's our ES6 code example.

function speak() {
  console.log(...args);
}

speak(1, 2, 3);

Here is that same code transpiled into ES5 compatible code.

"use strict";

function speak() {
  var _console;

  (_console = console).log.apply(_console, args);
}

speak(1, 2, 3);

(Technically, this is the loose transpilation because otherwise the above snippet would be eleven times longer.)

Using `fetch` in old browsers

We love the Fetch API because it means we can avoid using XMLHttpRequest. Instead of providing a callback, we can use lovely Promises.

Here's a fetch call that prints out the status code of a request.

fetch('https://httpstat.us')
    .then(res => console.log(res.status))

If we tried running that in Internet Explorer 11 (or Edge 13, or Chrome 39, and so on) we would get an error.

window.fetch would likely evaluate to undefined. We might get an error that looks like Uncaught TypeError: window.fetch is not a function.

It's easy to check one-off functionality for a feature on Can I use — here's fetch https://caniuse.com/#search=fetch. It isn't viable to check every feature that your code might use, so that's why we use things like @babel/preset-env.

@babel/preset-env is a smart preset that allows you to use the latest JavaScript without needing to micromanage which syntax transforms (and optionally, browser polyfills) are needed by your target environment(s). This both makes your life easier and JavaScript bundles smaller!

An even cooler feature of Babel is that it allows us to specify which platforms we want to support.

Why not just support all platforms? Because then the JavaScript bundle we send to our users would get larger and larger every year and website performance would grind to a halt.

Babel's Browserslist Integration lets us forget about version numbers and instead use handy shortcuts. Let's say we wanted to support 99.75% of browsers and no dead browsers. We can add "browserslist": "> 0.25%, not dead" to our package.json file.

However, you might be reading this because you found out that @babel/preset-env doesn't include a fetch polyfill. This is an open issue on GitHub (#9160). It looks like it's going to stay this way.

Babel only adds polyfill for ECMAScript methods (defined at https://tc39.github.io/ecma262/). Since fetch is defined by a web specification (https://fetch.spec.whatwg.org/), you need to add the polyfill yourself.

That's okay because we can use github/fetch to polyfill it for us.

Either by replacing all instances of fetch.

import 'whatwg-fetch'

window.fetch(...)

Or on a case-by-case basis.

import { fetch as fetchPolyfill } from 'whatwg-fetch'

window.fetch(...)   // use native browser version
fetchPolyfill(...)  // use polyfill implementation

Using `fetch` in Node.js

The Fetch API is common and people are fluent with it. It's productive if they can use it in all the JavaScript they write. Many people think it's available in Node.js by default. It's not but there's a package for that (™).

node-fetch/node-fetch let's us use the API we're fluent with to make HTTP calls on the back-end. Underneath, it uses native Node.js functionality.

// CommonJS
const fetch = require('node-fetch');

// ES Module
import fetch from 'node-fetch';

If you're looking for an isomorphic solution (this means using the same code in the browser and in Node.js) then you'll want Jason Miller's isomorphic-unfetch (but not for React Native, see #125) or Leonardo Quixada's cross-fetch.

These solutions will figure out which environment you're in and choose the correct polyfill.

Join 300+ people signed up to my newsletter about code and how I write it!

I tweet about tech @healeycodes.

The Flow of Knowledge in a Team (or Why I Dig Tickets)

Andrew Healey — Tue, 02 Jun 2020 07:40:16 +0000

Engineering tickets often end up meaning a lot. For the product but also emotionally for the individual. It's the engineer's job to be involved in the whole process from before the ticket even exists. Engineers refine the ticket, estimate the ticket, and shuffle it through columns on a sprint board until customers receive a feature.

The way products are built varies at every company. A team I sit metres away from at work do things so differently that there would be a learning curve if I joined them. But the concepts remain the same.

The fun part about writing this will be hearing about all the different ways y'all make software.

Feature or fix?

A ticket can be a feature that needs to be completed, a bug that requires solving, or a chore that needs doing (e.g. code clean-up, documentation creation). A spike is a type of ticket that means the situation is unclear. Perhaps it's because there are unknowns about the technical implementation. Perhaps what's required is checking where a tricky bug is originating from. A spike can sometimes mean: go and figure out what tickets need to be created to solve this problem and then create those tickets (the hydra of tickets).

In the Agile methodology, tickets shouldn't be time boxed but spikes are one of the exceptions. Without a time-limit, when does the engineering stop if unknowns are involved? It's far better to write down your findings and get your team's input. Sometimes spikes are easier than they seem and the development work can be completed within the time-limit and everyone is happy.

Where do tickets come from?

Here are some ways that tickets can come into existence from nothing.

A customer complains about a bug and a support engineer documents it
A project manager or team lead plans a piece of the project
Some technical debt is created while working on another ticket
A developer realizes that their current ticket requires additional parts
Another department (e.g. data scientists) require some work to be done

Where do tickets live?

Most tickets exist in a sad place called the backlog. It's sad because there are always tickets in there that you want to complete. Things that genuinely improve the product (or your ease of working on the product).

Backlogs usually increase in size until they become unmanageable. At which point they can be filtered and trimmed by certain criteria. Won't do is a common reason to close a ticket — as in: we realized that there is an issue here but we don't have any plans to actually fix the problem. I've seen date cut-off limits where after a year of a ticket existing if it hasn't been completed then it's put out of its misery and archived. A well trimmed backlog is the sign of a well managed engineering team.

Tickets are always archived, never deleted. If there once was a bug that then becomes a bigger bug in the future, you'll wish there was a trail to follow when it falls on your lap.

Tickets are brought out of the backlog into a refinement section. These tickets are then discussed in a backlog refining meeting. Questions are raised and answered before the ticket can continue on its journey. Sometimes the questions can't be answered and a comment is added. A comment might tag someone with a TODO like @alice to gather more requirements from (other team who requested the feature). This ticket will be looked at again in the next meeting.

In a backlog refinement meeting, you might ask:

Is this possible?
Is this within our area of responsibility (is this another team's job)?
Roughly, what steps are involved in solving this?
How long will this take?

A good rule for completing projects is: don't do something if you don't actually need to do it. This sounds silly until you realise how often it is broken.

Estimation

Planning poker is a ritual that helps get the average estimation for a ticket from a group of engineers. It works best when everyone reveals their estimations at the same time as this removes biases. Planning poker cards are often used because everyone can flip them over at the same time.

The unit of time is 'points' which can seem abstract at first. It's common to hear things like oh that's just a one-pointer or hm, 13 points is a little high, can we break that up into smaller tickets?.

Planning poker cards have a modified Fibonacci sequence on them. 1, 2, 3, 5, 8, 13, 20, 40, and 100. Why? The International Scrum Institute has a good summary:

The reason for using the Fibonacci sequence is to reflect the uncertainty in estimating larger items. A high estimate usually means that the story is not well understood in detail or should be broken down into multiple smaller stories. Smaller stories can be estimated in greater detail.

When tickets are pointed they can be placed into sprints in a sprint planning meeting (What is a Sprint in Scrum?). A sprint is a time frame where a goal is worked towards by completing tickets. Traditionally, a working product is demoed to stakeholders at the end of a sprint so that it can be worked on iteratively taking feedback into account.

One sprint follows another. At the end of a sprint there are reviews and retrospectives to ensure that the ways of working are productive and pleasant for everyone involved.

Interacting with tickets

When I pick up a ticket, I move it to a column called In Progress. This automatically assigns me to the ticket and puts my avatar in the corner of the card. This stops people doing duplicate work and helps product managers and quality assurance engineers see who is working on what.

Sprint boards usually have a column for tickets that are in review. GitHub notifications can get lost in a sea of other emails so being able to see what work I can review for my teammates really helps me.

The act of dragging these small cards in the sprint board's UI feels significant. Moving the ticket through the columns, left-to-right, indicates and broadcasts the ticket's progress. The final column, which means the work is done (meaning deployed to production or handed off to another team) is a happy place.

Good tickets are written so they can be scanned efficiently. They use Markdown to achieve this by marking sections and highlights. Bullet pointed lists are used in abundance. Good tickets also probably have:

A user story — the requirements from the POV of a user
Technical requirements if relevant
Links to other relevant tickets or previous work
For front-end tickets, links to designs
Tags! Tags give the team visibility into blockers and future problems

I like to keep tickets updated so that stakeholders don't need to chase me for an update. I do this by leaving comments about my progress.

I've normally been lucky at companies I've worked at in that code reviews are integrated to the issue tracking system. So 'attaching' a pull-request to a ticket is effortless and is achieved with a single click or comment.

Document the world

I like documentation. I like the flair that people add. Emojis, GIFs (practical or memes), and embedded videos. I spend a lot of time thinking about the best way to communicate ideas and problems — and I'm very happy that software engineering has allowed me to work on writing for humans. Lean writing that conveys a point. The least words required.

Some days I will be adding comments to technical designs, or explaining a pull-request that grew in size and requires a lot of context to understand. Other days I'll be breaking down a technology for the wider team by assuming the reader has minimal coding knowledge.

If you go the extra mile to throw up an internal documentation page after solving a problem, I appreciate you. I see you editing the onboarding documentation, keeping it tight, the instructions clear and friendly, and I thank you.

Join 300+ people signed up to my newsletter about code and how I write it!

I tweet about tech @healeycodes.

Adding Comments to Gatsby with Netlify Serverless Functions + GitHub

Andrew Healey — Sat, 09 May 2020 09:59:56 +0000

I wanted to accept user comments on a Gatsby website and store them on GitHub. As in, I wanted the comments to go straight into a file called comments.json in my repository. So I could use something as simple as

import comments from "../comments.json"

in my site's code. Without any databases. No third-party plugins making tens of networks requests.

Netlify serverless functions allowed me to use GitHub's API to make this repository change with the data from a submitted comment. It also hid my secret API credentials.

I built a prototype — healeycodes/gatsby-serverless-comments — that uses this flow:

👩 User enters a comment and clicks submit.
⚙️ A serverless function receives the data and hits GitHub's API.
🔧 It reads the existing comments.json , appends the new comment, and saves.
🚧 A new commit triggers a Netlify build.
✅ The new version of the website is deployed!

The new comment is visible to users ~30 seconds ⏰ after the first click.

The serverless function

Let's pick through the serverless function that receives the user's comment. It will make use of some constants that can be set through Netlify's website on settings → deploys.

The function is written with Node.js and exports a handler function, which is explained in Netlify's documentation.

// comment.js

const fetch = require("node-fetch")

const auth = process.env.GITHUB_PAT_TOKEN
const repo = process.env.GITHUB_REPO
const user = process.env.GITHUB_USER
const api =
  "https://api.github.com/repos/" +
  user +
  "/" +
  repo +
  "/contents/src/comments.json"

exports.handler = async function(event, context, callback) {
  // Use the Contents API from GitHub
  // https://developer.github.com/v3/repos/contents/#get-contents
  const existingFile = JSON.parse(
    await fetch(api, {
      headers: {
        // Pass some kind of authorization
        // I'm using a personal access token
        Authorization:
          "Basic " + Buffer.from(user + ":" + auth)
            .toString("base64"),
      },
    }).then(res => res.text())
  )

  // The file's content is stored in base64 encoding
  // Decode that into utf-8 and then parse into an object
  let comments = JSON.parse(
    Buffer.from(existingFile.content, "base64").toString("utf-8")
  )

  // This is the user submitted comment
  // Perhaps we would do some validation here
  const newComment = JSON.parse(event.body)

  // Update the comments
  comments.push({
    author: newComment.author,
    email: newComment.email,
    message: newComment.message,
    date: Date.now(),
  })

  // Use the Contents API to save the changes
  const res = await fetch(api, {
    method: "PUT",
    headers: {
      Authorization:
        "Basic " + Buffer.from(user + ":" + auth).toString("base64"),
    },
    body: JSON.stringify({
      message: "New comment on " + new Date().toDateString(),

      // Turn that object back into a string and encoded it
      content: Buffer(JSON.stringify(comments)).toString("base64"),

      // Required: the blob SHA of the existing file
      sha: existingFile.sha,
    }),
  }).then(res => res.text())

  callback(null, {
    statusCode: 204,
  })
}

Potential downsides

What if someone spams comments on your website? Well, you'll hit your build time limits pretty fast.

There's also a small window (10-100s of milliseconds between API calls) where two people comment at the same time and the older comment will be overwritten.

The fix for both of these is to alter our serverless function to open a pull-request with the comment change. Comments are now delayed but we've protected ourselves from malicious behavior and we can also screen comments for appropriateness. We won't lose any data but might rarely need to handle merge conflicts.

My Netlify review

Netlify are betting big on Jamstack applications. It's a bet I would make too.

Their developer experience (DX) is up there with the best right now. It's rare I read about a product just working and then it ends up doing so! Recently, Netlify's snappy deployments let me rush out changes to fix live issues within minutes.

What does this mean for their future success? Well, Tiny.cloud points out that:

DX is kind of a big deal. Developers can play a huge role in the uptake of your product, especially when you consider they are likely to provide guidance on what tools their organization should invest in, even though the final decision usually happens at the executive level.

Netlify's developer tooling lets me create prototypes like the one you're reading about without having to mess with configuration. My Gatsby website is hosted with their generous free tier and transferring and hosting it has been hiccup-free.

I recommend them.

Join 300+ people signed up to my newsletter on code and personal growth!

I tweet about tech @healeycodes.

Hacking Together an E-ink Dashboard

Andrew Healey — Tue, 28 Apr 2020 20:57:59 +0000

I realized I was asking Google Assistant the same questions over and over again. Like What's the current weather? or What's on my calendar for today? So I set out to build a small dashboard for my Raspberry Pi that I could check instead.

I bought a Waveshare 2.7inch e-Paper HAT (B) for ~$20. This version comes with a printed circuit board that slots right onto the general-purpose input/output (GPIO) pins of a Raspberry Pi. You can also use the included wires to avoid blocking the unused GPIO pins.

I followed the installation instructions from the Waveshare wiki and fired up one of the hello world programs — there are versions in C and Python. The steps to get this going, as well as the example programs, are quite accessible compared to other Raspberry Pi hardware accessories.

I started picking through the source code of the Python examples to understand how epd-library works. The version B of this screen is tri-colored, it can display white, black, and red, and the resolution is 264x176. The examples start by clearing the screen (filling it with white) and then create two layers which are Image objects from the Pillow library. One of the layers is the black content and the other layer is the red content. These are then passed to the render function EPD::display.

I've extracted and commented the key parts of examples/epd_2in7b_test.py.

# initialize the EPD class
epd = epd2in7b.EPD()
epd.init()

# start with a clear (white) background
epd.Clear()

# define a simple font
font24 = ImageFont.truetype(os.path.join(picdir, 'Font.ttc'), 24)

# create empty image objects/layers
HBlackimage = Image.new('1', (epd.height, epd.width), 255)
HRedimage = Image.new('1', (epd.height, epd.width), 255)

# render the black section
black_draw = ImageDraw.Draw(HBlackImage)
black_draw.text((1, 1), top_text, font=FONT_LARGER, fill=0)

# calculate the depth of the black section
top_depth = black_draw.textsize(top_text, font=FONT_LARGER)[1] + 5

# render the red section under the black section
red_draw = ImageDraw.Draw(HRedImage)
red_draw.text((1, top_depth), bottom_text, font=FONT_SMALLER, fill=0)

epd.display(epd.getbuffer(HBlackImage), epd.getbuffer(HRedImage))

For the prototype, I'll show the current weather and some news headlines from the BBC. Open Weather and News API both have generous free-tiers. For each of these features, I'll write functions that return text which I can create layers from.

I'll be using formatted strings from Python 3.5 (as opposed to the superior f-strings) because that's what Raspian (the default Raspberry Pi operating system) comes with.

This function talks to Open Weather using PyOWM, a wrapper library.

def get_weather():
    '''
    Get the formatted weather description.
    E.g. `Richmond, GB: Clear sky`
    '''
    owm = pyowm.OWM(OPEN_WEATHER_KEY)
    observation = owm.weather_at_place(LOCATION)
    w = observation.get_weather()
    detailed = w.get_detailed_status().capitalize()

    temp_data = w.get_temperature('celsius')
    current_temp = str(temp_data["temp"])
    min_temp = str(temp_data["temp_min"])
    max_temp = str(temp_data["temp_max"])
    text = '{}: {}\n{}°C ({}°C - {}°C)'

    return text.format(LOCATION, detailed, current_temp, min_temp, max_temp)

News API sadly doesn't have an up-to-date library available so I use requests to make an HTTP call and parse the data. I can display about three headlines (including an extra wrapped line for each). The text needs to be manually wrapped to stop it going off-screen.

def get_news(limit=3):
    '''
    Get some news headlines from the BBC formatted in a list.
    E.g. `- Acclaimed Swedish author Per Olov Enquist dies
    - PM's return to work 'a boost for the country'
    - 'Myth that Sweden has not taken serious steps'`
    '''
    url = 'https://newsapi.org/v2/top-headlines?sources=bbc-news&apiKey={}'.format(NEWS_API_KEY)
    resp = requests.get(url=url)
    data = resp.json()
    articles = [article['title'] for article in data['articles'][:limit]]

    wrapper = textwrap.TextWrapper(width=33)
    text = ''
    for headline in articles:
        text += '- {}\n'.format(wrapper.fill(text=headline))
    return '{}\n'.format(text)

With these two sections of text, I create the image layers. I want the weather to be on the black-colored layer with a slightly larger font. The news will be underneath on the red-colored layer. Here's a function that takes an instance of the EPD class as well as two sections of text.

def show_text(epd, top_text='', bottom_text=''):
    '''
    Given an EPD instance, and sections of text, render the text.
    '''
    HBlackImage = Image.new('1', (epd2in7b.EPD_HEIGHT, epd2in7b.EPD_WIDTH), 255)
    HRedImage = Image.new('1', (epd2in7b.EPD_HEIGHT, epd2in7b.EPD_WIDTH), 255)

    # render the black section
    black_draw = ImageDraw.Draw(HBlackImage)
    black_draw.text((1, 1), top_text, font=FONT_LARGER, fill=0)

    # calculate the depth of the black section
    top_depth = black_draw.textsize(top_text, font=FONT_LARGER)[1] + 5

    # render the red section under the black section
    red_draw = ImageDraw.Draw(HRedImage)
    red_draw.text((1, top_depth), bottom_text, font=FONT_SMALLER, fill=0)

    epd.display(epd.getbuffer(HBlackImage), epd.getbuffer(HRedImage))

The two Image objects contain simple data and don't 'know' what color they are. They look like this when rendered to bitmap format.

Notice the space that the second layer leaves for the first layer so that the text doesn't overlap.

The main function of this program makes sure that we only display the dashboard during hours when someone will be awake. I want to avoid ghosting which the documentation warns about. The EPD class that gets passed to the show_text function is initialized here.

def main():
    try:
        epd = epd2in7b.EPD()
        logging.info("Init and Clear")
        epd.init()
        epd.Clear()


        show_text(epd, top_text=get_weather(), bottom_text=get_news())

        # leave the dashboard up for half an hour
        time.sleep(60*30)

        logging.info("Clear...")
        epd.init()
        epd.Clear()

    except IOError as e:
        logging.info(e)
        sys.exit()

    except KeyboardInterrupt:    
        logging.info("ctrl + c:")
        epd2in7b.epdconfig.module_exit()
        sys.exit()

while True:
    t = datetime.datetime.now()

    # rest the screen outside this range
    if 7 <= t.hour <= 22:
        # update the dashboard
        main()
    else:
        time.sleep(30)

Let's see it in action!

I'm not sure if I'll flesh this prototype out into an open source project so for now the full source code is in a gist.

Final thoughts

I didn't invest any time in working with the four buttons on the left of the screen (they're labelled KEY1, KEY2, etc. in the above picture) but I think it would be worthwhile to integrate them in a fleshed out dashboard — they connect as standard GPIO buttons.

I've learned that e-ink screens are worth the investment for me and I might look into getting a much larger version. The viewing angle is like that of a Kindle and although it's small it's beautiful to look at and has a more 'real' feel than a laptop or smartwatch screen.

I look forward to seeing this technology progress and the commercial applications that arise. I saw that Whole Foods Market were using them in-store back in 2016. It would be nice to have less LEDs staring at me whenever I leave the house!

Join 250+ people signed up to my newsletter on programming and personal growth!

I tweet about code @healeycodes.

How I'm Working Remotely

Andrew Healey — Sun, 26 Apr 2020 15:04:58 +0000

Madeline and I moved our living room around so our desks could be up against the blinds in the lightest area. We have our full work setups here. We carried the gear home in a duffle bag and went back a week later for the chairs which, while being amongst the most ergonomic objects I've ever touched, rattled like heck as we wheeled them home.

When we have meetings at the same time, one of us normally goes into another room. It depends on how much we're speaking. When we're both talking we tend to get louder and louder, without realizing it, until we're almost shouting.

This is where I listen to lofi hip hop streams and build software and record GIFs to make my pull requests all pretty. My MacBook Pro 2019 is paired with a small £12 bluetooth keyboard because my work keyboard won't fit comfortably on the kitchen table.

Last week, I was in the other room and Madeline pranked me by turning the brightness and volume up and down on my MacBook via this keyboard (which I had left behind). I was in a casual call with someone at the time and I was describing, while bewildered, all of the weird symptoms until I heard Madeline stifling laughter on the other side of the door.

I spend a lot of time on Slack, Google Hangouts, and Discord. On Slack, my team has a private team channel for work memes and things that relate to the sprint. I find that when working remote, it's best to have conversations in channels rather than private messages because this shares knowledge and context among the team. Messages in channels are also searchable by the team (even though Slack's search feature leaves much to be desired). I found that it was worth learning a few Slack shortcuts to increase my productivity e.g. CMD+K to quickly navigate, UP ARROW to edit your last message.

We use Hangouts for our regularly scheduled meetings, all synced by Google Calendar. Hangouts has slightly worse quality than Zoom and seems to use up more battery. Discord has been great for pair programming. The stream feature is being improved almost every day. Discord is better for when I'm working closely with someone on a feature but we only need each other every few minutes. In between these moments we can go silent or just idly chat.

I've been quite into videos from a train driver's point of view. They're long and uneventful and beautiful. I leave these 12hr streams on my third screen. These videos are often from Norway, where there is the slow television movement.

The usual meeting schedule, and the cadence of our sprints, is the same as before we all went remote. Before, we each worked 1-2 days remotely each week so the transition hasn't been too bad. I have been trying to be positive and I've been looking forward to reviewing our remote practices for when we get back to the office. It's always been a priority for me to make sure that anyone remote is included fully in discussions and activities at the office.

One trick for connecting the remote person to office life is to let them lead the daily standup while they share their screen (usually viewing the sprint board). This ensures that the remote person understands everything that is going on and feels comfortable asking for clarifications. It ensures that everyone is conscious of the microphone setup at the office too (for when there is one shared microphone e.g. a standup meeting). Alternatively, everyone at the office can dial into the meeting separately to make it a level playing ground.

At lunch Madeline and I usually go for runs in Richmond Park. We work hard to keep two meters+ away from people. It's been beautiful to watch the season change into Summer.

I have been drinking more coffee — cold brew and instant — especially in the mornings. We discovered the joy of barista-style oat milk (shaking the carton plenty before pouring) and it has ruined all other types of oat milk for us! It's apparently made from liquid oats (I looked it up but still don't know what it means) which keeps the milk from separating when the coffee is left still.

Deploying code has been scarier while remote. At the office there is always someone around to help with problems even though the way of grabbing them is the same (we message on Slack if someone's headphones are in). However, the only problem I've run into is GitHub being down, which can affect our CI/CD systems. I'm lucky that Madeline works at the same company as me. She's in a different team but we have been sneaking in some pairing sessions and helping each other out. I've learned more about the area that her team works in. Our teams sit a few meters away at the office but our processes are surprisingly different. It's been fun to learn what works for them and then to take it back to my team. Madeline and I have been speeding up each other's features by calling out questions across the kitchen table when we're stuck.

A junior co-worker told me that they are googling more and getting less stuck because they aren't embarrassed by the idea of other people looking at their screen. I told them that I sometimes barely understand what is happening on my screen let alone anyone else's screen in the vicinity. They also told me they are learning quicker because they are seeking out help less.

My work tickets lately have been all over the stack, touching a little bit of everything. It's been fun and keeps things fresh. I hope soon to be able to deep dive into a technology like I did a few months ago when we shipped a Vue.js prototype.

We have a moodboard at the office — a big whiteboard off in one corner with a grid of days and people's names. We draw pictures, or write words, to describe our days so we can look back in retrospective meetings (once per two-week sprint). Before, when it was rarer to be remote, we sometimes had people describe their moodboard drawings for someone else to quickly sketch up which I enjoyed doing even when my doodles came out goofy. Now, we have moved to a fully remote moodboard with Miro. Which means we can use GIFs! On Miro, you can also see ghost-like cursors, belonging to faraway engineers, move around the shared board.

Our plant growing has extended to things that are consumable. With garlic, onion and spring onion, as well as pak choy, we hope to grow enough to make one meal within lockdown! It is meditative to care for them. I have lost count of the things that are growing in this apartment. And I'm thankful that Madeline is the main waterer.

The bi-monthly hackday was held remotely and Madeline and I spun up a QUAKE III: Team Arena server from scratch and deployed it so that we can use it for the #gaming Slack channel at work. We used to have the server hosted on someone's laptop. Our humorous presentation was about client/server latency and why the person hosting the game had an unfair advantage. We also started building a Vue.js app to help us keep track of goods around the house that we buy on different schedules e.g. kitchen roll, and black bin bags but ran out of time to fully flesh it out.

I've been reading more in the evenings. I bought The Electric State by Simon Stålenhag after admiring his art on Twitter. I've been reading articles from Lobsters, Hacker News, as well as The New Yorker — I really liked The Friendship That Made Google Huge.

Hope y'all are staying safe 👋

Join 250+ people signed up to my newsletter on programming and personal growth!

I tweet about code @healeycodes.

Twitter Ticker Tape with a POS58 Receipt Printer

Andrew Healey — Sat, 11 Apr 2020 09:16:23 +0000

Screens tend to keep me awake in the evening so I wondered if I could print out my Twitter home timeline, live, as I read a book in the evening.

Here's the final project in action:

I was gifted a receipt printer for Christmas to use with my Raspberry Pi. A POS58 USB Thermal Receipt Printer also known as a ZJ-5890K. It's sold by a few different brands under different names.

I couldn't get the drivers to work on Linux, MacOS, or Windows. I was stuck until I found vpatron/usb_receipt_printer — a guide that walks through setting the device up on a Raspberry Pi. It shows how to write to the USB port directly (a tactic I previously tried and failed). The included demo program solved all of my printing problems. Thanks Vince Patron!

Tweepy

Twitter has a powerful API but it's a little cryptic at first glance. Tweepy is a Python library for accessing the Twitter API — it has good documentation and community support.

In order to use Tweepy, we need to create an application on Twitter's Developer platform to get the keys and tokens we need to authenticate.

We will have three files in total.

config.py — for managing our keys.
poller.py — for polling Twitter's API via Tweepy, and converting tweets to text.
posprint.py — for calling the receipt printer.

We save the keys in a separate file. A better way of managing keys is for your application to access them at runtime through environment values. This can be safer (it avoids keys accidentally being pushed to GitHub) and lets our application use different keys in different environments (e.g. local versus live production deployment).

# config.py

consumer_key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
consumer_secret = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
access_token = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
access_token_secret = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

Twitter has a streaming data API but it was harder to set up than the chosen solution.

We ask for the latest 20 tweets every minute and check that we haven't seen them before. Then we format, clean, and print the new ones.

# poller.py

import tweepy
from emoji import demojize
from time import sleep
from collections import OrderedDict

import config
import posprint

auth = tweepy.OAuthHandler(
    config.consumer_key,
    config.consumer_secret
)
auth.set_access_token(
    config.access_token,
    config.access_token_secret
)

api = tweepy.API(auth)

# we use an ordered dictionary so we can remove old tweets
# from the data structure. Our program may be long-running
# and we don't want to run out of memory
seen_statuses = OrderedDict()

while True:
    public_tweets = api.home_timeline(tweet_mode='extended')
    for status in public_tweets:

        # we're interested in statuses we haven't seen before
        if status.id not in seen_statuses:

            # store the statuses we use
            seen_statuses[status.id] = status

            # dump old statuses that can't appear again
            if len(seen_statuses) > 20:
                seen_statuses.popitem()

            user = status.user.screen_name + ':\n'

            # get the full tweet text regardless of type
            # https://github.com/tweepy/tweepy/issues/935
            if hasattr(status, 'retweeted_status'):
                tweet = user + status.retweeted_status.full_text
            else:
                tweet = user + status.full_text

            # trim unprintable characters that POS58 can't output
            # by turning emojis into textual representation.
            # 😅 becomes :grinning_face_with_sweat:
            tweet = demojize(tweet).encode(
                encoding='ascii',
                # throwaway non ascii characters
                errors='ignore'
            ).decode()

            posprint.output(tweet)

    # twitter restricts statuses/home_timeline to once per minute
    sleep(65)

Let's modify Vince's demo program vpatron/usb_receipt_printer to export the function we call above — posprint.output(tweet) — with our textual tweet.

# posprint.py

import usb.core
import usb.util
import textwrap

"""Demo program to print to the POS58 USB thermal receipt printer. This is
labeled under different companies, but is made by Zijiang. See
http://zijiang.com

MIT License — Copyright (c) 2019 Vince Patron
"""

# In Linux, you must:
#
# 1) Add your user to the Linux group "lp" (line printer), otherwise you will
#    get a user permissions error when trying to print.
#
# 2) Add a udev rule to allow all users to use this USB device, otherwise you
#    will get a permissions error also. Example:
#
#    In /etc/udev/rules.d create a file ending in .rules, such as
#    33-receipt-printer.rules with the contents:
#
#   # Set permissions to let anyone use the thermal receipt printer
#   SUBSYSTEM=="usb", ATTR{idVendor}=="0416", ATTR{idProduct}=="5011", MODE="666"

def output(data):
    # find our device
    # 0416:5011 is POS58 USB thermal receipt printer
    dev = usb.core.find(idVendor=0x0416, idProduct=0x5011)

    # was it found?
    if dev is None:
        raise ValueError('Device not found')

    # disconnect it from kernel
    needs_reattach = False
    if dev.is_kernel_driver_active(0):
        needs_reattach = True
        dev.detach_kernel_driver(0)

    # set the active configuration. With no arguments, the first
    # configuration will be the active one
    dev.set_configuration()

    # get an endpoint instance
    cfg = dev.get_active_configuration()
    intf = cfg[(0,0)]

    ep = usb.util.find_descriptor(
        intf,
        # match the first OUT endpoint
        custom_match = \
        lambda e: \
            usb.util.endpoint_direction(e.bEndpointAddress) == \
            usb.util.ENDPOINT_OUT)

    assert ep is not None

    # print!
    lines = textwrap.wrap(data, width=30)
    for line in lines:
        ep.write(line + '\n')
    ep.write('\n\n\n\n')

    # reattach if it was attached originally
    dev.reset()
    if needs_reattach:
        dev.attach_kernel_driver(0)
        print('Reattached USB device to kernel driver')

Our program is started by running poller.py.

If everything went correctly, our POS58 printer should output tweets as they arrive in the following width-restricted format which avoids breaking words where possible.

jessfraz: Ive started using
Pocket to keep articles I want
to read later and also
combined it with a bot to read
RSS feeds. Its alright so far.
What are some tools like this
you cant live without?

Thanks again to Vince Patron for his comprehensive guide!

Join 250+ people signed up to my newsletter on programming and personal growth!

I tweet about code @healeycodes.

Book Review: The Developer's Guide to Content Creation

Andrew Healey — Mon, 09 Mar 2020 21:25:01 +0000

I'm in year two of technical blogging and I've learned a lot. I have far far more views and comments than I thought I would at this point. At the beginning I was throwing my thoughts into the void and it felt good. Reading The Developer's Guide to Content Creation reminded me of those roots. It inspired me.

I love seeing people blog about their passions and thoughts especially when it's on a personal website. Personal websites are cool and weird and make the internet more enjoyable for me. If there's a link in your bio there's a high chance I'll click it.

So I'm very happy, on this quiet and unusually cold Sunday, that there are resources for those who are starting out on this journey!

The Best Beginner Introduction to Blogging

Without a doubt The Developer's Guide to Content Creation is the best resource for those looking to start writing content for developers. Before reading it, I had a shortlist of topics that I thought such a resource should cover and it nailed every one. Morillo's writing is clear and engaging. She tells us how to write content that is accessability in presentation but also in content. She reminds us how not to write as well:

"Just follow these simple steps"
"It’s easy to put together"
"The answer is obvious"

I’ve realized that many writers of technical content use these words to make the subject matter seem less intimidating to the reader. They genuinely want to remove friction for their users, but they’re using language that makes the reader feel like the problem is with them.

Rich in Resources

The Developer's Guide to Content Creation also contains copious amounts of worksheets to inspire you, help you strategize, and put together a content calendar.

There are resources linked throughout the book that compliment each section. There's also an example of how to insert resources into an article in a more descriptive manner. To mirror one of Morillo's examples, take GitHub's trending page.

"GitHub has a list of trending repositories. Check it out here."
"Search through GitHub's list of trending repositories ."

For a reader, the second version flows better.

Morillo recommends comprehensive resources like DigitalOcean's Technical Writing Guidelines as well as tools that take seconds to set up and run on your writing.

AlexJS is an open-source tool that will help you "spot insensitive, inconsiderate writing."

Hemingway App is an online editor that helps you simplify your writing. It highlights confusing sections, areas that are too long, and more.

To add my own, reading other technical blogs can be a good resource and I'd recommend Hacker Noon's shortlist for Personal Developer Blog of the Year to get started. Drew DeVault posts great content and will occasionally pay out $20 to people starting a technical blog on an open platform (i.e. not Medium). Google have also released a free course on technical writing although I believe it's aimed more towards those writing technical documentation.

The Shape of a Post

Morillo talks a lot about how technical blogs are consumed and how they appear to users.

Before you start writing, you need to understand how people read online. People don’t read every word and every sentence; they scan. They’re scanning for the most relevant information and things that pop out at them will command their attention.

Thinking about the shape of a post can help during the editing stage. It's a way to cut through complexity and grab readers who would have otherwise failed to skim read blocks of text and given up. It also helps those who come to your post only looking for that one line of code that will unblock them.

We're reminded that people scan for "keywords, section headings, bullets, anchor text (when hyperlinked), words that are bolded or italicized for emphasis".

The Developer's Guide to Content Creation covers topics that were getting dusty in the back of my mind as well as delivering a nuggets of wisdom. Even as I write this post, I'm considering Morillo's advice. It's for this reason that I would recommend this book to anyone writing content for developers no matter their experience. It has changed the way I will approach writing for my personal blog as well as for my company's blog.

Following this book, Morillo is currently writing The Developer's Guide to Book Publishing — I'm looking forward to it!

Join 200+ people signed up to my newsletter on programming and personal growth!

I tweet about code @healeycodes.

DEV Article Analysis

Andrew Healey — Sun, 19 Jan 2020 16:33:08 +0000

My second passion alongside coding is writing. Whenever I can combine them, I do.

This weekend I used the DEV API to build a tool that analyzes your articles over time. It's an interactive website where you can enter DEV usernames and get text statistics shown in pretty graphs. The metrics that I chose were reading level, ease of reading, and sentiment analysis.

It's open-sourced at healeycodes/dev-article-analysis!

Readability

The Flesch–Kincaid readability tests were developed in the U.S. Navy in the 70s. These scales are used to measure and compare books, newspapers, and articles. In Florida, they are used to encourage readable language in insurance policies.

Florida Statutes Section 627.4145 (1) (a) The text achieves a minimum score of 45 on the Flesch reading ease test as computed in subsection (5) or an equivalent score on any other test comparable in result and approved by the office;

Flesch-Kincaid Grade scores text at a U.S. grade level. A score of 10.4 means that a tenth grader would be able to understand the article.

Flesch reading ease is a similar test with different weightings. It results in a score generally ranging between 0-100. Higher means easier to read. Wikipedia has a table which relates score to grade level with some notes.

Technical articles, especially those with code excerpts, get wide-ranging scores. However, a user's writing style (e.g. a similar amount of code per article) means that a reasonable line can be drawn over time, perhaps leading to some basic conclusions. text-readability was used to calculate these scores in the project.

Sentiment

Sentiment analysis helps us understand the emotional polarity of a text. For example, the phrase I love cats, but I am allergic to them. gives a comparative score of 0.1111111111111111. To calculate this, a list of words (and emojis) are given weightings. -5 to 5, positive to negative. The only tokens in our sentence with scores are { allergic: -2 }, { love: 3 }. We take the number of tokens, 9, and calculate (3 + -2) / 9 to find the comparative score. This example is used in the documentation of sentiment.

The DEV API provides the text of an article in HTML or markdown, neither of which is friendly to analysis. This is why remove-markdown is used to reduce the body_markdown value to plaintext.

Interaction

Users enter a DEV username. As the articles are retrieved, a progress status (and a random positive emoji) are sent to the page via WebSocket. This means the user isn't left sitting looking at a spinner.

I normally use Chart.js to render basic line charts but wondered if there were a simpler solution. I tried ApexCharts.js and didn't run into any problems. The online documentation is good and I searched and found the solution to one of my errors in a GitHub Issue. However, I might as well have used Chart.js because the level of customization I required meant the syntax for both libraries was near-identical.

The charts can be hovered to see article titles and their scores for each metric.

When arriving at the website, the graphs are already filled with precalculated data from my own DEV username (healeycodes). This is to help visitors better understand what the end result of their interaction will be.

Deployment

I developed this Node/Express application on Glitch. Their cloud IDE and console lets me move fast and get feedback from people seconds after I've made a change. You can export the project to a GitHub project (it creates a glitch branch which you can open a pull request from).

Sometimes I want to upload image files and in the past I found this tricky. Glitch wants you to use their asset mechanism for this, which makes exporting the whole project to GitHub harder than it needs to be. My workaround is to upload the image to the project using the asset mechanism, copy the long URL they provide, wget this URL in the console which downloads it into the project, and finally using mv to change the name of the image.

This often brings the cloud IDE out of sync but entering refresh in console fixes this.

Join 200+ people signed up to my newsletter on programming and personal growth!

I tweet about code @healeycodes.

Gatsby Sparks Joy

Andrew Healey — Sun, 12 Jan 2020 10:37:07 +0000

I migrated from Jekyll to Gatsby recently and so far I've had a really neat time. The whole process took about a week of casual coding (a few hours here and there). The Gatsby ecosystem enabled me to quickly add a few features to my blog that I thought were missing; dark mode, better syntax highlighting, and the ability to design with components.

Gatsby starters are boilerplate Gatsby sites maintained by the community. One of the reasons I love them is that they use Semantic HTML. This is great because:

It helps with search engine optimization — web crawlers are able to understand which parts of your pages are important.
It helps with accessibility — for people who use non-traditional browsers and screenreaders.
It helps with maintenance — I was able to pick up a starter and understand what the different parts of the template referred to due to the semantic tags.

Take this example from gatsby-starter-blog — the most popular starter and the base for my current blog (in-line styling removed).

<article>
  <header>
    <h1>{post.frontmatter.title}</h1>
    <p>{post.frontmatter.date}</p>
  </header>
  <section dangerouslySetInnerHTML={{ __html: post.html }} />
  <hr />
  <footer>
    <Bio />
  </footer>
</article>

I've seen quite a few beginner web development resources that skip on semantic HTML and encourage what I'll call 'div-spamming'. The HTML5 spec weighs in on this issue.

Authors are strongly encouraged to view the div element as an element of last resort, for when no other element is suitable. Use of more appropriate elements instead of the div element leads to better accessibility for readers and easier maintainability for authors.

Coming from Jekyll

I started blogging with Jekyll a year ago because I hosted my blog on GitHub pages and it was the static site generator with the least friction. It was a great choice at the time as it enabled me to get up and running straight away.

I've seen many people warning others (engineers in particular) to avoid rolling their own blogging solutions. The advice is that you should start writing and publishing first. This is because building a blog can function as procrastination and who knows if you actually enjoy blogging (the activity) or the idea of having blogged (the achievement).

With Jekyll, I used basic markdown and transferring written content to Gatsby wasn't too hard. Images had to be moved from one disorganized folder into separate folders. URLs were a bit of a pain and took 1.5hrs of manual work. I wanted all of my old posts to keep their location on the web so I added a front matter tag called path to override the default naming scheme. My old URLs were too long and included categories (which I'm still to implement) so the path scheme from now on will be the title only.

I extended onCreateNode in gatsby-node.js. I'm not sure if this is the best practice way to implement this feature but it works great so far.

exports.onCreateNode = ({ node, actions, getNode }) => {
  const { createNodeField } = actions

  if (node.internal.type === `MarkdownRemark`) {
    // Check to use legacy path
    const slug = node.frontmatter.path || createFilePath({ node, getNode })
    createNodeField({
      node,
      name: `slug`,
      value: slug,
    })
  }
}

Syntax highlighting

Code excerpts show up in a lot of my posts and I like them to be easy to parse.

I installed gatsby-remark-prismjs for syntax highlighting and was up and running in about an hour with another hour spent tinkering styles to match my light/dark mode toggle. I use New Moon Theme by Tania Rascia for my code excerpts. I couldn't find a version of the theme for PrismJS so I extracted the styling from Tania's (MIT-licensed) blog. My site's general color theme is custom.

One of the reasons I'm mentioning plugins is that I found it hard to integrate them with Jekyll and I feel like it wasn't just my inexperience with Ruby that was holding me back. Perhaps it's due to the hype surrounding Gatsby which means there are up-to-date resources. I've contributed one (small) open-source fix to the Jekyll project and I would still recommend it for anyone looking for a sensible system for HTML/CSS that has wide community support e.g., GitHub pages, Netlify, etc. If you want to avoid JavaScript, Jekyll is the way to go.

For my light/dark mode I use gatsby-plugin-dark-mode which works well out of the box and has good (but not great) documentation. For theme-toggling, I researched and found that a common pattern was to declare CSS variables in body scope and then to override these variables in class scope. This way, the dark class can be added to the <body> tag which means dark CSS variables take precedence due to CSS Specificity. Classes are then toggled on and off the <body>, storing the preference in local browser storage.

body {
  --bg: #eaeaeb;
  --textNormal: #414158;
}

body.dark {
  --bg: #21212c;
  --textNormal: #eaeaeb;
}

Designing with components

The first React component I wrote for my blog was for wrapping the <ThemeToggler /> from gatsby-plugin-dark-mode into a component. It switches between a sun and a moon to let the user know which theme can be switched to. The base for this is the example code from the docs.

<ThemeToggler>
  {({ theme, toggleTheme }) => (
    <label style={{ cursor: `pointer`, opacity: 0.8 }}>
      <input
        style={{ display: `none` }}
        type="checkbox"
        onChange={e => toggleTheme(e.target.checked ? "dark" : "light")}
        checked={theme === "dark"}
      />
      {theme === "dark" ? `☀️` : `🌑`}
    </label>
  )}
</ThemeToggler>

I've never used React as part of a blogging solution. I like the hierarchical UI approach that's encouraged. Including CSS-in-JS makes sense for the scale of my website. It's easier for me to reason about and quickly tinker with.

How I deploy

My website source exists in a GitHub repository. I write in markdown in VS Code, commit, and push. Netlify is connected to the repository and builds and deploys every commit to master. The build process takes 2m50s (30s locally).

I previously used Netlify for Jekyll and setting up either static site generator involved about 10 clicks and entering one or two build commands. The walkthrough covers everything.

Overall, the Gatsby experience has been very enjoyable. Everywhere I look in my online bubble (Twitter, DEV, lobste.rs) people are talking about Gatsby. It's nice to be part of the crowd.

I'm also happy that my site (despite being React-based) works fine without JavaScript enabled (barring the theme toggle, which I might hide with <noscript> styling).

Join 200+ people signed up to my newsletter on programming and personal growth!

I tweet about code @healeycodes.

Hardest JavaScript Puzzle I've Ever Solved

Andrew Healey — Sun, 17 Nov 2019 20:20:01 +0000

I love code puzzles. Always have. My deskmate told me about a puzzle that no one in the office had been able to solve when they shared it around. I was intrigued because of the challenge but also because it was a JavaScript puzzle.

I'd understand if it were a Project Euler problem (they can be mathy) or perhaps if no one had found the optimal solution to a LeetCode 'Hard' (I've seen some problems that once upon a time were research papers).

The puzzle took me two weeks to solve. I became stuck almost instantly and then it hung around in the back of my mind until the solution came to me. First, let's take a look at the easier version of the problem which helped me unlock the harder version.

Don't scroll too fast unless you want spoilers.

Codewars: Multi Line Task++: Hello World

Write a function f that returns Hello, world!.
Every line must have at most 2 characters, and the total number of lines must be less than 40.

Without the line restriction. The solution is:

function f () { return 'Hello, world!'; }
// or
const f = _ => 'Hello, world!';

I started splitting up the code and shortened the variable declaration. We can throw away the const and allow the function to exist in the global scope. We can also use template strings to break up the string into multiple lines.

Errors incoming.

f
=_
=> // Uncaught SyntaxError: Unexpected token '=>'
`H
el
l,
 w
or
ld
!` // This string has newline characters in it!

My next idea was to define the function inside an object and then retrieve the function out of the object.

f=
{g
()
{
return 'Hello, world!'; // Too long!
}}
[`
g
`[
1]
]

// 'Beautified'
f = {
        g() {
            return 'Hello, world!';
        }
    }
    [`
g
` [
        1
    ]]

No errors, I was declaring a function but now I couldn't figure out how to return from the inner function without using the return keyword. It felt like I was close but I wasn't. I was still stuck on defining the string without newline characters as well.

Finding inspiration: JSFuck

JSFuck is an esoteric and educational programming style based on the atomic parts of JavaScript. It uses only six different characters to write and execute code.

Reading through this project's source code really opened my mind to some parts of JavaScript that never really come up unless you're doing something like writing a library or code golfing.

Once I figured out how to remove the newline characters from the Hello, world! message (escaping with a backslash \) everything else fell into place. I could now use square brackets [] on objects to run pretty much anything. However, one of the requirements was to keep the total line count under 40 (one of the reasons why using JSFuck encoding was out of the question).

My idea was to create a new function by calling Function.prototype.bind on a String method. I used String.prototype.trim because it had the shortest name (and conveniently got rid of any trailing newlines too).

f=
''
[
'\
t\
r\
i\
m'
][
'\ // Get a new function where `this` is "Hello, world!"
b\
i\
n\
d'
]` // Tagged template, see below
H\
e\
l\
l\
o\
,\
 \
w\
o\
r\
l\
d\
!`

I also used tagged templates to pass Hello, world! as an argument to bind.

Tags allow you to parse template literals with a function. The first argument of a tag function contains an array of string values.

Let's take it up a level to the harder version that started this journey!

Codewars: Multi Line Task∞: Hello World

Write a function f that returns Hello, world!.
Every line must have at most 1 character, and the total number of lines must be less than 145.

Without having first solved the two-characters-per-line version I don't think I would have come close to answering this version.

The solution I went for is the same, we use bind on trim and pass the message as an argument (without template tags this time). To access the String object we use []+[] which evaluates to "".

Since we can no longer escape the newline character from within a template string we have to use a workaround. The property names (trim, bind) and the message (Hello, world!) have to be built with concatenated variables.

We use destructuring assignment on a template string and use empty slots in the first array to 'skip' assigning the newline character to anything. Like this:

[,a,,b,,c] = 'xHxix!' // avoid the "x"s
a + b + c // Evaluates to: "Hi!"

I didn't optimize the solution any further once it passed the requirements. It's been left verbose to better explain what's going on (for example, we only need one "l" in the template string).

[ // With destructuring assignment, start declaring variables
,
t
,
,
r
,
,
i
,
,
m
,
,
b
,
,
i
,
,
n
,
,
d
,
,
H
,
,
e
,
,
l
,
,
l
,
,
o
,
,
c // Comma
,
,
s
,
,
w
,
,
o
,
,
r
,
,
l
,
,
d
,
,
x // Exclamation mark
]
=
`
t
r
i
m
b
i
n
d
H
e
l
l
o
,

w
o
r
l
d
!
`
f // Start declaring our function
=
( // This evaluates to "" or, the String object
[
]
+
[
]
)
[ // `trim`
t
+
r
+
i
+
m
]
[ // `bind`
b
+
i
+
n
+
d
]
( // Can use parentheses or template tag syntax to call `bind`
H
+
e
+
l
+
l
+
o
+
c
+
s
+
w
+
o
+
r
+
l
+
d
+
x
)

I'm definitely taking a break from language-specific coding puzzles — give me logic over syntax! However, I'm glad I scratched this itch.

Join 150+ people signed up to my newsletter on programming and personal growth!

And tweet about tech @healeycodes.

DEV Community: Andrew Healey

Creating Randomness Without Math.random

Number Generation

More on Pseudorandom Number Generators

Entropy Required

So, Mission Accomplished?

A Practical Introduction to Graphs (Network Diagrams)

What's in a Graph?

Displaying Graphs

Shortest paths

Polyfilling the Fetch API for Old Browsers and Node.js

Examples of polyfilling and transpiling

Using fetch in old browsers

Using fetch in Node.js

The Flow of Knowledge in a Team (or Why I Dig Tickets)

Feature or fix?

Where do tickets come from?

Where do tickets live?

Estimation

Interacting with tickets

Document the world

Adding Comments to Gatsby with Netlify Serverless Functions + GitHub

The serverless function

Potential downsides

My Netlify review

Hacking Together an E-ink Dashboard

Final thoughts

How I'm Working Remotely

Twitter Ticker Tape with a POS58 Receipt Printer

Tweepy

Book Review: The Developer's Guide to Content Creation

The Best Beginner Introduction to Blogging

Rich in Resources

The Shape of a Post

DEV Article Analysis

Readability

Sentiment

Interaction

Deployment

Gatsby Sparks Joy

Coming from Jekyll

Syntax highlighting

Designing with components

How I deploy

Hardest JavaScript Puzzle I've Ever Solved

Codewars: Multi Line Task++: Hello World

Finding inspiration: JSFuck

Codewars: Multi Line Task∞: Hello World

Using `fetch` in old browsers

Using `fetch` in Node.js