DEV Community: Mark Lennox

Differential Privacy - Sharing Insights Privately, In Public

Mark Lennox — Mon, 28 Oct 2019 17:57:58 +0000

This is a cross-post from my personal blog

In a previous article I explained there is no such thing as anonymised data. If you don't want a privacy breach, don't release your dataset - in any form. As I explained, even statistical and demographic summaries of the data can be used in so-called reconstruction attacks to reveal information about individuals in the dataset.

Photo by Josh Rose on Unsplash

But for some datasets - such as census data - it is a legal obligation to release statistical analysis. It is also a legal obligation to preserve the privacy of citizens who participated in the census, but releasing the required statistical analysis will breach privacy...

You're damned if you do, and damned if you don't!

computer science got us into this mess, can it get us out of it? - Latanya Sweeney

Privacy in the context of statistical analysis

Before we dive into what differential privacy is, we should look at what we - as potential dataset creators - should consider if we care about privacy.

Keep it private

It bears repeating - there is no such thing as anonymised data, so we know we must never release the raw data.

Differential privacy is not concerned with data warehousing nor security, we'll just wave our hands here and say "the data is never public".

Protect the individuals

We want to ensure that the presence, or lack, of an individual in a dataset will not expose any details of that individual - usually easier with a large dataset.

Limit the number of queries

While so-called reconstruction attacks cannot be used against differentially private functions, allowing unlimited queries is a de-facto privacy breach. The number of queries should be limited because we know each query results in a loss of privacy.

If we set an upper limit to how much privacy loss is acceptable, and we quantify the privacy loss of each query, then we have the basis of a privacy budget to limit queries from individual researchers.

Differential privacy

Differential privacy mathematically guarantees that

anyone seeing the result of a differentially private analysis will essentially make the same inference about any individual's private information, whether or not that individual's private information is included in the input to the analysis

Not only that, but the results of differentially private statistical analysis are immune to reconstruction attacks, linkage attacks, re-identification, and can be combined with other datasets - present and future - without exposing privacy.

That is amazing.

How do we do this?

We add noise.

But how much noise should be added? We want our dataset to be useful, but also preserve privacy. If we add a small amount of noise the statistical results will be very accurate, but the privacy of individuals will be threatened. If we add a lot of noise then the privacy of individuals is guaranteed, but the statistical accuracy will suffer and the data will be useless.

In order to achieve the most accurate statistical results while preserving privacy, it is vitally important to accurately calculate the minimum amount of noise.

Privacy loss parameter

Each query made against a differentially private system will result in a loss of privacy, but as curators of the dataset we choose to set a maximum acceptable level of privacy loss using whatever criteria are relevant to the application. ε-differential privacy quantifies this maximum acceptable loss of privacy using the privacy loss parameter ε (epsilon).

ε - privacy loss parameter

The larger the value of ε the more privacy is lost. The variable ε quantifies the privacy loss for any query against an ε-differentially private system. A suitable value for ε could be between 0.1 and 1, although in practice much higher values are in use.

Sensitivity determines noise level

To determine how much noise should be added to the results of a particular query we need to figure out the sensitivity of that query. Sensitivity is a measure of the influence an individual's data can have on the output of a query. The higher the sensitivity, the higher the level of noise required to preserve privacy.

Imagine a small dataset containing the data of only 100 people. This means each person contributes roughly 1% of the total information. Clearly any query will have high sensitivity as the dataset is so small. We would need to add a lot of noise to preserve privacy.

Even in larger datasets queries can have high sensitivity. A query such as "billionaires with blue eyes, living in Florida, and taller than 1.8 metres" will have high sensitivity no matter how large the dataset.

It's straightforward to calculate the worst-case sensitivity of any query but more difficult to calculate the exact sensitivity of a particular query. This results in a higher level of noise than required to preserve privacy, and reduces accuracy of statistical analysis.

The accurate calculation of the sensitivity of queries is an important area of research as it aims to squeeze more accuracy from statistical queries against datasets while preserving privacy.

Privacy Budget And Composability

Each query made against an ε-differentially private function results in a loss of privacy, and as we said the dataset creator chooses the maximum acceptable privacy loss. It is possible for users querying the dataset to choose a privacy loss for each query that suits their needs as long as that privacy loss is less than the acceptable maximum privacy loss set by the dataset curator.

Researchers can choose to make many queries with low privacy loss, say 1 / 10 of ε, resulting in lower accuracy but higher privacy (ie. lower privacy loss for the individual) for each query, or choose a higher privacy loss and make fewer queries but with higher accuracy.

Privacy budget

Broadly speaking, you can track your privacy budget by adding together the fraction of ε used in each query. If you have made 10 queries each with a privacy loss which is 1/10 ε, then you can make no more queries as you have used your privacy budget.

A useful visualisation of this process is a hidden image revealed by uncovering random squares where the size of the square is determined by ε. You can see that smaller fraction of ε allows for more queries while still preserving the mystery of the hidden image. Queries using a larger fraction of ε would very quickly reveal the image.

Photo from Arif Riyanto on Unsplash

Most queries, as described above, are known as sequential queries and simply add their privacy loss to the total used so far. There are also parallel queries that only add a single unit of privacy loss no matter how many queries are made in parallel.

Composability

The result of each query is differentially private, and the combination of query results is also differentially private - a valuable feature of differential privacy called composability.

Composability means that while each subsequent query does increase privacy loss, the queries cannot be combined in a way that will lead to privacy loss greater than the sum of all the privacy loss from queries up to that point. As mentioned before, ε-differentially private statistical analysis is immune to reconstruction attacks. If you make 10 queries with privacy loss ε you will only ever have 10ε privacy loss - incredible!

Group privacy

What we have been talking about up to this point has applied to the details of a single person. What if we are concerned with a group of people, such as a family? The same principle of composability applies to groups of people. The privacy loss of an individual is ε and the privacy loss of a group of people is kε where k is the number of people in the group.

Accuracy

We're adding noise, so how useful can any statistical analysis be?

The outputs from ε-differentially private functions are noisy. If you make the same query twice you'll get two different answers. Remember that smaller ε means larger privacy, and thus less accuracy, which in practical terms means more variation between results from the same query.

The query sensitivity has a large part to play in the accuracy of any analysis. We know queries against small datasets have high sensitivity, and thus require relatively more noise to preserve privacy. Similar queries against large datasets have relatively lower sensitivity and need less noise to preserve privacy.

Querying larger datasets result in more accurate results while preserving the same level of privacy.

Privacy washing

Just because a company is open about its approach to privacy does not mean it has your best interests at heart. I can forsee a lot of companies convincing people that their data is private and "safe", encouraging them to share more - this is just privacy-washing the fact that they are data mining their customers.

Apple, Google, and the U.S. Census are high-profile examples of ε-differential privacy in the real-world. The U.S. Census has been very open and clear about how they implement differential privacy, as has Google with regard to its RAPPOR technology. Apple, however, have been less forthcoming.

Despite data-mining issues, companies can badly implement differential privacy. This can either be from simple incompetence or by choice. It takes time and money to properly implement differential privacy - as mentioned before data security is essential. A company might choose to cut costs and knowingly use a poor implementation that leaks their customer's privacy.

No such thing as anonymous data

Mark Lennox — Wed, 02 Oct 2019 07:35:03 +0000

New York City publicly released anonymised data of almost two million taxi rides - including fares and tips - in order to facilitate research into transport, traffic, and civic planning by independent researchers. However, they made a mistake - they only replaced the tax medallion number with the MD5 hash of the number. MD5 hashes are computationally weak which allowed a privacy researcher Vijay Pandurangan to re-identify every taxi driver in the dataset exposing their home address, and income.

Image provided by Luke Stackpoole

Not only that, other researchers showed that the addresses of celebrities could be exposed by cross-referencing images of them entering or leaving taxis in paparazzi pictures.

Despite the fact that the cause of this disastrous privacy breach was actually due to using a weak hashing algorithm to mask the highly sensitive taxi medallion number, as Cynthia Dwork says

de-identified data isn't

By this she means that either the data is not de-identified (anonymous), or is not data!

De-identified? Anonymous?

Even if efforts are made to achieve anonymised datasets - masking information, generalising, or even deleting directly and indirectly identifying information, privacy can still be jeopardised. Even if everything is done perfectly with no mistakes, privacy is never guaranteed. Once a dataset is public, it is open to privacy attacks including re-identification attacks, differencing, record linkage - all of which I'll briefly explain the various attacks below.

Any dataset is also open to attacks from unknown or future attacks. A dataset might have been processed using rules that should guarantee privacy, but that guarantee has either failed immediately or will fail in the near future either because of increases in computing power, new techniques, or new datasets from unrelated organisations that can be used to reveal identities.

Difference attack

If you have some knowledge about an individual, you can use that to examine multiple statistics in which the individual's data is included. In short, a differencing attack can occur where multiple questions on their own do not risk re-identifying an individual, but if two or more of the questions are asked they risk exposing an individual's identity.

For instance, if we know an individual is male, and vegetarian and attends a dinner where aggregate data of attendees is released, we can directly identify the individual if he is the only man to order a vegetarian meal. This may seem trivial, but the full data record from the event for the individual could be leveraged in a separate attack in different datasets.

This is a trivial example, easily noticed and understood by humans. Algorithms can quickly process large datasets, combining multiple datapoints, easily exposing an individual's details. A simple task for the algorithm but impossible for a human to notice and avoid before a dataset is released.

Re-identification attacks

If an attacker knows even a single piece of information about an individual, they can use that to identify an individual's information in a publicly available, anonymised dataset.

In 2006 Netflix published their user's movie-ranking information. The data had been anonymised - de-identified - by replacing the users names with random numbers, and changing their personal details. Arvind Narayanan and prof. Vitaly Shmatikov managed to re-identify individuals in the dataset using publicly available IMDB movie ratings. In fact, it was discovered that knowing the date of two public movie reviews was enough to have a nearly 70% chance of re-identifying an individual in the so-called anonymised dataset.

Record Linkage

Record linkage happens when an attacker can connect anonymised, often unrelated, datasets in order to reveal the identity of an individual. This is possible when datasets contain the same information about an individual, called indirect identifiers, or quasi-identifiers.

Quasi-identifiers do not directly identify an individual, but when combined with other quasi-identifiers will identify that individual.

A famous example of a linkage attack ocurred in Massachussets in 1997. There was some controversy over patient information released publicly, despite assurances from the governer William Weld that all direct identifiers had been deleted. Famously Latanya Sweeney took this publicly available, and supposedly anonymised data, and compared it with another publicly available dataset - the voting register, which she obtained for $20. With these two sets of data she was able to find the Governer's personal medical records. The reason this attack succeeded was the patient's zip code, date of birth, and sex, were untouched in the dataset of hospital attendance.

Image provided by John Barkiple

Reconstruction attack

This last attack is probably the most dangerous because it works not by using the dataset directly, but by relying on statistics from the dataset. Researchers often release statistical aggregations from their publicly available datasets - percentage of people who are married, single, divorced, or income bands, gender, etc. These seem innocuous, but every piece of aggregated data filters the possible set of records that could have contributed to that statistic.

Simson Garfinkel - Senior Scientist on the U.S. Census Bureau team for disclosure avoidance - showed that it is possible to reconstruct the personal data of individuals from a summary of the mean and median age, and frequency count broken down by some demographics - gender, income, marital status.

Even if the source dataset is never exposed to the public, the more statistics that are released, the more individual's details will be exposed.

Is honesty the end of privacy?

To gain value from gathered data the answers must be accurate, however as Cynthia Dwork and Aaron Roth point out in "The algorithmic foundations of Differential Privacy"

"overly accurate answers to too many questions will destroy privacy in a spectacular way"

What is the answer then? If everyone lies, all data gathered is useless, which is bad news for your you and your doctor. If you do let him know it hurts when you cough then you'll get proper treatment, but how do you know your personal data is safe? What if it helps others to share statistics your data contributes to? There is a choice to be made between never sharing your data or allowing it to be shared in a safe way with an informed choice on how much privacy risk you'll face for sharing different levels of detail.

Differential Privacy offers a solution to the problem of balancing privacy and generating useful statistical output from datasets - something I'll explain in another article.

The cover image of this article was provided by Clint Adair

Face off : Three(.js)

Mark Lennox — Mon, 23 Sep 2019 22:30:49 +0000

This is a cross-post from https://www.webpusher.ie/2019/09/23/face-off

I've been interested in photogrammetry which involves lots of photos, but what is the least number of photos required? Well, it turns out that all you need is one photo.

In this article I'll quickly run through how to generate a 3d mesh from a photograph, and then use netlify to publish a threejs based model viewer to see the final result. The process is entirely manual at the moment!

How to create a 3d mesh?

After some failed experiments with mobile photogrammetry apps I decided it was too difficult to easily capture many photos.

OpenCV

I then tried OpenCV Stereo SGBM which uses stereo photos, compares the differences and uses geometry, and some advanced techniques, to estimate a depth map, then colour it uses the original image.

This was very promising when using the example photos, but despite spending a lot of time on it, I found it impossible to get even barely passable results. The image below shows headphones resting on an orange book - and that is the best angle to view it from!

This is definitely something I will come back to. I didn't try it at the time, but there is a tool to help tweak the parameters. The other disadvantage is that building OpenCV is an arduous process.

Machine Learning to the rescue!

I used the popular PRNet CNN (convolutional neural net) to generate a wire mesh, which met with immediate success!

There is an accessible(ish) explanation of how the process works.

Is this a fake?

More like a deep fake!

PRNet is trained to generate a UV map, which encodes both volume and shape of a 3d object. Three important elements can be generated from this UV map

sparse feature point prediction - eyebrows, eyes, nose, mouth, lips, and chin
dense feature point prediction
full 3-dimensional mesh model, with colors from the input image.

Show me the code!

It's not just an academic paper there is fully working example code

It takes a little bit of work to get this running I'd recommend using my Pyenv configuration advice from a previous article.

I stuck to the requirements to get this working - Python 2.7 - so I didn't investigate if more recent versions of Python would work. If you do bump your Python version be aware that as of publishing (September 2019) dependencies of TensorFlow limit Python to version 3.6.

$ pyenv virtualenv 2.7.14 prnet
$ pyenv local prnet

You'll also need to install dlib.

$ pip install --upgrade dlib

I'd recommend upgrading pip and setuptools to the latest version before you install the project dependencies

$ pip install --upgrade pip setuptools

Now you can install the dependencies

$ pip install -r requirements.txt

Generating the mesh

You'll need some images saved in a known directory. Once you have that, simply run:

$ python demo.py -i {the directory holding images with faces} -o {output directory for the obj files}

You'll probably see lots of warnings (I did) which would likely be reduced or removed by upgrading your Python version, but I'm not doing to until later in the project!

In your output folder you should see a .obj file for each image you provided. Note, you will only see one face per image - there is no automatic support for multiple faces.

Viewing the output

I used the free and capable MeshLab to immediately open and view the models - I laughed at my stupid face spinning before me!

Even though it was not a goal of my mystery side-project, I wanted to leverage my webdev skills to put this online for all to see - everyone should be laughing at my stupid spinning face!

Threejs and Netlify to the rescue!

I use Netlify to publish my Gatsby based blog, so I knew it was going to do the heavy-lifting of deploying my new 3d monster gallery.

I didn't want to be messing about with bundling the 'app' so I just wrote a little shell script to copy the threejs library and associated support code from the node_modules directory to the src folder where I could import it into my es6 modules.

The netlify build settings I used are:

Repository : github.com/mlennox/three-dee-mee
Base directory : Not set
Build command : sh prepare.sh
Publish directory : src

The full, working code is on github and the actual working demo is available at https://thirsty-hypatia-f27848.netlify.com/ and looks like this (it may take a few seconds to initialise)

Just click the gurning visages in the row of thumbnails to load the associated three-d mesh, and you can control the hideous visage by clicking, dragging, and scrolling to zoom.

(npm) publish, and be damned!

Mark Lennox — Sun, 16 Jun 2019 14:00:33 +0000

This article is a cross-post from my personal blog

I recently published my first npm package and my colleague, Attila Vágó, suggested I write up some notes on the process. I'll point out that I am not an expert on this (I don't even play one on TV!) so you should take this with a pinch of salt.

I'm going to outline why you might want to publish an npm package, then show you how easy it is, and finally, an overview of the configuration options you need to know.

Motivation - why would you ever publish an npm package?

fun!

It's fun to learn!

forking an existing package

Maybe an existing package doesn't do what you need. Maybe it has been abandoned. Maybe you need to add some new configuration and port it to es6 at the same time. Whatever the reason, you want it? you can rebuild it. You have the technology.

code sharing

You might want to share some code between your own projects. Publishing it as an npm package makes that easier. If you need to restrict access to your published library you can also publish your package to a private registry so that only you and your team can install it - not all the options cost money.

Publishing your package in three easy steps

Let's jump right in!

It is actually really easy to publish an npm package. I'll show you how to do that first and then show you how to properly configure your package so you publish only the files you want, and provide a good description of how to install and use your package.

1. Login to your npm account from the terminal

You'll need to have an account, so if you don't have one, visit https://www.npmjs.com/signup and create one.

On the terminal logging in is straightforward

$ npm login
# prompts you for your email and password

You only need to login once after starting / restarting your terminal.

2. Increment the version of your package

Unless this is the first time you've published, you'll have to increment the version of the package or npm will reject it as it expects an updated version every time you publish.

{
  "version": "1.0.0" // increment to 1.0.1
}

3. Publish!

The command below will publish your package

$ npm publish

# npm notice 
# npm notice 📦  my-colourful-sorter@1.0.1
# npm notice === Tarball Contents === 
# npm notice 1.5kB package.json       
# npm notice 1.1kB LICENSE            
# npm notice 2.0kB README.md          
# npm notice 1.5kB src/sort.js       
# npm notice 865B  src/fuzzy.js
# npm notice === Tarball Details === 
# npm notice name:          my-colourful-sorter                     
# npm notice version:       1.0.1                                   
# npm notice package size:  3.1 kB                                  
# npm notice unpacked size: 6.9 kB                                  
# npm notice shasum:        d2ffe8d1c9f8f284319efca30e610edee091d993
# npm notice integrity:     sha512-zbG1FeJIr3Ec1[...]EZaLTYm8TKaEg==
# npm notice total files:   5                                       
# npm notice 
# + my-colourful-sorter@1.0.1

Congratulations! I told you it would be easy.

Now for the slightly hard stuff - configuration.

Configuration - package.json

This is where the magic happens. The package.json file contains all the details npm needs to properly package and publish your library to npm, and then install to the developer's project when they choose to use it.

Apart from your dependencies, the top three properties of the package.json file you really need for publishing are:

files
name
version

There are further properties that are optional but usually required for anything more than a simple package:

description
keywords
main / browser

Finally, there are some less important fields that are also useful - I won't be covering them here:

engine
homepage
bugs
licence

Configuration - three is the magic number

files

This is optional, but I consider it the most important field in package.json, which is why I am putting it first in this list. As it suggests, this property determines what files will be included in your npm package. You will see a lot of suggestions to use .npmignore to blacklist whatever files you do not want included in the package - you probably shouldn't use it at all. Anyway, using the files field is recommended by npm.

The files field allows you to whitelist files by directly referencing them or using globbing patterns. These will override what is in your .gitignore and .npmignore file. As an example:

// package.json
{
  "files": [
    "src/**/*.js",
  ],
}

Will include all the js files in the root and sub-directories of the src directory. The ** pattern matches any subdirectory paths.

The patterns are not just a whitelist, you can exclude files too, making the files field very powerful. As an example, I like to have my unit tests in the same folder as my source files and these can be excluded using:

// package.json
{
  "files": [
    "src/**/*.js",
    "!src/**/*.test.js"
  ],
}

Regardless of what you have in the files field, these files are always included:

package.json
README
CHANGES / CHANGELOG / HISTORY
LICENSE / LICENCE
NOTICE
The file you specify in main field (see below)

name

This is the package name, and also what will be used to import your package into a module, so best to keep it short and memorable. It also needs to be unique, but you should search on https://npmjs.com to check your package name has not already been taken.

You can have a shorter package name if you scope your package name with your own npm username. If you want to call your npm package colors - a package that already exists - you can, if you use a scoped package name.

@paddingtonbear/colors

Installing this in a project would be just as simple.

$ npm install @paddingtonbear/colors

version

This is semver compliant. Keep it simple, start at 1.0.0. Every time you publish your package to npm you will need to increment the version - you can use the handy npm version command for this. If in doubt, increment the patch number each time you publish.

$ npm version patch

This will increment your version 1.0.0 package to version 1.0.1.

// package.json
{
  "version": "1.0.1"
}

There are npm libraries that help you publish to npm - the excellent np, for instance - but I won't be covering them here.

Optional configuration

description

Your package's elevator pitch. Keep it short and to the point, this is what appears in the search results on npmjs.com.

The description should tell you exactly what problem the package solves.

This is a bad example.

A super-duper npm package I enjoyed writing and hope you will enjoy using it to sort colourful and other types of lists

Ah, now this is just what I need for my wallpaper database project!

Configurable list sorter that allows sorting by colour

keywords

Any keywords you provide will be used to improve search results. Try and include useful keywords - node and nodejs won't help, for instance. In our hypothetical example these might be a good choice

{
  "keywords": [
    "color",
    "sort",
    "rainbow",
    "color-blind",
    "protanomaly",
    "deuteranomaly",
    "tritanomaly"
  ]
}

main / browser

When your npm package is imported into a module the main field defines what is returned.

For instance, if you define main as follows

// package.json
{
  "name": "my-colourful-sorter"
  "main": "src/sort.js"
}

Where the src/sort.js file in your npm package is

// src/sort.js
export default function doIt() { /* magic happens */ }

If you install your package in a project and import it into a module in that project, the import will be the doIt method from the src/sort.js file:

// wallpapers/index.js
import colourfulSorter from 'colourful-sorter';
    :
colourfulSorter(someList); // this will execute the `doIt` function in your src/sort.js file

The browser property is essentially the same as main but intended for libraries that will be used in the UI.

README

Your readme file is an essential part of your npm package. It's what helps a developer decide if your package is worth installing. At the very least it should describe what your package does, how to install it, and how to use it. Also, it's optional, but common to include badges highlighting the status of your CI pipeline.

description

You can't go wrong by including the name, and quoting the description from your package.json. Add a short paragraph expanding on the description and you're done.

Configurable list sorter that allows sorting by colour.

This sorter allows sorting colour names based on the hue 
of the colour they represent. It can be configured to sort 
in reverse order and can sort based on hues as affected 
by colour-blindness - protanomaly, deuteranomaly, or tritanomaly.

installation and requirements

Developers need to know if a certain version of nodejs is required, and whether the package is best used as a production or development dependency.

Requires Node 8 or higher. 

npm install --save @paddingtonbear/colors

How to use

You should give some working examples to highlight how to configure and use your npm package.

import colorsort from `@paddingtonbear/colors`;

const colorList = ['mango', 'chartreuse', 'vermillion', 'cyan', 'lavender'];

// ['vermillion', 'mango', 'chartreuse', 'cyan', 'lavender']
const result = colorsort(colorList, { order: colorsort.reverse });

API

The usage section should highlight some different ways of using your npm package, and even if you only have a small number of options you should still include an API section to explain them. An incomplete example is shown below.

colorsort(array, [options])
Returns an array of colours sorted in rainbow order.

options
Should be provided as an object literal. The properties and 
values are listed below

order
Can be `colorsort.normal` or `colorsort.reverse`

etc.

Author

You should probably provide your name, non-private email, twitter, linkedin, blog, etc.

Author
Michelin Mann <mich@tires.fr>
- [@tirechap](https://twitter.com/tirechap)

Licence

Finally, you should list the licence you applied to your repo - the MIT licence is a good choice if you are not sure what to use.

How?
Github allows you to add a licence to your repo when you create it. This will add a LICENCE file to your repo that includes the standard MIT licence text.

Github has more information about choosing and applying a licence

README template

I've provided a template README.md file that you can download and use in your own projects.

My first npm package - eslint-formatter-complexity

Mark Lennox — Mon, 06 May 2019 15:56:14 +0000

Note: this is a cross-post from my personal blog https://www.webpusher.ie/2019/05/04/eslint-npm-package

I've just published my first npm package - eslint-formatter-complexity. It's an eslint formatter that uses complexity metrics (see below) to highlight the files in your codebase that are in most need of a refactor.

The code is available in a public repo https://github.com/mlennox/eslint-formatter-complexity - issues and pull requests are welcome.

This is a continuation of work from a previous article 'Find the most complex code using eslint'.

How does it work?

The formatter receives the eslint results, finds the complexity metric related rules that have been broken (see below), sorts the offending files in order according to how many errors and/or warnings were generated, and outputs the list to the console. This differs greatly from the standard eslint output which just lists the files and their broken rules as they are passed to it based on file structure.

A typical output from the formatter is shown below.

The complexity metrics I refer to are measured by the following eslint rules

Why?

For fun!

Also, I'm working on a series of blog posts related to abstract syntax trees and a few articles about eslint so I have been aiming to properly write up the code and release it as I go along.

I have a theory that code standards work best when automated; not really a controversial theory it has to be said.

Automated enforcement of code standards helps create consistent code, speeds up code reviews, helps reduce errors, and should also facilitate developers in maintaining a mental model of the state of their codebase. I released the eslint-formatter-complexity package primarily for the last point - to help developers visualise the complexity of their codebase.

Listing the files in descending order of complexity helps to emphasise which files are in need of a refactor. Ideally it should be added as a step in your CI build, with the results of each build added to quality gates and communicated to the entire team.

Issues

If you have any questions, suggestions for enhancements, or if you find any bugs, please raise them on the repo

https://github.com/mlennox/eslint-formatter-complexity/issues

AST selectors rule

Mark Lennox — Tue, 30 Apr 2019 21:30:07 +0000

My previous article on abstract syntax trees ran through a quick, but relatively broad, overview of syntax trees and how to manipulate them.

This second article will show you how to use a basic knowledge of abstract syntax trees to enforce code standards by adding simple ESlint rules implemented only using AST selectors, requiring no javascript!

Rule - 'no-restricted-syntax'

Eslint provides a no-restricted-syntax rule that allows you add simple rules using AST selectors - which are very similar to CSS selectors.

I'll run through a couple of examples in this article

enforce const over var
ensure FormattedMessage never renders a span

Examples provided here can be found in the AST Selectors folder in the accompanying github repo https://github.com/mlennox/abstractsyntaxforfunandprofit

AST selectors are implemented using esquery. Also, the eslint documentation on selectors is indispensable as a reference.

Const not var

I'll use an example from a previous article - enforce the use of const instead of var. There is already an excellent 'no-var' rule built-in to eslint. This is implemented as an eslint plugin, which requires some effort to write!

However, we can reproduce most of the functionality of the no-var plugin using only AST selectors. As I've already mentioned, AST selectors are based on CSS selectors and won't be a challenge if you have worked with CSS before. I'll explain the construction of the rule in a way that is accessible to those with no knowledge of CSS selectors.

Using the very simple variable declaration below to test against, we'll write an AST selector that will enforce the 'no var' rule in our IDE.

var willIt = true;

To start, we'll need to remind ourselves of the structure of the AST for a simple var variable declaration.

Firstly, lets try and state the problem in english

highlight, as an error, any VariableDeclaration of kind var

Simple enough.

Creating the selector

Firstly, we need to know how to select our variable declaration. Remember, the node type for our variable declaration is simply VariableDeclaration. The AST selector we use is a node type selector - which is simply the type of the node, like so

VariableDeclaration

Next, as we are selecting against all the nodes in the abstract syntax tree for every file in your codebase, we need to refine our selection to only those of kind var.

The kind we refer to is an attribute of the VariableDeclaration node.

We can select all nodes that have a kind attribute using the following selector

[kind]

And to select any kind attribute that has the value var we expand the selector like so

[kind='var']

Now we have a selector that will select all kind attributes with the value var, but we only want to select VariableDeclaration nodes that have that attribute and value, so:

VariableDeclaration[kind='var']

This is our final selector, but how do we add that to our list of eslint rules?

Adding the rule

To apply the rule to our codebase we add the example no-restricted-syntax rule to the rules section of the .eslintrc.js config file

"rules": {
    "no-restricted-syntax": [
      "error", "VariableDeclaration[kind='var']"
    ],
}

This produces the following error in VS Code

I think you'll agree that Using 'VariableDeclaration[kind='var'] is not allowed is a really bad error message.

Custom error message

Eslint supports a custom message for rule violations, so let's add that

"rules": {
    "no-restricted-syntax": [
      "error",  {
        "selector": "VariableDeclaration[kind='var']",
        "message": "All variables must be declared as 'const', do not use 'var'"
      }
    ],
}

This looks a lot better and the added structure to the configuration has the bonus of easier maintenance of your custom eslint rules.

What about a more complex example?

React JSX internationalisation - FormattedMessage

If you use react-intl you will be familiar with the FormattedMessage component that facilitates localised messages in your app.

The FormattedMessage component wraps the message in a span by default.

<FormattedMessage id={`someMessageId`} />
// results in : <span>some message text</span>

You can avoid the span by using this construction instead

<FormattedMessage id={`someMessageId`}>{text => text}</FormattedMessage>
// results in : some message text

I don't like it when spurious HTML is added to my layout, so let's write an eslint rule to ensure it doesn't happen. As before we'll state our problem goal in plain english

highlight, as an error, any FormattedMessage that does not contain child elements

We make a very reasonable assumption here that any children will use the general approach that we require, for example

    :
    :
<FormattedMessage id={`someMessageId`}>
  {labelText => (
    <MyComponent
      label={labelText}
      props={this.props}
      />
  )}
</FormattedMessage>
<FormattedMessage id={`anotherMessageId`}>
  {messageText => this.renderSomeStuff(messageText)}
</FormattedMessage>
    :
    :

This saves us from having to consider the types and format of the child components.

AST explorer + JSX = problem

The ever useful AST explorer does not handle JSX so we'll need to use a different approach to visualise the abstract syntax tree.

Babel parser with jsx plugin

The helper file showTree.js is included in the github repo but you cannot run this helper function from the repo root:

cd ASTselectors/FormattedMessage
node showTree.js

This will turn the stateless react component in the file basicReact.js into a JSON abstract syntax tree. We can use this to try and visualise how we might build a selector that selects only the FormattedMessage nodes that have no {text => text} child function.

Visualising the tree structure

The simplified abstract syntax tree for the second FormattedMessage in the file basicReact.js is shown below.

Note that the structure is relatively complex - a generic JSXElement as a parent container with the attributes openingElement and closingElement containing instances of the FormattedMessage tags themselves and the children of the JSXElement are a JSXEXpressionContainer containing the anonymous arrow function AST for {text => text}

{
  "type": "JSXElement",
  "openingElement": {
    "type": "JSXOpeningElement",
    "name": {
      "type": "JSXIdentifier",
      "name": "FormattedMessage"
    },
    "attributes": [ /* not important to us */ ],
    "selfClosing": false
  },
  "closingElement": {
    "type": "JSXClosingElement",
    "name": {
      "type": "JSXIdentifier",
      "name": "FormattedMessage"
    }
  },
  "children": [{
    "type": "JSXExpressionContainer",
    "expression": {
      "type": "ArrowFunctionExpression",
      "params": [{
        "type": "Identifier",
        "name": "text"
      }],
      "body": {
        "type": "Identifier",
        "name": "text"
      }
    }
  }]
}

As usual a graphic representation of the simplified abstract syntax tree shows the hierarchy much more clearly.

We won't be using the correctly structured FormattedMessage AST as a reference when building our selector, I supply this to as a reference to ensure we don't construct a selector that will also select a properly constructed FormattedMessage.

Now Lets compare that to the self-closing FormattedMessage. A simplified version of the JSON AST is shown below

{
  "type": "JSXElement",
  "openingElement": {
    "type": "JSXOpeningElement",
    "name": {
      "type": "JSXIdentifier",
      "name": "FormattedMessage"
    },
    "attributes": [ /* not important to us... */ ],
    "selfClosing": true
  },
  "closingElement": null,
  "children": []
}

Constructing the selector - approach 1 : JSXElement has no child elements

Referring to the JSON AST, we can see the parent JSXElement has no child elements we can select on that basis

{
  "type": "JSXElement",
  "children": []
}

The selector is simple enough, we want to select the JSXElement where the children attribute is empty.

JSXElement[children='']

Its important to note here that the children attribute is slightly confusing as the children it refers to are the children of the openingElement / closingElement. In regard to the AST selectors, the openingElement and closingElement themselves are the direct descendants (yes, children - hence the confusion) of the parent JSXElement. So armed with this information we know we can use descendant selectors to select the JSXOpeningElement

JSXElement[children=''] JSXOpeningElement

This is still too specific. We are still selecting many elements, we only want to select FormattedMessage elements inside a JSXElement that has an empty children attribute.

Once again, some explanation is required. As far as AST selectors are concerned, the direct descendants of the JSXOpeningElement in the abstract syntax tree are not the components referred to in the children attribute of the parent JSXElement but the JSXIdentifier referred to in the name attribute of the JSXOpeningElement.

Because the name attribute of the JSXOpeningElement is not a simple string it is not possible to use the attribute selector, as they only allow simple matching rules. For instance, the example below, or similar variations, would not work

// bad! does not work!
JSXOpeningElement[name='JSXIdentifier.name=FormattedMessage']

As far as the AST selectors are concerned, the name attribute element is a descendant element and can be selected using a descendant selector paired with an attribute selector that matches the all important string FormattedMessage.

JSXElement[children=''] JSXOpeningElement JSXIdentifier[name='FormattedMessage']

This will select the self-closing FormattedString components in the codebase and will ignore those that wrap components. Success!

But wait, there's more - this can be simpler.

The selector does not gain any specificity from using the JSXOpeningElement. We already know that the parent JSXElement indicates there are no child components, so we don't need to worry that our selector is going to select the JSXClosingElement as it is not there. We can simplify the selector by removing the reference to JSXOpeningElement.

JSXElement[children=''] JSXIdentifier[name='FormattedMessage']

And our final rule, in place in the eslint config

"error", {
  "selector": "JSXElement[children=''] JSXIdentifier[name='FormattedMessage']",
  "message": "Please use {text => text} function as child of FormattedMessage to avoid spurious span"
}

Constructing the selector - approach 2 : JSXOpeningElement is self-closing

There is a different approach we can take that only selects against the opening element itself without requiring reference to the parent JSXElement with an empty children attribute. Look at the JSON AST of the JSXOpeningElement.

{
  "type": "JSXOpeningElement",
  "name": {
    "type": "JSXIdentifier",
    "name": "FormattedMessage"
  },
  "attributes": [ /* not important to us */ ],
  "selfClosing": true
},

The important property here is selfClosing if it is true, as it is here, it means there is no closing tag and therefore no child components.

Instead of selecting the parent JSXElement we can now directly select the JSXOpeningElement that is self-closing.

JSXOpeningElement[selfClosing=true]

And we already know how to filter our selected components to a FormattedMessage by using a descendant selector combined with an attribute selector.

JSXOpeningElement[selfClosing=true] JSXIdentifier[name='FormattedMessage']

The final eslint config would be

"error", {
  "selector": "JSXOpeningElement[selfClosing=true] JSXIdentifier[name='FormattedMessage']",
  "message": "Please use {text => text} function as child of FormattedMessage to avoid spurious span"
}

Conclusion

AST selectors can be very useful in providing a simple way of adding a new ESlint rule, and they also leverage any existing CSS selector knowledge you may have. However, they suffer the same limitations as CSS selectors and quickly become cumbersome for what should be relatively simple selections. The selection of a node based on the contents of the attributes of the children of a sibling node is common, but not simple to achieve using AST selectors; while there is an adjacent and descendant selector there is no previous selector.

The next post in this series will look at writing "proper" ESlint plugins that are much more flexible, and useful.

I, for one, welcome our eslint overlords

Mark Lennox — Sun, 21 Apr 2019 16:48:22 +0000

Functioning teams usually agree to code standards intended to make life easier for everyone. Unfortunately, we constantly flout those agreed rules - to our detriment. Not only does this lead to less manageable code, it can also lead to friction and discord within your team.

I think Alexander Pope put it best

Ah ne'er so dire a Thirst of Glory boast,
Nor in the Critick let the Man be lost!
Good-Nature and Good-Sense must ever join;
To err is Humane; to Forgive, Divine.

Ideally, we'd like to offload the contentious aspects of code review to an objective third-party that knows and understands our agreed rules. If we can rely on trusted code to critique code we write then we can all hate that code instead of each other.

Fundamentally, using code to check our code is faster and (usually) free from errors or bias. This is why we use linting tools, driven by the rules we agree on pre-defined metrics such as line-length, cyclomatic complexity, nested callbacks and others I've mentioned in an article on finding the most complex files using eslint.

Underline it with a rule

There are already lots and lots of existing rules https://eslint.org/docs/rules/#best-practices that you should be using right now. If you use eslint you probably already use a number of these rules either commonly by using eslint:recommended or airbnb. Most teams should add their own rules or use overrides to change or remove airbnb/recommended rules they don't agree with.

A good approach to use when first introducing eslint rules is to regularly discuss their current use and how well they support your code standards; are the parameters properly tuned (what level of cyclomatic complexity should you find acceptable? only your team will know...), should they be warnings or errors, and which new rules should be introduced, and which are getting in the way and should be removed. Eventually, it will be a rare event for you to revisit your eslint config.

Don't break the rules - make your own

It is likely that you'll have situations where the standard eslint rules are insufficient. What I'm suggesting here is a small step beyond using pre-defined rules - you can write your own eslint plugins to enforce your own code standards. There are two main approaches to writing your own eslint rules - both of which I will cover in future articles in the series on abstract syntax trees:

AST selectors
writing a proper eslint rule

Rules are agreed to as a team and built as a team

Everyone needs to be involved in deciding which rules to use as well as implementing and maintaining the configuration of existing rules. Happy, functioning teams are built on solid communication and a large dose of cross-functional capability - the latter would be important in successfully employing this approach. Practically, it would quickly become a bottleneck if only a small number of developers know how to write the eslint rules, they won't get updated as frequently and they can quickly become a hinderance resulting in them being removed from the codebase.

Also, more to the point, if a small number of people are responsible for writing the linting rules they will be blamed rather than the code - however irrational that might be!

Can code snippets be used to enforce standards?

Yes, but no.

Code snippets are great, but there are a number of drawbacks. The main stumbling block, in my opinion, is affordance; developers need to know the snippets are there, and they need to know how to use the snippet.

Also, this is compounded by the fact that many teams do not enforce use of a single IDE. For instance, my team uses a mix of Webstorm, VS Code, and Sublime. The overhead in adding, maintaining and documenting code snippets for each of those editors is just more friction to the adoption of code snippets.

As much as possible, code standards checking should be automated, the point being to strengthen the weakest link in the chain: the developer. Relying on a developer to pick a code snippet when writing code, or notice that a snippet should have been used when reviewing code does not address the issue that snippets are meant to fix.

Not that snippets are not useful - use them if you like them, I do, but don't expect other developers to do the same.

Know your place

In short, to make your life easier, shorten your code reviews (a bit), and promote harmony and good-will within your team (well, probably), let the rules bind you and bend the knee to the eslint overlords!

Riding rough-shod over your eslint rules

Mark Lennox — Sun, 21 Apr 2019 16:43:45 +0000

In HMH we have adopted fairly strict eslint rules regarding code complexity. I've written about eslint complexity metrics before, and will no doubt do so again! The rules work well for our code, but do tend to get in the way when we are writing tests.

Tests need code standards too!

I'm not saying you should abandon all pretense to code standards for unit test code, but by their nature tests are often larger, more verbose, and complex than the code they are targeting. We've adopted a few approaches to try and keep the tests flexible and maintainable by breaking them into multiple files, adding a dash of SOLID principles and generally using our common sense.

Even with all that, our test files are a mess of squiggly green lines telling us we have failed to write cromulent code. If only we could relax the rules for our test code!

Overrides to the rescue

Luckily, we can indeed have eslint apply different rules to different files.

For us, we know that the test files are consistently named somemodule.test.js so the associated eslint config will be:

{
  "rules": {
    /* rules here will apply to all files */
    "complexity": [
            "error",
            6
        ],
        "max-nested-callbacks": [
            "error",
            2
        ],
        "max-lines": [
            "warn",
            80
        ],
  },
  "overrides": [
    {
      "files": ["*.test.js"],
      "rules": {
        // we've changed all the errors to warnings
        // and relaxed the rules
        "complexity": [
            "warn",
            12
        ],
        "max-nested-callbacks": [
            "warn",
            5
        ],
        "max-lines": [
            "warn",
            200
        ],
      }
    },
    {
      "files": ["*.any.other.files.js"],
      "rules": {
        /* add suitable rule configs here */
      }
    }
  ]
}

As you can see you can have multiple overrides for different file types, although we only override the unit test rules.

Abstract Syntax Trees for fun and profit

Mark Lennox — Mon, 08 Apr 2019 21:07:57 +0000

Part One - an overview

This article is a cross-post from my blog

This is part one of a series of articles about abstract syntax trees and their use in javascript. The scope of this article is a quick introduction to ASTs, babel plugins and some simple 'toy' examples.

I will present the information and instruction using description, code/json and a diagram wherever possible in a bid to make the subject matter easier to comprehend for a wider range of types of learners.

Scope of this article

This is a very light introduction to abstract syntax trees and the transformation of very simple code. Future articles will deal with real-world code, investigate static analysis, and look at approaches to creating babel plugins that could be useful in your own codebase, also - codemods.

Code

A github repo accompanies this article https://github.com/mlennox/abstractsyntaxforfunandprofit

what are abstract syntax trees

These useful data structures represent the abstract structure of source code regardless of the language. This is possible because despite the syntactic differences, all languages have a very large overlap in terms of the code structure they express: variable assignment, conditions, logic branching etc.

An abstract syntax tree can be used to facilitate static analysis of code, rewriting code, compiling code (transform from one language to another), or very commonly in web development - transpiling code (transform from one language to another with similar level of abstraction ie. typescript to javascript, or es6+ to es5).

In this article I'll show some examples of simple source code presented as abstract syntax trees, and also give a working example (see the repo) by building simple babel plugins to transform basic code

Purity of abstraction

The AST examples I'll be showing are not pure abstractions as they contain metadata relating to the source code and the elements are named to reflect javascript syntax. In all respects they are abstract syntax trees and closely follow the EStree spec

The JSON representations in this article were generated by the AST explorer listed in the useful resources section below.

Useful references

Javascript AST viewer - https://astexplorer.net/

Developer docs for babel plugin development - babel plugin handbook

Babel type reference - https://babeljs.io/docs/en/next/babel-types.html

AST examples

I'll provide some examples here to help visualise the resulting structure when code is parsed into an abstract syntax tree.

The first will change instances of var to const and the second will transform an array into an object.

Simple variable assignment

If we take the simple javascript code snippet below and process it with an AST parser.

const willIt = true;

The resulting AST can be expressed in a number of ways, most usefully as JSON . The snippet of code above transformed to an AST is represented by the following JSON.

{
  "type": "Program",
  "body": [
    {
      "type": "VariableDeclaration",
      "start": 0,
      "end": 20,
      "loc": {
        "start": {
          "line": 1,
          "column": 0
        },
        "end": {
          "line": 1,
          "column": 20
        }
      },
      "declarations": [
        {
          "type": "VariableDeclarator",
          "start": 6,
          "end": 19,
          "loc": {
            "start": {
              "line": 1,
              "column": 6
            },
            "end": {
              "line": 1,
              "column": 19
            }
          },
          "id": {
            "type": "Identifier",
            "start": 6,
            "end": 12,
            "loc": {
              "start": {
                "line": 1,
                "column": 6
              },
              "end": {
                "line": 1,
                "column": 12
              },
              "identifierName": "willIt"
            },
            "name": "willIt"
          },
          "init": {
            "type": "BooleanLiteral",
            "start": 15,
            "end": 19,
            "loc": {
              "start": {
                "line": 1,
                "column": 15
              },
              "end": {
                "line": 1,
                "column": 19
              }
            },
            "value": true
          }
        }
      ],
      "kind": "const"
    }
  ],
  "sourceType": "module"
}

The JSON is composed of a series of nodes each with a type property. The JSON below strips all but the type properties from the JSON above.

{
  "type": "Program"
  "body": {
    "type": "VariableDeclaration"
    "declarations": [
      {
        "type": "VariableDeclarator",
        "id": {
          "type": "Identifier"
        },
        "init": {
          "type": "BooleanLiteral"
        }
      },
    ]
  }
}

You'll also notice each node contains location data that refers to the position of the associated expression in the source code.

{
  "type": "VariableDeclaration",
  "start": 0,
  "end": 20,
  "loc": {
    "start": {
      "line": 1,
      "column": 0
    },
    "end": {
      "line": 1,
      "column": 20
    }
  },
}

Also note, and this is the key point of interest for us, the variable declaration signifies the kind of variable - in this case a const.

{
  "type": "VariableDeclaration",
  "kind": "const"
}

The graphical representation of the hierarchical nature of the tree is much easier to grasp.

Assigning an object

let favouriteBelt = {
  material: "leather",
  length: 40
};

The JSON in this case is much more complex. I've omitted some of the properties for clarity.

{
  "type": "VariableDeclaration",
  "declarations": [
    {
      "type": "VariableDeclarator",
      "id": {
        "type": "Identifier",
        "name": "favouriteBelt"
      },
      "init": {
        "type": "ObjectExpression",
        "properties": [
          {
            "type": "ObjectProperty",
            "key": {
              "type": "Identifier",
              "name": "material"
            },
            "value": {
              "type": "StringLiteral",
              "value": "leather"
            }
          },
          {
            "type": "ObjectProperty",
            "key": {
              "type": "Identifier",
              "name": "length"
            },
            "value": {
              "type": "NumericLiteral",
              "value": 40
            }
          }
        ]
      }
    }
  ],
  "kind": "let"
}

And the graphical representation.

You can see that the hierarchy breaks down into familiar arrangements of nodes despite the relative increase in complexity compared to the simple value assignment.

Transforming code

Hopefully, now you have some idea of what an AST looks like and how it relates to the source code. Next, I'll show how you can transform the source code using the AST. The familiar babel library provides all the tools necessary to parse, transform and re-generate source code so for simplicity the examples provided will be babel plugins.

One caveat, due to the way babel references plugins, these particular plugins can't easily be integrated into your codebase - ideally you would need a publish npm package. The alternative would be to write scripts to move the 'plugin' to a location accessible to babel.

Babel plugins and traversing the AST

Babel plugins use the visitor pattern, an abstraction which facilitates adding extra functionality to objects without requiring a refactor of the original object. The idea is that the object can 'accept' a visitor function that can alter the properties and, as we'll see, the structure of the object.

As the AST is traversed each node gets passed to the babel plugin, a simplified version of which is shown below - an object literal implementing a visitor property which consists of an object of methods named to match the node it should process. The example here has implemented a visitor that will act on all ArrayExpression nodes.

const ourCustomVisitor = {
  visitor: {
    ArrayExpression(path, state) {
      // do stuff
    },
  },
};

When the AST is traversed, data about all corresponding nodes will be passed into the corresponding handler method - the order in which they are passed in, how at what point in the hierarchy and how previous transformations might affect the code are concepts that need to be addressed with real-world code, but the simple, 'flat' examples in this article are chosen to focus on basic concepts.

It is not true to say that the each matching node itself is passed to the handler, each node handler is passed two parameters, path (which does contain the node) and state,which are detailed below.

Path

The path is an object that represents the link between nodes. As you alter the AST babel will update the paths between all nodes.

If we take the following example of an ObjectProperty and the child StringLiteral value

{
  type: "ObjectProperty",
  value: {
    type: "StringLiteral",
    value: "gummi bears"
  }
}

The path that represents the relationship between the nodes would be:

{
  "parent": {
    "type": "ObjectProperty",
      :
  },
  "node": {
    "type": "StringLiteral",
    "value": "gummi bears"
      :
  },
    :
}

In this case node is the current element being handled in a StringLiteral handler in the plugin's visitor:

{
  visitor: {
    StringLiteral(path) {
      // path is:
      // {
      //   "parent": {
      //     "type": "ObjectProperty",
      //       :
      //   },
      //   "node": {
      //     "type": "StringLiteral",
      //     "value": "gummi bears"
      //        :
      //   }
      //    :
      // }
    }
  }
}

Metadata and methods

The path also contains metadata and methods to allow deleting, adding, or updating nodes within the tree.

In the arrayToObject example in the accompanying repo we use path.replaceWith inside an ArrayExpression handler to replace the node defining an array with a node defining an object.

ArrayExpression(path) {
  // ArrayExpression has a property 'elements' that contains the array elements
  const objectProps = path.node.elements.map((element, index) => {
    return new t.objectProperty(new t.stringLiteral(`${index}`), element);
  });

  path.replaceWith(new t.objectExpression(objectProps));
}

State

This contains details of the plugin - including the visitor declaration, pre and post methods. It also contains details of the file being parsed, the state of the AST etc. These can all be accessed within the plugin. The most commonly used state property is opts.

Options

If you are running the plugin as part of your babel stack, rather than through the 'runner' in the associated repo, you can provide options to the babel plugins using your .babelrc file

{
  plugins: [
    ["customPlugin", {
      "doIt": true,
      "decorate": "paint"
    }]
  ]
}

These options will be available in the plugin from state.opts.

state.opts === {
  doIt: true,
  decorate: "paint"
}

Plugin Examples

Keep in mind these are very simple examples that use a single variable assignment meaning we don't need to worry about scope, depth of code blocks etc. Future examples in other articles will use more complex code.

A good starting template for babel plugins is shown below

module.exports = function({ types: t }) {
  return {
    visitor: {
      // add handlers here
    },
  };
};

Convert `var` to `const`

In this example, I want to build a simple babel plugin to replace any instance of var with const in the example code - only var should be affected.

// this 'var' should be replaced with a 'const'
var itemOne = ['items', 'things', 'gizmos', 'widgets'];

// this will stay a 'let'
let itemTwo = ['tchotchke', 'stuff', 'yokes'];

The AST for the itemOne variable assignment is presented below. The AST below has all the location information removed for clarity.

{
  "type": "VariableDeclaration",
  "kind": "var"
  "declarations": [
    {
      "type": "VariableDeclarator",
      "id": {
        "type": "Identifier",
        "name": "itemOne"
      },
      "init": {
        "type": "ArrayExpression",
        "elements": [
          {
            "type": "StringLiteral",
            "value": "items"
          },
          {
            "type": "StringLiteral",
            "value": "things"
          },
          {
            "type": "StringLiteral",
            "value": "gizmos"
          },
          {
            "type": "StringLiteral",
            "value": "widgets"
          }
        ]
      }
    }
  ],
  "leadingComments": [
    {
      "type": "CommentLine",
      "value": " this 'var' should be replaced with a 'const'",
    }
  ]
}

The node we are interested in is the top-level node VariableDeclaration, so lets add a handler for that in the babel plugin

module.exports = function({ types: t }) {
  return {
    visitor: {
      VariableDeclaration(path) {
      },
    },
  };
};

We need to recall that the path is not the node, but the relationship between nodes and metadata etc. To get at the VariableDeclaration node we reference path.node.

Let's have a quick look at the AST again, focussing on the point of interest for us

{
  "type": "VariableDeclaration",
  "kind": "var",
    :
}

We want to update the kind of variable declaration from a var to const. The only other valid option is of course let. Babel will let you update that to anything you like, which seems like an oversight, I'm actually not sure why they don't throw an error, or limit the values in some way.

The updated plugin that updates the variable declaration to const and ensures that only var will be affected. I've removed the types destructuring as I don't use it in this plugin.

module.exports = function() {
  return {
    visitor: {
      VariableDeclaration(path) {
        if (path.node.kind === 'var') {
          path.node.kind = 'const';
        }
      },
    },
  };
};

You can run this example yourself from the accompanying repo. Assuming you have installed the dependencies with npm install the command to run the transformation is

node compile.js varToConst vars.source.js

Try messing with the code, adding console.log to see the structure of the path, change the code in vars.source.js to see how the result is affected.

Object from Array

While this is slightly more complex than the 'var to const' example, it is still fairly simple. I'll include some diagrams to be sure the transformation is clear.

First, the source code that we will transform.

// we'll convert this from an array to an object literal
// that uses the position in the list as the key
const coins = ['thrupenny', { name: 'penny', value: 'a penny, ya dope' }, 2];

Once the transformation is complete we want to end up with the following.

const coins = {
  "0": 'thrupenny',
  "1": { name: 'penny', value: 'a penny, ya dope' },
  "2": 2
};

This means that we will need to replace the ArrayExpression with an ObjectExpression and convert each of the elements of the ArrayExpression into an ObjectProperty.

The AST of the source code is below, with some properties removed for clarity.


{
  "type": "VariableDeclaration",
  "declarations": [
    {
      "type": "VariableDeclarator",
      "id": {
        "type": "Identifier",
        "name": "coins"
      },
      "init": {
        "type": "ArrayExpression",
        "elements": [
          {
            "type": "StringLiteral",
            "value": "thrupenny"
          },
          {
            "type": "ObjectExpression",
            "properties": [
              {
                "type": "ObjectProperty",
                "key": {
                  "type": "Identifier",
                  "name": "name"
                },
                "value": {
                  "type": "StringLiteral",
                  "value": "penny"
                }
              },
              {
                "type": "ObjectProperty",
                "key": {
                  "type": "Identifier",
                  "name": "value"
                },
                "value": {
                  "type": "StringLiteral",
                  "value": "a penny, ya dope"
                }
              }
            ]
          },
          {
            "type": "NumericLiteral",
            "value": 2
          }
        ]
      }
    }
  ],
  "kind": "const"
}

Also, a simplified diagram of the AST showing each element - the ObjectExpression in the second element has also been simplified for clarity.

I'm interested in the elements of the ArrayExpression. I will take each element and construct an ObjectProperty that uses a StringLiteral of the array index of the element as the Identifier and uses the element itself as the value. Focussing on the first element in the array

// const coins = ['thrupenny', { name: 'penny', value: 'a penny, ya dope' }, 2];

  {
    "type": "StringLiteral",
    "value": "thrupenny"
  },

The index is zero, so the ObjectProperty - constructed here using babel.types - looks like

const firstArrayElement = path.node.elements[0];
const firstObjectProperty = new t.objectProperty(new t.stringLiteral(`0`), firstArrayElement);

Although the other elements are different types, the approach is the same. The elements don't need any extra processing to convert them to a different type so we can convert the Array elements into Object properties in one step, using Array.map

const objectProps = path.node.elements.map((element, index) => {
  return new t.objectProperty(new t.stringLiteral(`${index}`), element);
});

A simplified diagram of the resulting AST is shown below. The blue elements have all been created by the code outlined above:

The last step is to replace the ArrayExpression node with an ObjectExpression constructed using the new array of ObjectProperty. Luckily the path includes a number of methods to help in transforming the AST, including replaceWith(replacementNode) which swaps the current node for the node provided as a param.

Constructing the ObjectExpression is simple

const objectExpression = new t.objectExpression(objectProps);

Then I can use the replaceWith method to swap out the ArrayExpression for the new ObjectExpression

path.replaceWith(objectExpression);

Which will generate the expected result

const coins = {
  "0": 'thrupenny',
  "1": { name: 'penny', value: 'a penny, ya dope' },
  "2": 2
};

You can run this example yourself from the accompanying repo. Assuming you have installed the dependencies with npm install the command to run the transformation is

node compile.js arrayToObject array.source.js