DEV Community: Thibaut Tiberghien

Deploy private source maps with Netlify

Thibaut Tiberghien — Fri, 11 Oct 2019 11:00:07 +0000

Originally posted on Medium on Mar 10, 2019.
Photo by Markus Spiske on Unsplash.

Source maps are great, they let you ship minified, optimised code to your users while enabling you to map things back to your original source code in error logs and while debugging. You will typically use a build system to generate your minifed code along with the source maps and deploy them together on your host. You can also use this build step to “uglify” your code in order to protect valuable intellectual property present in your frontend code. When that happens, you can no longer afford to deploy the source maps publicly. This post details how I deployed our source maps with team-only access.

Private hosting of source maps

We chose to deploy our source maps on Google Cloud Storage, in a bucket configured for access to authenticated users from our domain only. This is really convenient as we are using G Suite for our emails and can therefore leverage an authentication system that’s already in place. To do this, set permissions using Access Control Lists (ACLs) on your bucket.

For example, head to Permissions for your bucket, then Add members, enter yourdomain.com as the member and add the role Storage Object Viewer to let anyone in your organization access source maps when their browser is logged in.

Pointing the source maps to your bucket

Source maps are automatically fetched by the browser whenever needed, to do this the browser relies on an address indicated at the end of the minified file. It usually looks like this:

//# sourceMappingURL=index.e0d98431.js.map

By default, most build systems will co-locate source maps with the minified files and link them using their relative path. We will need to change the URL to the one of the bucket we created.

Here, your mileage may vary depending on the tools you use, but the concept remain the same. We use React for our frontend code, with create-react-app for an easy setup. We had to modify the Webpack configuration to use the SourceMapDevToolPlugin to set a custom URL. We use react-app-rewired to modify the Webpack configuration without ejecting.

// config-overrides.js
const webpack = require('webpack')

function setSourceMaps (config, env) {
  config.devtool = false
  if (!config.plugins) {
    config.plugins = []
  }
  config.plugins.push(
    new webpack.SourceMapDevToolPlugin({
      append:
        env === 'development'
          ? '\n//# sourceMappingURL=[url]'
          : '\n//# sourceMappingURL=https://storage.cloud.google.com/<bucket-name>/[url]',
      filename: '[file].map'
    })
  )
  return config
}

module.exports = {
  webpack: (config, env) => {
    config = setSourceMaps(config, env)
    return config
  }
}

From there, your build will inject the new URL at the end of the minified file:

//# sourceMappingURL=https://storage.cloud.google.com/<bucket-name>/index.e0d98431.js.map

Deploying the source maps with Netlify

We use Netlify to deploy and host our frontend code behind their Application Delivery Network (ADN). We previously built the code and let Netlify handle the rest. Our build script was something like:

“build”: “react-app-rewired build”

Now we call a shell script that handles the build and upload the source maps:

“build”: “./build.sh”

# build.sh

#!/bin/bash
# build
yarn react-app-rewired build
# install gcloud cli
echo "Installing gcloud sdk"
curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-237.0.0-linux-x86_64.tar.gz
tar -xzf google-cloud-sdk-237.0.0-linux-x86_64.tar.gz
./google-cloud-sdk/bin/gcloud --version
# authenticate to gcloud
echo "Authenticating to gcloud"
echo $GCLOUD_KEY_FILE > /tmp/gcloud_keyfile.json
./google-cloud-sdk/bin/gcloud auth activate-service-account --key-file /tmp/gcloud_keyfile.json
# upload sourcemaps
echo "Uploading sourcemaps to gcs"
./google-cloud-sdk/bin/gsutil cp build/**/**/*.map gs://<bucket-name>/
# delete sourcemaps
echo "Deleting sourcemaps from netlify deploy"
rm build/**/**/*.map

You may be using another Continuous Integration (CI) tool like TravisCI but the concept will be exactly the same.

Build your code as usual
You must install the GCloud SDK (which includes gsutil, the cloud storage CLI) in a non-interactive fashion by getting it from archived versions. This also ensures you use a fix version of the SDK and future updates don’t break your CI pipeline.
You must authenticate to Google Cloud. They recommend using a service account for CI jobs. See how to create one from the docs. This will generate a JSON file which you need to provide during your deploy job. As we do not want to put this file in the repository for security reasons, I suggest storing its content as en environment variable in Netlify (or the CI tool you use) and building the file from the environment variable. (that’s what $GCLOUD_KEY_FILE is all about)
From there the hardest is done. Use gsutil to upload the source maps and do not forget to delete them from the build folder before letting Netlify deploy that folder.

BONUS: Upload code and source maps to Sentry

If you’re using Sentry as error reporting solution, you may also want to upload both the minified code and private source maps to Sentry to get the best error stack traces you can. To do that, we’ll simply add a section to create a Sentry “release” in our build.sh file. Below are the extras to add right before deleting the source maps.

# build.sh

# ... build and push to gcloud ... #

# install sentry cli
echo "Installing and configuring sentry cli"
curl -L -o sentry-cli https://github.com/getsentry/sentry-cli/releases/download/1.40.0/sentry-cli-Linux-x86_64
chmod u+x sentry-cli
./sentry-cli --version
export SENTRY_ORG=<sentry-org>
export SENTRY_PROJECT=<sentry-project>
# create sentry release and upload source maps
echo "Creating release"
./sentry-cli releases new ${COMMIT_REF::7}
echo "Setting release commits"
./sentry-cli releases set-commits --commit "<github-org>/<github-repo>@$COMMIT_REF" ${COMMIT_REF::7}
echo "Link deploy to release"
./sentry-cli releases deploys ${COMMIT_REF::7} new -e $SENTRY_ENV
echo "Uploading source maps"
./sentry-cli releases files ${COMMIT_REF::7} upload-sourcemaps build/static/js/ --rewrite --url-prefix '~/static/js'
echo "Finalizing release"
./sentry-cli releases finalize ${COMMIT_REF::7}

# ... delete source maps ... #

Notes:

The name of the release is your choice, I use the short SHA of the commit ${COMMIT_REF::7} but you can use anything, just make sure to adapt it in every sentry-cli command.
$SENTRY_ENV is an environment variable I use to name my release (dev, staging, prod). It is set by branch in the netlify.toml file as described in the docs.
upload-sourcemaps will look for .js and .js.map files in the provided directory, here build/static/js/ where our JS assets are bundled by webpack. The rewrite and url-prefix options ensure that sentry uses its own uploaded artifacts instead of trying to get the one from cloud storage.

Hope this has been a useful post. Happy devops ;).

GraphQL Interfaces (and Union Types) with Prisma and Yoga

Thibaut Tiberghien — Fri, 11 Oct 2019 10:22:31 +0000

Originally posted on Medium on Apr 2, 2018.
Photo by Clint Adair on Unsplash.

What is GraphQL?

GraphQL is an API query language that came out of the Facebook team and has been taking over the internet recently. It gets its strength from being built around a strongly typed API contract which defines exhaustively the data in your API as well as its schema, how to request for it, and so on. It supports deeply nested querying with controlled hydration and lets API clients combine data from different sources or models, all into a single query. With GraphQL, you get exactly the data you want, formatted the way you want, and in a single query, solving several problems of traditional REST APIs. Moreover, the API contract concept enables a wide variety of powerful developer tools, some of which I describe below.

My GraphQL Stack

Prisma, by the amazing team at Graphcool, is sort of a GraphQL ORM, it takes your data schema defined in the SDL (Schema Definition Language) and generates a database and API for it. The extensiveness of the generated API for (nested) CRUD operations is just amazing. You can deploy your database service in their cloud or using docker on your infrastructure. On top of this, Prisma comes with bindings which provide a convenience layer for building GraphQL servers on top of Prisma services.
graphql-yoga, also by Graphcool (these guys are on 🔥), is the simplest way to build GraphQL servers. It is based on or compatible with most of the de facto standard libraries for building GraphQL servers in Javascript, but it takes the angle of improving developer experience by making everything easier to setup, with sensible defaults and a more declarative approach to configuration. It covers more or less the whole GraphQL spec, even up to WebSockets support for Subscriptions.
GraphQL Playground, also by Graphcool (wuuut? 😱), is a web-based GraphQL client / IDE which supercharges your development workflow by introspecting your API contract to provide an automatic and interactive documentation for it as well as a query interface with auto-completion and validation against your schema. It is packed with nifty little features and is a go-to tool for anything GraphQL.
Apollo Client, by the geniuses at Apollo, is probably the best GraphQL client available. It is compatible with every major frontend platform, and focuses on getting your data inside UI components without taking care of all the plumbing to get it. I love its declarative data fetching approach for React, and the advanced data loading features it supports. e.g. caching, loading, optimistic UI, pagination, etc. The devtools are a great addition to your developer experience as well.

Now to Interfaces…

Some Context

The GraphQL schema specification supports Interfaces and Union Types. An Interface is an abstract type that includes a certain set of fields that a type must include to implement the interface, while Union Types allow for the grouping of several types without the sharing of any structure.

For any non-trivial data structure, you will most probably need to leverage these constructs to model your data. The problem is:

Prisma does not support Interfaces or Union Types yet. There are open issues for each of them — see Interface and Union Type.
graphql-yoga supports both of them but their usage is not yet documented, which makes it hard to actually implement anything. I opened an issue to know more a while back and this post is where it led me.

My approach

Since Prisma only supports Types and Enums at the moment, we have to find a way to model our data without using Interfaces in Prisma. We can however use Interfaces on the GraphQL server (graphql-yoga) so that the client facing API is properly structured and users get to request data across types using Inline Fragments.

This leaves us with 2 options:

Storing all data with optional type-specific fields under one type (the interface) in Prisma, and then splitting the data back between the primitive types in the app server.
Storing the data in each primitive type on Prisma, and stitching things for queries on the app server.

The problem with option 2 is that you loose the consistency in pagination. How do you get the last 20 items for the interface? How many of each primitive type should you request? You could do 20, sort them, and take 20, but that seems inelegant to me.

So I picked option 1, let’s see how to implement it. I’ll give code snippets following the schema used in the docs.

Prisma Workaround

Basically, we want to merge all primitive types as a single “interface” type. Type-specific fields must be optional since they will not be available for every entry, and they are prefixed with the name of the primitive type to make sure they are unique. In the docs, we have:

# datamodel.graphql
interface Character {
  id: ID!
  name: String!
  friends: [Character]
  appearsIn: [Episode]!
}

type Human implements Character {
  id: ID!
  name: String!
  friends: [Character]
  appearsIn: [Episode]!
  starships: [Starship]
  totalCredits: Int
}

type Droid implements Character {
  id: ID!
  name: String!
  friends: [Character]
  appearsIn: [Episode]!
  primaryFunction: String
}

Our workaround schema is:

# datamodel.graphql
type DbCharacter {
  # interface
  id: ID!
  name: String!
  friends: [Character]
  appearsIn: [Episode]!
  # custom fields: Human
  human_starships: [Starship]
  human_totalCredits: Int
  #custom fields: Droid
  droid_primaryFunction: String
}

Mapping interfaces in graphql-yoga

As desired, we declare in the schema for the client facing API the same interface and primitive types as in the docs. We also copy the schema of the dbCharacters query generated by Prisma as the characters query for our client facing API. This could probably be more refined. The return type is however changed to our interface, hence returned items should be mapped to a primitive type on which type-specific inline fragments can be used.

# src/schema.graphql
# import {} from "./generated/prisma.graphql"

type Query {
  characters(
    where: DbCharacterWhereInput
    orderBy: DbCharacterOrderByInput
    skip: Int
    after: String
    before: String
    first: Int
    last: Int
  ): [Character!]!
}

interface Character {
  id: ID!
  name: String!
  friends: [Character]
  appearsIn: [Episode]!
}

type Human implements Character {
  # interface
  id: ID!
  name: String!
  friends: [Character]
  appearsIn: [Episode]!
  # custom fields
  starships: [Starship]
  totalCredits: Int
}

type Droid implements Character {
  # interface
  id: ID!
  name: String!
  friends: [Character]
  appearsIn: [Episode]!
  # custom fields
  primaryFunction: String!
}

In order to map items returned by Prisma to a primitive type, we need to provide a type resolver for our interface at the root of our resolvers object. I have separated the declaration of interface resolvers into a separate file and import it with object destructuring into the resolvers object. See the __resolveType example in the interfaces.js file. This is a simplistic example showcasing how to resolve types. You would implement yours according to the specific business logic of your data.

// src/resolvers/index.js
const { interfaces } = require('./interfaces')
const { Query } = require('./Query')

module.exports = {
  ...interfaces,
  Query
}

// src/resolvers/interfaces.js
const interfaces = {
  Character: {
    __resolveType (obj) {
      // resolve the type of the incoming interface data
      if (obj.primaryFunction) {
        return 'Droid'
      } else {
        return 'Human'
      }
    }
  }
}

module.exports = { interfaces }

The last thing to do is to implement the client API for the interface. It is backed by the corresponding API from Prisma but we need to translate I/Os between the 2 schemas. The resolver for the characters query is implemented in the Query.js file, which is pretty classic. The implementation details are as follow:

We must make sure all fields selected for the primitive types in the query are requested from Prisma. To do this I have written a utility function called makeSelection into interfaces.js which takes the info object from the resolver and parses the query AST (GraphQLResolveInfo) to generate the string selection sent to Prisma. This modifies the selection to make sure all fields nested in Inline Fragments such as ...on Droid { primaryFunction } will be queried from Prisma as normal prefixed fields, e.g. droid_primaryFunction. The code for this method was pretty much trial and error while inspecting the info object and mapping it to the expected selection to send to Prisma. Disclaimer: the code covers only the queries I have have been needing and might need additions to cover all use-cases. Note also that I’m not an expert with ASTs so there might be a better way to do this, please suggest in the comments if you know one.
We must format the objects received from Prisma back to their expected form in the client API schema. I use another utility function called formatPrimitiveFields, also available in interfaces.js which takes a field such as droid_primaryFunction and remove the primitive type prefix.

// src/resolvers/Query.js
const { makeSelection, formatPrimitiveFields } = require('./interfaces')

const Query = {
  characters (parent, args, ctx, info) {
    return ctx.db.query
      .dbCharacters(
        args,
        makeSelection(info)
      )
      .then(formatPrimitiveFields)
  }
}

module.exports = { Query }

// src/resolvers/interfaces.js
const R = require('ramda')

const interfaces = {...}

const unpackSelectionFromAST = R.map(s => {
  switch (s.kind) {
    case 'Field':
      if (!s.selectionSet) {
        return s.name.value
      } else {
        return `${s.name.value} { ${unpackSelectionFromAST(
          s.selectionSet.selections
        )} }`
      }
    case 'InlineFragment':
      switch (s.typeCondition.kind) {
        case 'NamedType':
          return R.compose(
            R.map(field => `${R.toLower(s.typeCondition.name.value)}_${field}`),
            R.reject(R.startsWith('__')), // apollo client compatibility (__typename)
            unpackSelectionFromAST
          )(s.selectionSet.selections)
        default:
          console.error(`${s.typeCondition.kind} unknown in selections AST`)
          break
      }
      break
    default:
      console.error(`${s.kind} unknown in selections AST`)
      break
  }
})

const makeSelection = (info) =>
  R.compose(
    fields => `{ ${R.join(',')(fields)} }`,
    R.reject(R.isNil),
    R.flatten,
    unpackSelectionFromAST,
    R.prop('selections'),
    R.prop('selectionSet'),
    R.head,
    R.prop('fieldNodes')
  )(info)

const formatPrimitiveFields = R.map(
  R.compose(
    R.fromPairs,
    R.map(([k, v]) => [R.replace(/^.*_/, '', k), v]),
    R.toPairs
  )
)

module.exports = { interfaces, makeSelection, formatPrimitiveFields }

Unions Types are not directly covered in this post but they are pretty similar to the __resolveType approach for Interfaces.

Code snippets are written for node 8 and above.

If you’re using Apollo Client, note that interfaces and unions in inline fragments are not resolved properly out of the box. You need to setup a custom fragment matcher based on the api schema. This is explained in details in the docs.

My Recommended Free Resources to Learn React

Thibaut Tiberghien — Fri, 11 Oct 2019 07:55:37 +0000

Originally posted on Medium on June 18, 2017.
Photo by Artem Sapegin on Unsplash.

Over the past few months, I’ve been learning React slowly, trying to build an in-depth understanding of its large ecosystem. This post lists the resources I found most useful to get a solid foundation in this area. I have definitely missed out on some links or people, so please feel free to reply with your own favourites and recommendations.

Note: I strongly recommend coding along to get the most out of these.

React Fundamentals, by Tyler McGinnis

This is the best place to start in my opinion, because it begins with React out of its ecosystem to make you feel the pain points solved by each tool it introduces. That way you won’t feel overwhelmed by too much tooling all at once, and you will then understand the reason for the rather large tooling lineup.

The course covers the JS bits you might miss, React (with and without JSX), the different kinds of React components, ES6 classes, React Router, Babel, Webpack. It offers good explanations right from the basics up to best practices. And you are lucky as it has been updated as of React v15.5.

Redux docs & egghead.io courses by Dan Abramov

Redux is not mandatory with React, but I am personally a huge fan of how it keeps the code really clear and self-documenting, by decoupling concerns around state management.

[EDIT] I am primarily not using Redux anymore, and prefer MobX-State-Tree (more details further down). However, I would still highly recommend learning Redux. You’ll become a much better developer just understanding the underlying principles to Redux. Read more about this in this excellent article.

There are 2 ways to go about this list — video lectures or docs tutorials. You can pick the method you prefer, but I do find useful to do both and this is the order I’d recommend (I prefer reading first, so feel free to swap the video/docs order):

Redux docs, read the “Introduction” and “Basics” sections and treat the Todo List example as your first tutorial.
Getting Started with Redux course on egghead.io builds upon the Todo List example.
Redux docs, read the “Advanced” and “Recipes” sections and treat the Reddit API example as your second tutorial.
Building React Applications with Idiomatic Redux course on egghead.io is a continuation of the first egghead.io course.

See the note on MobX-State-Tree below, which is currently my favorite state management library. I would still recommend learning Redux, as it is an essential part of the community’s fundamentals and is an excellent way to understand simple functional programming concepts which will make you a better developer in the long run.

Other tools and libs

Having gone through the above, you should be pretty confident by now to build a typical React + Redux app. The best at this point would be to pick a side project of your own to further explore things first-hand. My chosen project was a React + D3 playground which I wrote about in this post. Below are some tools and libs that I’ve been playing with and would recommend learning.

Immutable.js & Reselect

These tools are great for performance optimizations using pure components. They help to ensure the props you pass to your components are immutable, allowing to drop out of unnecessary render cycles. If you haven’t already, read the part on Immutability from the redux docs. When using Redux, use Immutable.js to make your whole store immutable and Reselect to make computed props optimal with memoized selectors.

Styled Components

This is my favorite CSS-in-JS solution. It has been thought to be the way to go with CSS in the component age. It uses ES6 tagged template literals to allow JS-powered CSS writing, with dynamic theming, props-based styling, etc. It has full CSS support and allows you to basically write CSS along your components with close to no learning curve. Classes are automatically generated and styles are nicely encapsulated. This means your component are really portable without risking class conflicts inherent to big apps.

Create React App

This is the best way to get down to coding without having a ton of configuration to handle. It is supported by the React team and is more or less the React CLI people dream of. It provides a great developer experience, with out-of-the-box support for most of the features you might desire, as well as options for building your app for production.

Redux-Saga

Redux does not support side-effects in its actions / reducers. A simple way to get around this limitation is Redux Thunk, a Redux middleware for dispatching functions. If you want a more complex but elegant approach, Redux-Saga is pretty darn amazing. It is another Redux middleware based on the Saga pattern, which helps you implement your side-effects calls as ES6 generators. You declare them in a central place and they are executed reactively to pure Redux actions.

Recompose

Recompose is a React utility belt which I use mainly to write Higher-Order Components and compose behaviours into components in neat ways. It is very well introduced in the Higher Order Components with Functional Patterns Using Recompose egghead.io course by Tim Kindberg.

If you want to know more about HOCs, there is also a great post by Franleplant entitled React Higher Order Components in depth.

MobX

Redux can sometimes be an overkill. Yet, you might still want some easy-to-use state management solution that is more featured than setState(). I like MobX for such situations. MobX turns your state into an observable, allows for computed values and automatic reactions (side-effects) based on actions, and most importantly turns React components into reactive components automatically.

MobX-State-Tree (MST)

Despite its name, MST is very different from MobX. It was built by the same people and leverages MobX under the hood for the observable and reactivity part. However, MST is a lot more opinionated and full-fledged than the minimal MobX. With MST, you need to provide the shape (type information) of you state “tree” (i.e. the models, actions, etc.) which is automatically leveraged to generate a living tree of immutable, structurally shared snapshots of your state. This can then be used to support time travel, hot module reloading, and other developer goodness that Redux got us accustomed to, but sparing us from the boilerplate.

I see MST as a mature solution for managing state which combines the goodness of immutability, the ease of use of mutable structures, the convenience of type checking, the separation of concerns of dedicated state stores, and the natural performance of observable structures. It is to state what React is to UI (paraphrasing Daniel Earwicker). It is the best developer experience (DX) I’ve had managing state. 🔥

I’d recommend learning in depth about MobX from the docs, before diving in on MST with the excellent egghead course by Michel Weststrate.

Final good reads

Having played around with the above on your own, you should start to feel pretty solid. I would then recommend the following resources to wrap things up.

React docs. Yes, it probably sounds odd to finish here. While I find the above resources provide a better pace to get started with React, the docs are a great reference for everything-React and make for a good conclusion chapter.
Watch the Worst ‘Hello World’ ever by mpj, which, given all your newly acquired knowledge, should make you laugh. Disclaimer: this is not the way you should be coding!

I would like to highlight that the Complete Intro to React course by Brian Holt on Frontend Masters is really great as well. It is not part of the resources listed above as it is not free, but I do recommend it.

Shout-out to Clevertech, my employer who sponsored the Frontend Masters subscription. Check out what we are up to and come join us!

Some great React-ers to follow on Twitter

In alphabetical order, let’s not play favourites!

React + D3.js: Balancing Performance & Developer Experience

Thibaut Tiberghien — Fri, 11 Oct 2019 07:33:01 +0000

Originally posted on Medium on May 17, 2017.

Let’s put it out there, I love dashboards. I find the way they help you gain a rapid understanding out of complex information really interesting. I have written real-time data visualisations in the past, but always thought that complete dashboards were really hard to implement. That was until I learnt React a while back, and then it clicked: I had finally found a tech that would make building dashboards easier and save the developer’s sanity. I recently started on a side project to try and get React and D3 to integrate seamlessly, with two main goals: render performance and developer experience (DX).

Below is a quick showcase of the mock dashboard I built. The data here is not important since the focus is really on the technical integration of both libraries. You can find a live demo hosted on ∆ now and the code on GitHub.

Motivation

There are many great things about integrating React and D3. You can build yourself a custom library of reusable charts backed by React, which means better render performance with React’s reconciliation, and opportunities for lifecycle performance tweaks. Additionally, you bring all the new shiny possibilities of the modern web to your D3 charts: great developer tools, server-side rendering, an array of excellent state management options, data selectors and immutability, CSS-in-JS, …

Of course, you can use some of these things without React but it is so much easier when the path is all tar road with proper documentation.

The problem

The problem lies with integrating React and D3 seamlessly. Both libraries are built upon data-driven DOM manipulation where the DOM is taken care of for you. So without careful precautions, React would not accept well getting its stuff moved around. It’s like trying to change some code convention in a project managed by that OCD colleague of yours (that might be me). Ouch!

So I read a bit here and there and compared the options available. Below is what I found and thought. I hope you will find this interesting or even helpful. I’m still learning all this, so do drop a response whether you want to send appreciation, highlight a misunderstanding on my end, or point me in a better direction.

React + D3: What is the best way?

TL;DR: Looking into integrating React and D3 seamlessly, I first tried to disable React in D3 land (1), then to use D3’s helpers only and to render charts as JSX with React (2), to finally settle on the react-faux-dom approach (3).

Solution 1 — To each its (DOM) land!

The first and simplest idea I have encountered is to basically disable React in D3 land. There are a few techniques to do so, either by rendering an empty <div/> with React which becomes D3’s scope, or by returning false in the shouldComponentUpdate() lifecycle method.

My main issue with this rather effective solution is that you lose all the goodness brought by React into D3 land. In particular, you get slower rendering performance by doing heavy DOM manipulation that React’s reconciliation algorithm could have shaved milliseconds off. You also loose all the tooling and the DX provided by React that you probably started to love (see Motivation). And for the last point, I will just go ahead and quote Oliver Caldwell, whom I completely agree with.

Many solutions involve stepping out of the React tree for that specific component, which does work, but leaves you with a little island of mutable DOM, festering away inside your tree. It just doesn’t feel quite right to me.
— Oliver Caldwell, D3 within React the right way

Solution 2 — D3 for the maths, React for the DOM

This is probably the most widespread technique at the time of writing. It consists in only using D3’s helpers to prepare the data, the axes, etc. and then feed all of that to React to be rendered. This means you don’t actually use D3’s data binding, but handle it yourself with React by specifying a key for all your SVG elements. This is something you sometimes have to do in D3 too, when the binding is not trivial enough for D3 to figure it out alone. The big change here is that you will render your SVG elements as JSX, instead of using the familiar d3.(...).append(). There is a great post by Dan Scanlon on Hackernoon about this approach.

This method provides good performance overall, but my main issues here are with the DX. Firstly, the visualization code is extremely different from vanilla D3 code. This introduces several disavantages in my opinion.

Having to draw out the SVG and axes myself in JSX feels really awkward at first, I’m not sure how long I would take to get used to it and whether I would ever like this way of doing things.
It undeniably stamps your code “React”, making it harder to extract it from its component in case it ever becomes useful. I worry here about framework lock-in, since the volatility of JS frameworks is rather high as compared to D3's.
It becomes time-consuming to code from example (or port existing code), since you have to convert all your vanilla D3 code to JSX. This is important for me as it is my default process for implementing D3 visualizations, and I’m probably not alone considering the 20K+ examples available.
The learning curve for D3 developers is steep and I am not sure if it is worth the cost, at least not for every team.

Another issue with this solution is that since D3’s data binding is not used, we also lose the enter-update-exit pattern and therefore D3 transitions. I consider D3 transitions and animations as a great part of D3's value proposition. This is what powers many techniques for creating rich user experiences. This, added to the reduced DX, makes it hard for me to really embrace this approach.

Solution 2b — Enter/exit with React, update with D3

This solution was described in an excellent Medium post by Shirley Wu. It builds upon solution 2 but mixes in a bit of solution 1. The idea is still to use D3’s helpers and JSX to render SVG elements, except that now the elements rendered by React are rendered without attributes, and D3 is used to add their attributes. So the line of ownership between React and D3 is not at the element level like in solution 1, but at the attributes level. Although small, the difference is key to getting D3 transitions back. Attributes being handled by D3, we can add an enter() method called in componentDidMount() and an update() method called in componentDidUpdate(). Each of these methods can use typical D3 code to position, style, and transition elements.

There are some caveats to this approach:

As declared in the post introducing this idea, exit() transitions are not supported without bringing in React’s TransitionGroup.
Since React does not keep track of attributes, we have to manually implement state comparison to detect when the component should update in order to call the update() method performing D3 transitions. This basically means that we implement React’s job for it because we intentionaly bypassed it.
This approach still has all the DX issues inherent to solution 2.
I found the implementation too complex for a simple chart. I believe this is due to the need to split the code according to the line of ownership between React and D3, instead of splitting it into logical units.

Solution 3 — Feed D3 a fake DOM that renders to state

This is the solution I found the most elegant so far, and it is what powers the demo at the beginning of this post. It is based on react-faux-dom, made by Oliver Caldwell who detailed the idea on his blog. The concept is that D3 is fed a fake DOM which implements all methods it would expect the DOM to have. That fake DOM is manipulated by D3 and then automatically rendered as React elements stored into the component’s state where React can pick up changes and kick-off an update, including lifecycle methods and reconciliation as you would expect.

I found this approach elegant because both D3 and React are used without alienation.

Except for feeding the faux DOM node to D3 instead of using a selector as you normally would, vanilla D3 code can be used. This means no framework lock-in, easily port existing code or start from example, and no learning curve for D3 developers.
The full D3 API is supported, with transitions, animations, mouse events, etc.
React’s component lifecycle and render methods are being used, and changes made by the D3 are picked up and reconciled seamlessly. Hence, you get to enjoy the typical render performance of React components.
SVG elements are automatically transformed into React elements and are inspectable in the devtools.
The implementation is compatible with server-side rendering, so you get isomorphic charts at no cost.

Overall, this solution has restored my faith is having a great DX when using D3 visualizations in React components, while making the most out of React’s render performance.

Performance tricks

In this section, I will describe some techniques I have used to improve the render performance of my playground dashboard. The basic idea is that D3 updates are more expensive than React re-renders. Indeed, without resorting to performance-motivated tricks to decompose you D3 code, each time D3 processes some update it needs to recompute all the chart helpers and check all the data to possibly update the bound elements. Also D3 updates will trigger a new render cycle of the component anyway. So how can we avoid D3 updates? TL;DR: Only update D3 on new data or on resize events.

Extract tooltips to React

Tooltips are typically something I prefer extracting from D3 into React land. Being usually displayed on mouse hover and hidden on mouse out, their update rate is much higher than that of the underlying data. This means recomputing helpers and checking over the data is pointless and it makes tooltips prime candidates for Reactification — if that’s even a word.

To extract tooltips to React, I add mouseover and mouseout event listeners to SVG elements, in which I setState the hover value so that React can kick-off a render cycle on updates. I often use setTimeout() in the mouseout callback, and then clearTimeout() in the mouseover callback to avoid the flickering between hovers caused by the margin/space between the graph elements. This also lets me use CSS animations to translate tooltips. The tooltip is then rendered directly in JSX, using D3 helpers for positioning if necessary. You can simply share the helpers in the component’s scope using the this keyword. Also, we must be careful to avoid updating D3 when the hover changes in state. To do so, I omit hover from the state’s shallow comparison done in componentDidUpdate. Now, that’s a lot to take in without code so here you go with a simplified code excerpt and feel free to dive in the full source on GitHub.

class Chart extends React.Component {
  // ...
  componentDidUpdate (prevProps, prevState) {
    const stripState = p => _.omit(p, ['hover'])
    if (!shallowEqual(stripState(this.state), stripState(prevState))) {
      this.renderD3()
    }
  }
  // ...
  setHover (hX) {
    this.setState({
      hover: hX
    })
  }
  // ...
  computeTooltipProps (hX) {
    const hoveredData = _.map(this.props.data, 'values').map(d =>
      _.find(d, {x: hX})
    )
    return {
      style: {
        top: this.y(_.sum(_.map(hoveredData, 'y'))),
        left: this.x(hX)
      },
      content: `${hX}: ${_.map(hoveredData, 'y').join(', ')}`
    }
  }
  // ...
  render () {
    return (
      <div>
        {this.state.chart}
        {this.state.hover &&
          <Tooltip {...this.computeTooltipProps(this.state.hover)} />
        }
      </div>
    )
  }
  // ...
  renderD3() {
    // ...
    // make x and y helpers available to JSX for tooltips positioning
    const x = d3.scale
      .ordinal()
      .domain(this.props.xDomain)
      .rangeRoundBands([0, width], 0.08)
    this.x = x
    const y = d3.scale.linear().domain([0, yStackMax]).range([height, 0])
    this.y = y
    // ...
    // add mouse event listeners
    let rect = layer.selectAll('rect').data(d => d.values)
    rect
      .enter()
      .append('rect')
      .attr('x', d => x(d.x))
      .attr('y', height)
      .attr('width', x.rangeBand())
      .attr('height', 0)
      .on('mouseover', d => {
        clearTimeout(this.unsetHoverTimeout)
        this.setHover(d.x)
      })
      .on('mouseout', d => {
        this.unsetHoverTimeout = setTimeout(
          () => this.setHover(null),
          200
        )
      })
  }
}

Handle styling updates in a parent component

If you decide to go with dynamic styling for your charts — for example by reducing the opacity of non-hovered values, or by letting users change colors dynamically — you should certainly not go through a D3 update to do so. Instead, add a CSS class to your SVG elements that includes a key to the data and/or group they represent, and then handle styling outside of D3 land using your favourite CSS-in-JS tool. I personally am a huge fan of styled-components.

Going further with this idea, if you are building a dashboard or anything that gets you maintaining multiple charts in your codebase, you might want to share the parts of the state that dictate your charts styling into a parent component — I love Redux for state management, but pick anything that works for you. You can then apply styling on that parent component, and it will be shared by all chart components in its subtree. For example, in my playground dashboard, none of the chart components need rendering when the user picks a new color from the pallet, it’s all handled by rendering the dashboard component. Similarly hovering the barchart does not re-render the scatterplot although it looks like it does; the dashboard takes care of setting the opacity on filtered data. This also has the advantage that you code your styling once and it is handled for all your chart components, so you have one less thing to manage in your charts code.

Use pure components, immutability and memoized selectors

This is not really specific to React+D3, but since I’m on performance tricks, I might as well mention it. You can make big wins in render performance by reducing the need for React to even render your components (recompute the virtual DOM) and perform the reconciliation when you know there is nothing to update. There are a few techniques that you should employ together to do this.

React components normally update when their parent component does or if their props or state change. You can extend React.PureComponent instead of React.Component and your component will only update if the shallow comparison of its state and props shows differences. See the docs for details.
Because deep comparison can be expensive in Javascript, especially with visualizing large datasets, pure components only perform a shallow comparison. This means your component’s state and props are compared by reference to their previous self. In order to use pure components effectively, you should be sure to make your state and props immutable. One option to do this is the awesome immutable.js which, being a Redux user, I simply apply on my entire Redux store at initialization. I then make sure to apply immutable modifications to the store in my reducers.
Props are passed down from parent components or containers, they are often calculated by these components from the state. You need to make sure that new values are not recomputed when the state has not changed. To do so, you can use memoized selectors with reselect, a “selector” library for Redux. Reselect only computes new props values when the underlying state has changed, and return the reference to the previous value if the state has not changed, making it a perfect fit for pure components and immutable redux stores.

That’s all folks!

It has been an exciting challenge trying to get the best of React and D3 in a seamless developer experience while keeping performance in mind. A great thank you to the authors of the articles I posted above for getting much of the problem stated and for providing some great answers. A huge shout out to Oliver Caldwell for masterminding the react-faux-dom approach. I hope to see it evolve, improve further, and get the community attention that I think it deserves. I leave the rest to the comments section. We can hopefully see some interesting ideas and debate about this these techniques.

Edit: a recent article by Marcos Iglesias is a great addition to this with a look at more charting libs for React and D3, it’s at https://www.smashingmagazine.com/2018/02/react-d3-ecosystem.

Digital Ocean: Setup DNS, Nginx web hosting & G Suite for emails

Thibaut Tiberghien — Fri, 11 Oct 2019 05:21:36 +0000

Originally posted on Medium on May 8, 2016.

YMMV: The code below is given for the environment and use-case I had at the time of writing, adapt it for your requirements.

Stack

Web hosting on a Digital Ocean droplet, running Ubuntu 16.04
Nginx web server, serving www.planecq.com from /var/www/planecq.com/html
G Suite for email
DNS settings on Digital Ocean

1. DNS

Forward DNS management to Digital Ocean

Go to your domain name registrar, search for the DNS settings and set the following external nameservers:

ns1.digitalocean.com
ns2.digitalocean.com
ns3.digitalocean.com

This may take a while to propagate, up to 48h, but usually around an hour or two for me.

Direct web traffic for your domain to your droplet

Login to cloud.digitalocean.com, go to Networking, then Domains. Add a domain and link it to the relevant droplet. After that, click “view” for your newly added domain and add a CNAME record for www pointing towards your main A record, in my case planecq.com., with the trailing dot. You should be good to go.

Direct email traffic for your domain to G Suite

Login to cloud.digitalocean.com, go to Networking, then Domains. Click “view” for your domain, then “MX”, then hit the “Add Gmail MX records”. You’re done!

2. Web Hosting

Setup Nginx to host your website

Install Nginx:

sudo apt-get update
sudo apt-get install -y nginx
sudo update-rc.d nginx defaults

It should be running after install, you can check with

sudo service nginx status

Add the site’s config file (/etc/nginx/sites-available/planecq.com)

server {
       listen       80;
       listen       [::]:80;
       server_name  planecq.com;
       return       301 $scheme://www.planecq.com$request_uri;
}

server {
       listen 80 default;
       listen [::]:80 default;

       root /var/www/planecq.com/html;

       index index.min.html index.html;

       server_name www.planecq.com;

       gzip_static on;
       gzip_proxied no-cache no-store private expired auth;
       gzip_http_version 1.0;

       location / {
                try_files $uri $uri/ =404;
       }
}

Note: you can only have one default server, so in case you are co-hosting several websites, remember to choose the one you want.

Enable your site:

sudo ln -s /etc/nginx/sites-available/planecq.com /etc/nginx/sites-enabled/planecq.com

Disable the default Nginx site:

sudo rm /etc/nginx/sites-enabled/default

You can play around with more settings in your site’s config file or in Nginx config (/etc/nginx/nginx.conf) to affect all of your websites.

You can now copy your website’s files to /var/www/planecq.com/html and it should work once the DNS is propagated. You can already try to hit the site using the IP of your droplet.

Deployment of static websites to Digital Ocean using TravisCI

Thibaut Tiberghien — Fri, 11 Oct 2019 05:06:42 +0000

Originally posted on Medium on May 8, 2016.

YMMV: The code below is given for the environment and use-case I had at the time of writing, adapt it for your requirements.

Stack

Code on GitHub, master branch deploys to prod, and staging branch deploys to a subdomain for preview and QA.
Web hosting on a Digital Ocean droplet, running Ubuntu 16.04. Using Nginx web server, serving www.planecq.com from /var/www/planecq.com/html.
TravisCI for the automatic deployment when git pushing to specific branches. It pushes to a bare git repo at /var/www/planecq.com/.git. From there, a post-receive hook checks out the latest code to the html folder.
Gulp used for the build and test workflow

1. Create an encrypted private key for travis (on your dev machine)

Install Travis CLI if needed:

gem install travis
travis login

Create the encrypted key and public key in your local repo:

cd ~/Sites/planecq.com
touch .travis.yml
ssh-keygen -t rsa -N "" -C "travis@planecq.com" -f travis_rsa
travis encrypt-file travis_rsa --add
rm travis_rsa
cat travis_rsa.pub # << copy this for later

2. Setup Travis on the droplet

Create a passwordless travis user on the droplet, setup the generated access key, and give it access to the folder where the website is hosted:

sudo adduser --disabled-password --gecos "" travis
sudo chown -R travis:travis /var/www/planecq.com
sudo su travis
mkdir ~/.ssh
chmod 700 .ssh
emacs .ssh/authorized_keys
"copy content of previous cat, save, exit"
chmod 600 .ssh/authorized_keys
exit

3. Prepare remote repository on the droplet

sudo su travis
cd /var/www/planecq.com
mkdir .git
cd .git
git init --bare
cd hooks
emacs post-receive
"copy the content of the hook posted below, save, exit"
chmod +x post-receive
exit

post-receive hook:

#!/bin/sh
git --work-tree=/var/www/planecq.com/html/ --git-dir=/var/www/planecq.com/.git checkout -f

4. Travis config

.travis.yml

language: node_js
node_js:
  - 4.3.1
env:
  global:
  - NOKOGIRI_USE_SYSTEM_LIBRARIES=true # speeds up installation of html-proofer
addons:
  ssh_known_hosts: webhost.planecq.xyz
branches:
  only:
  - master
  - staging
before_install:
  - rvm install 2.2.2
  - openssl aes-256-cbc -K $encrypted_b0b2958c016f_key -iv $encrypted_b0b2958c016f_iv -in .travis/travis_rsa.enc -out ~/.ssh/travis_rsa -d
  - chmod 600 ~/.ssh/travis_rsa
install:
  - gem install html-proofer
  - npm install -g gulp
  - npm install
script:
  - gulp build && gulp test
deploy:
  skip_cleanup: true
  provider: script
  script: .travis/deploy.sh
  on:
    all_branches: true
notifications:
  email: false
  slack:
    secure: rWg9[...]x69

gulp build

This is where the website is compressed for production. I have written a separate post covering that.

gulp test (extract of gulpfile.js)

var exec = require('child_process').exec;
var gulp = require('gulp');
var runSequence = require('run-sequence');
var bootlint  = require('gulp-bootlint');

// Validate html, links, etc.
gulp.task('html-proofer', function(done) {
  execute('htmlproofer ./index.min.html --check-html --check-favicon --check-external-hash', {}, done);
});

// Validate bootstrap
gulp.task('bootlint', function() {
  return gulp.src('./index.html.min')
    .pipe(bootlint({
      stoponerror: true
    }));
});

// Full test task
gulp.task('test', function(cb) {
  runSequence('html-proofer', 'bootlint', cb);
});

// Util to execute external command
function execute(cmd, opts, done) {
  console.log(cmd);
  exec(cmd, opts, function(error, stdout, stderr) {
    console.log(stdout);
    console.error(stderr);
    done(error);
  });
}

.travis/deploy.sh

#!/bin/bash

# print outputs and exit on first failure
set -xe

if [ $TRAVIS_BRANCH == "master" ] ; then

    # setup ssh agent, git config and remote
    eval "$(ssh-agent -s)"
    ssh-add ~/.ssh/travis_rsa
    git remote add deploy "travis@webhost.planecq.xyz:/var/www/planecq.com"
    git config user.name "Travis CI"
    git config user.email "travis@planecq.com"

    # commit compressed files and push it to remote
    rm -f .gitignore
    cp .travis/deployignore .gitignore
    git add .
    git status # debug
    git commit -m "Deploy compressed files"
    git push -f deploy HEAD:master

elif [ $TRAVIS_BRANCH == "staging" ] ; then

    # setup ssh agent, git config and remote
    eval "$(ssh-agent -s)"
    ssh-add ~/.ssh/travis_rsa
    git remote add deploy "travis@webhost.planecq.xyz:/var/www/planecq.xyz"
    git config user.name "Travis CI"
    git config user.email "travis@planecq.com"

    # commit compressed files and push it to remote
    rm -f .gitignore
    cp .travis/deployignore .gitignore
    git add .
    git status # debug
    git commit -m "Deploy compressed files"
    git push -f deploy HEAD:master

else

    echo "No deploy script for branch '$TRAVIS_BRANCH'"

fi

.travis/deployignore

This file replaces .gitignore for deploy only and lets the following files to be committed and deployed (files generated by gulp build and normally ignored in the repository):

gulp/
index.min.html
*.gz