DEV Community: Ronen Lahat

The Thirteen-Factor Team

Ronen Lahat — Wed, 02 Mar 2022 13:01:54 +0000

Originally published on Medium

Three years ago I joined the AT&T R&D center as a full-stack developer. I was welcomed directly into a brand new project, where I coincidentally witnessed a restructuring of the company's engineering practices. As I was getting my photo id and laptop, seating arrangements changed, the department rebranded, QAs were being trained as developers, and developers started assuming operational tasks.

During these three years we moved fast and morale was high, even though we had to assume new responsibilities and learn new skills, often within a short time period. We grew professionally while witnessing rewarding metrics from almost 18 million devices in production.

As the dust settled, I thought about the maxims and methodologies that allowed us to build a good, testable product, easy to refactor without breaking, and with a fast deployment to production. These are some of them.

Developers as Feature Owners

A developer is not just a stop in the moving assembly line of a product, but a feature artisan from conception to delivery. The developer should be given technological autonomy and a sense of ownership of the feature. This begins in developers themselves slicing a feature into tasks, and by contributing in all stages of the software lifecycle, including observation and maintenance. They know best how to tackle the feature's challenges and bring it to fruition.

Given autonomy we chose the latest technologies, versions, and conventions, which above all kept us engaged and motivated to continuously refactor.

Contribute in All Areas

After the developer built a feature, they are the most knowledgeable about it and can contribute in all its lifecycle across testing, deployment, and maintenance. The more they contribute, the more they'll understand the big picture and take ownership of features.

Beyond full-stack, a developer can contribute in operational tasks by managing cloud services, administering repositories, orchestrating clusters, writing scripts to automate changes, running deployments, writing libraries for CI tools, and creating metrics widgets and dashboards.

We might have a lot of learning resources available, but jumping onboard is what makes really us feel professional progress and accomplishment.

Infrastructure as Code

The way to give developers the power to contribute in all areas is by making everything code. Define all infrastructure as code, versioned and committed into the same repository as the application code, and give developers the credentials to work on it like all other code. This includes cloud resources, templates and configuration files for the Kubernetes cluster, the CI pipeline, build scripts.

We had a hard working DevOps team to support developers, but gradually we picked up many operational and infrastructure responsibilities.

Automate All Processes

Deployment from one environment to another should be automated, defined in code, never depending on someone's memory (and never before weekends). The same is true for updates and creation of services and databases, etc. This can be encouraged with multiple staging environments, where deployments from one to the other serve as practice and quality assurance.

Let the Pipeline and Application Code Co-Evolve

Development must begin with a good CI/CD pipeline, and developers should fix and improve it as their application's home. When developers work on the pipeline, they can find creative and elegant solutions to issues by tweaking the application, thus making it easier to build, test and deploy.

Our pipeline originated from a dedicated DevOps team which worked very closely with us, and committed its code into the application's repo. With time, our pipeline and application code co-evolved into ever greater robustness and synergy.

Pipelines are notoriously "antifragile," and keeping them stable can be frustrating. Fixing failures is part of the process. Developers learn through them to debug their apps and learn its quirks, avoiding similar issues in production.

Smash the Monolith

When the project was just a blueprint and a few PoCs, the architects defined an API for us. The project had many parties and teams, but we all talked a shared vocabulary. Each could test the API on their end, and decouple themselves by mocking it.

Splitting your app into components and your back-end into micro-services and serverless functions encourages decoupling, which reduces team dependencies. When in doubt, split. It's easier to merge components later than separate when conjoined. Think about creative ways to split your front end into separate deployments as well with micro-frontends.

Monolith No, Monorepo Yes

Ironically, although we decoupled many components, we still want to keep them together in source. We started with one repo for each microservice and for each client, but we found ourselves cloning them in one big folder and writing awkward scripts to link and build them together. As features spanned separate repositories, we had to open multiple pull requests which in turn triggered multiple builds.

This is what monorepo tools are designed to solve, we use Lerna for all our JavaScript repos, and we're never going back.

Bite-Sized Pull Requests, Short-Lived Branches

Some call it trunk-based development. It's easier and faster to merge changes when changes are small. Small Pull Requests are easier for the developer, the reviewers, and other developers pulling changes. Drastic code changes bode bugs, and often come with technical debt that no one wants to pay.

Knowing pull requests are a continuous work in progress helps in code reviews. Nevertheless all pull requests should include tests and feature toggles, to avoid technical debt.

In our case, we deleted the branch after merges. Before shipping, we would create a release branch with a version number and test it thoroughly. Nothing would get merged to it except bug fixes. After release we created a git tag for future reference.

Reduce Hoops, Shorten Feedback Loops

As the software development cycle advances, changes become harder and time estimations longer. What a developer can detect immediately in their local environment can take hours in the CI builds, or days once deployed, sometimes weeks.

Test as earlier and often as possible, use quality gates in the pipeline. The "merge" button was greyed out for us unless two reviewers approved it and the build returned green. This includes code analysis, lint, security scans, unit tests, end-to-end automation, mutation tests, etc. With coverage thresholds. All can also run locally, and most third-party code scanning services offer a CLI or IDE plugin.

Containerize

For micro-services, this is one of the best ways to shorten feedback loops. The runtime that works on the developer's machine is the one that gets tested by the CI/CD pipeline, and is eventually shipped to production in the form of a Kubernetes pods. If a developer says "it works in my machine" and they use a container, we can be confident that it'll behave the same way in production.

Backend developers should see the container as the application's deliverable unit, not the folders with compiled files within. While both backend and client developers can isolate their application's environment in a declarative way, and share that environment with the rest of the team. In one of our projects we used Nix for this purpose.

Gamify Team Excellence, We Like to Show Off

The gamification of the office TV dashboards was accidental. We displayed pipeline build results - which were either green or red - together with various quality metrics, organized by teams. It turned into a competition which also improved team cohesion. Teams also showcased their work at demos, which was an opportunity for all involved to take pride in their contributions and congratulate each other.

When we started working from home we lost the effect of shared office TV dashboard. Build results and metrics turned into individualized alerts and notifications, but the demos and the congratulations didn't change.

We're All Learners, We Make Rookie Mistakes

We all want to excel professionally, but this sometimes comes in the form of perfectionism. One of the subtle signs of perfectionism is the fear of under-performing in fields in which we're inexperienced. A seasoned Java developer may refrain from writing JavaScript, as they might not know its quirks and feel awkward. This harms your ability to contribute to all processes and see the big picture.

Forgive yourself and others and allow yourself to be a rookie again, we have your back.

Give Kudos, Show You Care

This can't be automated, but it can become second nature. It's a basic human need to be recognized and feel that one's work is meaningful. This feels true when busy managers join demos and give feedback. A short email with "the client loved this feature" goes a long way. Likewise, think about congratulating a team member or welcoming someone new.

All the automation scripting and pipeline fixing can be tedious and mechanical, but it's the people which make it humane, feeling recognized and meaningful at the end of the day.

Future

As a developer, I'd like to "commit" what works. But it's impossible to define "methodology as code" like infrastructure, it's a soft science involving people. Behavior cannot be programmed, but it can be measured, fostered and encouraged.

There are many ways to improve our current working practices. Some involving data-driven decision making, and some by adding even more automation, often in creative ways.

With data, we can research trends and predict targets to aid in time and effort estimations, identify bottlenecks, and find ways to further decouple dependencies, automate, and reduce feedback loops.

Recently, we improved pull request pickup time by automatically pinging the relevant reviewers. This reduced the nudging we had from each other to review our pull requests, while clicking on the message conveniently redirects us to review the code. The message even includes estimated review time based on the type of file changed and its changes.

This wasn't a script someone in our team wrote, but part of a dedicated project on its own, currently scaling up. The project is also working to combine all dashboard, metrics, scores, and quality scan results into one big-data framework, a project on which I'm currently participating. We aim to foster a common language for the organization, a language with actionable feedback for teams, hopefully resulting in a healthier software lifecycle, and developers motivated to give their best work.

Twitter: @ronenl

How Nix-Shell Saved Our Team’s Sanity

Ronen Lahat — Wed, 27 Oct 2021 15:00:23 +0000

Originaly published at: https://medium.com/att-israel/how-nix-shell-saved-our-teams-sanity-a22fe6668d0e

We're developing a large React Native app which relies heavily on native components already written in Java, C++, and Objective-C. This means that we needed to develop, build, and test many different platforms on complex developer environments and build tools, that change often with platform updates.

This became a burden for our teams, with knowledge spread across many developers, installation guides, readme files, and internal wiki pages. It became expected that installations would take several days, and even a minor change in a dependency version resulted in inconsistent builds with obscure error messages.

Some suggested Dockerizing the environment, but after several attempts, Nix became our tool of choice. Nix allows us to share the same development environment across Linux and macOS with exact dependencies for tools such as CMake, Ninja, Android NDK, etc. With Nix installed, when opening the repository the developer is greeted by all the required dependencies available in their shell. We use Linux for android builds and macOS for both android and apple builds.

So, What’s Nix?

Nix is both a package manager and build tool. Generally, these two are separate things, such as RPM and Make. This unity becomes useful with Nix’s source deployment model in which packages are built from source. Most of the time the package is substituted transparently for a cached binary from a server (as long as the hash of the build instructions is the same.)

Nix prioritizes consistency, and to achieve this it forces you to declare all dependencies and inputs explicitly while sandboxing the build environment from your shell environment and the internet. Not only is the package built from source, but also its dependencies and their dependencies, which can depend on each other, as nodes in a graph mesh.
Nix-env, the Package Manager

With nix-env you can manage user environments. nix-env creates an abstraction layer over bin directories in your PATH with symlinks to /nix/store. As it uses symlink references it can do several important things:

It keeps track of versions of your environment, and in O(1) time it can roll back to a different version by changing the symlink to a previous profile.
Installations and uninstallations are atomic. The later version isn’t referenced until the installation is complete.
As dependencies are not installed in a global folder, multiple users on a machine cannot override or compromise each other’s dependencies, and therefore are allowed to install packages without privileges.

This is possible because each version of a package is installed in a different directory under /nix/store and erasing a dependency doesn’t remove it from disk until it is completely de-referenced and garbage-collected.

Nix takes versioning into its own hands by hashing the build instructions and its input. Even the slightest change constitutes a new version, as the hash is different. Components reside in the Nix Store, alongside all their dependencies, as:

/nix/store/f2rrk276criwxn19bf82cglym4dkv9gr-ninja-1.9.0.drv
/nix/store/iwm3knkdi294rj50w9ai5rdwaglgr362-ninja-1.9.0/

The last characters are the human-readable name attribute. Nix-env is managed with the nix-env command and the .nix-profile directory.

Installation Issue on Mac

Nix can either be installed for a single user (who owns /nix) or as multi-user (in which root owns /nix). However, on a Mac neither will work anymore, as the root filesystem (anything under /) has been read-only since macOS 10.15. Nix can’t trivially change the path for the Nix Store, as all their binary cache has been compiled with /nix/store as its path. The current workaround is to change the path but mount it as an unencrypted (encrypted at rest) APFS Volume.

$ sh <(curl -L https://nixos.org/nix/install) --darwin-use-unencrypted-nix-store-volume --daemon

The installation will explain what it will do, and will request super-user access, which it will call dozens of times. This is how the Nix Store Volume looks with Disk Utility:

And here it is in Finder:

Nix Store Volume in Finder. For some reason, the Unix timestamp is at 0 (and I’ve given out my timezone).
Nix-shell, the Virtual Environment

It was however nix-shell that made an impact for us. With nix-shell, we can create virtual environments per-project, without having to install dependencies on a user or system level with nix-env.

Just add a shell.nix file in your project. Then, when you enter nix-shell, the environment and all the dependencies are ready for use. This file is, of course, committed to source control and shared among all developers. The file lists dependencies, environment variables, and shell hooks to be run when loaded.
Example shell.nix file with two different Nixpkgs sources.

This can be further integrated into the shell with Direnv, which automatically activates the environment when the directory changes; and Lorri, a daemon process that monitors the project’s shell.nix for changes, and automatically reloads the environment if it has. Niv eases the dependency management of a project with a sources.json file, like a higher-order package manager for Nix-shell.

Some prefer the use of Nix-shell over Nix-env for entire user-level environments, as it can be controlled in an isolated, declarative way. Home Manager enables the configuration of user-specific (non-global) packages and “dot-files.” See what you can do in the NixOS wiki. Finally, Nix-drawin enables the configuration of your Mac the way NixOS does with a configuration.nix file.

Nix-shell can be extended into your OS with the tools above, but it can also be used in a narrower, specific way. It’s possible to run commands in Nix-shell without entering its interactive shell with:

nix-shell --run "node ./index.js".

And it’s possible to specify Nix-shell as an interpreter for a file with a shebang at the top of the file:

#! /usr/bin/env nix-shell
#! nix-shell -i real-interpreter -p packages...

The above file will be executed inside of nix-shell, along with its environment.

Nix-build, the Build Tool

Nix-build is a build manager with correctness in its top priority. That is, all builds will be identical given the same build tools and inputs.

Build managers take sources, such as source code and dependencies, and invoke generators such as compilers, to create derivates such as binaries. Both sources and derivates are components. This is the task of tools like Make, CMake, Ant, or Gradle.

Nix builds are based on a derivation, which is a set that lists exact (hashed) dependencies and exact (hashed) build scripts, which look like this:

Derive([("out","/nix/store/winl36i87aydwj5qgrz0nbc7kq3w0yzi-user-environment","","")],[],["/nix/store/kygr761f08l1nanw27lfxkg8qibf0qn1-env-manifest.nix"],"builtin","builtin:buildenv",[],[("allowSubstitutes",""),("builder","builtin:buildenv"),("derivations","true 5 1 /nix/store/9nqninr2aaicvmq83q10d5a1hwagbzyc-hello-2.10 true 5 1 /nix/store/df26nnjiw55rvv6mxy4kapps9h4kfvw7-niv-0.2.19-bin true 5 1 /nix/store/f3swypnb5zi5yd3w7k2ycwyv6b3sv8fa-direnv-2.28.0 true 5 1 /nix/store/vgdizqicd30k4183ssq7g6i07dvys6xl-home-manager-path true -10 1 /nix/store/4023c0ymrxsg1x36jxmnircqjl1y9fkq-nodejs-14.17.6"),("manifest","/nix/store/kygr761f08l1nanw27lfxkg8qibf0qn1-env-manifest.nix"),("name","user-environment"),("out","/nix/store/winl36i87aydwj5qgrz0nbc7kq3w0yzi-user-environment"),("preferLocalBuild","1"),

Nix Expressions, the Language

The above is a minified version of its human-readable version, written functionally with a Nix Expression:

https://gist.github.com/ronenlh/c2c9ca9ed319bfadd212f2eb15e29629#file-default-nix

The entire file is a single function. Lines 1 to 6 describe a set, passed as the only parameter. The set defines all the dependencies needed to build the component. : in line 6 defines the beginning of the function’s body.

The entire body is a call to stdenv.mkDerivation, which will minimize the data into the derivation written above. rec is a function that will enable recursion inside the data set, allowing the definition of values in terms of other keys in the set.

For didactic purposes, the syntax could be rewritten as a JavaScript lambda as:

({ stdenv, ... }) => stdenv.mkDerivation(rec({ ... }))

The value for src is retrieved from a remote URL and validated with a hash. src is the expected key for the standard build tool, which will perform the standard autoconf (./configure ; make ; make install) shell script.

It’s possible to experiment with the Nix language in its interactive shell.
Nixpkgs, the Package Repository

The above function is not yet callable, as we don’t have the parameters for the function. We can achieve the same result with another rec which recursively defines the needed components and its parameters. e.g.,

rec {
  lib1 = import package1/default.nix { };
  program2 = import package2/default.nix { inherit lib1; };
}

This turns all dependencies into a dependency graph, and as long as they are acyclical, Nix can build all of them. This set can be abstracted with the callPackage function. This is how it is done in the Nix Packages Collection in this amazing file all-packages.nix.

This file is queried implicitly when we install a package in the form:

nix-env -i hello

This is the equivalent of:

nix-env -f .../all-packages.nix -i hello

Both will build and install hello. Nix will represent all the dependencies as a graph and build them as needed. It’s important to note that Nix is Lazy: The parameters are not evaluated until called, which means that dependencies will not be built until (or if) needed.

The file for all-packages can be changed using the nix-channel command. Channels are sorted by stability status.
How Can I Install a Specific Version of a Package with Nix?

The Nixpkgs repository includes the latest versions of packages (according to the selected stability branch). Packages depend on each other and are built as a whole. To pin a specific version of a dependency, you must switch to a different revision of Nixpkgs altogether. A great utility to reverse-search a Nixpkgs revision according to a package’s version is Lazamar’s Nix Package Search.

It’s best practice to always pin your build dependencies to a specific revision of Nixpkgs, for consistency (as you’d do with Docker), and to update to the latest version of Nixpkgs on Nix-env, according to your selected Nix-channel (as you’d do with Homebrew).

Other Nix Tools

NixOS — using the primitives listed above, builds and configures an entire Linux distribution. The whole of NixOS is defined inside Nixpkgs repository, which is incredible.
NixOps — related to Cloud deployment, deploys NixOS system configurations to remote machines, as well as provisions cloud resources.
Hydra — CI tool that periodically checks out the source code of a project, builds it, tests it, and produces reports for developers. Hydra is used to check the stability status of the Nix channels.
Flakes —an upcoming feature that will remove much of the hassle of pinning dependencies with syntactic sugar. Each dependency’s commit hash will be stored inside a flake.lock file. This is intuitive for NPM/Yarn or Cargo users.

So, Why Not Docker?

Nix and Container engines such as Docker are two very different tools. One is a package and build manager, the other is a resource isolation mechanism that virtualizes the host’s operating system. Both have great caching mechanisms behind them, and both can be used for consistent environments on Linux machines. See below about how Replit migrated from Docker to Nix.

The main abstraction of Docker is the Container: a loosely isolated, lightweight, portable, and encapsulated environment that contains everything needed to run the application. The container — which is runnable — is described by a read-only Image. The image is created by a Dockerfile, where each directive creates a separate Layer, tagged by its cryptographic hash and cached.

Like layers, images can be built one on top of the other and vertically stacked, e.g., the official Node image is built on top of the tiny Alpine Linux image. Your node app would probably be stacked on top of the node image.

Layers of Docker node image (node:slim) from Docker Hub

Containers define the implementation of an image or a layer in terms of another, its parent. Nix creates new functionality by assembling or composing dependencies. Nix requires dependencies to be explicit, and these dependencies are black-boxed and consumed through their interface.

However, Dockerfiles don’t have to be linear. Multi-stage builds introduce a new abstraction: the stage. Docker’s new BuildKit traverses stages from the bottom (of the target stage) to top in a graph data structure, skipping unneeded ones, and building stages concurrently where applicable.

Graph of BuildKit’s Multi-stage build, starting from the bottom (the target stage) to the top, discarding unneeded stages. From ‘Dockerfile Best Practices’ talk: https://youtu.be/JofsaZ3H1qM?t=1169

Favor Composition Over Inheritance

It’s difficult to change layers in Docker, as we’re not sure what each component does or how it will affect the lower layer. Also, developers are disincentivized from changing higher layers as they risk rebuilding all the lower layers in the Dockerfile. This is also a performance bottleneck in terms of concurrency, as Docker builds layers in sequence, and unneeded stages will be unnecessarily built and then discarded.

Docker has a great advantage which is immediately familiar to developers and ops alike. Nix originated as a Ph.D. thesis and it sometimes feels like that. But a design that doesn’t take change into account risks major redesign in the future. Docker hashes machine states, Nix hashes the precise components of a build. As explained earlier, the two tools serve different purposes.

In our case, we were building a library for a client app, so there was no need to ship a machine container as would’ve been the case when developing a Node microservice in Kubernetes. We just needed to share a consistent build environment to create replicable builds. Furthermore, with nix-shell, we can still use our local XCode and the rest of macOS’s walled garden for our tvOS and iOS builds.

The Case of Replit

Replit is a collaborative in-browser IDE with support for a huge number of languages. Replit started with a separate Docker image for each language, but concluded that it was simpler and more efficient to use a single monolithic image: Polygott. This has become a huge burden to maintain, in their own words, as “every new package creates a new exciting way things can break.”

With Nix, Replit users themselves can define infinite combinations of sandboxed environments without the need to maintain a monolithic Docker image. Each machine has /nix/store (with all the binaries cached) mounted, so the instantiation of their environment is immediate.

How Does it Compare with Homebrew?

Homebrew is an incredible tool which has become second nature for most macOS users. Installations work out of the box and is intuitive to use.

Like Nix, Homebrew builds from source unless it finds a “bottle,” that is, a pre-built binary. Similarly — and for the same reason — Homebrew has to be installed into a default path (/opt/homebrew on Apple Silicon or /usr/local on Intel) to enjoy pre-build binaries. This folder is referred to as the cellar.

Homebrew uses Ruby for its formulae, which provides instructions and metadata for Homebrew to install a piece of software. A formula is defined as a class that inherits from Formula. This follows the object-oriented paradigm, unlike the functional Nix derivations which are defined with a function.

class Wget < Formula
  homepage "https://www.gnu.org/software/wget/"
  url "https://ftp.gnu.org/gnu/wget/wget-1.15.tar.gz"
  sha256 "52126be8cf1bddd7536886e74c053ad7d0ed2aa89b4b630f76785bac21695fcd"

  def install
    system "./configure", "--prefix=#{prefix}"
    system "make", "install"
  end
end

Homebrew can be used in Linux (formerly Linuxbrew), although Linux distributions often have popular package managers. Similar to nix-channels, brew uses “Taps,” which are third-party repositories.

The immense popularity of Homebrew in Mac gives it an advantage over Nix’s build reliability and thoughtful dependency graph. Most installations are pre-built and “just work.”

Conclusion

From a marketing perspective, I find that Nix lacks branding and distinctive names for their services (except for Hydra and Flakes), which makes it difficult to search for documentation. Nix has merged Nix and NixOS documentation, so trivial beginner searches about nix-env easily lead to solutions about the modification of configuration.nix, which is only applicable to NixOS.

The use of /nix/store has been a bit non-conventional on the part of Nix, as it breaks the FHS guidelines. It would have been more appropriate to put it under /var somewhere. I don’t think macOS follows FHS, but now the root (/) level is read-only in macOS, and Nix had to scratch their head to find workarounds.

Nix isn’t as intuitive as other build tools, but it excels at correctness. As such it aims to have the rigor of science and shows the hard work from academia. It has been embraced by the communities of functional languages such as Haskell and NixOS has piqued the interest of the entire Linux community.

How I Switched from TypeScript to ReScript

Ronen Lahat — Wed, 20 Jan 2021 19:59:02 +0000

A glimpse into a more civilized (yet challenging) tool in the JavaScript ecosystem

Article originally published at Medium

This is not evangelism of ReScript or a one-to-one comparison with TypeScript. I love TypeScript. I decided to rewrite a small TypeScript+React+Jest side project into ReScript.

ReScript is not new. In a way it’s as old as JavaScript itself. ReScript is a rebranding of ReasonML (Facebook) and BuckleScript (Bloomberg), which wrap OCaml on both ends. The former is an interface of the OCaml syntax, while the latter makes sure to compile the AST into JavaScript. ReasonML was created by Jordan Walke, the creator of React. ReasonML still exists as a parallel project to ReScript, with a slightly different syntax and mission.

ReScript syntax compiling into OCaml Abstract-Syntax-Tree, and BuckleScript compiling into readable, optimized JavaScript

ReScript is not just a rebranding: it’s a ReasonML which freed itself of the yoke of the OCaml ecosystem. By doing so, it forfeited compilation to native code and OCaml library interop, but gained a freer syntax which further resembles JavaScript to embrace its developers, eager for better tools.

First Impression

My first attempt was to just install ReScript on my project, start the watcher, rename an easy file into .res and be guided by the errors. I immediately learned that refactoring into ReScript is not “breadth-first” but “depth-first.” Simply renaming the file extension won’t work, as the compiler stops completely at type errors.

In TypeScript one can gradually assign types and interfaces to dynamic types, while tagging some as unknown or any. Depth-first means that you start with one small function, or one small React component, and write it properly. If all the types are right — and with mathematical precision— your code will compile into JavaScript.

While TypeScript often transpiles into unreadable code, it’s good practice to keep an open tab on the auto-generated js file from ReScript. You’ll be pleasantly surprised by the speed of transpilation, the conciseness and readability of the code, and the performance of such code. If the ReScript code compiled, it means its types are safe and sound, so it can optimize away all the noise.

The only exception I saw to readability and performance of the generated JavaScript was in curried functions. All functions in ReScript are curried by default, and some of them generate code which imports a Currying library. This didn’t happen often, and currying can be disabled.

But what about TypeScript? Inter-operation with JavaScript code is trivial, but importing and exporting types from TypeScript (or Flow) can be more complex, and it creates two sources of truth: one for ReScript types and another for TypeScript.

GenType, described below, auto-generates a typed tsx file from your ReScript code which you can import into other modules. This helped for exporting ReScript types, but it’s not possible to import TypeScript ones. The automation of type conversions eased the problem of the two sources of truth.

Furthermore, the generated ts code uses CommonJs require syntax, which break when using native ECMAScript module support. I also had to tweak my tsc to not transpile the auto-generated tsx into a fourth (!) source file:

.res ReScript source code.
.bs.js compiled JavaScript, which you can ignore in your source control
.gen.tsx auto-generated by GenType, which import the compiled JavaScript code and re-export it with proper types. Also add to your .gitignore.
.gen.jsx accidentally transpiled by TypeScript, delete it and reconfigure your tsconfig.json.

I first rewrote my algorithms, since they didn’t have any third-party imports to inter-operate with, and the import syntax was daunting for me at first. Some teams go for a data-first strategy, or a UI-first one (as Facebook did in 2017 for Messenger.com, rewriting 50% of the codebase).

Types

ReScript is part of the statically typed functional programming language family, which means it’s not compiling. Just kidding, it means it uses the Hindley-Milner type algorithm, which deduces types with 100% certainty and can prove it mathematically as long as your variables are immutable (and a few other language design choices). TypeScript on the other hand tries to do it’s best at finding a common type for all your usages.

This might blow your mind as a TypeScript user, but the following ReScript function is fully statically typed:

let add = (a, b) => a + b

ReScript knows with provable certainty that a and b are both int and that the function returns an int. This is because the + operator only works on two int and returns an int . To concatenate two strings you’d use ++ and for two floats use +.. To combine two different types you need to convert either of them. Also, no semicolons.

If you’re like me and like to type your code as you prototype, you can do so as you’d expect:

let add = (a: int, b: int): int => a + b

The generated JavaScript code in both cases is the same (ReScript v8.4.2):

'use strict';
function add(a, b) {  
    return a + b | 0;  
}
exports.add = add;

Notice how I didn’t specify any module exports but the resulting code did. This shows how everything in the module/file is exported by default. The JavaScript function itself is not type safe, so importing it in a JavaScript module and using it there won’t have all the advantages of ReScript.

You can try it for yourself in the official playground.

Generating TypeScript

To interoperate with TypeScript with proper type information you’ll use third-party genType. Add it as a devDependency and annotate the module export you want to generate with @genType (in previous versions you’d surround annotations with square brackets).

// MyModule.res
@genType  
let add = (a,b) => a + b

This will result in the following TypeScript. Notice how the generated TypeScript imports the generated JavaScript MyModule.bs.js file:

// MyModule.gen.tsx
const MyModuleBS = require('./MyModule.bs');
export const add: (_1:number, _2:number) => number = MyModuleBS.add;

GenType generates a one-line re-export of your generated .bs.js file, with proper TypeScript typing. From this example you’ll notice two more things:

Every file is a module.
Everything is exported.

Here’s an example repo genTyping to TypeScript with React.

For using TypeScript types, see “Importing TypeScript Types” below.

Records

There is only one type which does need a type declaration, which is the record type. A type declaration will look like this and produces no JavaScript code:

type student = {  
  age: int,  
  name: string  
}

Types must begin with a lowercase! If we prepend it with @genType, the generated TypeScript will look like this:

// tslint:disable-next-line:interface-over-type-literal_  
export type student = {  
    readonly age: number;  
    readonly name: string  
};

If you’re wincing at the lower-cased type breaking all your conventions you can rename the type on conversion with @genType.as("Student"). This will add another line of code below the previous one:

export type Student = student;

Also it includes a tslint ignore line, which I hope they switch soon to eslint as the former is deprecated.

These are record types, not ReScript objects (don’t misuse the string type on them). As soon you type something like foo.age ReScript will know that foo is of type student. In case there’s another record with and age field, it will infer it’s the last one declared. In that case you might want to explicitly annotate the type.

In the case you don’t want that much ceremony, you can use the object type and index it with a string: student["age"]; then you don’t need to declare a type.

Furthermore you can use student as a variable name, so student.age is a valid expression, TypeScript would scream at something like this. Variables (that is, bindings) and Types live in a separate namespace, so a student of type student an be written as student: student.

Nominal Typing

Record types have “nominal typing” similar to Java or C#, as opposed to TypeScript’s “structural typing.” This is why interfaces are so important in TypeScript, and are used much more than Types. TypeScript doesn’t really care about “what you are”, it cares about “how you look.”

For instance, if there’s another type, say, teacher with the same fields of a student, you cannot assign a student to somewhere expecting a teacher:

// defined first  
type student = {  
  age: int,  
  name: string  
}

// defined last  
type teacher = {  
    age: int,  
    name: string  
}

// t is a teacher  
let t = {  
    age: 35,  
    name: "Ronen"  
}

let s: student = t // Error!

You’d get a colored error saying:

We've found a bug for you!
//...This has type: teacher
Somewhere wanted: student  
FAILED: cannot make progress due to previous errors.  
>>>> Finish compiling(exit: 1)

Unlike TypeScript’s tsc compiler, bsb won’t begrudgingly continue its transpilation work into working JavaScript. It will stop with a non-zero exit code, and you have to fix the issue in order to make any progress.

Optionals

One of the features I most like in modern TypeScript (or future JavaScript) are the optionals. They make working with nullable types easy and concise:

const something: string = foo?.bar?.baz ?? "default";

something will be the content of baz if it reached that far, or be "default".

There are no null or undefined in ReScript. But we can work with nullable values using the Variant option. But how can we get the elegance of the above TypeScript code? I tried to answer this question but, we can’t, currently. Not enough sugar.

As with other functional languages, we can use a myriad of interesting library functions. Some of Belt utility functions are:

Belt.Option.Map will execute a function on the optional value if it exists, or return None.
Belt.Option.getWithDefault will return a default if the optional is None.
Belt.Array.keepMap will trim away all None values from an array.

But for this case, the best option is with Pattern Matching:

let baz = switch foo {  
   | Some({ bar: Some({ baz: baz })}) => baz  
   | None => None  
}

There isn’t yet a sugared syntax for optionals; the optional operators are very new to TypeScript as well.

The important quality of pattern matching is that the compiler will complain if there’s any case — doesn’t matter how deeply nested — you haven’t addressed. It’s best practice for most cases.

Pipes

Pipes are great. They compile this code:

person  
  ->parseData  
  ->getAge  
  ->validateAge

Into this:

validateAge(getAge(parseData(person)));

Previous versions used a triangle operator |>. The difference is in where to shove the data: as the first parameter, as the arrow does, or as the last parameter, as the deprecated triangle does. More about this.

Notice that in the case of a one-parameter function we don’t write the unit, that is (). This is a common beginner’s mistake. In the case of multiple parameters, the value gets passed as the first one and the other parameters begin with the second one.

This is especially important in a functional language, since we lose some of the elegance of calling methods in objects.

What would be a JavaScript method call such as map:

myArray.map(value => console.log(value));

Has to be written functionally in ReScript as:

Belt.Array.map(myArray, value => Js.log(value))

But can be rewritten as:

myArray -> Belt.Array.map(value => Js.log(value))

As a newcomer I try to find a use for it anywhere I can, which can lead to the bad practice of rewriting code around it to impress my coworkers. To use it on JavaScript libraries you’ll have to write the correct bindings for them. This is one thing I’d like to see in JavaScript. Here are a few stage-1 proposals.

By the way, if you’re not using Fira Code then you’re missing out on a lot of the pipe’s aesthetics.

Promises

This was very frustrating for me. I love using modern async and await syntax in my code, which ReScript didn’t implement yet. I had to go back into thinking about then and resolve, which made simple code look complex.

The following code:

const getName = async (id: number): Promise<string> => {  
    const user = await fetchUser(id);  
    return user.name;  
}

Is de-sugared into:

const getName = async (id: number): Promise<string> =>   
    fetchUser(id).then(user => user.name);

Now consider then to be a function in the Js.Promises module instead of a method, which accepts fetchUser(id) as its last parameter, and you can write it like this:

let getName = (id) =>  
    Js.Promise.then_(  
        user => Js.Promise.resolve(user.name),  
        fetchUser(id))

Typed as Js.Promise.t<string>, and with arrow pipe syntax for readability, the above function can be written as:

let getName = (id): Js.Promise.t<string> =>  
    fetchUser(id) |> Js.Promise.then_(  
        user => Js.Promise.resolve(user.name))

The Promise library still uses the old convention of passing the data as the last argument, so in order to use the newer arrow pipe, an underscore has to be placed in the proper location.

Here are examples for Promises written in the (almost-identical) ReasonML syntax.

The ReScript team promised (no pun intended) to implement a Promise API revamp with their own async and await.

Import JavaScript Modules

If you’re writing only in ReScript you don’t need to bother with imports or exports, and this is done under the hood. Every file is a module and everything in it is exported. If you only want specific things exported you do so with an interface file. To import JavaScript modules however, the syntax can get complicated.

To import dirname from the path module, you’d write:

@bs.module("path") external dirname: string => string = "dirname"

the elements of an import from JavaScript files

Then use it accordingly:

let root = dirname("/User/github") // returns "User"

For ReasonReact this became particularly tiresome, as I had to define inline modules for each React Component, and reexport the default export as the “make” function, paying attention to named parameters such as “children.” Here I imported the Container from react-bootstrap and used it in ReasonReact:

module Container = {  
    @bs.module("react-bootstrap/Container")  
    @react.component  
    external make: (~children: React.element) => React.element = "default"  
}
@react.component  
let make = () => <Container> ...

Redex

For this case I can get the bindings from redex, and add it as a dependency both to my package.json and my bsconfig.json. I can then import it with open ReactBootstrap at the top of my file. This is similar to DefinitelyTyped, where you can find high-quality type definitions for TypeScript.

For this case however I ran into an error, as the package I needed was not updated to the latest version. I had to fork it and manually update it to react-jsx version 3.

Importing TypeScript Types

You can’t import a type from TypeScript and use it in ReScript, you have to re-declare it. However, you can link the type you created to the original TypeScript one for correct inter-operation. Here’s an example with Node.js’ fs module:

@genType.import(("fs", "Dirent"))  
type dirent

Notice that I passed a tuple to import, not an argument list. This will link my type dirent to fs.Dirent, and will generate the following TypeScript:

import {Dirent as $$dirent} from 'fs';_// tslint:disable-next-line:interface-over-type-literal_  
export type dirent = $$dirent;

You can declare the entire type, in case you need to use its properties, or leave it as is.

Because of the syntax overhead of TypeScript-ReScript inter-operation, I recommend doing it as little as possible, using each language in separate areas of your app.

ReasonReact

ReasonML (now ReScript) was created by Jordan Walke, the creator of React. Reason+React pushes the React philosophy further by utilizing the language syntax and features for ReactJS’s programming patterns.

ReasonReact provides smooth JS interop and uses built-in language features to integrate into UI framework patterns left unaddressed by ReactJS, such as routing and data management. Using them feels like “just using Reason.”

The documentation for ReasonReact still uses the old syntax, so things like:

[@react.component]

Needs to be changed into:

@react.component

If you want to use the old syntax, just change the file extension to .re instead of .res.

ReasonReact is stricter than ReactJS, mainly in its use of types (e.g., strings need to be used with React.string()in JSX. Other than this, the React.useState returns a proper tuple instead of an array, the way it was originally intended. Finally, React Components are rendered through a make function, and prepended with @react.component (I added @genType as well for TypeScript generation):

For the example, I imported this component into a React TypeScript file:

// index.tsx
import { make as Demo } from "./pages/Demo.gen";
// ...<Demo name={"Foo"} />

Which, when rendered, looks like this:

In case we don’t want GenType for TypeScript generation, we just import Demo.bs instead.

Testing

In order to write tests in ReScript, and thus test your code directly, you can use bs-jest, which provides ReScript bindings to Jest. If you prefer, you can also use the slightly less mature bs-mocha. You can also test the generated JavaScript or TypeScript files with no extra configuration.

Since ReScript is in the JavaScript ecosystem it makes little sense to create specialized testing tools for ReScript, and the direction seems to be in developing bindings for JavaScript testing tools.

With bs-jest, you have to name you can’t name your file foo.spec.res, only with a valid module name, such as foo_spec.res. Jest will run on the compiled folder, by default inside lib/js. Also, assertions are not executed immediately, but instead returned by the function and run at the end of the suite. It’s a functional way to thing about tests. Consequently, you can only write one assertion per test, which is best practice anyway.

Tooling

ReScript devs did well on prioritizing the plugin for VSCode, which works really well. With the ReScript’s watcher running, you’ll see your Type errors underlined in red, with a descriptive bubble on hover. You also get type hints, formatting, and jumps to definitions. There’s also official support for Vim (both plain Vim and Coc Language Server) and Sublime.

Screen capture from rescript-vscode.

The Community

A few times in my coding career I had to work with small communities, and I always loved it. I developed smart-contracts in Solidity, some database queries in the functional language Q, and Roku channels in BrightScript. You end up working with Slack/Discord/Gitter open and code together with the few others going through your similar problems. You don’t even bother checking StackOverflow for answers.

This forces you to read and reread the official documentation and examples, as you don’t want to look like dumb in the chatroom. Also, you’re part of a community maintained by real people, where you can always contribute something interesting, and even shape its development.

Not all communities are alike, of course. I personally found the ReasonML/ReScript community to be welcoming. ReScript has an official forum where you can communicate asynchronously and with a permanent paper record you can search. The core team consists of a handful of developers with public Twitter accounts, and there’s an official blog. I found however that the community hangs around in the ReasonML’s Discord server, in an unofficial ReScript room.

Finally, there’s ReasonTown, “a podcast about the ReasonML language and the community that makes it good,” ReasonConf’s YouTube channel, and Redex, to find bindings for your libraries.

Conclusion

The switch is not easy; a refactor of an existing app is even more difficult given its fatal stop on the first issue. This will certainly hinder its adoption. Popular transpilers, such as TypeScript, SCSS, or CoffeeScript garnered adoption by its ease. Just copy-paste your code — or rename your file — and you’re done.

This is different. ReScript, as with other statically typed functional languages, aims at changing the way code is approached at a fundamental level. I believe we’ll see a greater adoption of functional programming in the future, eventually becoming the default for some industries. This is due to the mathematical approach to types, formal verification of a program’s correctness, and given immutability: less moving pieces and mental mapping.

We’re already at the first stage of adopting a “functional style” in the ecosystem with map, filter, reduce functions in JavaScript. ReScript represent the next — hybrid stage — of a properly functional language from the ML family which compiles to the industry’s standard JavaScript.

Functional programming at its core takes itself seriously. It’s mathematical, formal, and doesn’t comply with hacks. It aspires to deals with truths, not processes. Writing a «functional style» in JavaScript only whets one’s appetite for more, as the language brings one’s good intentions down, not up. ReScript, while frustrating, might be the precision tool for a more civilized future in the ecosystem.