DEV Community: Gio

Pitfalls When Adding Turborepo To Your Project

Gio — Tue, 15 Feb 2022 14:23:09 +0000

Originally posted at https://engineering.caribouwealth.com

We at Caribou have recently adopted a new TypeScript monorepo stack for our app frontends using turborepo.

Issues faced with our original monorepo setup

As our number of apps and codebases grew, we decided that we wanted to:

Save time and money on ever increasing build times
- Build times were increasing dramatically as we went from 2 apps in our monorepo to 4. The original monorepo setup would naively deploy all apps inside the project on every push to GitHub. Once we got to 4 projects, the build times got really out of hand.
Enable the granular tracking of individual application deployments for our metrics monitoring
- We strive to do 5 releases per week on average (across all of our projects), and we need to track whether we're hitting these targets or not.
Add CircleCI as a layer between GitHub and Netlify to manage our CI pipeline
- Our other repos were already on CircleCI, so this allowed us to unify our CI/CD process.

As we’ve faced multiple roadblocks undergoing this transition, we decided to record them for the benefit of developers at Caribou or anyone else undertaking similar endeavours.

Starting point and stack choice

We started from a flat file system containing multiple apps that were all located in the project’s root folder. The directory structure itself needed work.

Researching & The Design Document Phase

At Caribou, net-new functionality or highly complex additions to our systems must go through a design document process.

We wrote a design document outlining our requirements and how the new stack would meet them. Our requirements were not complicated. We wanted to rebuild and deploy only those parts of the monorepo that have changed, and add our desired checks on CircleCI.

We had a first look at monorepo management. We knew Lerna was a popular choice, but Turborepo had recently been acquired by Vercel and seemed highly promising. It purported to be very fast but simpler than Lerna, and one of our engineers had a positive experience with it.

After a few days of playing around with Turborepo, we concluded that its simple and intuitive API was sufficient justification to proceed with it as our tool of choice.

Turborepo works with one of Yarn, npm, or pnpm workspaces. We already used npm as a package manager, so in order to keep things familiar we went with npm workspaces.

Finally, we already used CircleCI for our backend CI, so we wanted to keep using
CircleCI on the frontend.

Setting up the npm workspace

This is easily done inside the root package.json.

Run `npm install` to create the symlinks inside `node_modules`

One thing to note is to not forget to rerun npm install at the project root (we initially did...). If you forget, npm won’t create the symlinks to your workspaces/packages inside node_modules, and you won’t be able to use absolute paths to other modules inside your imports.

npm v7 is needed or the IDE/compiler can't resolve modules

Even if you run npm install, only npm 7 and up support workspaces. There is no straightforward way to enforce developer npm version although it is not impossible, so you might want to document the version requirement in your root README. A developer without npm 7+ will end up with unresolved modules in their editor.

New commands to install dependencies and run scripts

When using npm packages, you must keep in mind that the commands to install dependencies and run scripts are different.

Assuming a sub-package named blog, installing the dependency neverthrow is done by running this command at the monorepo root:



# DON'T do that anymore
npm install neverthrow
# Do this instead
npm install --workspace blog neverthrow
# or for short
npm i -w blog neverthrow

Running the start script from the blog subpackage is done with the following:



# Don't do that anymore
npm run start
# Do this instead
npm run --workspace blog start
# or for short
npm run -w blog start

Separating dependencies

One detail which was not immediately obvious during the transition is that the root package.json should only contain dev dependencies. (It doesn’t need to be all of them, either.) We initially thought we should keep common dependencies in the root package.json. This caused React errors from having multiple instances of React running.

Another thing to note is you should never see a package-lock.json inside a sub-package’s folder. This means the npm install command was run inside it, which is incorrect! Delete the resulting package-lock.json as well as the node_modules it newly installed. When using npm workspaces, all dependencies live in the root node_modules.

Import resolution after transitioning

We use webpack for our build pipeline, and found out that webpack was sometimes resolving modules that tsc couldn’t. This is problematic, as we wanted to use tsc for our CI checks! After experimentation, I found that imports must adhere to the following format:

Absolute imports from the current package must not be prefixed with the package’s name, i.e. if you are currently inside ha-dash (the name of one of our sub-projects within the monorepo) you must write import { whatever } from 'src/components and not import { whatever } from 'ha-dash/src/components'.
- The src may be skipped by setting that package’s baseUrl to src in its tsconfig.json
Absolute imports from other packages must be written as {package_name}/src/some_module
- Unfortunately we haven’t found how to skip the /src/ for cross-package imports yet. This solution seemed promising but it causes the typescript compiler to hang for some reason.

While transitioning and changing import paths, I’ve often used Linux shell loops like the following:



# make sure your current directory is the package you wish to perform changes in
# commit your current repo state so you can undo in case of mistake!
for file in **/**.{ts,tsx}; do
  sed -i -e "s?from 'survey-manager-src/?from '?g" $file;
done

while in the survey-manager directory, I ran this command to change all instances of from 'survey-manager-src/ to from '.

Failing tests

We use jest for tests, and found that in order for tests to work in our setup we needed each package to contain a babel.config.js file including '@babel/preset-react'. This may be applicable to your pipeline, too!

CircleCI

Saving turbo cache artifacts between builds

Turborepo stores build artifacts at node_modules/.cache in order to restore files which do not need to be rebuilt.



build:
    executor: caribou
    resource_class: xlarge
    steps:
      - checkout
      - attach_workspace:
          at: .
      - restore_cache:
          keys:
            - previous-build-{{ .Branch }}
      - run:
          name: "Build apps"
          command: npx turbo run build
      - save_cache:
          key: previous-build-{{ .Branch }}
          paths:
            - node_modules/.cache
      - persist_to_workspace:
          root: .
          paths:
            - apps/

The important sections here are restore_cache and save_cache. Basically this looks for any turborepo cache saved by CircleCI named previous-build-{name_of_current_branch}. Then turbo will know what packages it needs to rebuild.

The persist_to_workspace section is important, as it lets the next step (deploy) have access to the built files.



deploy:
    executor: caribou
    steps:
      - checkout
      - attach_workspace:
          at: .
      - run:
          name: "Deploy to netlify"
          command: ./deploy.sh ${CIRCLE_BRANCH} ${CIRCLE_SHA1}

Saving dependencies between builds

While you are at it, you can cache npm dependencies between runs. The strategy is slightly different:



install-deps:
    executor: caribou
    steps:
      - checkout
      - restore_cache:
          keys:
            - npm-deps-{{ checksum "package-lock.json" }}
            - npm-deps-
      - run:
          name: "Install Dependencies"
          command: |
            echo "Node version: $(node -v)"
            echo "npm version: $(npm -v)"
            npm install
      - save_cache:
          key: npm-deps-{{ checksum "package-lock.json" }}
          paths:
            - node_modules
      - persist_to_workspace:
          root: .
          paths:
            - node_modules

We use npm-deps-{{ checksum "package-lock.json" }} this time, to look for cached node modules from runs of any branch that had the same package-lock.json. If none is found, we simply get the latest cached node_modules. Then npm install is run anyway, so that any missing package is added.

⚠️ The netlify CLI cannot use same URL prefixes as automatic branch deployments

https://github.com/netlify/cli/issues/1984#issuecomment-862554734

If you’ve previously used automatic netlify deployments by branch, then you might be used to having URLs formatted as {branch-name}--{site-name}.netlify.app.

As soon as you’ve used this feature once, you can no longer use that subdomain with the Netlify CLI. We had to move to other prefixes using the Netlify CLI --alias option. The documentation says to “avoid” using the same prefix as branch names, but doesn’t say why... now you know! Here is the GitHub issue about this.

Only deploying the individual apps which turbo rebuilt

This is something which the documentation for the netlify CLI doesn't tell you, so you won't find out until you actually run it: the netlify CLI compares the newest build's file hashes with the previous build's hashes, and requests only those files which have changed. In other words, you can safely use the netlify CLI to trigger deployments of all your packages, and netlify will only ever receive those files which have changed.

However, if you are using something less sophisticated than netlify, here's a bash script I wrote before I realised that netlify already took care of this. This script will parse the turbo build output and only redeploy apps which turbo deemed necessary to rebuild.



# Save the turbo output with this command:
# $ npx turbo run build 2>&1 | tee .build_output

APPS=("blog" "client-dashboard" "admin-panel")

deploy_app() {
  app_name=$1
  # your deployment command here
}

for app in ${APPS[@]}; do
  case "$(cat ./.build_output)" in
    *"${app}:build: cache miss, executing"*) deploy_app "$app" ;;
    *"${app}:build: cache bypass, force"*) deploy_app "$app" ;;
    # Uncomment the first *) line to force deployment
    # *) deploy_app "$app" ;;
    *) echo "turbo did not rebuild $app, not deploying." ;;
  esac
done

And for whoever it might help, our netlify deploy function:



# Those environment variables are set in CircleCI

site_id_of() {

  case "$1" in

    ha-dash) echo "$HA_DASH_NETLIFY_ID" ;;

    fa-dash) echo "$FA_DASH_NETLIFY_ID" ;;

    planner) echo "$PLANNER_NETLIFY_ID" ;;

    survey-manager) echo "$SURVEY_MANAGER_NETLIFY_ID" ;;

  esac

}

deploy_app() {

  app_name=$1

  if [ "$BRANCH" = "production" ]; then

    branch_option=""

  else

    branch_option="--alias staging-branch"

  fi

  # --prod argument applies to staging too

  npx netlify deploy </span>

    --auth=$NETLIFY_AUTH_TOKEN </span>

      --dir=./apps/$app_name/build </span>

    --message="$BRANCH deployment of $GIT_HASH" </span>

    --prod </span>

    --site=$(site_id_of "$appName") </span>

    $branch_option

}

Conclusion

Do you have experience transitioning to monorepo management tools? Do you see anything we can improve? Let us know! I hope this log of some of the challenges making the transition can be helpful to some of you. Happy hacking!

Did you enjoy this post? We're hiring!

We have several open roles across Ops, Design, Marketing and Engineering!

Get Paid To Work On Open Source

Gio — Wed, 21 Jul 2021 01:37:35 +0000

Cover: Tianyi Ma (Unsplash)

Hey! I'm the maintainer of neverthrow; one of the more popular implementations of a Result type for the TypeScript ecosystem.

I am trying out a "bounty" program whereby I pay you to fix issues or implement new features.

What Is `neverthrow` or a "`Result` Type"?!

For a gentle introduction about what a "Result type" even is, check out my blog post on type-safe error handling in TypeScript.

It's a small library, but dense with advanced concepts:

Generics
Functional programming and immutability
Algebraic Data Types
Mapped Types and advanced typescript black magic

Check out the post on GitHub for details on how to participate .

Challenge: Create a `pad` function without using loops!

Gio — Tue, 20 Apr 2021 14:28:32 +0000

Photo Credit: Kylie Fitts / www.kyliefitts.com & https://unsplash.com

In any language, implement a function pad that takes a value and conditionally pads it with n number of padding:

const padded = pad({
  value: '👋',
  padding: '*',
  requiredLength: 4,
})

console.log(padded) // --> ***👋

//////////
// Case 2: do not pad a value whose length is equal to `requiredLength`
//

const padded = pad({
  value: '👋👋👋👋',
  padding: '*',
  requiredLength: 4,
})

console.log(padded) // --> 👋👋👋👋


//////////
// Case 3: do not overwrite a value that is longer than `requiredLength`
//

const padded = pad({
  value: '👋👋👋👋👋👋',
  padding: '*',
  requiredLength: 4,
})

console.log(padded) // --> 👋👋👋👋👋👋

Submit your solutions down below! 👇👇👇

Remember, your solution cannot use any sort of loop construct such as while, do, or for!

WARNING: Here is my solution in typescript.

Type Constraints In TypeScript

Gio — Thu, 12 Nov 2020 16:14:04 +0000

Photo credit: https://unsplash.com/photos/ar2dUoleWYA

Suppose you're a back end API developer and you need a way to guarantee at compile time that you're only sending json-serializeable data out of your API.

You have a send function that takes some data and sends it to a API consumer.

const send = <T>(data: T): Promise<null> => {
  // magic...
  // or more realistically
  // calls express' res.json() method!
}

And you're trying to prevent a dev from trying to send the following out of your api:

send({
  makesNoSense: () => { console.log('woops!') },
  myDate: new Date(), 
})

The above would be stringified (i.e. serialized) under the hood into { myDate: 'iso-date-string' }. Functions aren't part of the JSON spec and thus would be removed entirely. And Dates are automatically cast to strings which is not a very efficient way of sending timestamps down the network (hint: you want unix timestamps in the form of an integer).

Woops! Looks like a dev forgot to call a function and also forgot to call Date.getTime 😭

So how do we prevent this sort of thing?

Type Constraints To The Rescue

A type constraint is a "rule" that narrows down the possibilities of what a generic type could be.

For example, in the the send definition above, we declared a type variable T that is not constrained at all. Which is why we were able to call send with values that aren't JSON serializeable.

// This compiles ... API would send `{}`
send(new Set([1,2,3]))

We can thus narrow the possibilities of the generic type T to allow allow JSON values as follow:

const send = <T extends JSONValues>(data: T): Promise<null> => {
  // ...
}

The only difference is that now we've appended extends JSONValues to the type variable declaration for T.

In plain english T extends JSONValues translates to, "T is a generic type that must conform to the definition of JSONValues".

What's JSONValues?

It's defined as this:

type JSONValues
    = number
    | string
    | null
    | boolean
    | { [k: string ]: JSONValues }
    | JSONValues[]

... Yup, this is the entire JSON specification in 7 lines of code! 🤯

Now, if I call send(new Set([1,2,3])) I will get a type error. Whaaaat?!?!

Now you can guarantee at compile-time that you will only send valid data to your JSON API consumers :)

Live demo

Conclusion

Type constraints are a very powerful way to supercharge your typescript codebases.

For each generic type variable that you'd like to constrain, you would append the extends SomeTypeName to the definition. Example:

const myFn = <T extends JsonValues, U extends FinancialData>() => {...}

Hope that helps!

Shameless Plug

Liked this post?

I stream functional programming, TypeScript and Elm development every Tuesday at 10am on Twitch!

https://www.twitch.tv/vimboycolor

Implement a type-safe version of Node's Promisify in 7 lines of TypeScript

Gio — Mon, 31 Aug 2020 19:54:25 +0000

type Callback<A> = (args: A) => void;

const promisify = <T, A>(fn: (args: T, cb: Callback<A>) => void): ((args: T) => Promise<A>) =>
  (args: T) => new Promise((resolve) => {
    fn(args, (callbackArgs) => {
      resolve(callbackArgs);
    });
  });

Check out this video that demonstrates type inference in action!

I'm using type-variables T and A to generecally implement this function over the original function's arguments, and the callback function's arguments.

Exercises:

Did I need to define an inline anonymous function for the second argument of fn? How else could I have invoked fn?
Note that my Callback type isn't a typical error-first callback like in a lot of node APIs (this is just because the functions I'm trying to promisify aren't error-first callbacks). So I'll leave it to you as an exercise to refactor my promisify function to reject when an error on an error-first callback is not null ;)

Introducing Elm Antd

Gio — Sun, 05 Jul 2020 15:43:13 +0000

Hey friends,

I'd like to share a project I've been working on that is finally in a presentable state (but still in its infancy)!

The project is called Elm Ant Design.

The repo can be found here.

I built Elm Ant Design in such a way that it encourages contributions from all developers, even those who don't yet know Elm!

Project Goals

Attain 100% feature parity with React's implementation of Ant Design (which can be found here)
Create a space that allows developers of all skill levels to be able to easily contribute to an open source project
- You'll see in various parts of the documentation that I make reference to contributing to the project
- I've also created a large list of "good first issue"s and "help wanted" tags on github (link)

Don't Know Elm? No Problem!

Elm is a tiny language compared to JavaScript. To learn 80% of Elm takes a lot less time than learning 80% of JS.

Secondly, there is a small portion of the code that is written in JS:

Webpack configuration is in JS
Visual tests using Cypress and Percy are written entirely in JS as well

So you can contribute to Elm Ant Design without even knowing Elm!

"But Elm is not a very popular language, why would I want to learn Elm?"

Elm will make you a better _____ developer. Whether your main language is Python, Java, C#, JavaScript. The fact that Elm is a pure functional language means that you're forced to think in a different way about your programs. And this new mental model will help you write more predictable code in your main language - I guarantee this (I have experienced this effect first hand with my job, as I'm primarily a JS & TypeScript developer).

I would appreciate any and all feedback!

Chaining Failable Tasks

Gio — Sat, 07 Mar 2020 17:36:20 +0000

originally posted on my blog

This post assumes familiarity with TypeScript.

In my previous post, Type-Safe Error Handling In TypeScript, I introduced a npm package to model failure at the type level.

If you're not familiar with neverthrow, here's a quick rundown (feel free to skip this tiny intro by clicking here):

The package introduces a functional alternative to throwing exceptions
- By getting rid of throwing exceptions, you make your error handling logic pure!
- This is the standard approach in many other languages such as Rust, Elm and Haskell to name a few. This isn't some random wild experiment I invented.
neverthrow has a Result type that represents either success (Ok) or failure (Err)

Result is defined as follows:

type  Result<T, E>
  =  Ok<T, E>
  |  Err<T, E>

Ok<T, E>: contains the success value of type T

Err<T, E>: contains the failure value of type E

Usage:

Create Ok or Err instances with the ok and err functions.

import { ok, err } from 'neverthrow'

// something awesome happend

const yesss = ok(someAwesomeValue)

// moments later ...

const mappedYes = yesss.map(doingSuperUsefulStuff)

You can access the value inside of Err and Ok instances as follows:

if (myResult.isOk()) {
  // if I didn't first call `isOk`, I would get a compilation error
  myResult.value
}

// or accessing values
if (myResult.isErr()) {
  myResult.error
}

This quick rundown doesn't do the package justice, so I highly recommend you check out my previous post that really walks you through the package.

...

A while back, I got feedback (link to github issue) from two users that this module wasn't very ergonomic when it came to Results wrapped inside of a promise.

This post is dedicated to covering the problem, and the solution to it.

The Problem

Let's suppose we're working on a project that has 3 async functions:

getUserFromSessionId
getCatsByUserId
getCatFavoriteFoodsByCatIds

And here are the type signatures for each of these functions:

type GetUserFromSessionId = (sessionUUID: string) => Promise<Result<User, AppError>>

type GetCatsByUserId = (userId: number) => Promise<Result<Cat[], AppError>>

type GetCatFavoriteFoodsByCatIds = (catIds: number[]) => Promise<Result<Food[], AppError>>

Let's also assume that you're a developer tasked with leveraging these functions in order to get all of the favorite foods of all of the cats owned by a single user.

By taking a close look at the type signatures of these functions, we can start to see how we might go about implementing our task:

First call getUserFromSession
then get the User and use that value to call getCatsByUserId
then get all of the cats (Cat[]) and call getCatFavoriteFoodsByCatIds by passing it an array of cat ids

The issue is that the values we need (User, Cat[] and Food[]) are wrapped inside of Promise and Result.

First Attempt At A Solution

Let's see how we might implement this naively.

The neverthrow api has a asyncMap method and andThen method that we could use to solve this:

// imagine we have a sessionId already

const result1 = await getUserFromSessionId(sessionId)

// result2 is a Result<Result<Cat[]>, AppError>, AppError>
const result2 = await result1.asyncMap((user) => getCatsByUserId(user.id))

// need to get the inner result using `andThen`
// now catListResult is Result<Cat[]>, AppError>
const catListResult = result2.andThen((innerResult) => innerResult)

// result3 is
// Result<Result<Food[], AppError>, AppError>
const result3 = await catListResult.asyncMap(
  (cats) => getCatFavoriteFoodsByCatIds(cats.map((cat) => cat.id))
)

// so now we need to unwrap the inner result again ...
// foodListResult is Result<Food[], AppError>
const foodListResult = result3.andThen((innerResult => innerResult))

Holy boilerplate! That was not fun. And super cumbersome! There was a lot of legwork required to continue this chain of async Result tasks.

... If there were only a better way!

Using Result Chains! 🔗

Version 2.2.0 of neverthrow introduces a wayyy better approach to dealing with this issue.

This is what it would look like

import { chain3 } from 'neverthrow'

// foodListResult is Result<Food[], AppError>
const foodListResult = chain3(
  getUserFromSessionId(sessionId),
  (user) => getCatsByUserId(user.id),
  (cats) => {
    const catIds = cats.map((cat) => cat.id)
    return getCatFavoriteFoodsByCatIds(catIds)
  }
)

That's it.

Check out the API docs here.

Obviously the above example is quite contrived, but I promise you that this has very practical implications. As an example, here's a snippet from my own side project where I use the chain3 function:

chain3(
  validateAdmin(parsed.username, parsed.password),
  async (admin) => {
    const sessionResult = await session.createSession(admin)

    return sessionResult.map((sessionToken) => {
      return {
        sessionToken,
        admin
      }
    })
  },
  ({ sessionToken, admin }) => Promise.resolve(
    ok(AppData.init(
      removePassword(admin),
      sessionToken
    ))
  )
)

There are 8 different chain functions, each of which only vary in their arity (the number of arguments that the functions take).

chain: takes 2 async Result tasks
chain3: takes 3 async Result tasks
chain4: takes 4 async Result tasks
chain5: etc
chain6: etc
chain7: etc
chain8: etc

The beautiful thing about this chain API is that it retains the same properties as synchronous Result.map chains ... namely, these async chains short-circuit whenever something at the top of the chain results in a Err value 😍

A useful way to think of the chain api is to think of it as the asynchronous alternative to the andThen method.

I've had this issue noodling in my head for a while. Eventually in that same github issue I mentioned at the top of this post, I proposed an approach to chaining many async computations with a set of utility functions.

Before committing to that solution, I started dogfooding this approach through my own side project. After a few days of using this chain API, I concluded that it was in fact quite good and ergonomic.

This API is heavily tested and well-documented!

Cheers!

Type-Safe Error Handling In TypeScript

Gio — Sun, 05 May 2019 18:46:45 +0000

Originally posted on my blog

We've all been there before. We write a function that has to deal with some edge case, and we use the throw keyword in order to handle this situation:

type ResponseData = {
  statusCode: number
  responseBody?: ResponseBody
}

const makeHttpRequest = async (url: string): Promise<ResponseData> => {
  if (!isUrl(url)) {
    throw new Error(
      'Invalid string passed into `makeHttpRequest`. Expected a valid URL.'
    )
  }

  // ...
  // other business logic here
  // ...

  return { ... } // ResponseData
}

Now imagine a month later, you are working on your project when you or a colleague forget to wrap makeHttpRequest inside of a try / catch block.

Two things happen here:

The compiler is no longer able to tell you whether your code is safe from runtime errors. In other words, using throw is not typesafe. And is just as dangerous as any. They both dilute the benefits of using TypeScript in the first place.
Because neither the compiler nor the types tell you that makeHttpRequest can fail (read: throw), you will eventually get a runtime error. This is a waste of time, money and happiness for everyone. People begin to ask why they're using TypeScript if the compiler isn't helping them with something so basic as adding a try / catch block.

So the question is:

How do we encode failability into the typesystem?

First, let's begin by acknowledging that throw is not typesafe. We must use a different approach in order to get the TypeScript compiler on our side.

What if we had a type or interface which represented the outcome of a computation that might fail?

Our type would represent two simple outcomes:

Success case: Which would return / contain contain a "success value" (i.e. ResponseData in the case of makeHttpRequest)
Failure case: Which would return / contain helpful information about why the failure occurred

Let's call our type somethething intuitive like, Result. Let's call our Success variant Ok and our Failure variant Err.

Thus, if we were to formalize our type into code, it would look something like this:

type Result<T, E>
  = Ok<T, E> // contains a success value of type T
  | Err<T, E> // contains a failure value of type E

Going back to our makeHttpRequest function, we would want to encode the potential for failure into the typesystem.

Thus makeHttpRequest would have the following signature:

makeHttpRequest(url: string): Promise<Result<ResponseData, Error>>

And the function definition would look something like this:

// utility functions to build Ok and Err instances
const ok = <T, E>(value: T): Result<T, E> => new Ok(value)

const err = <T, E>(error: E): Result<T, E> => new Err(error)

const makeHttpRequest = async (url: string): Promise<Result<ResponseData, Error>> => {
  if (!isUrl(url)) {
    return err(new Error(
      'Invalid string passed into `makeHttpRequest`. Expected a valid URL.'
    ))
  }

  // ...
  // other business logic here
  // ...

  return ok({ ... }) // Ok(ResponseData)
}

Of course err(new Error('...')) seems a little tedious. But here are some things you should know:

The argument of the err function must be of type E, or you'll get a compile error (type mismatch) between the type inside of err and the return type of makeHttpRequest (where the E type is represented as an Error instance).
- Relatedly, I just chose Error as the type for E for the sake of simplicity ... meaning E could be anything you want! More on this in a bit!
The user of makeHttpRequest can use this function without fear that it might randomly throw. No more runtime errors 🚀
The author of the makeHttpRequest function also doesn't have to worry about writing and updating documentation every time a new edge case appears that would cause the function to throw an error. All of the behaviour of the function is encoded in the return type. Relatedly, the type serves as documentation now: "makeHttpRequest is an asynchronous function that can either succeed with ResponseData or fail with a Error."

... "But wait, how do I get the T value or E value that is wrapped inside of a Result<T, E>?"

Great question. Let me show you how. We're going to use a package I made [aptly] named neverthrow.

> npm install neverthrow

import { ok, err, Result } from 'neverthrow'

// we'll keep this simple
type ResponseBody = {}

interface ResponseData {
  statusCode: number
  responseBody?: ResponseBody
}

const makeHttpRequest = async (
  url: string
): Promise<Result<ResponseData, Error>> => {
  if (!isUrl(url)) {
    return err(new Error(
      'Invalid string passed into `makeHttpRequest`. Expected a valid URL.'
    ))
  }

  // ...
  // other business logic here
  // ...

  return ok({ ... }) // Ok(ResponseData)
}

So we're currently at the same place we were at with the last code snippet, except this time we're using the neverthrow package.

If you were to read through the neverthrow documentation you'd see that a Result has a .map method which takes the T value inside of a Result and converts it into anything you want.

Here's an example:

import { makeHttpRequest } from './http-api.ts'

const run = async () => {
  // unwrap the Promise
  // at this point
  // we have a Result<ResponseData, Error>
  const result = await makeHttpRequest('https://jsonplaceholder.typicode.com/todos/1')

  result.map(responseData => {
    console.log(responseData)
  })
}

run()

But wait, what if the result variable contains an E value? in other words, it's an Err instead of an Ok.

Well, again, the docs for neverthrow show you how to handle this situation too ... Just use mapErr!

import { makeHttpRequest } from './http-api.ts'

const run = async () => {
  // unwrap the Promise
  // at this point
  // we have a Result<ResponseData, Error>
  const result = await makeHttpRequest('https://jsonplaceholder.typicode.com/todos/1')

  result.mapErr(errorInstance => {
    console.log(errorInstance)
  })
}

run()

The most beautiful thing about Results is that they are chainable! Here's the above code in a more realistic example:

import { makeHttpRequest } from './http-api.ts'

const run = async () => {
  // unwrap the Promise
  // at this point
  // we have a Result<ResponseData, Error>
  const result = await makeHttpRequest('https://jsonplaceholder.typicode.com/todos/1')

  result
    .map(responseData => {
      // do something with the success value
    })
    .mapErr(errorInstance => {
      // do something with the failure value
    })
}

run()

There is a lot more you can do with a Result type (check out the docs), but maping is the most important part of the API.

Making Your Types More Intuitive

If you start using Result a lot in your return types, you might notice two things:

The meaning of the Results is not very clear
- Example: The Result of a databse query might be something like Promise<Result<T, DbError>> while the Result of a network call might be something like Promise<Result<T, NetworkError>>.
The types are really long and verbose. For example, above we had a Promise<Result<ResponseData, Error>> ... this is a somewhat intimidating type!

To solve both issues, you could leverage type aliases!

Here's an example. Instead of having a function return a generic Result with a DbError as the E type, why not alias this type to something much more intuitive?

type DbResult<T> = Result<T, DbError>

Now your function can just return Promise<DbResult<T>>. It's a lot more succinct!

Further, your type now encodes meaning. The above type says that there's something async going on that could fail and I know that it's dealing with the database. Neat!

Here's a real-world example from one of my projects:

handler: (req: Request, res: SessionManager) => DecodeResult<Promise<RouteResult<T>>>

So handler is a function that does a few things:

It does some decoding / deserializing of incoming data
It then does something async which generates a RouteResult with some data of type T

I know exactly what's going to happen by only reading the types. And the beautiful thing is that I won't ever get runtime errors because none of my code throws (and all the 3rd party libs I depend on have been wrapped to return Results as well).

Summary

Avoid using throw if you can.
- Users of your API are not required to catch (the compiler doesn't enforce it). This means you will eventually hit a runtime error ... it's just a matter of time
- Using throw forces you to maintain documentation which will eventually become stale
- If you do manage to maintain your documentation, a lot of people won't bother reading completely through the documentation, and won't realize that your functions throw in certain scenarios
Encode the potential for failure into your types using Results
- The types are self documenting, and they cannot become "stale"
- Users are given a friendly API that lets them deal with failable values in a safe way (using map and mapErr)
There's a fully-tested and type-checked npm package for this called neverthrow ... try it out!

My Over-Engineered Blogging Setup

Gio — Mon, 08 Oct 2018 17:48:24 +0000

Originally posted at gdelgado.ca

In this post I describe how I updated my Pelican blog setup.

A while ago, HTTP requests to my website stopped working for some reason. It was an error with my CDN (cloudflare). This, to me, was a sign from the gods that it was time to do something unnecessarily complex to fix the problem.

Seriously though; there were some things I wanted to fix for a long time that I hadn't gotten around to doing. The current CDN issue + a long weekend seemed like the right time to do this.

tldr:

Created a custom docker image with (sass + python and all the pelican dependancies)
Dockerized the whole project using the aforementioned docker image
Set up automated deploys (using CircleCI) to deploy the blog to Netlify

Dependency Hell

There were some things that were bugging me about the way Pelican blogs are set up for local development. The pelican docs suggest that you install Virtualenv. That worked fine until I decided I wanted a better developer experience with CSS. I switched over to SASS for my CSS, but that meant that I had to install ruby since it's a ruby gem.

Ok so now I have python, virtualenv and Ruby globally installed just so that I may run the blog locally. This complicates automated deploys as well since you have to install these dependancies to ensure building the blog succeeds.

Docker to the Rescue

I really do not like having dependancies globally installed. So I figured this would be a fun opportunity to play around with Docker.

I'll spare you all the intermediate versions of my docker setup and get to what you currently see in the repo.

There are three docker-related files:

Dockerfile-push
Dockerfile
docker-compose.yml

Dockerfile-push contains the instructions to create a reusable image (I reuse this image during the CI process. More on this later). This image lives at https://hub.docker.com/r/giorgio14/pelican/ ... The image is fairly large, I know. So what? Sue me.

Now that I have a base image to run pelican and all its dependencies, I can create a new image from it that runs a development server. That's where Dockerfile comes in. In it I:

Create an image FROM my custom giorgio14/pelican image
COPY the relevant files and directories into the image
Bind my OS's port 8080 onto the container's port 8080
run a make script which runs a live-reloading server on port 8080

There's just one problem. If I were to create a container from this image, the container creates copies of my files, but it doesn't link back to them. In other words; the container has its own copies that are completely unrelated to the ones on my OS. This means that in spite of having a live-reloading server that listens to file changes, no files will actually change within the container.

We need volumes. This is a mechanism that allows a container to have access to a host OS's files and directories. I use volumes so that I may edit anything inside the theme and content directories and have those changes propagate to the running docker container. Once the changes occur within the container, then the python server running inside the container can regenerate the blog and serve the updated files.

For this reason, I use the docker-compose.yml file. In it, there's a volumes key that describes which local directories I want to mount onto the running container.

Now that this is all done, I can simply run:

docker-compose up

And docker will:

Pull and generate the required images
Run make devserver PORT=8080 (which comes from the Dockerfile) inside the container

Now I have a schnazzy dev setup that does not need python, virtualenv, ruby or anything :) ... Only Docker!

Automated deploys

Now that I have a reusable docker image hosted on hub.docker.com, I can leverage it to make my CI steps super easy.

My goal was for my website to update on every new push to the master branch on my repo. I chose CircleCI for its brand, great documentation (took me less than a day to figure out how to use it), and free pricing 🙃.

I basically just followed the documentation to connect my github repo to circle and had the "hello world" pipeline running shortly thereafter.

Here are the relevant files associated with circle. It took me a few tries and tweaks to get the pipeline running as I wanted, so I created a bash script that allowed me to run pipelines without pushing commits to github. Here's the relevant doc from circle.

I'll mention some high-level points about the config.yml file. Note that I use the custom image I made above. No need to install ruby, do apt-get, or anything. It's all just ready to go with the pelican cli installed as well.

Since I want to deploy to netlify, I have to install their cli. I chose their current stable cli (repo here) instead of their node-based one since I was familiar with it already.

Then I generate a production build of the blog using make publish.

For the sake of future debugging I figured I would save the output of the make publish command in case something strange ever occurred. Circle's UI allows me to access the zipped directory if I ever need to.

Finally I deploy to netlify.

I've set up a custom domain on netlify by creating a CNAME record that points to the randomly-generated netlify URL. More info here on netlify custom domains.

Still to be figured out

I really want to have comments on my website, but I definitely don't want to use Disqus. When I get a chance I'll give Staticman a shot.

Staticman + Pelican Tutorial

... and that's it!

Zero dependancy Pub / Sub system with PostgreSQL

Gio — Sun, 27 May 2018 15:54:59 +0000

Photo by Chen Hu on Unsplash.

At Setter we have a 2nd generation API server that handles:

API requests coming in from internal software
API requests from our customer-facing iOS and Android applications
Webhook requests from 3rd party services

This is all quite typical stuff.

Communicating With Systems Outside Of Our Control

In the process of handling some of these requests, we have to communicate with 3rd party services.

One example being customer order approvals, in which we have to send the customer an email to confirm the order and provide a sort of feedback loop to the user.

So in this case, the flow looks like:

Receive API request from mobile app
Process API request (which will involve make some DB inserts / updates)
Dispatch API request to 3rd party email provider (in our case we use Postmark and we highly recommend it)

By making API requests directly from our system, we've now reduced the certainty of success, and introduced incomplete states. For example, Postmark (the email service provider we use) could be down for routine maintenance, and hence a request to their service could fail at unpredictable times. This introduces an incomplete state in that the customer will never receive an email to let them know that their order was indeed processed and acknowledged.

This sort of thing has happened a few times at our company.

Eliminating The Dependancy on 3rd Party Services

Currently we're undergoing an internal RFC processes to decide how we're going to decouple 3rd party services from the core of our system.

I took the lead on this particular RFC (although I've had lots of guidance from my colleagues while writing it), and in this post I discuss the bulk of it.

What I'm proposing at our company is that we leverage the technologies we already have (PostgreSQL & NodeJS) in order to not increase system complexity - as opposed to using a tool such as RabbitMQ (not to say that RabbitMQ is bad).

By using PostgreSQL's LISTEN / NOTIFY features, you have everything you need in order to have a high-performance, fault-taulerant pub / sub system.

I went ahead and created an example app that implements this system - feedback welcome!

https://github.com/gDelgado14/pg-pubsub

Here are the relevant parts of the example project (as found in the README.md):

Implementing the "Pub" in Pub / Sub

The migrations folder contains the schemas / triggers / and SQL functions necessary to implement the publishing aspect of the system.

More specifically,a db_events table is created which stores messages sent into the pub sub system. Further, there is a trigger made that executes a sql function on any insertion into the db_events.

Implementing the "Sub" in Pub / Sub

Inside src/services/client.ts, I use the pg module to:

Connect to the db
Listen to "pub_sub" events being invoked from within postgres (which I've defined in the migrations)
Invoke any asynchronous functions associated with the various events that can occur in the system.

Now you can subscribe to any event you want. You can define the events in your code. It really helps if you use a statically-typed language (which is why I implemented the example in TypeScript) so that your message payload is always consistent to the message the payload is associated to.

You can see some example channel / payload combinations inside src/services/client.ts. For example, if you publish an sms message, the payload going in and out of the pub / sub system will always be { msg: 'some string' }

Another awesome aspect of this system is that you can choose to run your subscription logic on a different machine / runtime / language. This is because it's postgres that's sending messages into the runtime. In my example I kept it simple and had both the publishing and subscribing happening in the same app, but it doesn't have to be that way if you don't want it to be!

Handling failure gracefully

One thing I haven't thought enough about is how to handle message processing failures.

Example: Say I publish an email message into the pub / sub system and a subscriber tries to process the message by, say, sending an API request to Postmark and Postmark is down. How should I best manage this?

I think implementing an exponential back-off retry might be the right approach.

Would love to hear your thoughts on this!

The Economics of JS

Gio — Tue, 03 Apr 2018 15:51:36 +0000

tldr;

The democratization of the web platform has brought about an influx of JS alternatives - some of which will eventually overtake JS as the de-facto tool for writing user interfaces on the web.

JavaScript Has Reached Its Zenith

It is easy to forget JS's early history. The language was built by one person in just a few days without the knowledge that the "internet browser" would eventually dominate the software landscape, thus making JS the default language for reaching millions of users. Economics, not JavaScript's inherent features, are what lead it to become so prominent. There was no alternative way to add interactivity and delight to web pages. JS had a monopoly on front-end languages.

Since its inception, JS has continued to evolve; pushing the limits of what could be accomplished while also improving the ergonomics of the language by leaps and bounds. We're spoiled in this day and age when dealing with asynchronicity, but we all remember a time when callback-hell was the norm. The barrier to creating complex apps has progressively declined.

But there are only so much changes one can make, and features one can add to a language, before any additions become only marginally beneficial. This is simply an economic law as real as the gravitational force.

I think the last game-changing update to JS was async / await. Everything after that has been nice, but not revolutionary. In essence; JS will only get marginally better, but will stay fundamentally as it is for the foreseeable future.

At its optimal state, JS is good to use, but it is not the tool to solve every conceivable task as some people are lead to believe.

The Democratization of the Web Platform

As JS's evolution slows down, the web platform is undergoing a democratization. You no longer need to use JavaScript to deliver a web application to your users. This was not the case at the outset of the web platform. The monopoly is no more, and the market for front-end programming languages is starting to resemble something closer to a free market; fueled by a large supply of alternative languages.

Technologies such as WebAssembly are opening the door to solving problems which were historically restricted to the domain of JavaScript - Languages that arguably deal much better with software complexity at a large scale.

Software Complexity and JS

As JS apps got more and more ambitious, the need to manage software complexity increased. Unit tests are no longer enough. Linting is no longer enough.

Out of this need, Flow and TypeScript emerged to help bring typesystems into JS, and hence help increase software correctness.

Herein lies a hint that we've reached the limits of JS's capabilities. We're forcing types on an untyped language. And these aren't toy projects - there's a lot of demand for this featureset; just look at the downloads per day for each of the respective npm packages.

I thank the heavens for TypeScript. It has made my day job a lot less stressful. But it's not without its rough edges:

// Detect dark theme var iframe = document.getElementById('tweet-974324130479718400-0'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=974324130479718400&theme=dark" }

The above tweet represents the TypeScript ecosystem fairly in my opinion. The sheer amount of outdated or simply improperly written types is astounding. As I mentioned in the twitter thread: I would have considered contributing to the DefinitelyTyped project but from the looks of it, I get the impression that it's a lost cause.

The prolific use of any in TypeScript is saddening. By using any you have thrown all type safety out the window. Now you're writing a Java-esque version of JS, that has some type safety but it's not guaranteed because of the any escape hatch.

I think this is partly because of how fast the JS ecosystem moves: For every typedef written, 3 new packages come out and one week later the aforementioned typedef becomes outdated due to a major version bump (I am only estimating, so please correct me if I am way off). With this sort of pace it's impossible to keep up. The rationale becomes, "I'll fix the types later, but for now I'll just say everything is an any."

Or how about this?

So not only are typedefs either just wrong and littered with any, but they're also outdated. Sometimes by several versions.

So here we are with a plethora of very large JS projects, and the current solution is to throw a type system on top of a dynamic language. This makes total sense for projects that are too large to undergo a total rewrite ... But what about all the smaller projects? Why not just use a whole other (better) language altogether?

A paradigm shift

Nowadays there is lots of talk about functional programming. Many have realized the perils of object oriented architectures and we are slowly seeing a shift toward functional programming and stateless architectures.

This shift isn't inherently bad for JS as it has FP features, however, most JS programmers have no idea how to code functionally.

Further, much like in typescript, if you allow for escape hatches, you will use escape hatches. That is to say; if you can use a for loop to get a feature out quicker than thinking ahead of time of a more functional (and longer lasting) approach, then you'll eventually succumb to the temptation. Again, this is economics at play: It's much easier to follow the path of least resistance, and writing code imperatively generally requires less consideration.

Moving away from JS

As mentioned above, the web platform's opening up to new languages is evidence of demand for better guarantees around software complexity.

Now that there are capable alternatives to writing web applications in languages other than JS, we'll start seeing growing usage of these languages for serious projects. Most notably ReasonML within Facebook.

These are the compile-to-JS languages that I'm aware of, and I'm sure there are many more which deserve a shot-out. The point being that there is clearly a trend here. Many people are unsatisfied with JS's ability to write complex software.

That's not to say you can't write complex software in JS. It's just way harder to do so.

These are the sorts of things you have to deal with when writing apps in JS:

No type system (and if you use Flow and TS, have fun dealing with their verbosity - not to mention the insidious usage of any)
Quirks around the language (don't use == or you'll implicitly coerce types! Don't use arguments as it's not actually an array! What does this mean in this context?)
Highly fragmented package ecosystem. There are many alternative packages to solving the same problem:
- "Oh you want to unit test? Well, just use mocha + chai + sinon. Or alternatively Jest. Or Ava. Or Tape. Or ..."
- Which one is the right one? Only an expert JS dev could tell you.
- Now you need a bundler (Webpack is the current standard - which is a glorified compiler)
- Are you using CommonJS or AMD modules?
- Are you transpiling your code?
- Which version of Node are you using?

JS is a bit of a catch 22: It's probably one of the easiest languages to get started with, however its simplicity means that it's also one of the toughest languages to master. The amount of discipline and skill required to build a healthy and maintainable codebase is testament to this.

The simplicity in getting started with JS defers complexity further down the time horizon. You'll shoot yourself in the foot sooner or later because there are basically no restrictions to what you can or cannot do in the language. And then you're left staring at a cesspool of code-smells with nothing to help you.

// Detect dark theme var iframe = document.getElementById('tweet-976836311943385088-802'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=976836311943385088&theme=dark" }

Hmm, maybe JS isn't conducive to maintainability in the first place?

Tell a novice JS dev, "write clean code" and let me know how that pans out. Conversely, writing in, say Elm or Rust is a lot saner. You have a compiler that helps you. It ensures that your code will run as you intend it to and it provides you feedback as you go. It's materially harder to write unclean code in many other languages relative to JS.

Do you want to refactor a large chunk of your JS codebase? I sure hope you have written enough unit tests and your ESLint config is there to catch other errors (so much so that you've essentially done what a compiler would have done for you out of the box).

NPM == jungle full unpredictable packages

You don't need to unit test your package / library before publishing it to npm.
You also don't have a compiler in JS to provide guarantees that the package you wrote isn't going to crash.

So the packages you download from npm are basically at your own peril. This is why you need a downloads counter in npm. The logic being, "If others are downloading this package, then surely it's safe to use". But obviously this is not always the case as edge cases often take a long time to surface.

This is in stark contrast to any of the package ecosystems in strictly typed languages (crates.io, Hackage, Elm-Package, etc...). You don't need a downloads counter because you know the package had to be compiled before even landing on the package ecosystem. Consider pre-upload compilation a sort of litmus test which sets a consistent expectation of quality across the board.

Another benefit with these other languages is that they are sustained by communities consisting of very skilled developers (the influx of devs coming from code bootcamps doesn't help alleviate the code quality problem plaguing npm). That's not to say that you don't have amazing devs in JS-land (you do), but the distribution of talent in JS has a massive standard deviation.

Equilibrium in the Web Platform

In the back-end ecosystem, there is complete freedom to choose whichever language best solves the task at hand. There is no monopoly unlike the front-end world. But I've mentioned already, this is no longer the case, and over time, we will see the advent of incredibly large and complex front-end applications built without JS (or with JS as a minor component to the app: Much like the concept of ports in Elm)

This is known as equilibrium in economics and it's about time we reached it in the front end development ecosystem.

DEV Community: Gio

Pitfalls When Adding Turborepo To Your Project

Issues faced with our original monorepo setup

Starting point and stack choice

Researching & The Design Document Phase

Setting up the npm workspace

Run npm install to create the symlinks inside node_modules

npm v7 is needed or the IDE/compiler can't resolve modules

New commands to install dependencies and run scripts

Separating dependencies

Import resolution after transitioning

Failing tests

CircleCI

Saving turbo cache artifacts between builds

Saving dependencies between builds

⚠️ The netlify CLI cannot use same URL prefixes as automatic branch deployments

Only deploying the individual apps which turbo rebuilt

Conclusion

Did you enjoy this post? We're hiring!

Get Paid To Work On Open Source

What Is neverthrow or a "Result Type"?!

Challenge: Create a `pad` function without using loops!

Submit your solutions down below! 👇👇👇

Type Constraints In TypeScript

Type Constraints To The Rescue

Conclusion

Shameless Plug

Implement a type-safe version of Node's Promisify in 7 lines of TypeScript

Exercises:

Introducing Elm Antd

Project Goals

Don't Know Elm? No Problem!

"But Elm is not a very popular language, why would I want to learn Elm?"

Chaining Failable Tasks

The Problem

First Attempt At A Solution

Using Result Chains! 🔗

Type-Safe Error Handling In TypeScript

Making Your Types More Intuitive

Summary

My Over-Engineered Blogging Setup

Dependency Hell

Docker to the Rescue

Automated deploys

Still to be figured out

Zero dependancy Pub / Sub system with PostgreSQL

Communicating With Systems Outside Of Our Control

Eliminating The Dependancy on 3rd Party Services

Handling failure gracefully

The Economics of JS

tldr;

JavaScript Has Reached Its Zenith

The Democratization of the Web Platform

Software Complexity and JS

A paradigm shift

Moving away from JS

NPM == jungle full unpredictable packages

Equilibrium in the Web Platform

Run `npm install` to create the symlinks inside `node_modules`

What Is `neverthrow` or a "`Result` Type"?!