Eugene Ghanizadeh

Posted on Dec 4, 2020

From Callback Hell to Callback Heaven

#javascript #typescript #node #async

Remember the time when a lot of JavaScript code looked like this?

router.put('/some-url', (req, res) => {
  fs.readFile(filePath(req), (err, file) => {
    if (err) res.status(500).send()
    else {
      parse(file, (err, parsed) => {
        if (err) res.status(500).send()
        else db.insert(parsed, err => {
          if (err) res.status(500).send()
          else res.status(200).send()
        })
      })
    }
  })
})

For those fortunate enough to not remember those days, this was called a callback hell, for obvious reasons. Fortunately, we have moved on, and these days equivalent code most likely looks something like this:

router.put('/some-url', async (req, res) => {
  try {
    const file = await fs.readFile(filePath(req));
    const value = await parse(file);
    await db.insert(value);
    response.status(200).send();
  } catch {
    response.status(500).send();
  }
})

Of course now that we have async/await and Promises, it is easy to blame the callback hell era on lack of syntactic features of JS at the time and move on. But I do think there is value in reflecting back, analyzing the core issue, how it was solved, and what's to learn from it all.

The Issue

Lets look back at the overall structure of the hellish example above:

doX(args, (err, res) => {
  if (err) { ... }
  else {
    doY(args, (err, res) => {
      if (err) { ... }
      ...
    })
  }
})

A glaring issue here is that most of whats on the screen is dedicated to not-really-important stuff:

doX(args /*, (err, res) => {
  if (err) { ... }
  else {*/
    doY(args /*, (err, res) => {
      if (err) { ... } */
      ...
    /*}*/)
  /*}*/
/*}*/)

For comparison, this would be the equivalent structure for the modern not-hellish version:

/* try { */
  /*await*/ doX(args)
  /*await*/ doY(args)
  ...
/*} catch { }*/

The commented bits in both versions are indicative of the same things: doX() and doY() are async functions, and also there might be some errors. In the hellish version though, you need to spend much more space for these side notes, which results in a far less readable code.

👉 Note that we could trim the boilerplate and restructure the code into a more readable format without additional syntax too. Historically speaking, that IS what happened, in the form of Promise libraries (which then got standardized and a bit of more love with some syntactic support):

doX(args)
.then(() => doY(args))
.then(() => ...)
.catch(() => { ... })

doX(args)
/*.then(() =>*/doY(args)/*)*/
/*.then(() =>*/.../*)*/
/*.catch(() => { ... })*/

An important difference between this code and the hellish code is that in the hellish code, important stuff and boilerplate stuff are extremely intertwined, while with promise libraries, they are neatly separated, making the code easier to read even in cases where amount of boilerplate is almost the same:

// without promises:
doX(args/*, (err, res) => { ... }*/)

// with promises:
doX(args)/*.then(() => { ... })*/

// with async/await:
/*await*/ doX(args)

Promises also provide other important features that help with async programming ergonomics, most importantly:

Promises are automatically flattened when chained.
Promises are shared.

However, I think these properties, although beneficial, aren't as important as the aforementioned separation. To illustrate that, lets create an experimental promise library that just does the separation and nothing else, and see how it fares.

The Experiment

So initially, we started with functions looking like this:

doX(args, (err, res) => {...})

The callback here is the main boilerplate (and namesake of our hell), so easiest separation is to take it out of argument list of doX(), and put it in a deferred function instead:

doX(args)((err, res) => {...})

☝️ This is basically a change in how doX is implemented, from this:

function doX(args, callback) {
  // do stuff
  // maybe do more
  callback(undefined, 42)
}

To this:

function doX(args) {
  // do stuff
  return callback => {
    // maybe do more
    callback(undefined, 42)
  }
}

In other words, we just changed the convention from:

accept a callback as last argument

to:

return a function that accepts a callback as argument

Our separation convention seems to not have helped much on its own, as we still have the same amount of boilerplate. However, it did open the door for simplistic utilities that help us take away the boilerplate. To see that, let me first introduce the pipe() utility:

function pipe(...cbs) {
  let res = cbs[0];
  for (let i = 1; i < cbs.length; i++) res = cbs[i](res);
  return res;
}

Simply put, this:

pipe(a, b, c, d)

equals to this:

let x = a
x = b(x)
x = c(x)
x = d(x)

In the not so distant future, pipe() might even get incorporated into JavaScript itself, which would look like this:

a |> b |> c |> d

Anyways, the pipe() operator allows us to neatly transform the function returned by the (new convention) doX() (which remember, is a function accepting a standard callback), without having to write callbacks manually. For example, I could create a then() utility as follows:

export function then(f) {
  return src => {
    src((err, res) => {
      if (!err) f(res)
    })

    return src
  }
}

and with these utilities, my async code would transform from this:

doX(args)((err, res) => { ... })

to this:

pipe(
  doX(args),
  then(() => { ... })
)

or better yet (with pipeline operator incorporated):

doX(args) |> then(() => { ... })

which looks a lot like a standard promise library:

doX(args).then(() => { ... })

I could also create a simple catch() utility:

function catch(f) {
  return src => {
    src((err) => {
      if (err) f(err)
    })

    return src
  }
}

Which would give me async codes like this:

doX(args)
|> then(() => doY(args))
|> then(() => ...)
|> catch(() => { ... })

doX(args)
/*|> then(() =>*/ doY(args) /*)*/
/*|> then(() =>*/ ... /*)*/
/*|> catch(() => { ... })*/

Which is as succinct as promise libraries, with almost no effort. Better yet, this method gives us extensibility as well, as we are not bound to a set Promise object and can create / use a much wider range of utility functions:

function map(f) {
  return src => cb => src((err, res) => {
    if (err) cb(err, undefined)
    else cb(undefined, f(res))
  })
}

function delay(n) {
  return src => cb => src((err, res) => {
    if (err) cb(err, undefined)
    else setTimeout(() => cb(undefined, res), n)
  })
}

and start getting a bit wild:

doX(args)
|> then(() => doY(args))
|> map(yRes => yRes * 2)
|> delay(200)
|> then(console.log)

Real-life Examples

Ok so it seems like a simple change of convention allowed us to create utilities and libraries that provide the same convenience provided by promise libraries (and almost similar to async/await syntax). To get a better perspective, let's look at real-life examples. For this purpose (and mostly out of curiosity), I have created an online playground with an implementation of our experimental lib.

First off, let's take a look at our original example, which looked like this in its most hellish version:

router.put('/some-url', (req, res) => {
  fs.readFile(filePath(req), (err, file) => {
    if (err) res.status(500).send()
    else {
      parse(file, (err, parsed) => {
        if (err) res.status(500).send()
        else db.insert(parsed, err => {
          if (err) res.status(500).send()
          else res.status(200).send()
        })
      })
    }
  })
})

This is how the modern JavaScript version looks like:

router.put('/some-url', async (req, res) => {
  try {
    const file = await fs.readFile(filePath(req));
    const value = await parse(file);
    await db.insert(value);
    response.status(200).send();
  } catch {
    response.status(500).send();
  }
})

And this is how our new callback convention code looks like:

router.put('/some-url', (req, res) => {
  fs.readFile(filePath(req))
  |> map(parse)
  |> flatten
  |> map(db.insert)
  |> flatten
  |> then(() => res.status(200).send())
  |> catch(() => res.status(500).send())
})

The convention gets us pretty close to the convenience of async/await. There is a little nuance though: see the flatten utility used twice in the middle? That is because unlike promises, our callbacks are not flattened while chaining. We assumed that parse() is also async, i.e. it also returns a promise-ish. map(parse) then maps the result of readFile() to a new promise-ish, which should be flattened to resolved values before being passed to db.insert(). In the async/await code, this is done by the await keyword before parse(), and here we need to do it with the flatten utility.

P.s., the flatten() utility is also pretty simplistic in nature:

function flatten(src) {
  return cb => src((err, res) => {
    if (err) cb(err, undefined)
    else res((err, res) => {
      if (err) cb(err, undefined)
      else cb(undefined, res)
    })
  })
}

Let's also take a look at another example: here, we want to fetch some Pokémon info from PokéAPI and log its abilities:

fetch('https://pokeapi.co/api/v2/pokemon/ditto')
|> map(res => res.json())
|> flatten
|> then(res => console.log(res.abilities))

async(() => {
  let res = await fetch('https://pokeapi.co/api/v2/pokemon/ditto')
  res = await res.json()
  console.log(res.abilities)
})()

Conclusion

So to recap, these seem to have been the main issues resulting in callback hells:

Lots of boilerplate code
Boilerplate code severely intertwined with important code

As per our little experiment, addressing the second issue in the simplest manner (just separating boilerplate code and important code with no other change) was pretty key: it allowed us to bundle boilerplate code into small utility functions and reduce the ratio of boilerplate code and important code, making it (almost) as convenience as a heavy-handed solution such as adding new syntax to the language itself.

This notion is particularly important: you might have ugly implementation details and boilerplate that you cannot get rid of, but you can always bundle it together and separate it from actual important code, and doing this even in the simplest manner opens the door for turning a hellish situation into a heavenly one.

It is also notable that the same methodology is applicable to other, similar problems we are facing today. While we have (mostly) solved the issues of asynchronous functions, newer constructs such as asynchronous streams (which are like async functions but with many, possibly infinite outputs instead of one) keep creeping into our toolbox and demand similar problem solving.

P.s. the name callback heaven actually comes from the callbag specification, which is like our new callback convention but for streams instead of promises. If you enjoyed this post, be sure to check it out as well.

Top comments (6)

Bogdan Zaharia • Dec 15 '20

Really nice idea. I guess that, since the map |> flatten pattern is pretty common, a flatMap would come in handy.
Also, somehow related is this article series. The link is to the last post, which links to the first two.

Eugene Ghanizadeh • Dec 15 '20

yes, I think thats what actual libraries like RxJS typically have, and for that particular reason (though I do like the explicitness of flatten operator in callbags for example).