Actually, callbacks are fine

Asad Saeeduddin — Wed, 28 Nov 2018 15:42:19 +0000

...Wat?

This is an article about how the most well known villain in the JS universe isn't really evil, just misunderstood.

Going to hell in a callbasket

I'm not going to dig too deep into the background of the term "callback hell"; instead I'll just point you to this nice article that explains the problem and some typical solutions. If you're unfamilar with the term, please go read that article; I'll wait.

Ok. So we're going to copy and paste the problematic code from the article, and then we're going to see how we might solve the problem without using promises and async/await:

const verifyUser = function(username, password, callback) {
  dataBase.verifyUser(username, password, (error, userInfo) => {
    if (error) {
      callback(error);
    } else {
      dataBase.getRoles(username, (error, roles) => {
        if (error) {
          callback(error);
        } else {
          dataBase.logAccess(username, error => {
            if (error) {
              callback(error);
            } else {
              callback(null, userInfo, roles);
            }
          });
        }
      });
    }
  });
};

Flattening the pyramid

If we look at the code, we notice that every time we perform an asynchronous operation, we have to pass a callback to receive the result. Because we're defining all the result-receiving callbacks inline as anonymous functions, we end up with this huge pyramid of doom.

As a first step, let's perform a simple refactoring where we just copy and paste each anonymous callback function into a separate variable, introducing curried arguments to explicitly pass around variables that were being captured from the surrounding scope:

const verifyUser = (username, password, callback) =>
  dataBase.verifyUser(username, password, f(username, callback));

const f = (username, callback) => (error, userInfo) => {
  if (error) {
    callback(error);
  } else {
    dataBase.getRoles(username, g(username, userInfo, callback));
  }
};

const g = (username, userInfo, callback) => (error, roles) => {
  if (error) {
    callback(error);
  } else {
    dataBase.logAccess(username, h(userInfo, roles, callback));
  }
};

const h = (userInfo, roles, callback) => (error, _) => {
  if (error) {
    callback(error);
  } else {
    callback(null, userInfo, roles);
  }
};

If nothing else it's certainly a little flatter, but we now have some new problems with this code:

The if (error) { ... } else { ... } business is being repeated everywhere
Our variable names for our intermediate expressions are meaningless
verifyUser, f, g and h are all tightly coupled to each other, since they reference each other directly

Seeing the pattern

Before we deal with any of those issues though, let's note some similarities between these expressions.

All of these functions accept some data and a callback parameter. f, g and h additionally accept a pair of arguments (error, something), of which only one will be a non-null/undefined value. If error is non-null, the functions immediately feed error to callback and terminate. Otherwise, they use something to do some more work, causing callback to eventually be fed a different error, or null and some result value.

Keeping these commonalities in mind, we'll embark on a program of refactoring our intermediate expressions so they look increasingly similar.

Cosmetic changes

I find if statements really verbose, so we'll take a moment now to replace all these if statements with ternary expressions. Since the return values are all being discarded anyway, this doesn't cause any change in the behavior of the code.

I'm also going to reduce the visual noise by shortening the repetitive error and callback variables to e and cb respectively:

const verifyUser = (username, password, cb) =>
  dataBase.verifyUser(username, password, f(username, cb));

const f = (username, cb) => (e, userInfo) =>
  e ? cb(e) : dataBase.getRoles(username, g(username, userInfo, cb));

const g = (username, userInfo, cb) => (e, roles) =>
  e ? cb(e) : dataBase.logAccess(username, h(userInfo, roles, cb));

const h = (userInfo, roles, cb) => (e, _) =>
  e ? cb(e) : cb(null, userInfo, roles);

Currying aggressively

Because we're about to start performing some serious gymnastics with function parameters, I'm going to take this opportunity to curry all the function arguments that can be curried. This introduces uniformity and facilitates further refactoring.

We can't easily curry the functions which accept a pair of arguments (e, xyz), since the underlying dataBase API (which is opaque to us) requires the callback to simultaneously accept a possible error and a possible result. But all other occurrences of multi-parameter functions can (and will) be eliminated by currying.

We'll start with the dataBase methods:

// Curried wrapper around the `dataBase` API
const DB = {
  verifyUser: username => password => cb =>
    dataBase.verifyUser(username, password, cb),

  getRoles: username => cb =>
    dataBase.getRoles(username, cb),

  logAccess: username => cb =>
    dataBase.logAccess(username, cb)
}

Now we will replace all usages of dataBase with wrapped operations from DB, and curry any remaining multi-parameter functions. Additionally, we'll replace the cb(null, userInfo, roles) in h with cb(null, { userInfo, roles }), so that a callback always receives precisely two arguments: a possible error and a possible result.

const verifyUser = username => password => cb =>
  DB.verifyUser(username)(password)(f(username)(cb));

const f = username => cb => (e, userInfo) =>
  e ? cb(e) : DB.getRoles(username)(g(username)(userInfo)(cb));

const g = username => userInfo => cb => (e, roles) =>
  e ? cb(e) : DB.logAccess(username)(h(userInfo)(roles)(cb));

const h = userInfo => roles => cb => (e, _) =>
  e ? cb(e) : cb(null, { userInfo, roles });

Turning it inside out

Let's do some more refactoring. For reasons that will become clear momentarily, we're going to pull all the error checking code "outwards" one level. Instead of each step doing its own error checking, we'll use an anonymous function that receives the error e or result v of the current step, and forwards the result and callback to the next step if there are no problems:

const verifyUser = username => password => cb =>
  DB.verifyUser(username)(password)((e, v) =>
    e ? cb(e) : f(username)(cb)(v)
  );

const f = username => cb => userInfo =>
  DB.getRoles(username)((e, v) =>
    e ? cb(e) : g(username)(userInfo)(cb)(v)
  );

const g = username => userInfo => cb => roles =>
  DB.logAccess(username)((e, _) =>
    e ? cb(e) : h(userInfo)(roles)(cb)
  );

const h = userInfo => roles => cb => cb(null, { userInfo, roles });

Note how the error handling has entirely disappeared from our final function: h. It simply accepts a couple of parameters, builds up some composite result from them, and immediately turns around and feeds the result into a given callback. Let's rewrite h to show this more clearly:

const h = userInfo => roles => {
  const result = { userInfo, roles };

  return cb => cb(null, result);
}

The cb parameter is now being passed in various positions, so for consistency, we'll move around the arguments so that all the data goes first and the callback goes last:

const verifyUser = username => password => cb =>
  DB.verifyUser(username)(password)((e, v) =>
    e ? cb(e) : f(username)(v)(cb)
  );

const f = username => userInfo => cb =>
  DB.getRoles(username)((e, v) =>
    e ? cb(e) : g(username)(userInfo)(v)(cb)
  );

const g = username => userInfo => roles => cb =>
  DB.logAccess(username)((e, _) =>
    e ? cb(e) : h(userInfo)(roles)(cb)
  );

const h = userInfo => roles => {
  const result = { userInfo, roles };

  return cb => cb(null, result);
}

verifyUser and f now look almost identical. They both:

Receive some data and a callback
Perform some asynchronous operation
Receive an error or a value
If the result is an error, immediately pass it to the callback
Otherwise, pass the successful result and callback into some further step (<next step>(v)(cb))

g is very similar, but there is a twist. Instead of receiving a v argument and passing it on to the next step if there are no problems, it unconditionally discards any successful result and passes only the callback to the next step.

To smooth out this wrinkle, we will rewrite g so that it imitates the other two functions and passes on its (undefined) result. To deal with the unwanted result, we will introduce a dummy argument to the "next step", so that it discards whatever was passed:

const g = username => userInfo => roles => cb =>
  DB.logAccess(username)((e, v) =>
    e ? cb(e) : (_ => h(userInfo)(roles))(v)(cb) // the "next step" discards the result
  );

Now it follows the same formula as verifyUser and f. For clarity, let's explicitly copy the asynchronous operation and "next step" of each function into local variables:

const verifyUser = username => password => {
  const task = DB.verifyUser(username)(password);
  const next = f(username);

  return cb => task((e, v) => e ? cb(e) : next(v)(cb));
}

const f = username => userInfo => {
  const task = DB.getRoles(username);
  const next = g(username)(userInfo);

  return cb => task((e, v) => e ? cb(e) : next(v)(cb));
}


const g = username => userInfo => roles => {
  const task = DB.logAccess(username);
  const next = _ => h(userInfo)(roles);

  return cb => task((e, v) => e ? cb(e) : next(v)(cb));
}

const h = userInfo => roles => {
  const result = { userInfo, roles };

  return cb => cb(null, result);
}

Do you see the pattern?

Factoring out the pattern

By this point it is hopefully obvious that there is something very repetitive is happening. It looks like someone has copied and pasted code for handling errors and threading around callbacks into every function. Of course, this is deliberate; we have refactored our way to a unified pattern, so that we may copy and paste the repetition out.

Now, in one fell swoop, we can move all the error handling and callback thread business into a pair of helper functions:

const after = task => next =>
  cb => task((e, v) => e ? cb(e) : next(v)(cb));

const succeed = v =>
  cb => cb(null, v);

Our steps turn into:

const verifyUser = username => password =>
  after
    (DB.verifyUser(username)(password))
    (f(username));

const f = username => userInfo =>
  after
    (DB.getRoles(username))
    (g(username)(userInfo));

const g = username => userInfo => roles =>
  after
    (DB.logAccess(username))
    (_ => h(userInfo)(roles));

const h = userInfo => roles =>
  succeed({ userInfo, roles });

The error handling and callback threading has disappeared!

It's a good idea to pause here for a second. Try to inline the definitions of after and succeed into these new expressions, to convince yourself that they are equivalent to the ones we refactored away.

Ok, so we're getting warmer! f, g and h don't seem to be doing much of anything anymore though...

Pruning dead weight

...so let's get rid of them! All we have to do is to work our way backwards from h and inline each function into the definition that references it:

// Inline h into g
const g = username => userInfo => roles =>
  after(DB.logAccess(username))(_ =>
    succeed({ userInfo, roles })
  );

// Inline g into f
const f = username => userInfo =>
  after(DB.getRoles(username))(roles =>
    after(DB.logAccess(username))(_ =>
      succeed({ userInfo, roles })
    )
  );

// Inline f into verifyUser
const verifyUser = username => password =>
  after(DB.verifyUser(username)(password))(userInfo =>
    after(DB.getRoles(username))(roles =>
      after(DB.logAccess(username))(_ =>
        succeed({ userInfo, roles })
      )
    )
  );

We can use referential transparency to introduce some temporary variables and make it a little more readable:

const verifyUser = username => password => {
  const auth  = DB.verifyUser(username)(password);
  const roles = DB.getRoles(username);
  const log   = DB.logAccess(username);

  return after(auth)(u =>
    after(roles)(r =>
      after(log)(_ =>
        succeed({ userInfo: u, roles: r })
      )
    )
  );
};

And there you have it! This is quite concise, doesn't repeat any error checking, and is roughly analogous to the Promise version from the article we linked earlier. You invoke verifyUser like so:

const main = verifyUser("someusername")("somepassword");
main((e, o) => (e ? console.error(e) : console.log(o)));

Final code

// Tools for sequencing callback APIs
const after = task => next =>
  cb => task((e, v) => e ? cb(e) : next(v)(cb));

const succeed = v =>
  cb => cb(null, v);

// Curried wrapper around the `dataBase` API
const DB = {
  verifyUser: username => password => cb =>
    dataBase.verifyUser(username, password, cb),

  getRoles: username => cb =>
    dataBase.getRoles(username, cb),

  logAccess: username => cb =>
    dataBase.logAccess(username, cb)
}

// Our implementation
const verifyUser = username => password => {
  const auth  = DB.verifyUser(username)(password);
  const roles = DB.getRoles(username);
  const log   = DB.logAccess(username);

  return after(auth)(u =>
    after(roles)(r =>
      after(log)(_ =>
        succeed({ userInfo: u, roles: r })
      )
    )
  );
};

The M-word

Are we done? Well, some of us might still find the code in verifyUser a little too triangular. There are ways to fix this, but in order to explain how I first have to fess up to something.

I didn't independently discover the definitions of after and succeed in the process of refactoring this code. I actually had the definitions up front, since I copied them from a Haskell library where they go by the name of >>= and pure. Together, these two functions constitute the definition of the "continuation monad".

Why is this relevant? Well, it turns out that there are many handy ways to sequence together monadic computations that don't suffer from the pyramid-of-doom effect.

To illustrate, let's start by formatting the definition of verifyUser a little bit differently:

const verifyUser = username => password => {
  const auth  = DB.verifyUser(username)(password);
  const roles = DB.getRoles(username);
  const log   = DB.logAccess(username);

  return
    after(auth) (u =>
    after(roles)(r =>
    after(log)  (_ =>
    succeed({ userInfo: u, roles: r }))));
};

If you squint and ignore the parentheses, you might notice the similarity between this definition and the following Haskell function:

-- In Haskell, function application does not require parentheses,
-- and binary functions may be applied infix
verifyUser :: Username -> Password -> IO (UserInfo, Roles)
verifyUser username password =
  let
    auth  = DB.verifyUser username password
    roles = DB.getRoles username
    log   = DB.logAccess username
  in
    auth  >>= \u ->
    roles >>= \r ->
    log   >>= \_ ->
    pure (u, r)

This pattern of using >>= and functions to introduce new variables captured from the steps of a monadic computation is so common that there is special syntax sugar for it, called "do-notation". Here is the same computation in Haskell written with do-notation:

verifyUser' :: Username -> Password -> IO (UserInfo, Roles)
verifyUser' username password =
  let
    auth  = DB.verifyUser username password
    roles = DB.getRoles username
    log   = DB.logAccess username
  in
    do
      u <- auth
      r <- roles
      _ <- log
      pure (u, r)

Although we do not have general purpose do-notation in JS (perhaps we should!), there are various ways to simulate it. A detailed explanation of monads and do-notation is beyond the scope of this article, but for illustrative purposes, here is one way to write verifyUser in JS with a simulated do-notation library:

const { mdo } = require("@masaeedu/do");

// `Cont` is our implementation of the continuation monad
const Cont = monad({ pure: succeed, bind: after });

const verifyUser = username => password => {
  const auth  = DB.verifyUser(username)(password);
  const roles = DB.getRoles(username);
  const log   = DB.logAccess(username);

  return mdo(Cont)(({ u, r }) => [
    [u, () => auth ],
    [r, () => roles],
        () => log   ,
        () => Cont.pure({ userInfo: u, roles: r })
  ]);
};

This is well and good, but it is also worth noting that some monadic computations have a "fixed" structure, i.e. they might not utilize the result of previous steps to decide what to do next. Since such computations have no real need for explicitly binding over and naming the results of intermediate steps, they can be built up more conveniently by "traversing" a fixed container of the steps, which will eventually produce a corresponding container of results.

Luckily for us, our example is just such a "fixed structure" computation, in that each step is independent of the results of previous steps. This means it can also be written in the following, more concise ways:

const verifyUser = username => password => {
  const auth  = DB.verifyUser(username)(password);
  const roles = DB.getRoles(username);
  const log   = DB.logAccess(username);

  // Applicative lifting
  const f = u => r => _ => ({ userInfo: u, roles: r });
  return Cont.lift(f)([auth, roles, log]);
};

const verifyUser = username => password => {
  const auth  = DB.verifyUser(username)(password);
  const roles = DB.getRoles(username);
  const log   = DB.logAccess(username);

  // Traverse a dictionary of continuations into a continuation of a dictionary
  return Obj.sequence(Cont)({
    userInfo: auth,
    roles: roles,
    _: log
  })
};

A detailed analysis of all the ways to build up monadic and applicative computations is beyond the scope of this post, but suffice it to say that there are a number of powerful, elegant tools for synthesizing computations in an arbitrary monad. By recognizing that our callback-based model of asynchronicity is monadic (specifically, that it corresponds to the continuation monad) and witnessing the relevant monad operations, we can apply these general purpose tools to async programming.

Conclusion

Okay, we made it! What are the takeaways? I hope I've managed to persuade you of the following:

Referentially transparent refactoring is a powerful technique for eliminating repetition and discovering useful patterns
"Callback hell" is not a problem innate to callbacks, but to a particular calling discipline for callback-based APIs. With the right approach, callback-based APIs can be concise and elegant to work with
The concept of a "monad" in a programming context is not (merely) academic mumbo jumbo, but is a useful tool for recognizing and exploiting patterns that arise naturally in everyday programming

Further work

I have deliberately avoided introducing type signatures or concepts like monads until the very end of the post in order to keep things approachable. Perhaps in a future post we can re-derive this abstraction with the monad and monad-transformer concepts foremost in our minds, and with special attention to the types and laws.

Acknowledgements

Many thanks to @jlavelle, @mvaldesdeleon and @gabejohnson for providing feedback and suggestions on this post.

DEV Community: Asad Saeeduddin