Using promises in IndexedDB

This is part 3 of my IndexedDB tutorial series. You can find Part 1] here and Part 2 here. The code for this tutorial on GitHub is here.

In my last IndexedDB tutorial, we refactored the IndexedDB functions on a sticky note store to take in callbacks so that we could use them in automated tests. We could use those callbacks to guarantee that our IndexedDB actions run in consecutive order. But with that callback style our Jest test, doing just four IndexedDB actions, ended up looking like this:

test('we can store and retrieve sticky notes', function(done) {
  setupDB('FORWARD_TEST', function() {
    addStickyNote('SLOTHS', function() {
      addStickyNote('RULE!', function() {
        // Now that our sticky notes are both added, we retrieve them from
        // IndexedDB and check that we got them back in the right order.
        getNotes(reverseOrder=false, function(notes) {
          expect(notes).toHaveLength(2);
          expect(notes[0].text).toBe('SLOTHS');
          expect(notes[1].text).toBe('RULE!');
          done();
        });
      });
    });
  });
});

It does the job, but as you're writing more intricate tests with even more IndexedDB actions, the callback pyramid we've got will get even bigger, which means more cognitive load on people reading and maintaining your code.

It'd be great if instead of having each IndexedDB function be the last function's callback, we could have code that looks more like the actions are happening in a sequence:

test('we can store and retrieve sticky notes', function(done) {
  setupDB('FORWARD_TEST');
  addStickyNote('SLOTHS');
  addStickyNote('RULE!');
  let notes = getNotes(reverseOrder=false);

  // Check that we got back the sticky notes we exepcted

  done();
});

One way we can have code that works similar to that is by having our IndexedDB functions chain together using promises instead of callbacks. Although there are ways to clean up callback-based code, I personally find that promise-based IndexedDB code is easier to reason about, which is why I use it in my own IndexedDB functions. So in this tutorial, I'll show how to promisify callback-based IndexedDB code.

This tutorial assumes you have some familiarity with promises in JavaScript. You can read about promises at this tutorial from Mozilla Developer Network.

Overview of promises

Like callbacks, promises are a way of handling asynchronous actions, telling JavaScript what you want your code to do after an action completes, without blocking the JavaScript runtime's thread.

With promises, instead of passing a callback into an asynchronous function to run after it completes, like you would downloading data in jQuery:

$.get('sloth.txt', function(data) {
  console.log(data);
});
console.log(`This code still runs while we're waiting for our sloth`);

You would make a Promise object and pass your callback into its .then method like in the fetch API:

I mean this fetch API!

fetch('sloth.txt').then(function(res) {
  console.log(res.text());
})

The callback in a fetch API promise's .then method is run once our download completes, just like when the callback you pass into $.get() is run. So it's a similar pattern, but one advantage of promises is that you can chain asynchronous functions returning promises together, like this:

fetch('/my-profile-data').
  then(function(res) {
    // Get the URL of the user's profile picture based on what's in the data we
    // got with our first fetch call, and then run fetch on that URL. We
    // return a promise for when that fetch completes, so this promise can be
    // chained with the callback below
    let profilePicURL = res.json()["profilePicURL"]
    return fetch(profilePicURL);
  }).then(function(res) {
    console.log(res.text());
  });

That means in a test we would be able to make our code look like this, making it much more clear that our functions are running in a sequence:

setupDB().
  then(() => addStickyNote('SLOTHS').
  then(() => addStickyNote('RULE!')).
  then(() => getNotes(reverseOrder=false)).
  then((notes) => { /* Here we run assertions on the notes we get back */ });

So in order to get our IndexedDB functions to use promises, the idea is that we need to have each of those functions return a Promise object so that the next IndexedDB action can be run in the promise's .then. If we do that, we will be able to chain together all of our IndexedDB actions.

Promisifying setupDB

First step is to start with promisifying setupDB. Inside the callback version of setupDB, we have all our code for setting up our database and creating our object store. The code looked like this:

function setupDB(namespace, callback) {
  if (namespace != dbNamespace) {
    db = null;
  }
  dbNamespace = namespace;

  // If setupDB has already been run and the database was set up, no need to
  // open the database again; just run our callback and return!
  if (db) {
    callback();
    return;
  }

  let dbName = namespace == '' ? 'myDatabase' : 'myDatabase_' + namespace;
  let dbReq = indexedDB.open(dbName, 2);

  // Fires when the version of the database goes up, or the database is created
  // for the first time
  dbReq.onupgradeneeded = function(event) {
    db = event.target.result;

    // Create an object store named notes, or retrieve it if it already exists.
    // Object stores in databases are where data are stored.
    let notes;
    if (!db.objectStoreNames.contains('notes')) {
      notes = db.createObjectStore('notes', {autoIncrement: true});
    } else {
      notes = dbReq.transaction.objectStore('notes');
    }
  }

  // Fires once the database is opened (and onupgradeneeded completes, if
  // onupgradeneeded was called)
  dbReq.onsuccess = function(event) {
    // Set the db variable to our database so we can use it!
    db = event.target.result;
    callback();
  }

  // Fires when we can't open the database
  dbReq.onerror = function(event) {
    alert('error opening database ' + event.target.errorCode);
  }
}

As we saw in the last tutorial, the "end" of this action is when either dbReq.onsuccess fires, running the callback function, or its onerror fires, making an alert popup. The idea is that once we get an event triggering the onsuccess handler, that means the db variable is set, and our database is created.

To convert this callback-based IndexedDB function to a promise-based function, we need to follow this pattern, which you can find the full code changes for in Commit 1:

Step 1: Wrap the whole body of setupDB in an anonymous function that we pass to the Promise constructor.

function setupDB(namespace) {
  return Promise((resolve, reject) => {
    if (namespace != dbNamespace) {
      db = null;
    }
    dbNamespace = namespace;

    // ...
  });
}

This way, all of the main code will still run, but now setupDB returns a Promise object, rather than returning nothing and running the callback when it completes.

Step 2: Replace all calls to our request's callback with calls to resolve(). This would be two places: the callback in the if statement for when the db variable is already set:

  if (db) {
-    callback();
+    resolve();
    return;
  }

and the callback for dbReq.onsuccess, which runs once the database is open.

  dbReq.onsuccess = function(event) {
    // Set the db variable to our database so we can use it!
    db = event.target.result;
-    callback();
+    resolve();
  }

The resolve and reject parameters on the function we passed to the promise constructor are used to indicate when the asynchronous action is done. For example,

setupDB().then(callback);

means that if our IndexedDB action succeeds, then we resolve and then we run the callback to do our next action in the promise's .then.

Step 3: Replace the code handling our IndexedDB request/transaction's onerror and onabort methods with a call to reject():

    dbReq.onerror = function(event) {
-      alert('error opening database ' + 'event.target.errorCode');
+      reject(`error opening database ${event.target.errorCode}`);
    }

This means that if we get an error running our database request, then the promise rejects and the callback we pass into the promise's catch method will run. For example in the code:

setupDB().then(callback).catch((err) => { alert(err); })

setupDB has its then callback run if our IndexedDB transaction succeeds, or it runs its catch callback if it fails.

Step 4: Since we changed the function signature of setupDB, now anywhere that was calling setupDB(callback) will need to be changed to setupDB.then(callback).

In our codebase, this means in index.html, when we run setupDB and then get and display our notes, we would run:

    <script type="text/javascript">
-      setupDB(getAndDisplayNotes);
+      setupDB('').then(getAndDisplayNotes);
    </script>

Now we have a promisified setupDB, so if we wanted to set up the database and then put a sticky note in, we would run code like:

setupDB('').then(() => addStickyNote('SLOTHS')

Pretty good, but in our tests, we added more than one sticky note to our database. That means in our tests we'd want to chain multiple calls to addStickyNote in a promise chain. So to do that, addStickyNote will need to return a promise after that.

Promisifying addStickyNote

Converting our addStickyNote function to a promise functino follows the same pattern as we had in setupDB; we wrap the body of the function in the Promise constructor to make it return a promise, we replace our calls to the callback with calls to resolve, and we replace our error handing with a call to reject.

For addStickyNote, you can see the whole change in Commit 2, but the part of the code we're most interested in is below:

    tx.oncomplete = resolve;
    tx.onerror = function(event) {
      reject(`error storing note ${event.target.errorCode}`);
    }

As you can see, our transaction's oncomplete callback is set to just our resolve function, and our onerror callback now just rejects with the error we got.

Although it looks kind of funny, tx.oncomplete = resolve is completely valid JavaScript. resolve is a function, and when the transaction to add a sticky note to the database completes, tx.oncomplete runs, so that means that resolve runs.

Now that we have addStickyNote returning a promise, we could chain addStickyNote calls together like this:

setupDB().
  then(() => addStickyNote('SLOTHS')).
  then(() => addStickyNote('RULE!'));

This promise chain reads reads "setup our database, then when it's ready add the sticky note 'SLOTHS', and finally once that's ready, add the sticky note 'RULE!'". Each function in the then callback is a function that returns a Promise, and that's why each addStickyNote can be chained with another method.

Now, with our addStickyNote method ready to chain, in page.js, where we have the user interface function submitNote, we would chain it with getAndDisplayNotes like this.

function submitNote() {
  let message = document.getElementById('newmessage');
-  addStickyNote(message.value, getAndDisplayNotes);
+  addStickyNote(message.value).then(getAndDisplayNotes);
  message.value = '';
}

In the submitNote function, addStickyNote starts adding our message to the database, and when its promise resolves, we run getAndDisplayNotes to retrieve our sticky notes and display them. While our asynchronous code is running, we set the content of our web app's textarea to blank.

⚠️ One subtle pitfall I ran into with this, though, was trying to chain the calls together like this:

setupDB().
  then(addStickyNote('SLOTHS')).
  then(addStickyNote('RULE!'));

I thought this would be an even more slick way to call this function, and it looks like it'd work since addStickyNote returns a promise. While that function does indeed return a promise, the value of addStickyNote('SLOTHS') is not a function, it's the Promise object addStickyNote will have already returned.

This means that in setupDB().then(addStickyNote('SLOTHS!')), each call to addStickyNote has to run so it can evaluate to a value, so the function starts running while our db variable is still undefined.

By contrast, () => addStickyNote('SLOTHS') is a function returning a promise, rather than a promise itself, so if we pass our anonymous function into the promise's .then, that function won't start until setupDB's promise resolves.

Promisifying getNotes

We have just one function left to promisify: getNotes, and we're using the same technique once more, except this time there is one small difference.

In setupDB and addStickyNote, we weren't retrieving any data, so there was nothing we need to pass on to the next function; we could just run resolve() to let our next action run in our promise's then callback. However in getNotes, we are retrieving some data, which is our sticky notes, and we want to use our notes in the then callback.

To do this (you can see all of the changes for this in Commit 3), just like before, we run resolve where we previously ran callback. So our onsuccess callback will now look like this:

    let allNotes = [];
    req.onsuccess = function(event) {
      let cursor = event.target.result;

      if (cursor != null) {
        // If the cursor isn't null, we got an IndexedDB item. Add it to the
        // note array and have the cursor continue!
        allNotes.push(cursor.value);
        cursor.continue();
      } else {
        // If we have a null cursor, it means we've gotten all the items in
        // the store, so resolve with those notes!
-        callback(allNotes);
+        resolve(allNotes);
      }
    }

Like in our other functions, our request's onerror callback now just calls reject instead of calling alert.

req.onerror = function(event) {
-   alert('error in cursor request ' + event.target.errorCode);
+   reject(`error in cursor request ${event.target.errorCode}`);
}

This means that with our changes, getNotes now returns a Promise like our other IndexedDB functions. However, this is not a promise that resolves with no data, this is a promise that resolves with an array of sticky notes!

That means if our call to getNotes has a then callback, instead of giving then a function that takes in nothing, we can give then a function that takes in an array of sticky notes. Which is what we would do in the body of getAndDisplayNotes!

function getAndDisplayNotes() {
-   getNotes(reverseOrder, displayNotes);
+   getNotes(reverseOrder).then((notes) => { displayNotes(notes) });
}

Now when we run getNotes, it resolves with our list of sticky notes, so those are passed into our callback, which runs displayNotes with them.

Sweet! All of our functions directly touching IndexedDB now return promises, so next stop: Our test coverage!

Promisifying our IndexedDB tests

As we've seen in page.js and index.html, when we want to run promise-based IndexedDB actions consecutively, we have each action run in the last action's then. So now, instead of our callback pyramid in our Jest test, we would have our tests run a promise chain like this:

test('we can store and retrieve sticky notes!', function() {
  return setupDB('FORWARD_TEST').
    then(() => addStickyNote('SLOTHS')).
    then(() => addStickyNote('RULE!')).
    then(() => getNotes(reverseOrder=false)).
    then((notes) => {
      // Assertions on the notes we retrieved
      expect(notes).toHaveLength(2);
      expect(notes[0].text).toBe('SLOTHS');
      expect(notes[1].text).toBe('RULE!');
    });
});

On the first line of the function, we set up our database. setupDB returns a promise, and when it resolves, it then adds the sticky note SLOTHS to the database. then once that promise resolves, we add the sticky note RULE!. And then, in the following action in the promise chain, we run getNotes, knowing that both of our addStickyNote IndexedDB actions had completed.

Finally, when getNotes resolves, the then callback takes in the two sticky notes we retrieved, so we run our assertions on them. If they all succeed, that means our whole test passes, but if one of them fails, then the test fails.

As you can see, with this promise chaining, we don't need to keep pushing each IndexedDB action a couple spaces to the right in our text editor. Instead we're able to write it out more like a sequence of actions to run in order.

One thing that's important to notice about how we changed this function, by the way, is that the signature of the function we pass into test has changed slightly:

- test('we can store and retrieve sticky notes!', function(done) {
+ test('we can store and retrieve sticky notes!', function() {

Remember that the function we're passing in works with asynchronous code, so we need to have a way for our code to tell Jest that we finished running the test. So the done parameter solved that by being a function that we call after we finish running our assertions, indicating that the test is done. But why don't we need that done parameter with our new promise chain style? Let's take a closer look at the first line in our test:

test('we can store and retrieve sticky notes!', function() {
  return setupDB('FORWARD_TEST').

In our test, we don't just run our promise chain, we return it! And in Jest, if your test's callback takes in a function that returns a promise, then Jest knows the testing is complete when that promise resolves! So it's kind of like if Jest was saying

runOurTest().then(runOurNextTest)

Since the test for retrieving the sticky notes in reverse order looks the same, I won't show the promisified version of that test, but you can see it in Commit 4. If you run the test, you will see that:

The tests pass! Now let's do one more change to our tests, using the newer async/await keywords!

Running async/await like the cool kids!

async/await gives one more way of handling asynchronous code. Instead of running each action in a sequence by using .then callbacks like:

doAsynchronousAction().
  then(doAnotherAsynchronousAction).
  then(finallyRunThisCode);

async/await lets us write our actions run one after another, as if the functions weren't asynchronous at all!

await doAsynchronousAction();
await doAnotherAsynchronousAction();
finallyRunThisCode();

No code after an await in the function will run until the awaited promise completes. To me, I find this a much more natural way of writing sequences of asynchronous actions, since we aren't trying to run any of them simultaneously.

So with async/await introduced in commit 5, our first function would look like:

test('we can store and retrieve sticky notes!', function() {
  await setupDB('FORWARD_TEST');
  await addStickyNote('SLOTHS');
  await addStickyNote('RULE!');

  let notes = await getNotes(reverseOrder=false);
  expect(notes).toHaveLength(2);
  expect(notes[0].text).toBe('SLOTHS');
  expect(notes[1].text).toBe('RULE!');
});

We await setupDB completing, then we start adding the sticky note SLOTHS to our database, awaiting its completion, and when that completes, we await adding the sticky note RULE to our database.

It gets more interesting with retreiving our sticky notes with getNotes. Since getNotes returns a promise that resolves with some data, we can assign the data getNotes resolves with to a variable using await.

let notes = await getNotes(reverseOrder=false);

This line means that after we retrieve our list of sticky notes, those notes getNotes resolved with are now in the notes variable. So that means below that await, we can run our assertions on the sticky notes.

The await keyword has now abstracted away the idea that setupDB, addStickyNote, getNotes, and our assertions are supposed to be callbacks. And if we run this test, we will get:

Unfortunately, we have an error; await keywords can't be used inside regular functions. The reason why is because in a regular function, waiting for each action to complete would block the JavaScript runtime's single thread. Luckily, getting this to work is just a one-line fix:

- test('we can store and retrieve sticky notes!', function() {
+ test('we can store and retrieve sticky notes!', async function() {

Now instead of passing test a regular function, we are giving an async function. And since async functions implicitly return a promise, that means we still don't need a done parameter in our tests.

Run this test again, and you will get:

Passing tests! Now you've seen how to convert callback-based IndexdedDB functions to promise-based functions, how to chain them together, and how to write tests that use promise chains. I hope this has helped you with designing the IndexedDB code for your web app. Until next time,

STAY SLOTHFUL!

The sloth picture was taken by Eric Kilby, and it is licensed under CC-BY-SA 2.0.