Simple KV storage on top of indexedDB

If your client-side application ever needs to persist a larger portion of data, it's no longer suitable to put it inside a localStorage entry. The first thing that comes into mind is to use indexedDB. But then you have to manage transactions, versioning, etc.

Sometimes all you need may be a straightforward key-value store, that hides those intricacies inside, like:

const kv = await openKV("kv");

await kv.set(key, value);

const data = await kv.get(key);

What is stopping you from building one?
Let's get to work!

Opening up

Working with indexedDB API is not especially pleasant. It's taken straight from the primal age of JavaScript. If you're looking for a more civilized API to digest - check out idb - it enhances the indexedDB API with promises and shortcuts for common operations

But in this post, we're not afraid of the tears and pain of the past

Opening up (the database)

First, we need to open a new connection request. Then attach handlers for success and error events. Everything is enclosed within a compact Promise:

const STORE_NAME = "store";
const openKVDatabase = (dbName: string) =>
  new Promise<IDBDatabase>((resolve, reject) => {
    const request = indexedDB.open(dbName);

    request.onsuccess = () => {
      resolve(request.result);
    };
    request.onerror = () => {
      reject("indexedDB request error");
    };
    request.onupgradeneeded = () => {
      request.result.createObjectStore(STORE_NAME, { keyPath: "key" });
    };
  });

upgradeneeded event will be fired once the database is created (or when its version gets updated). Inside this handler, we can create our one and only store - the KV store. I've put STORE_NAME in a constant as we'll need to use it in multiple places later on

First blood methods

Let's scaffold a basic shape for get, set, and delete methods. They will correspond to indexDB objectStore operations consequently: get, put, and delete

export async function openKV<T = unknown>(dbName: string) {
  const db = await openKVDatabase(dbName);

  const openStore = () => {
    return db.transaction(STORE_NAME, "readwrite").objectStore(STORE_NAME);
  };

  return {
    async get(key: string) {},

    async set(key: string, value: T) {},

    async delete(key: string) {},
  };
}

openStore helper function opens up a new transaction and returns the handle for our KV store

Requests as promised

One more thing needs to be done before implementing the methods. objectStore methods return IDBRequest object. This object achieves the same goal as a promise (it's like a goofy version of it). Let's create a utility that will map them into promises - so we can await them:

function idbRequestToPromise<T>(request: IDBRequest<T>) {
  return new Promise<T>((resolve, reject) => {
    request.onsuccess = () => resolve(request.result);
    request.onerror = () => reject(request.error);
  });
}

Methods

  async get(key: string): Promise<T | undefined> {
      const pair: Pair | undefined = await idbRequestToPromise(
        openStore().get(key)
      );

      return pair?.value as T | undefined;
    },

    async set(key: string, value: T) {
      const pair: Pair = { key, value };
      await idbRequestToPromise(openStore().put(pair));
    },

    delete(key: string) {
      return idbRequestToPromise(openStore().delete(key));
    },

The Pair type used here is just:

type Pair<T = unknown> = { key: string; value: T };

You got to pump it up

As you probably noticed opening a new transaction every time we perform a single key value operation is suboptimal. Consider this snippet:

for (const item of arr) {
    kv.set(item.id, item);
}

To handle an array of 1000 items, we need to open 1000 transactions. If the operations are executed in a single task (triggered synchronously) as in the example above, grouping them into a single transaction (aka batching) could improve efficiency. Let's verify if this assumption holds true

Batching

To implement batching, we need to update the openStore function a little bit

const db = await openKVDatabase(dbName);
// Create 'store' variable to share it between calls
let store: IDBObjectStore | null;

const openStore = () => {
  if (!store) {
    store = db.transaction(STORE_NAME, "readwrite").objectStore(STORE_NAME);

    queueMicrotask(() => {
      // Finish the transaction after the current task ends
      store?.transaction?.commit();
      store = null;
    });
  }

  return store;
};

queueMicrotask allows running code after the current task has been executed (microtasks are run between regular tasks). Learn more here.

Testing

I used tinybench to prepare a basic test case like so:

Promise.all(arr.map((v) => kv.set(v, v)));

Where arr is a 1000-element array of strings

Results

Unsurprisingly there is a small improvement over the 1000 transaction version:

| 1000 transactions | batching     |
| ----------------- | ------------ |
| 7 (ops/sec)       | 32 (ops/sec) |

Transactions

Okay, so when I run queries synchronously, they will be put into a single transaction. But what about the original reason for inventing database transactions? It was to group queries together into one to ensure consistency. Check out this code:

async function inc() {
    await kv.set("x", (await kv.get("x")) + 1);
}

It would only make sense if both set and get operations would form a single transaction

Async / await tracking

Unfortunately, APIs like AsyncLocalStorage that are available in server runtimes including Node.js, Deno, and Bun, that would allow us to track async context are not (yet) available in the browsers. However, we can hook into async / await by leveraging custom Thenables and microtasks scheduling...

If you are interested in learning more about tracking asynchronous contexts, you can check out proposal-async-context - the official ECMAScript proposal that addresses this particular issue

...and then

Userland Thenables can be awaited just like Promises. The key difference if it comes to Thenables is that “then” method is always executed when used in async / await code. This allows us to intercept the async execution of the code and inject hooks before and after the continuation of the async / await block. Here's my attempt at doing that:

export class Thenable<T> {

  constructor(
    private promise: Promise<T>,
    private hooks?: {
      before?: () => void;
      after?: () => void;
    }
  ) {}

  then<U>(onFulfilled: (value: T) => U): Thenable<U> {
    return new Thenable(
      this.promise.then((value) => {
        this.hooks?.before?.();
        const result = onFulfilled(value);
        queueMicrotask(() => this.hooks?.after?.());
        return result;
      }),
      this.hooks
    );
  }
}

Notice how after hook is pushed into the microtask queue. It's because calling onFulfilled will push continuation to the queue itself - this way after hook is called after the continuation microtask

Sharing current transaction

Taking up before and after hooks, we can now make the current transaction accessible from within adjacent queries. Here's the type of the transaction object that will be shared:

type Transaction = {
  store: IDBObjectStore;
  committed?: boolean;
  lastQueried?: boolean;
};

commited and lastQueried flags will be used to implement auto-committing of the transaction. All queries will now be wrapped in a query function to handle sharing.

  const query = <R>(
    fn: (transaction: Transaction<T>) => Promise<R>
  ): Thenable<R> => {
    const transaction = (currentTransaction ??= {
      store: db.transaction(STORE_NAME, "readwrite").objectStore(STORE_NAME),
    });

    // Clear current transaction after current task
    queueMicrotask(() => {
      currentTransaction = null;
    });

    const result = fn(transaction);

    return new Thenable(result, {
      before() {
        // Resume transaction before the continuation
        currentTransaction = transaction;
      },
      after() {
        currentTransaction = null;
      },

    });

  };

And the example of usage:

    set(key: string, value: T) {
      // Wrap handler with query
      return query(async ({ store }) => {
        const pair = { key, value };
        await idbRequestToPromise(store.put(pair));
      });

    },

Auto-commiting

After the series of queries, it would be great to handle commiting automatically. The lastQueried flag will indicate if the queries were executed last microtasks:

  const query = <R>(
    fn: (transaction: Transaction<T>) => Promise<R>
  ): Thenable<R> => {
    const transaction: Transaction<T> = (currentTransaction ??= {
      store: db.transaction(STORE_NAME, "readwrite").objectStore(STORE_NAME),
    });


    // Running `query` will reset the flag
    transaction.lastQueried = true;

    queueMicrotask(() => {
      currentTransaction = null;
    });

    const result = fn(transaction);

    return new Thenable(result, {

      before() {
        currentTransaction = transaction;
        transaction.lastQueried = false;
      },

      after() {
        // If there were no new queries during the last microtask
        if (!transaction.lastQueried && !transaction.committed) {
          transaction.store.transaction.commit();
          transaction.committed = true;
        }
        currentTransaction = null;
      },
    });
  };

Admiring the results

Look at that and think:

async function inc() {
    await kv.set("x", (await kv.get("x")) + 1);
}

The function above now forms a single ACID transaction!

Summing Up

The indexedDB is not the easiest API to work with. It's not the fastest horse in the stable either. It's probably a good idea to use some of the popular wrapper libraries like idb or Dexie.js. That will simplify and streamline the process of working with it. There is also idbkeyval - super-simple key-value store (but without automatic batching and transactions 🙊). Still, implementing your own wrapper may be great fun and will definitely help you understand better how it works