DEV Community: Markus Ast

Store SQLite in Cloudflare Durable Objects

Markus Ast — Wed, 26 Jan 2022 09:04:27 +0000

This is my journey of creating a proof-of-concept of persisting an SQLite database in a Cloudflare Durable Object. It is not meant as a tutorial, but rather as something I myself can get back to it, to revisit what and why I did certain things.

tl;dr: A custom SQLite virtual file system and some WASM/WASI compilation magic allow to run SQLite on a Cloudflare Worker and persist it into a Durable Object. POC source can be found at github.com/rkusa/do-sqlite.

The Plan

Everything started with me wanting a way of saving an SQLite database close to my Cloudflare workers (just Workers in the following).

Where to store SQLite on Cloudflare?

While Workers can easily interact with anything that speaks HTTP, I'd like to be able to use something serverless, and preferably something that also lives on the edge, or at least close to it. Cloudflare offers two persistent storage products for its workers right now. There are Workers KV (KV in the following) and Durable Objects (DO in the following). Soon there is also R2 Storage, a S3 compatible object storage, which I am not going to cover here as I don't have access to it yet.

Both Workers KV and Durable objects are key-value stores. The main difference between both is their consistency guarantees. Workers KV is eventually-consistent while Durable Objects is strongly-consistent. All other differences are mostly a result of those different consistencies.

So which one to choose?

First, let's look at their limitations. KV can store values of up to 25 MiB with no total limit for paid accounts (KV Limits). DO has a total limit of 10 GB per account (can be raised by contacting Cloudflare) and a value limit of 128 KiB (DO Limits).

Neither 25 MiB nor 128 KiB is enough as an upper limit for an SQL database. Fortunately, this is not a deal-breaker, as we can split up the SQLite database file into blocks and persist one block as one value.

The SQLite file conveniently consists of one or multiple pages of the same size. The size of a page is a power of two between 512 and 65536 inclusive (see SQLite File Format). The default page size is 4096 bytes. One single read/write call of SQLite never crosses page boundaries. This makes splitting the file into blocks particularly easy.

The maximum page size fits both KV and DO, so neither of them is ruled out yet. Let's think about their consistencies next. If the database file is split across multiple key-value pairs, we have to make sure that all pages (all key-value pairs) are always up to date when accessing the database. This is not guaranteed to be the case with an eventually-consistent storage, so KV is not an option here.

With DO being strongly-consistent, we have our SQLite store. It of course comes with drawbacks compared to KV that are necessary to achieve the higher consistency guarantees. In a nutshell, one DO only exists once. It is unique. Calls to it are routed to its point-of-presence that owns it. This leads to higher latencies compared to KV. The latency depends on where a request hits the Cloudflare network and where the DO is located.

The DO introduction blog post also confirmed that DO is a reasonable choice for this use-case:

That said, every big distributed database – whether it be relational, document, graph, etc. – is, at some low level, composed of "chunks" or "shards" that store one piece of the overall data. The job of a distributed database is to coordinate between chunks.

We see a future of edge databases that store each "chunk" as a Durable Object. By doing so, it will be possible to build databases that operate entirely at the edge, fully distributed with no regions or home location. These databases need not be built by us; anyone can potentially build them on top of Durable Objects. Durable Objects are only the first step in the edge storage journey.

How to store the SQLite database file in a DO?

This is fortunately not as difficult as it might sound. SQLite already offers pluggable file systems - aka. virtual file systems (VFS). So I just have to write my own virtual file system, which stores each page of the database file into a separate DO key-value, and plug it into SQLite.

How to run SQLite on Workers?

SQLite is written in C, while workers is based on V8 isolates, so it mainly runs JavaScript. Fortunately, it also supports running WASM through initialising and calling WASM modules via JavaScript. Emscripten can be used to build WASM from C, but I'd rather use it through Rust (using rusqlite), so this is what I focus on right away. Workers can also be written entirely in Rust using worker-rs.

The initial plan

With that all out of the way, the initial plan is to write a worker using worker-rs. Use rusqlite to create and use the SQLite database. Save each page of the database file into a separate DO key-value and persisted into the DO using a custom virtual file system.

Compile SQLite to WASM for target `wasm-unknown-unknown`

With worker-rs this should just be a matter of using rusqlite and building the project via wrangler dev or wrangler publish. With a plain Rust project, it would be a matter of building via cargo build --target wasm32-unknown-unknown. I am using the bundled feature for the build:

rusqlite = { version = "0.26", features = ["bundled"] }

Does it work? Nope.

It doesn't like the usage of <stdio.h>.

warning: sqlite3/sqlite3.c:13699:10: fatal error: 'stdio.h' file not found
warning: #include <stdio.h>
warning:          ^~~~~~~~~
warning: 1 error generated.

Fortunately, there is an open PR to improve the WASM compatibility: rusqlite/rusqlite#1010.

Let's try using the fork instead:

rusqlite = { git = "https://github.com/trevyn/rusqlite.git", branch = "wasm32-unknown-unknown", features = ["bundled"] }

Which brings us to:

...
  CC = Some("/usr/local/opt/llvm/bin/clang")
...
error occurred: Failed to find tool. Is `clang` installed?

I am on a MacBook and /usr/local/opt/llvm/bin/clang indeed doesn't exist on my machine. My solution was to first install llvm via Homebrew:

brew install llvm

And then instruct Cargo via environment variables where my clang is located:

CC=/opt/homebrew/opt/llvm/bin/clang \
  cargo build --release --target wasm32-unknown-unknown

Still nope:

error occurred: Failed to find tool. Is `/usr/local/opt/llvm/bin/llvm-ar` installed?

Let's also tell it where llvm-ar is located:

CC=/opt/homebrew/opt/llvm/bin/clang \
AR=/opt/homebrew/opt/llvm/bin/llvm-ar \
  cargo build --release --target wasm32-unknown-unknown

Aaannnnd ... it builds!

Build the Virtual File System

Building a VFS for SQLite is a matter of subclassing three objects: sqlite3_vfs, sqlite3_io_methods, and sqlite3_file. A demo implementation that uses POSIX functions is provided, too.

With that, I had enough information to build sqlite-vfs, a Rust library that wraps all the necessary interactions with the C SQLite library and exposes a much simpler Rust trait. I'll not go into the details of this library here.

Implementing this trait is enough for a simple VFS. I only implemented as much as was necessary for my use-case (e.g. I skipped memory-mapped files). The trait is:

pub trait File: Read + Seek + Write {
    fn file_size(&self) -> Result<u64, std::io::Error>;
}

pub trait Vfs {
    type File: File;

    fn open(&self, path: &Path, flags: OpenFlags) -> Result<Self::File, std::io::Error>;
    fn delete(&self, path: &Path) -> Result<(), std::io::Error>;
    fn exists(&self, _path: &Path) -> Result<bool, std::io::Error>;

    fn access(&self, _path: &Path, _write: bool) -> Result<bool, std::io::Error> {
        Ok(true)
    }
}

The implementation of the trait for saving each page as a separate object works as follows.

The PagesVfs is a struct that is generic over the page size.

pub struct PagesVfs<const PAGE_SIZE: usize>;

It uses a Pages struct as the File associate type, which in return is a map of page index to Page, the count of pages in total, and the current offset for reading and writing. Each Page is simply its data and a dirty flag to keep track of whether it changed. The structs use the rather new Rust feature of const generics.

impl<const PAGE_SIZE: usize> Vfs for PagesVfs<PAGE_SIZE> {
    type File = Pages<PAGE_SIZE>;

    // ...
}

struct Page<const PAGE_SIZE: usize> {
    data: [u8; PAGE_SIZE],
    dirty: bool,
}

pub struct Pages<const PAGE_SIZE: usize> {
    count: usize,
    offset: usize,
    blocks: HashMap<u32, Page<PAGE_SIZE>>,
}

The implementation of the trait is rather straight forward. I'll spare you all the boring details. Just a summary of how reading and writing works.

For reading, I wrote a method to get the current page based on the current offset into the database file. This method is used in the std::io::Read implementation. It reduces everything down to a get_page method that takes an index and returns the page data for it. With a fixed block/page size, it is quite easy to get the current page (self.offset / PAGE_SIZE), as well as the offset into it (self.offset % PAGE_SIZE).

impl<const PAGE_SIZE: usize> Pages<PAGE_SIZE> {
    fn current(&mut self) -> Result<&mut Page<PAGE_SIZE>, std::io::Error> {
        let index = self.offset / PAGE_SIZE;

        if let Entry::Vacant(entry) = self.pages.entry(index as u32) {
            let data = Self::get_page(index as u32);
            entry.insert(Page {
                data: data.unwrap_or_else(|| [0; PAGE_SIZE]),
                dirty: false,
            });
        }

        Ok(self.pages.get_mut(&(index as u32)).unwrap())
    }

    pub fn get_page(ix: u32) -> Option<[u8; PAGE_SIZE]> {
        // TODO: retrieve page
    }
}

impl<const PAGE_SIZE: usize> Read for Pages<PAGE_SIZE> {
    fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
        let offset = self.offset % PAGE_SIZE;
        let page = self.current()?;
        let n = (&page.data[offset..]).read(buf)?;
        self.offset += n;
        Ok(n)
    }
}

For writing (the std::io::Write implementation), I only update the in memory state of the page and mark it as changed. Actually persisting the changes happens as part of the flush implementation. It iterates through all loaded pages and persists them by calling put_page.

impl<const PAGE_SIZE: usize> Write for Pages<PAGE_SIZE> {
    fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
        let offset = self.offset % PAGE_SIZE;
        let page = self.current()?;
        let n = (&mut page.data[offset..]).write(buf)?;
        page.dirty = true;
        self.offset += n;

        let count = (self.offset / PAGE_SIZE) + 1;
        if count > self.count {
            self.count = count;
        }

        Ok(n)
    }

    fn flush(&mut self) -> std::io::Result<()> {
        for (index, page) in &mut self.pages {
            if page.dirty {
                Self::put_page(*index, &page.data);
                page.dirty = false;
            }
        }
        Ok(())
    }
}

impl<const PAGE_SIZE: usize> Pages<PAGE_SIZE> {
    fn put_page(ix: u32, data: &[u8; PAGE_SIZE]) {
        // TODO
    }
}

The whole code can be found here: github.com/rkusa/wasm-sqlite/blob/main/wasm/src/vfs.rs. By reducing all down to the two methods get_page and put_page (to which I get back later in the post), the VFS ended up being a generic VFS for when you want to persist SQLite pages as separate files/objects. Using it anywhere else than a DO would just be a matter of providing a different implementation to those two methods.

Use the VFS in a DO

Using the VFS in a DO is – in theory – as simple as interacting with the DO's Transactional storage API inside of the get_page and put_page methods in the previous code block.

This, however, doesn't work. The VFS is synchronous while calls to the storage are asynchronous. For example, the get method has the following signature:

get<T>(key: string, options?: Object): Promise<T | undefined>

The worker runs, same as the browser, as a single thread using cooperative multitasking. It is thus impossible to await an asynchronous method in the synchronous context of the VFS. One might think one could just have an endless loop polling the promise (or future in the Rust context) until it is completed. But this is impossible in a single-threaded environment, since the endless loop would block forever and never switch to actually running any other async task. It would require at least two threads to do that (and it would be a bad idea even then).

Fortunately, the wasm-opt tool has an option to asyncify methods in a WASM. It, therefore, rewrites the WASM and allows to pause and resume the execution. This is exactly what I need to call async functions from within the sync context of the VFS.

It has one drawback though. I need control over how the WASM module is initialised so that I can use the Asyncify wrapper and provide the corresponding async imports. This is something I don't easily have control over using worker-rs, since worker-rs takes care of generating all the necessary JavaScript glue code for me.

So one step back. worker-rs cannot be used. At least for now – I am sure there is a way to make it work. But the easier way forward is to write the worker and DO in JavaScript and write a WASM module that only takes care of querying the SQLite database.

Target: `wasm32-unknown-unknown` vs `wasm32-wasi`

Before diving into the asyncification, I have to take full control over the WASM initialisation and thus its imports and exports.

While I could theoretically still build for wasm32-unknown-unknown, this target has a major drawback. A lot of libraries automatically use wasm-bindgen as soon as you build for it.

I want control over the module initialisation, which means that I am writing the glue code for JavaScript myself. This doesn't work if my dependencies assume wasm-bindgen and generate an endless amount of __wbindgen imports and exports. Let's inspect the build for wasm32-unknown-unknown.

wasmer inspect target/wasm32-unknown-unknown/release/do_sqlite.wasm

Excerpt of the output:

Imports:
  Functions:
    "__wbindgen_placeholder__"."__wbindgen_describe": [I32] -> []
    "__wbindgen_placeholder__"."__wbg_getRandomValues_98117e9a7e993920": [I32, I32] -> []
    "__wbindgen_placeholder__"."__wbg_randomFillSync_64cc7d048f228ca8": [I32, I32, I32] -> []
    "__wbindgen_placeholder__"."__wbg_process_2f24d6544ea7b200": [I32] -> [I32]
    "__wbindgen_placeholder__"."__wbindgen_is_object": [I32] -> [I32]
    "__wbindgen_placeholder__"."__wbg_versions_6164651e75405d4a": [I32] -> [I32]
    ...
  Memories:
  Tables:
  Globals:
Exports:
  Functions:
    "__wbindgen_describe___wbg_crypto_98fc271021c7d2ad": [] -> []
    "__wbindgen_describe___wbg_getRandomValues_98117e9a7e993920": [] -> []
    "__wbindgen_describe___wbg_modulerequire_3440a4bcf44437db": [] -> []
    "__wbindgen_describe___wbg_randomFillSync_64cc7d048f228ca8": [] -> []
    "__wbindgen_describe___wbg_msCrypto_a2cdb043d2bfe57f": [] -> []
    ...

I had to trim down the output a lot. There are 716 imports/exports in total that have the __wbindgen prefix.

So wasm32-unknown-unknown simply isn't going to work. I need kind of a neutral WASM target. Let's see what WASM targets we can choose from:

% rustup target list | grep wasm
wasm32-unknown-emscripten (installed)
wasm32-unknown-unknown (installed)
wasm32-wasi (installed)

The emscripten target also assumes JavaScript glue code from my experience, so wasm32-wasi is the only one left - so I am going with that.

Build SQLite for WASI for `wasm32-wasi`

If you are wondering, WASI is a modular system interface for WebAssembly. Head over to wasi.dev for a more detailed introduction.

There is some work required to get rusqlite to build for wasm32-wasi. Let's walk through all errors and their solutions.

cargo build --target wasm32-wasi

Similar to trying to build for wasm32-unknown-unknown, it complains about not finding stdio.h.

cargo:warning=sqlite3/sqlite3.c:13699:10: fatal error: 'stdio.h' file not found
cargo:warning=#include <stdio.h>
cargo:warning=         ^~~~~~~~~
cargo:warning=1 error generated.

The previously mentioned PR for wasm32-unknown-unknown compatibility solved this by including libc .c files from OpenBSD. My go to solution is different though. I prefer to build using the wasi-sdk (a WASI-enabled WebAssembly C/C++ toolchain).

At first we need to build the wasi-sdk. Here is a shell script I am using for that in various projects (you might have to install cmake and/or ninja: brew install cmake ninja):

#!/usr/bin/env sh
set -e
cd wasm
git clone --branch wasi-sdk-14 https://github.com/WebAssembly/wasi-sdk.git
cd wasi-sdk
git submodule update --init --recursive
NINJA_FLAGS=-v make package
tar -xzf dist/wasi-sdk-*.tar.gz -C dist

This clones and builds the wasi-sdk into ./wasm/wasi-sdk. The build toolchain can then be found at ./wasm/wasi-sdk/dist/wasi-sdk-14.0/bin, and the WASI sysroot at ./wasm/wasi-sdk/dist/wasi-sdk-14.0/share/wasi-sysroot. The build will take a while (~15min on my M1). Alternatively, if you are not on a Mac ARM, you can use a prebuild sysroot and binaries instead.

I found that the easiest way to get cargo to use all the build tools from the wasi-sdk is to prepend the PATH env var with the bin directory of the wasi-sdk build. While I usually prefer calling cargo directly and set up everything else in .config/cargo.toml, I couldn't find a way to set all build tools in .config/cargo.toml. So a Makefile it is. After putting everything together, the Makefile (at ./wasm/Makefile) looks like:

ifndef DEBUG
    cargo_build_flags += --release
endif

.PHONY: build
build:
    PATH="$(shell pwd)/wasi-sdk/dist/wasi-sdk-14.0/bin:${PATH}" \
    CFLAGS="--sysroot=$(shell pwd)/wasi-sdk/dist/wasi-sdk-14.0/share/wasi-sysroot" \
        cargo build --target wasm32-wasi $(cargo_build_flags)

With the Makefile above, make build creates a release build, and make build DEBUG=1 builds a debug build.

Let's try building again:

make build

The previous error is fixed, but a new one is thrown. It now complains about not finding pthread.h.

cargo:warning=sqlite3/sqlite3.c:27215:10: fatal error: 'pthread.h' file not found
cargo:warning=#include <pthread.h>
cargo:warning=         ^~~~~~~~~~~
cargo:warning=1 error generated.

The wasi-sdk doesn't have support for pthread.h yet (see WebAssembly/wasi-libc#209). A workaround for most libraries is to look at build flags that remove everything related to pthreads from the library you are trying to build.

In case of SQLite, I found various flags to achieve that. One being SQLITE_THREADSAFE=0, which disables code used to operate sqlite safely in multithreaded environments. I'd assume our WASM to be single-threaded, so I think I'd be safe to disable it. However, I didn't dig deeper into this, because adding this build flag brings us to the next set of errors:

cargo:warning=#error "WASI lacks a true mmap; to enable minimal mmap emulation, \

This looks a lot like OS-specific stuff that is not available in WASI. While looking into how to get rid of this code, I found the SQLITE_OS_OTHER flag. When this flag is set, built-in OS interfaces (for Unix, Windows, and OS/2) are omitted. Exactly what I was looking for.

Let's try building again.

make build

Aaannndd... it builds!

I also found out that with SQLITE_OS_OTHER being set, it doesn't seem to matter anymore whether SQLITE_THREADSAFE is set to 1 or 0 - both build.

There is a PR for rusqlite now, to fix its WASI compatibility (rusqlite/rusqlite#1116; someone beat me to it).

While it builds now, does it also run? Nope. Trying to run it fails with:

WebAssembly.instantiate(): Import #1 module=\"env\" function=\"sqlite3_os_init\" error: function import requires a callable

But this was to be expected, as the docs clearly state:

Applications must also supply implementations for the sqlite3_os_init() and sqlite3_os_end() interfaces. The usual practice is for the supplied sqlite3_os_init() to invoke sqlite3_vfs_register(). SQLite will automatically invoke sqlite3_os_init() when it initializes.

A minimal sqlite3_os_init would simply register a custom virtual file system:

#[no_mangle]
extern "C" fn sqlite3_os_init() -> i32 {
    ffi::sqlite3_vfs_register(CUSTOM_VFS, false as i32)
}

As a side-note, if sqlite3_os_init isn't registering a VFS, or if it isn't returning 0 (as in no error), you might get a rather unhelpful error like:

RuntimeError: unreachable
    at wasm://wasm/010bf20e:wasm-function[1716]:0x2a1b64
    at wasm://wasm/010bf20e:wasm-function[1764]:0x2a84cb
    at wasm://wasm/010bf20e:wasm-function[1782]:0x2aa2e3
    at wasm://wasm/010bf20e:wasm-function[1829]:0x2b3a23

With being able to build SQLite for wasm32-wasi, we can continue looking into how to initialise and use it.

WASI imports

When inspecting the build WASM module, you find various required imports:

wasmer inspect target/wasm32-wasi/release/do_sqlite.wasm

Imports:
  Functions:
    "wasi_snapshot_preview1"."random_get": [I32, I32] -> [I32]
    "wasi_snapshot_preview1"."clock_time_get": [I32, I64, I32] -> [I32]
    "wasi_snapshot_preview1"."fd_write": [I32, I32, I32, I32] -> [I32]
    "wasi_snapshot_preview1"."poll_oneoff": [I32, I32, I32, I32] -> [I32]
    "wasi_snapshot_preview1"."environ_get": [I32, I32] -> [I32]
    "wasi_snapshot_preview1"."environ_sizes_get": [I32, I32] -> [I32]
    "wasi_snapshot_preview1"."proc_exit": [I32] -> []

WASI offers a huge amount of methods. Fortunately, it is modular and our module will only require the ones that are actually used. We have to provide the once it uses or the WASM module isn't going to run otherwise.

While there is a WASI implementation for Workers: cloudflare/workers-wasi, I prefer to implement each import manually - especially when there are so few and especially while I am still experimenting. This helps me to keep the full picture of what's going on.

Providing my own implementation basically looks like:

const instance = await WebAssembly.instantiate(module, {
  wasi_snapshot_preview1: {
    // "wasi_snapshot_preview1"."random_get": [I32, I32] -> [I32]
    random_get(offset: number, length: number) {
      const buffer = new Uint8Array(
        instance.exports.memory.buffer,
        offset,
        length
      );
      crypto.getRandomValues(buffer);

      return ERRNO_SUCCESS;
    },

    // "wasi_snapshot_preview1"."clock_time_get": [I32, I64, I32] -> [I32]
    clock_time_get() {
      throw new Error("clock_time_get not implemented");
    },

    // ...

The imports are specified when instantiating the WASM module, and are added under the wasi_snapshot_preview1 namespace. As hinted in the snippet above (for clock_time_get), I usually throw a not implemented error for all imports at first, and only implement them when I see them being used.

The fd_write is a special one, because I usually only implement it for stdout and stderr and not any other file descriptor (so not for actual files):

fd_write(
  fd: number,
  iovsOffset: number,
  iovsLength: number,
  nwrittenOffset: number
) {
  if (fd !== 1 && fd !== 2) {
    return ERRNO_BADF;
  }

  const decoder = new TextDecoder();
  const memoryView = new DataView(instance.exports.memory.buffer);
  let nwritten = 0;
  for (let i = 0; i < iovsLength; i++) {
    const dataOffset = memoryView.getUint32(iovsOffset, true);
    iovsOffset += 4;

    const dataLength = memoryView.getUint32(iovsOffset, true);
    iovsOffset += 4;

    const data = new Uint8Array(
      instance.exports.memory.buffer,
      dataOffset,
      dataLength
    );
    const s = decoder.decode(data);
    nwritten += data.byteLength;
    switch (fd) {
      case 1: // stdout
        console.log(s);
        break;
      case 2: // stderr
        console.error(s);
        break;
      default:
        return ERRNO_BADF;
    }
  }

  memoryView.setUint32(nwrittenOffset, nwritten, true);

  return ERRNO_SUCCESS;
}

With that out of the way, it is time to get back to asyncifying the WASM module.

Asyncify WASM

As a recap, the asyncification is necessary to call the async storage APIs from the sync VFS.

The asyncification is achieved by unwinding and rewinding the local stack. With wasm-opt, this is done on a WASM level as post-build transformation. This makes it a language-agnostic approach. Once a WASM module is asyncified it needs to be used through the JavaScript wrapper Asyncify. A good introduction to asyncify can be found here: Using asynchronous web APIs from WebAssembly.

This is how it is used. First I had to add WASM imports, which correspond to external functions in Rust:

extern "C" {
    pub fn get_page(ix: u32) -> *mut u8;
    pub fn put_page(ix: u32, ptr: *const u8);
}

I am then using those imports in my VFS:

impl<const PAGE_SIZE: usize> Pages<PAGE_SIZE> {
    pub fn get_page(ix: u32) -> Option<[u8; PAGE_SIZE]> {
        unsafe {
            let ptr = crate::get_page(ix);
            if ptr.is_null() {
                None
            } else {
                let slice = slice::from_raw_parts_mut(ptr, PAGE_SIZE);
                slice[..].try_into().ok()
            }
        }
    }

    fn put_page(ix: u32, data: &[u8; PAGE_SIZE]) {
        unsafe {
            crate::put_page(ix, data.as_ptr());
        }
    }
}

With that integrated, I can build the WASM file as done previously and once build, post-process it via wasm-opt:

wasm-opt -O --asyncify --pass-arg asyncify-imports@env.put_page,env.get_page \
  wasm/target/wasm32-wasi/release/do_sqlite.wasm \
  -o dist/do_sqlite.wasm

The flags --asyncify --pass-arg asyncify-imports@env.put_page,env.get_page tell the tool that both the put_page and get_page imports are asynchronous.

I then have to initialise the WASM module using the Asyncify wrapper and provide async implementations of get_page and put_page:

const instance = await Asyncify.instantiate(module, {
  env: {
    async get_page(ix: number): Promise<number> {
      const page: Array<number> =
        (await storage.get<Array<number>>(String(ix))) ?? new Array(4096);

      const offset: number = await instance.exports.alloc(
        4096
      );
      const dst = new Uint8Array(
        instance.exports.memory.buffer,
        offset,
        4096
      );
      dst.set(Array.from(new Uint8Array(page)));

      return offset;
    },

    async put_page(ix: number, ptr: number) {
      const page = new Uint8Array(
        instance.exports.memory.buffer,
        ptr,
        4096
      );
      await storage.put(String(ix), Array.from(page), {});
    },
  },
});

That's it. The WASM module is now able to interact with the async storage APIs.

Increase Asyncify Stack Size

With all pieces together now, does it run? Yes and no. Depending on the query I've tested, I ran into yet another very helpful error:

RuntimeError: unreachable
    at wasm://wasm/010bf772:wasm-function[1833]:0x2b3c6e

Some println!() debugging later, I've found that the error happens once the WASM execution is resumed. When the WASM execution is paused, all locals at the point of pausing are saved. Before execution is resumed, all those locals are restored. Asyncify allocates memory to persist those locals between pausing and resuming. This memory has a size of 1024 bytes and is apparently simply not enough for this use-case.

The stack size is unfortunately not configurable. According to Asyncify's code, it assumes that the first 1024 bytes of the WASM memory are unused and uses that. I'd rather properly allocate memory for it. Since it is writing the stack start and end locations at an offset of 16, I simply overwrite those values with the location of a bigger and properly allocated memory location for now:

const STACK_SIZE = 4096;
const DATA_ADDR = 16;
const ptr = await exports.alloc(STACK_SIZE);
new Int32Array(exports.memory.buffer, DATA_ADDR, 2).set([
  ptr,
  ptr + STACK_SIZE,
]);

That's it. With that I had SQLite running in a Cloudflare Durable Object.

I am not sharing a link to my demo, as I haven't estimated the to be expected costs yet. But if you are curious, you can give it a try yourself using github.com/rkusa/do-sqlite.

Conclusion

Let me emphasise again: this is just an experiment. It is not in a state where I'd use it for an actual app yet. Here are some open points / concerns / future work:

I am setting SQLite's journal mode to memory right now, which could lead to data loss. I'd have to extend the VFS to be capable of also persisting the journal file into the DO.
I am not happy with the the overhead that comes with pausing/resuming via Asyncify.
I have a bad feeling about Asyncify's stack size. I could imagine that I'd have to increase it further the more the database grows / or the larger the query result is.
The 128MB memory size of Workers might be a limiting factor when the database grows to a certain size. A possible improvement here would be to use an LRU cache for the pages and basically dispose pages that have been read in the past, but were not used recently.
I need to make some more research and testing to be sure that the DO isn't writing to the storage concurrently.
The current solution isn't capable of running DB transactions across multiple calls to the DO (and might never be actually).
I'd need to make an estimation of the expected costs of running SQLite on a DO to feel safe actually deploying it for an app.
A replication between multiple DO's living on different Cloudflare locations would be neat (maybe based on litestream.io).

The code can be found here:

github.com/rkusa/sqlite-vfs: Build SQLite virtual file systems (VFS) by implementing a simple Rust trait.
github.com/rkusa/wasm-sqlite: SQLite compiled to WASM with pluggable page storage.
github.com/rkusa/do-sqlite: POC that persists SQLite in a Cloudflare Durable Object.

I'll continue experimenting.

WASM instead of C Dependencies?

Markus Ast — Fri, 14 Jan 2022 10:56:20 +0000

I have web applications written in Rust and Go that need some basic image processing (reading JPEGs, PNGs, writing JPEGs, WebPs, AVIFs and resizing). This is something I always struggle with, because most libraries for image processing are written in C (libpng, libwebp, mozjpeg; or higher-level ones like vips). While there are usually dependencies in each language build on top of those C dependencies, like bimg for Go, I don’t like having C dependencies in a Rust, Go or even Node.js projects.

Pain during builds

Let’s take bimg as an example (just and example, I had the same experience with similar dependencies in Rust, Go and Node.js):

go get -u github.com/h2non/bimg

Trying to get it fails on my system with:

# pkg-config --cflags  -- vips vips vips vips
Package vips was not found in the pkg-config search path.
Perhaps you should add the directory containing `vips.pc'
to the PKG_CONFIG_PATH environment variable
No package 'vips' found

So I have to get vips first:

brew install vips

Now I have a dependency that is not managed by the Go toolchain as the rest of my code. Installing it is different from OS to OS.

And does it work now? Nope.

% go get -u github.com/h2non/bimg
go build github.com/h2non/bimg: invalid flag in pkg-config --cflags: -Xpreprocessor

After allowing the cflag, I can finally get the dependency:

env CGO_CFLAGS_ALLOW="-Xpreprocessor" go get -u github.com/h2non/bimg

This is not the developer experience I am aiming for. Not for myself, not for my future self who forgot how it worked, and not for other devs that have to work with my projects.

Pain during deployments

This is highly subjective, but I really want my Dockerfiles to either be FROM scratch or FROM gcr.io/distroless/static (GoogleContainerTools/distroless). To qualify, my applications must compile to static binaries and must not require libc. The worst case I’d be fine with is FROM gcr.io/distroless/base (static compiled, but libc is fine).

If I want to build a minimal Docker image with a program that as an example depends on vips, I'd have to add a long list of shared objects to the image:

/opt/vips/lib/libvips.so.42
/usr/lib/libgobject-2.0.so.0
/usr/lib/libglib-2.0.so.0
/usr/lib/libintl.so.8
/lib/ld-musl-x86_64.so.1
/usr/lib/libexpat.so.1
/usr/lib/libheif.so.1
/usr/lib/libwebpmux.so.3
/usr/lib/libwebpdemux.so.2
/usr/lib/libwebp.so.7
/usr/lib/libpng16.so.16
/usr/lib/libjpeg.so.8
/usr/lib/libexif.so.12
/usr/lib/libgmodule-2.0.so.0
/usr/lib/libgio-2.0.so.0
/usr/lib/libffi.so.8
/usr/lib/libpcre.so.1
/usr/lib/libaom.so.3
/usr/lib/libde265.so.0
/usr/lib/libx265.so.199
/usr/lib/libstdc++.so.6
/usr/lib/libgcc_s.so.1
/lib/libz.so.1
/lib/libmount.so.1
/lib/libblkid.so.1

WebAssembly as an alternative solution?

So I was wondering if WebAssembly would be ready to act as a solution to provide C libraries to other ecosystems with less headaches. The questions I was wondering about are:

How much slower is WASM compare to a C binding?
Does WASM allow me to get rid of all C dependencies?

How much slower is WASM compare to a C binding?

WASM will be slower compared to C bindings, this is a fact I wasn’t wondering about. I just wanted to get a rough idea of how much slower. So I did a benchmark - as for all benchmarks take it with a grain of salt.

Testsetup: Encode a 6048x2048px big image using mozjpeg (mozjpeg-sys to be more specific), resize it down to 1008x665px using PistonDevelopers/resize, and decode it again using mozjpeg.

As a reference, when executing the test without going through WASM (so Rust directly compiled to an executable), the image transformation takes around 205ms. I compared this to running the same code compiled to WASM in different WASM runtimes in Go. The results are:

Wasmer LLVM runtime: 265ms
Wasmtime: 272ms
Wasmer Cranelift runtime: 275ms
wazero: 450000ms (7.5min)

Now I know that it is roughly 25% slower (for the first three) for that specific use-case. Since I’d heavily cache transformed images, I could life with the 25% slowdown.

Does WASM allow me to get rid of all C dependencies?

Yes, for Rust, as there are plenty of WASM runtimes written in Rust (Wasmer and Wasmtime for example). But unfortunately no for Go, as wazero was the only runtime I could find that is written in Go (and it doesn't work well enough for my use-case). Wasmer and Wasmtime are also available in Go, but only through CGO as both consume C APIs provided by the underlying Rust implementation. So I'd get rid of a C dependency by compiling it to WASM just to add a new C dependency to run the WASM.

Conclusion

I’d consider WASM as a good alternative for C dependencies for Node.js projects, if the use-case does allow for the slowdown compared to C bindings.

I personally have projects in Rust, Go and on Cloudflare Workers (V8). So my ideal solution would be a Rust project that:

I can consume as WASM in Rust or directly as a Rust dependency if I am fine with the C dependencies,
I can consume as WASM in Cloudflare Workers and
I can consume via C bindings in Go.

My inner monk is still struggling a bit with the third point. But I already have a POC for it and it at least reduces the amount of shared objects I have to provide to my final Docker image to libc and libgcc1, which would allow me to at least use gcr.io/distroless/cc as a base image.

GoogleChromeLabs/squoosh is actually pretty close to what I'd want, but isn't published as a Rust dependency and heavily relies on Node.js so doesn't work in Cloudflare Workers.

Minimal React Transition Hook (useTransition)

Markus Ast — Mon, 29 Nov 2021 21:26:32 +0000

I often find myself in the need of a very basic open/close transition in React for components like dialogs, side sheets, or dropdowns. The goto solution for a while seemed to be React Transition Group, which I never understood how to use properly. An excellent solution for animations is react-spring, but I'd consider it an overkill for a basic CSS powered open/close transition (but it's great for animations in something like an image viewer).

This is why I’ve ended up writing my own hook: react-css-transition-hook

It is used like this:

const { isOpen } = useMenu();
const [isVisible, props] = useTransition(isOpen, {
  entering: "transition ease-out duration-100 transform opacity-0 scale-95",
  entered: "transition ease-out duration-100 transform opacity-100 scale-100",
  exiting: "transition ease-in duration-75 transform opacity-100 scale-100",
  exited: "transition ease-in duration-75 transform opacity-0 scale-95",
});

if (!isVisible) {
  return null
}

return (
  <div {...props}>
    ...
  </div>
)

Isn’t that easy to understand and reason about just from reading the usage? Here is a complete example using the hook: Demo, Source.

The hook itself is just ~50 lines long (excluding the typings and doc comments) and so simple, it easily fits into this post:

export function useTransition(
  desiredState: boolean,
  opts: UseTransitionOpts
): [boolean, TransitionProps, TransitionStep] {
  const [currentState, setCurrentState] = useState(
    Boolean(desiredState && opts.disableInitialEnterTransition)
  );
  const [transition, setTransition] = useState<TransitionStep>(() =>
    desiredState ? "entered" : null
  );

  useEffect(() => {
    // exited -> entering
    if (!currentState && desiredState) {
      setCurrentState(true);
      setTransition("entering");
    }
    // entered -> exited
    else if (currentState && !desiredState) {
      setTransition("exiting");
    }
  }, [currentState, desiredState]);

  // Once the state changed to true, trigger another re-render for the switch to
  // the entered classnames
  useEffect(() => {
    switch (transition) {
      case "entering":
        setTransition("entered");
        break;
      case "exiting":
        setTransition("exited");
        break;
    }
  }, [transition]);

  const onTransitionEnd = useCallback(() => {
    if (!desiredState) {
      setCurrentState(false);
      setTransition(null);
    }
  }, [desiredState]);

  return [
    currentState,
    { className: transition ? opts[transition] ?? "" : "", onTransitionEnd },
    transition,
  ];
}

This is exactly what I wanted. Simple, small, no fancy magic - just using basic useState, useEffect, and useCallback hooks.

Let’s dissect its inner workings top-down.

Typically, when a component is closed, it is just not rendered anymore. This does not work well with a close transition, because it is necessary to keep the component in the DOM until the close transition finished. This is why the hook takes the desired state (visible or not; isOpen in the usage example above, and desiredState in the code above) as an input, and returns whether you should still render the component or not (isVisible in the example usage above, and currentState in the code below).

const [currentState, setCurrentState] = useState(
  Boolean(desiredState && opts.disableInitialEnterTransition)
);
const [transition, setTransition] = useState<TransitionStep>(() =>
  desiredState ? "entered" : null
);

When the hook is first used, it determines what the initial state is and also provides an option to skip the enter transition if it starts being visible right away. It also sets its initial transition state (transition), which is either entered, if the component is already visible, or null if it is not.

useEffect(() => {
  // exited -> entering
  if (!currentState && desiredState) {
    setCurrentState(true);
    setTransition("entering");
  }
  // entered -> exited
  else if (currentState && !desiredState) {
    setTransition("exiting");
  }
}, [currentState, desiredState]);

When either the current or desired states change, it updates the active transition accordingly:

Not visible right now (currentState === false), but should be shown (desiredState === true): Render the component and set entering (usually something like 0% opacity, or moved outside of the screen) as the active transition.
Visible right now (currentState === true), but should not be shown anymore (desiredState === false): Set active transition to exiting (often the same as entering, so something like 0% opacity, …) and keep the component for now.

For the open transition, the transition cannot be set to entered right away. It is always necessary to render the component with entering first so that there is a starting point for the transition to be based on. Example:

Render with 0% opacity, and once that is reflected in the DOM,
Set the opacity to 100% for the transition to start.

This is what the second useEffect is for.

useEffect(() => {
  switch (transition) {
    case "entering":
      setTransition("entered");
      break;
    case "exiting":
      setTransition("exited");
      break;
  }
}, [transition]);

The second useEffect cannot be integrated into the first one, because there needs to be a DOM update before the state changes of the second useEffect are applied. By separating them, the state changes from the first effect are reflected in the DOM before the whole hook is called again and applies the changes from the second effect. The second effect is thereby simply reacting on the changes from the first useEffect and kicks of the transition by moving from entering to entered, or from exiting to exited.

const onTransitionEnd = useCallback(() => {
  if (!desiredState) {
    setCurrentState(false);
    setTransition(null);
  }
}, [desiredState]);

It is necessary to know when the close transition finished so that the component can be removed from the DOM. This is achieved by a simple onTransitionEnd event handler. Once fired, it sets the current state to false and resets the transition to null.

That’s all there is to it.

Finally, as a small bonus, an advanced example of how to use it for a Radix UI Dialog-based side sheet:

import React, { PropsWithChildren, useCallback } from "react";
import * as Dialog from "@radix-ui/react-dialog";
import { XIcon } from "@heroicons/react/outline";
import { useTransition } from "react-css-transition-hook";
import classNames from "classnames";

export default function SideSheet({
  isOpen,
  dismiss,
  title,
  children,
}: PropsWithChildren<{ isOpen: true; dismiss(): void; title: string }>) {
  const handleOpenChange = useCallback(
    (open: boolean) => {
      if (!open) {
        dismiss();
      }
    },
    [dismiss]
  );

  const [isVisible, { className: contentClassName, ...props }, step] =
    useTransition(isOpen, {
      entering: "translate-x-full",
      entered: "translate-x-0",
      exiting: "translate-x-0",
      exited: "translate-x-full",
    });

  const backdropClassName = step
    ? {
        entering: "opacity-0",
        entered: "opacity-100",
        exiting: "opacity-100",
        exited: "opacity-0",
      }[step]
    : "";

  if (!isVisible) {
    return null;
  }

  return (
    <Dialog.Root open={true} onOpenChange={handleOpenChange}>
      <Dialog.Overlay
        className={classNames(
          "fixed inset-0 bg-black bg-opacity-50",
          "transition-opacity duration-500 ease-in-out",
          backdropClassName
        )}
      />
      <Dialog.Content
        className={classNames(
          "fixed inset-y-0 right-0 px-4 md:px-16 pt-8 pb-16",
          "w-screen max-w-[496px]",
          "bg-white overflow-auto",
          "transform transition-transform duration-500 ease-in-out",
          contentClassName
        )}
        {...props}
      >
        <header className="flex justify-between items-center mb-8">
          <Dialog.Title className="text-2xl m-0">{title}</Dialog.Title>
          <Dialog.Close asChild>
            <button>
              <XIcon className="h-6 w-6" aria-hidden="true" />
            </button>
          </Dialog.Close>
        </header>

        {children}
      </Dialog.Content>
    </Dialog.Root>
  );
}

Dynamic Gatsby Routes on Cloudflare Pages

Markus Ast — Wed, 06 Oct 2021 10:28:35 +0000

In this post, I'll be talking about getting dynamic routes – or more specifically the rewrites for dynamic routes – to work with Cloudflare Pages deployments.

Cloudflare Pages is still quite new and thus not as feature rich as other longer established JAMstack platforms. But what it lacks in features, it makes up in speed, which is why I desperately want to move some of my sites to it.

As an example in this post I'll be using an actual route I am using in one of my shops. It is a dynamic route that allows my customers to check the status of their order. The route has the following format:

/status/:orderId/:token

It contains the id of the order and a order-specific secret token to restrict access to this page to the ones who know the URL (only the customer). This page can obviously not be generated during built-time of the site. It instead has to fetch the current order state on the client-side based on the order id and token from the URL.

Since the shop in question is built in Gatsby, I'll be using actual code examples from Gatsby. But the approaches shown in this post can also be applied to (exported) Next.js and other static sites. In Gatsby, I am creating the dynamic route as follows:

createPage({
  path: '/status/:orderId/:token',
  matchPath: '/status/:orderId/:token',
  component: path.resolve('./src/templates/status.tsx'),
})

This creates the static /status/:orderId/:token.html when the site is built. We now have to make sure that all URLs that match /status/:orderId/:token (e.g. /status/1234/EvEf_HAp7oyb194ZekX39bIcwueV1) are rewritten to /status/:orderId/:token.html. We need a rewrite and not a redirect, because a redirect would change the URL in the browser's address bar and would also lose the actual order id and the token. While Gatsby would still be able to figure out that it has to load /status/:orderId/:token/index.html if not have the rewrite, it'd first show a 404 before doing a client-side redirect to the correct page. This is a bad UX and needs to be avoided.

On Netlify, I am simply using a rewrite in the _redirects file right now:

/status/*  /status/:orderId/:token/index.html  200

The status of 200 makes sure that this is a rewrite and not a redirect. While Cloudflare Pages also supports _redirects files, it only does for redirects (301 and 302) for now. We thus need another solution.

Option 0: Pages Functions

Option added to the post on the 2021-11-27.

Cloudflare released Pages Functions since this post was first created. Pages Functions are now the recommended approach. It is basically the same as Option 2: Cloudflare Worker just that it integrates nicely into your existing repository and doesn't require maintaining a separate worker.

The function for the rewrite is as simple as creating /functions/status/[order]/[token].ts. With the following content:

export function onRequestGet({ env, request }: EventContext<Env, string, unknown>) {
  return env.ASSETS.fetch(
    new Request(
      new URL('/status/:orderId/:token/', request.url).toString(),
      request
    )
  )
}

interface Env {
  ASSETS: Fetcher
}

Make sure that the /functions/ directory is at the root of your project (the root you specify in the Pages build settings). You don't have to transpire the function from TypeScript to JavaScript, Pages will automatically take care of it.

Option 1: Transform Rule

The first possible solution is to use a transform rule. In your Cloudflare Dashboard, select your site, go to Rules and create a new Rewrite URL transform rule. Setup the rule as shown in the screenshot below.

Make sure to use the matches regex operator to match from the beginning of the URI path. Don't use a contains operator because we don't want to rewrite the page-data.json URL from Gatsby which is located at /page-data/status/:orderId/:token/page-data.json in the example.

Also make sure to rewrite the URL to /status/:orderId/:token/, and not directly to /status/:orderId/:token/index.html, because Cloudflare would return a redirect to /status/:orderId/:token/index in the latter case (and a rewrite to /status/:orderId/:token/ if you try to rewrite it to /status/:orderId/:token/index).

If you are still on the Free plan, you don't have access to the matches regex operator. In this case, you can fall back to option 2.

Option 2: Cloudflare Worker

As a second solution, you can deploy a very simple Cloudflare Worker. The docs will get you started on how to create and deploy a worker, so I'll only show the relevant pieces here.

The worker itself is as simple as:

addEventListener("fetch", (event) => {
  event.respondWith(
    fetch(
      new Request(
        new URL("/status/:orderId/:token/", event.request.url).toString(),
        event.request
      )
    )
  );
});

It just takes the request, modifies its URL, and executes it. The worker doesn't check if the incoming URL starts with /status, since this can be done by deploying the worker to the corresponding route instead. Just add the route setting in your wrangler.toml accordingly:

route = "example.org/status/*"

Drawbacks?

What I don't like about both solutions is that they are not tied to the repository of my site. I prefer that each commit in my repo is a self-contained deployment, i.e., I can always check out a specific commit, run one deployment command and everything works as it was supposed to at this commit.

However, I could probably achieve this with both options. For Option 1 I'd have to add calls to Cloudflare's API to ensure that the transform rule exists as intended for the commit, and for Option 2 I could add the worker to the repository and update it as part of the site's deployment.

~~Another drawback is that both options require your domain to go through Cloudflare - but if you are using Cloudflare Pages, chances are good that it already does.~~

Fixed with Option 0 that was added on the 2021-11-27.

Can I move my shop to Cloudflare Pages yet?

No, unfortunately. I am doing i18n on built-time and thus have different deployments (and domains) for each language I support. Cloudflare Pages does not support multiple projects per repository yet.

DEV Community: Markus Ast

Store SQLite in Cloudflare Durable Objects

The Plan

Where to store SQLite on Cloudflare?

How to store the SQLite database file in a DO?

How to run SQLite on Workers?

The initial plan

Compile SQLite to WASM for target wasm-unknown-unknown

Build the Virtual File System

Use the VFS in a DO

Target: wasm32-unknown-unknown vs wasm32-wasi

Build SQLite for WASI for wasm32-wasi

WASI imports

Asyncify WASM

Increase Asyncify Stack Size

Conclusion

WASM instead of C Dependencies?

Pain during builds

Pain during deployments

WebAssembly as an alternative solution?

How much slower is WASM compare to a C binding?

Does WASM allow me to get rid of all C dependencies?

Conclusion

Minimal React Transition Hook (useTransition)

Dynamic Gatsby Routes on Cloudflare Pages

Option 0: Pages Functions

Option 1: Transform Rule

Option 2: Cloudflare Worker

Drawbacks?

Can I move my shop to Cloudflare Pages yet?

Compile SQLite to WASM for target `wasm-unknown-unknown`

Target: `wasm32-unknown-unknown` vs `wasm32-wasi`

Build SQLite for WASI for `wasm32-wasi`