Zephyr, Web Bluetooth and Accessibility

#zephyr #nrf52840 #webbluetooth #a11y

UI interaction using a normal mouse is something we all take for granted. However, for some, this can be a bit more cumbersome.

The nRF52840 Dongle from Nordic Semiconductor is a powerful little device, where the main SoC controls both USB and Bluetooth Low Energy connections.

Let's look at how simple firmware done with Zephyr RTOS running on the dongle, can provide affordable and easily customized USB mouse controls on any computer from any BLE connected phone/tablet.

The firmware

As the dongle should act as a bridge between a BLE connected device on one side and act as a standard USB HID mouse on the other, these features need to be enabled in the firmware.
Also, to enable easy control from a web application using Web Bluetooth, a simple GATT service must be created.

Note: In this post, I'll only go through an overview of key parts of the firmware (using C) and web application (using JavaScript). The full source for the PoC is available on GitHub and more details will be in separate posts (please comment if you're interested ;)).

Defining the GATT service structure in Zephyr:

/* Simple Mouse Service Declaration */
BT_GATT_SERVICE_DEFINE(sm_svc,
    BT_GATT_PRIMARY_SERVICE(&simplems_uuid),
    BT_GATT_CHARACTERISTIC(&simplems_move_xy_uuid.uuid, BT_GATT_CHRC_WRITE,
                BT_GATT_PERM_WRITE,
                NULL, write_move_xy, NULL),
    BT_GATT_CUD("Move XY", BT_GATT_PERM_READ),
    BT_GATT_CHARACTERISTIC(&simplems_buttons_uuid.uuid, BT_GATT_CHRC_WRITE,
                BT_GATT_PERM_WRITE,
                NULL, write_buttons, NULL),
    BT_GATT_CUD("Buttons", BT_GATT_PERM_READ),
);

The characteristics defined in the service can be seen as 'variables' that have properties tied to them, making them readable, writable and/or possible to be notified from (~ listen to changes). In this example, we are only interested in writing mouse movement and button events from the connected application.

In order for the client (web application) to find the dongle, running the firmware, we also need to set up BLE advertising:

static const struct bt_data ad[] = {
    BT_DATA_BYTES(BT_DATA_FLAGS, (BT_LE_AD_GENERAL | BT_LE_AD_NO_BREDR)),
    BT_DATA_BYTES(BT_DATA_UUID128_ALL, BT_UUID_SIMPLE_MOUSE_SERVICE),
};

...

err = bt_le_adv_start(BT_LE_ADV_CONN_NAME, ad, ARRAY_SIZE(ad), NULL, 0);
if (err) {
    printk("Advertising failed to start (err %d)\n", err);
    return;
}

printk("Advertising successfully started\n");

When a connected client (in our case, a web application using Web Bluetooth) wants to e.g. move the mouse cursor, dX and dY values are sent as 2 signed bytes to the simple_move_xy characteristic, handled by the write handler and sent to a registered callback function:

static ssize_t write_move_xy(struct bt_conn *conn,
                  const struct bt_gatt_attr *attr,
                  const void *buf, uint16_t len,
                  uint16_t offset, uint8_t flags)
{
    int8_t val[2];

    if (offset != 0) {
        return BT_GATT_ERR(BT_ATT_ERR_INVALID_OFFSET);
    } else if (len != sizeof(val)) {
        return BT_GATT_ERR(BT_ATT_ERR_INVALID_ATTRIBUTE_LEN);
    }

    (void)memcpy(&val, buf, len);

    if (sm_cbs->move_xy) {
        sm_cbs->move_xy(val[0], val[1]);
    }

    return len;
}

The callback function then converts it to a USB HID Mouse move 'event' and written to the host computer, where the USB dongle is connected:

static void sm_move_xy_cb(int8_t distx, int8_t disty) {
    printk("Move (x,y) = (%d, %d)\n", distx, disty);

    uint8_t statex = status[MOUSE_X_REPORT_POS];
    uint8_t statey = status[MOUSE_Y_REPORT_POS];

    statex += distx;
    statey += disty;

    if (status[MOUSE_X_REPORT_POS] != statex || status[MOUSE_Y_REPORT_POS] != statey) {
        status[MOUSE_X_REPORT_POS] = statex;
        status[MOUSE_Y_REPORT_POS] = statey;
        k_sem_give(&sem);
    }
}

...

while (true) {
    k_sem_take(&sem, K_FOREVER);

    report[MOUSE_BTN_REPORT_POS] = status[MOUSE_BTN_REPORT_POS];
    report[MOUSE_X_REPORT_POS] = status[MOUSE_X_REPORT_POS];
    status[MOUSE_X_REPORT_POS] = 0U;
    report[MOUSE_Y_REPORT_POS] = status[MOUSE_Y_REPORT_POS];
    status[MOUSE_Y_REPORT_POS] = 0U;
    err = hid_int_ep_write(hid_dev, report, sizeof(report), NULL);
    if (err) {
        printk("HID write error, %d\n", err);
    }
}

Note: This code is derived from a USB HID Mouse sample in the Zephyr project for the purpose of this simple PoC.

Web Driver

Different users of the controlling web application, might have very different needs. Some might want to control the mouse with a simple touch joystick (as is done in this example), while other might want to control the mouse by using voice commands or the accelerometer.

For this purpose, a simple web driver is created that can be embedded in any framework/web application. This driver takes care of the essential communication with the dongle, using Web Bluetooth for connection and command handling.

The full source is available on GitHub, but here are a few essential snippets, e.g. connecting to the device:

async scan() {
    const device = await navigator.bluetooth.requestDevice({
        filters: [
            { services: [SimpleMouseLinkUUID] },
            { name: 'Simple Mouse Link'}
        ]
    });

    if (device) {
        await this.openDevice(device);
    }
}

Getting access to the GATT service and write command handlers:

async fetchWriteCharacteristics(server) {
    const service = await server.getPrimaryService(SimpleMouseLinkUUID);
    this.#writeFuncs.move_xy = await service.getCharacteristic(SimpleMouseMoveXYUUID);
    this.#writeFuncs.buttons = await service.getCharacteristic(SimpleMouseButtonsUUID);
}

Sending mouse move commands:

move(x, y) {
    if (this.#writeFuncs.move_xy) {
        const xx = cap_val(x);
        const yy = cap_val(y);
        console.log("Mouse move:", xx, yy)
        if (xx || yy) this.#writeFuncs.move_xy.writeValue(new Uint8Array([xx, yy]));
    }
}

Full code and demo!

The full source for the firmware and web application can be found on GitHub and the live demo can be found here.

Please consider making request for features, report issues or just give a comment below :). I plan to go in more detail on how to get started with Zephyr for web connected hardware and more in followup posts.

For now, here is a small video capture of the demo web application in action, where the dongle is connected to a PC and the mouse is controlled by phone connected via Web Bluetooth (unfortunately, the UI showing the BLE connect dialog is not shown through the DevTools link):