DEV Community: Tom Holloway 🏕

Realtime Collaborative Drawing (part 4): Redis PubSub + WebRTC Signaling

Tom Holloway 🏕 — Mon, 21 Sep 2020 17:04:00 +0000

In any system that involves real-time communication, open connections, and messages that need to be routed among peers - you tend to run into the problem that not all of your connections will be able to run on a single server. What we need to do, instead, is setup a system that can route messages to any number of servers maintaining any number of connections.

In a previous article, our drawing program was recently refactored to use server sent events by maintaining a connection open. However, if we introduced another web server to create some load balancing, we run into the problem that the client connections may not be accessible across servers.

We can solve this by having a shared communication server/cluster that can handle routing all these messages for us. To do this we will be using the publisher-subscriber pattern and we will leverage redis to get this job done.

Redis

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes with radius queries and streams. Redis has built-in replication, Lua scripting, LRU eviction, transactions and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster

Redis is an incredible project, it's extremely fast and uses minimal cpu. It is a work of art that the project has also remained backwards compatible since version 1. The maintainer antirez (who recently has decided to move on) has created this project over a number of years and built it into something truly incredible. Redis supports just about all the data structures as first-class features and operations.

And it even supports cluster. You can even use it as a last-write-wins CRDT using roshi. At my work, we've used just about all of these features from queues, hyperloglogs, sorted sets, caching. In a previous project, I once used redis to build a click stream system using an sort of event sourcing model.

Redis PubSub

We're going to use a small feature of redis called pubsub to route our messages between connections on our server. Assuming you have a redis-server setup. You'll need to add redis as a dependency to our drawing app.

npm install --save redis bluebird

We're going to use bluebird to be able to promisifyAll to all the redis client functions. This will help us write our code with async/await instead of a number of callbacks.

/connect to SSE and Subscribe to Redis

Recall that our express server was simply keeping an in-memory cache of both connections and channels. We're going to first update our /connect function to instead subscribe to messages received from a redis pubsub client. To do this, we'll update the client creation code and add a redis.createClient. Then subscribe to messages received to our particular client id via redis.subscribe('messages:' + client.id). Whenever we receive messages via redis.on('message', (channel, message) => ...) we can simply emit them back to the server sent event stream.

var redis = require('redis');
var bluebird = require('bluebird');
bluebird.promisifyAll(redis);

app.get('/connect', auth, (req,res) => {
    if (req.headers.accept !== 'text/event-stream') {
        return res.sendStatus(404);
    }

    // write the event stream headers
    res.setHeader('Cache-Control', 'no-cache');
    res.setHeader('Content-Type', 'text/event-stream');
    res.setHeader("Access-Control-Allow-Origin", "*");
    res.flushHeaders();

    // setup a client
    let client = {
        id: req.user.id,
        user: req.user,
        redis: redis.createClient(),
        emit: (event, data) => {
            res.write(`id: ${uuid.v4()}\n`);
            res.write(`event: ${event}\n`);
            res.write(`data: ${JSON.stringify(data)}\n\n`);
        }
    };

    // cache the current connection until it disconnects
    clients[client.id] = client;

    // subscribe to redis events for user
    client.redis.on('message', (channel, message) => {
        let msg = JSON.parse(message);
        client.emit(msg.event, msg.data);
    });
    client.redis.subscribe(`messages:${client.id}`);

    // emit the connected state
    client.emit('connected', { user: req.user });

    // ping to the client every so often
    setInterval(() => {
        client.emit('ping');
    }, 10000);

    req.on('close', () => {
        disconnected(client);
    });
});

Also notice that I've added an interval to ping the client once every 10 seconds or so. This may not be completely necessary, but I add it to make sure our connection state doesn't inadvertently get cut off for whatever reason.

Peer Join, Peer Signaling

The only other functions we need to change are when a peer joins the room, when a peer is sending a signal message to another peer, and when a peer disconnects from the server. The other functions like auth, :roomId remain the same. Let's update the join function below. Note, we will need to keep track of a redis client that the server for general purpose redis communication.

const redisClient = redis.createClient();

app.post('/:roomId/join', auth, async (req, res) => {
    let roomId = req.params.roomId;

    await redisClient.saddAsync(`${req.user.id}:channels`, roomId);

    let peerIds = await redisClient.smembersAsync(`channels:${roomId}`);
    peerIds.forEach(peerId => {
        redisClient.publish(`messages:${peerId}`, JSON.stringify({
            event: 'add-peer',
            data: {
                peer: req.user,
                roomId,
                offer: false
            }
        }));
        redisClient.publish(`messages:${req.user.id}`, JSON.stringify({
            event: 'add-peer',
            data: {
                peer: { id: peerId },
                roomId,
                offer: true
            }
        }));
    });

    await redisClient.saddAsync(`channels:${roomId}`, req.user.id);
    return res.sendStatus(200);
});

In order to keep track of who is in a particular roomId, we will make use of redis sets and add the room id to the current user's set of channels. Following this, we lookup what members are in the channels:{roomId} and iterate over the peer ids. For each peer id, we effectively will be routing a message to that peer that the current user has joined, and we will route the peer id to the request.user. Finally, we add our request.user to the channels:{roomId} set in redis.

Next, let's update the relay code. This will be even simpler since all we have to do is just publish the message to that peer id.

app.post('/relay/:peerId/:event', auth, (req, res) => {
    let peerId = req.params.peerId;
    let msg = {
        event: req.params.event,
        data: {
            peer: req.user,
            data: req.body
        }
    };
    redisClient.publish(`messages:${peerId}`, JSON.stringify(msg));
    return res.sendStatus(200);
});

Disconnect

Disconnect is a bit more involved, since we have to clean up the rooms that the user is in, then iterate over those rooms to get the list of peers in those rooms, then we must signal to each peer in those rooms that the peer has disconnected.

async function disconnected(client) {
    delete clients[client.id];
    await redisClient.delAsync(`messages:${client.id}`);

    let roomIds = await redisClient.smembersAsync(`${client.id}:channels`);
    await redisClient.delAsync(`${client.id}:channels`);

    await Promise.all(roomIds.map(async roomId => {
        await redisClient.sremAsync(`channels:${roomId}`, client.id);
        let peerIds = await redisClient.smembersAsync(`channels:${roomId}`);
        let msg = JSON.stringify({
            event: 'remove-peer',
            data: {
                peer: client.user,
                roomId: roomId
            }
        });
        await Promise.all(peerIds.forEach(async peerId => {
            if (peerId !== client.id) {
                await redisClient.publish(`messages:${peerId}`, msg);
            }
        }));
    }));
}

Success!

Conclusion

Now that we've added support for Redis PubSub, we can scale our service to any number of server nodes (so long as we have a redis server that we can communicate between). Connections will remain open per node process while the messages and channel communication will be routed through redis to ensure that every message is delivered through the proper server sent event stream.

Thanks for following along!

Cheers! 🍻

CODE

If you're interested in the code for this series, check out my repository on GitHub below:

https://github.com/nyxtom/drawing-webrtc

Thanks again!

Realtime Collaborative Drawing (part 3): Simulating webkit force and using a color swatch with canvas and javascript

Tom Holloway 🏕 — Thu, 10 Sep 2020 17:17:24 +0000

I promised to give our drawing tool in canvas an upgrade, so let's get a look at that now. There's a few things that it currently doesn't allow such as:

Switching colors
Pressure values only available in webkit
Single line based brush tool

Color Swatch

From our previous article our color generation was simply a random function on load. I looked at a number of color generations and libraries and they all were quite a bit too much. Specifically many of these color picker libraries add code for converting between different color schemes like CMYK, RGB, HSL, HSV. Instead, an easier (one function) approach will be to simply create a swatch (similar to Google Docs).

All we need really to accomplish this is have an array of defined colors.

const swatch = [
    ["#000000", "#434343", "#666666", "#999999", "#b7b7b7", "#cccccc", "#d9d9d9", "#efefef", "#f3f3f3", "#ffffff"],
    ["#980000", "#ff0000", "#ff9900", "#ffff00", "#00ff00", "#00ffff", "#4a86e8", "#0000ff", "#9900ff", "#ff00ff"],
    ["#e6b8af", "#f4cccc", "#fce5cd", "#fff2cc", "#d9ead3", "#d0e0e3", "#c9daf8", "#cfe2f3", "#d9d2e9", "#ead1dc"],
    ["#dd7e6b", "#ea9999", "#f9cb9c", "#ffe599", "#b6d7a8", "#a2c4c9", "#a4c2f4", "#9fc5e8", "#b4a7d6", "#d5a6bd"],
    ["#cc4125", "#e06666", "#f6b26b", "#ffd966", "#93c47d", "#76a5af", "#6d9eeb", "#6fa8dc", "#8e7cc3", "#c27ba0"],
    ["#a61c00", "#cc0000", "#e69138", "#f1c232", "#6aa84f", "#45818e", "#3c78d8", "#3d85c6", "#674ea7", "#a64d79"],
    ["#85200c", "#990000", "#b45f06", "#bf9000", "#38761d", "#134f5c", "#1155cc", "#0b5394", "#351c75", "#741b47"],
    ["#5b0f00", "#660000", "#783f04", "#7f6000", "#274e13", "#0c343d", "#1c4587", "#073763", "#20124d", "#4c1130"]
];

The above color swatch should relatively correspond to the same colors you might see in Google Docs. Next, we need to generate a list of divs that will correspond to the given colors in this swatch.

const colorMap = swatch.flat();

let swatchContainer = document.querySelector('#color-picker');
let colorElements = {};
swatch.forEach(row => {
    let rowElem = document.createElement('div');
    rowElem.classList.add('hstack');
    row.forEach(c => {
        let elem = document.createElement('div');
        elem.classList.add('box');
        elem.style.backgroundColor = c;
        colorElements[c] = elem;
        rowElem.appendChild(elem);
    });

    swatchContainer.appendChild(rowElem);
});

Simple! Now let's go ahead and add a click behavior to this code such that the box will become active/inactive according to the current color.

elem.onclick = function (e) {
    colorPicker.dataset.color = c;
    colorPicker.style.color = c;
    if (colorElements[color]) {
        colorElements[color].classList.remove('active');
    }
    color = c;
    elem.classList.toggle('active');
    e.preventDefault();
};

Recall that our random color was generating a color based on the rgb and used Math.random() to do this. We will replace that code with the following in order to generate a random color within the existing swatch.

function randomColor() {
    return parseInt(Math.random() * colorMap.length);
}

var colorIndex = randomColor();
var color = colorMap[colorIndex];
var colorPicker = document.querySelector('[data-color]');
colorPicker.dataset.color = color;
colorPicker.style.color = color;
colorElements[color].classList.add('active');

Great! Let's get ahead and add the html to correspond to this setup. Ideally, our color picker should behave as a dropdown. Let's add the initial html there.

<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8">
    <title>Let's Draw Together</title>
    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/remixicon@2.5.0/fonts/remixicon.css">
    <link rel="stylesheet" href="/static/index.css">
    <link rel="alternate icon" type="image/png" href="/static/logo.png">
    <link rel="icon" type="image/svg+xml" href="/static/logo.png">
</head>
<body>
    <div class="flush vstack">
        <div class="menubar hstack">
            <a class="icon-link center">
                <i class="ri-lg ri-landscape-line"></i>
            </a>
            <div class="spacer"></div>
            <a class="icon-link active center" data-tool="pencil">
                <i class="ri-lg ri-pencil-fill"></i>
            </a>
            <a class="icon-link center" data-tool="rect">
                <i class="ri-lg ri-shape-line"></i>
            </a>
            <a class="icon-link center" data-tool="circle">
                <i class="ri-lg ri-checkbox-blank-circle-line"></i>
            </a>
            <a class="icon-link center" data-tool="text">
                <i class="ri-lg ri-font-size-2"></i>
            </a>
            <div class="spacer"></div>
            <div class="relative">
                <a class="icon-link center" data-color="#33ffff">
                    <i class="ri-lg ri-palette-line"></i>
                    <i class="ri-lg ri-checkbox-blank-fill center"></i>
                </a>
                <div id="color-picker" class="dropdown vstack">
                </div>
            </div>
            <div class="spacer"></div>
        </div>
        <div class="spacer app">
            <canvas></canvas>
        </div>
    </div>

    <script type="text/javascript" src="/static/load.js"></script>
    <script type="text/javascript" src="/static/draw.js"></script>
</body>
</html>

Notice that we are wrapping the data-color link in a relative class. Let's make sure we have the corresponding classes to handle this.

.relative {
    position: relative;
}

The behavior that I would like to replicate is that the dropdown shows up on hover (either hover in the swatch or hover on this data-color link).

:root {
    /** .... */
    --dropdown-background: #fff;
    --dropdown-shadow:  0px 0px 1px 0px rgba(0,0,0,0.5), 0px 2px 6px -5px rgba(0,0,0,0.75);
}

.dropdown {
    position: absolute;
    background-color: var(--dropdown-background);
    padding: 4px;
    box-shadow: var(--dropdown-shadow);
    border-radius: 4px;
    z-index: 1;
    margin-left: -80px;
    transition: all 0.25s ease-in-out;
}
.icon-link + .dropdown {
    opacity: 0;
    top: 8px;
    visibility: hidden;
    transition: all 0.25s ease-in-out;
}
.icon-link:hover + .dropdown, .dropdown:hover {
    opacity: 1;
    top: 16px;
    visibility: visible;
}
.icon-link:hover + .dropdown *, .dropdown:hover * {
    opacity: 1;
}

Finally, we need our box classes to handle hover/layout.

/** Color Picker */
.box {
    width: 24px;
    height: 24px;
    cursor: pointer;
}
.box:hover, .box.active {
    box-shadow: inset 0px 0px 2px 2px #fff;
    opacity: 0.75;
}

Great! Now we have ourselves a color swatch tool!

Simulating Webkit Force

Recall that if you are in Safari you can use the webkitForce property on the onwebkitmouseforcechanged event to get the current trackpad pressure value. Since this is proprietary we have no real way of accessing this value without either working in Safari (or in Swift for desktop apps). We can however, simulate this kind of value by using a keypress value to increase or decrease a force value. This will somewhat mimic a change as we are moving around our cursor.

var force = 1;
var mouseDown = false;

function move(e) {
    mouseDown = e.buttons;
    /** ... */
}

function key(e) {
    if (e.key === 'Backspace') {
        ctx.clearRect(0, 0, canvas.width, canvas.height);
    }
    if (mouseDown && e.key === 'ArrowUp') {
        force += 0.025;
    }
    if (mouseDown && e.key === 'ArrowDown') {
        force -= 0.025;
    }
}

window.onkeydown = key;

Now whenever we press the key up or down we can change the value while we are pressing down on the mouse!

Changing Colors with Left/Right Arrow Keys

One additional thing I'd like to add is the ability to change the colors on keypress as well. We can simply update the current color index to do this.

function key(e) {
    if (e.key === 'Backspace') {
        ctx.clearRect(0, 0, canvas.width, canvas.height);
    }
    if (e.key === 'ArrowRight') {
        colorIndex++;
    }
    if (e.key === 'ArrowLeft') {
        colorIndex--;
    }
    if (e.key === 'ArrowRight' || e.key === 'ArrowLeft') {
        if (colorIndex >= colorMap.length) {
            colorIndex = 0;
        }
        if (colorIndex < 0) {
            colorIndex = colorMap.length - 1;
        }
        if (colorElements[color]) {
            colorElements[color].classList.remove('active');
        }
        color = colorMap[colorIndex];
        colorPicker.dataset.color = color;
        colorPicker.style.color = color;
        colorElements[color].classList.toggle('active');
    }
    if (mouseDown && e.key === 'ArrowUp') {
        force += 0.025;
    }
    if (mouseDown && e.key === 'ArrowDown') {
        force -= 0.025;
    }
}

What about combining them at the same time? To do this, we'll need to change our keys for force changes to something like SHIFT and ALT. We still want to be able to control with the up/down arrows and we want to limit shift/alt to only when left or right is pressed.

    if (mouseDown && (e.key === 'ArrowUp' || (e.shiftKey && ['ArrowLeft', 'ArrowRight'].includes(e.key)))) {
        force += 0.025;
    }
    if (mouseDown && (e.key === 'ArrowDown' || (e.altKey && ['ArrowLeft', 'ArrowRight'].includes(e.key)))) {
        force -= 0.025;
    }

Awesome! If you liked this article, give me a follow and a like if you are so inclined. Also, feel free to check out my twitter where I post similar and other interesting stuff on physics, code, cs theory and more!

Cheers! 🍻

CODE

If you're interested in the code for this series, check out my repository on GitHub below:

https://github.com/nyxtom/drawing-webrtc

Realtime collaborative drawing (part 2): Server Sent Events + WebRTC Mesh Networks

Tom Holloway 🏕 — Tue, 08 Sep 2020 05:18:00 +0000

In a previous article, we created a realtime collaborative drawing tool with canvas and webrtc. We used simplepeer to get the job done and used WebSocket to communicate to a signaling server.

This works great, but it sort of glosses over the underlying implementation with some added abstractions and complexity that may not be necessary. We can simplify things by using SSE. As well, we'll take a closer look at WebRTC by utilizing the RTCPeerConnection directly.

By the end of this article we'll have:

Randomly generated ids for drawing rooms
Server Sent Events for our communication signaling channel
Fully connected mesh network of WebRTC data channels

Simplifying Server Communication with SSE

We were using WebSockets because we needed a way for the server to trigger callbacks on the clients whenever things like an offer, a peer joined, and any other signaling communication happened. Unfortunately, there are a number of edge cases that need to be handled:

Losing connection state to our server/client
WebSockets might not be supported (by default on load balancers)
Fallbacks to long-polling

This added complexity of WebSockets is such that usually you will just use something like socket.io to handle all these for us. Instead, we can use SSE to handle our communication from the server, and it uses just HTTP to do this.

By using SSE, we can gain the following benefits:

Data efficient, easily understood protocol over HTTP
Automatically multiplexed over HTTP/2
Can use just a single connection
Connections can be moved to a different server easily
No need for complicated load balancer configurations or issues with proxies or firewalls

Rooms and Mesh Networks

Our server functionality was mostly just passing messages along but we want to do a little bit more. We need to be able to orchestrate how our peers join the server. We also want to have a sharable room id so that other people can join. Let's take another look at creating our express server now.

The first thing we need to take care of is routing our users to a unique room. This will ensure that on page load we get our own unique drawing surface and to have others join, we simply need to just share that link.

var express = require('express');
var http = require('http');
var path = require('path');

const app = express();
app.use('/static', express.static(`${__dirname}/static`));

const server = http.createServer(app);

// starting index
app.locals.index = 100000000000;

app.get('/', (req, res) => {
    app.locals.index++;
    let id = app.locals.index.toString(36);
    res.redirect(`/${id}`);
});

app.get('/:roomId', (req, res) => {
    res.sendFile(path.join(__dirname, 'static/index.html'));
});

server.listen(process.env.PORT || 8081, () => {
    console.log(`Started server on port ${server.address().port}`);
});

Then in our static directory we have:

/static/index.html

<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8">
    <title>Let's Draw Together</title>
    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/remixicon@2.5.0/fonts/remixicon.css">
    <link rel="stylesheet" href="/static/index.css">
    <link rel="alternate icon" type="image/png" href="/static/logo.png">
    <link rel="icon" type="image/svg+xml" href="/static/logo.png">
</head>
<body>
    <div class="flush vstack">
        <div class="menubar hstack">
            <a class="icon-link center">
                <i class="ri-lg ri-landscape-line"></i>
            </a>
            <div class="spacer"></div>
            <a class="icon-link active center">
                <i class="ri-lg ri-pencil-fill"></i>
            </a>
            <div class="spacer"></div>
            <a class="icon-link center">
                <i class="ri-lg ri-palette-line"></i>
                <i class="ri-lg ri-checkbox-blank-fill"></i>
            </a>
            <div class="spacer"></div>
        </div>
        <div class="spacer app">
            <canvas></canvas>
        </div>
    </div>
</body>
</html>

/static/index.css

:root {
    --root-font-size: 12px;
    --standard-padding: 16px;

    --bg: #fafafa;
    --fg: #666;
    --menubar-bg: #fdfdfd;
    --active-color: #339999;

    --menubar-shadow: 0 8px 6px -6px #f4f4f4;
}

/** Reset */
html, body, nav, ul, h1, h2, h3, h4, a, canvas {
    margin: 0px;
    padding: 0px;
    color: var(--fg);
}
html, body {
    font-family: Roboto, -apple-system, BlinkMacSystemFont, 'Segoe UI', Oxygen, Ubuntu, Cantarell, 'Open Sans', 'Helvetica Neue', sans-serif;
    font-size: var(--root-font-size);
    background: var(--bg);
    height: 100%;
    width: 100%;
    overflow: hidden;
}
*, body, button, input, select, textarea, canvas {
    text-rendering: optimizeLegibility;
    -webkit-font-smoothing: antialiased;
    -moz-osx-font-smoothing: grayscale;
    outline: 0;
}

/** Utilities */
.hstack {
    display: flex;
    flex-direction: row;
}
.vstack {
    display: flex;
    flex-direction: column;
}
.center {
    display: flex;
    align-items: center;
}
.spacer {
    flex: 1;
}
.flush {
    height: 100%;
}
.icon-link {
    padding: calc(var(--standard-padding) / 2);
    margin: calc(var(--standard-padding) * -1) 0px;
    font-size: 1rem;
    position: relative;
    border-bottom: 2px solid transparent;
    top: 2px;
    cursor: pointer;
}
.icon-link:hover {
    color: var(--active-color);
}
.icon-link.active {
    color: var(--active-color);
    border-bottom: 2px solid var(--active-color);
}

/** Sections */
.menubar {
    padding: var(--standard-padding);
    box-shadow: var(--menubar-shadow);
    background: var(--menubar-bg);
}
.app {
    width: 100%;
}

Connecting to the Server Event Stream

A Server Sent Event Stream, in terms of HTTP, is not much different than a download that never finishes. We simply want to open up a connection to the server and establish this connection as a client that can be written to anywhere in the application. Let's add the code for that portion:

// store the connections from clients here
var clients = {};

function disconnected(client) {
    let index = app.locals.clients.indexOf(client);
    if (index > -1) {
        app.locals.clients.splice(index, 1);
    }
}

app.get('/connect', (req, res) => {
    if (req.headers.accept !== 'text/event-stream') {
        return res.sendStatus(404);
    }

    // write the event stream headers
    res.setHeader('Cache-Control', 'no-cache');
    res.setHeader('Content-Type', 'text/event-stream');
    res.setHeader("Access-Control-Allow-Origin", "*");
    res.flushHeaders();

    // setup a client
    let client = {
        id: uuid.v4(),
        emit: (event, data) => {
            res.write(`id: ${uuid.v4()}`);
            res.write(`event: ${event}`);
            res.write(`data: ${JSON.stringify(data)}\n\n`);
        }
    };

    clients[client.id] = client;

    // emit the connected state
    client.emit('connected', { user: req.user });

    req.on('close', () => {
        disconnected(client);
    });
});

In the above implementation, all we are doing is keeping the response connection around in the clients in order to respond to other messages and relay information from one client to another. To do this, all we have to do is write the headers as a response that we are going to use a text/event-stream and all subsequent writes can take the simple format described below.

Server Sent Events Format

The event stream is a simple stream of text data which must be encoded using UTF-8. Messages in the event stream are separated by a pair of newline characters. A colon as the first character of a line is in essence a comment, and is ignored.

Each message consists of one or more lines of text listing the fields for that message. Each field is represented by the field name, followed by a colon, followed by the text data for that field's value.

Server Sent Events consist of 4 available fields (one per line) separated by a colon. These include:

event

A string identifying the type of event described. If this is specified, an event will be dispatched on the browser to the listener for the specified event name; the website source code should use addEventListener() to listen for named events. The onmessage handler is called if no event name is specified for a message.

data

The data field for the message. When the EventSource receives multiple consecutive lines that begin with data:, it concatenates them, inserting a newline character between each one. Trailing newlines are removed.

The event ID to set the EventSource object's last event ID value

retry

The reconnection time to use when attempting to send the event. This must be an integer, specifying the reconnection time in milliseconds. If a non-integer value is specified, the field is ignored.

event: userconnect
data: {"username": "bobby", "time": "02:33:48"}

event: usermessage
data: {"username": "bobby", "time": "02:34:11", "text": "Hi everyone."}

event: userdisconnect
data: {"username": "bobby", "time": "02:34:23"}

event: usermessage
data: {"username": "sean", "time": "02:34:36", "text": "Bye, bobby."}

JWT (Json Web Tokens)

We need a quick way to identify which requests belong to which user in the website. For this, we're just going to use jwt. It's a quick pass at letting us make sure we have the right user and that subsequent peer requests can be correctly associated with that user.

First, make sure to add it as a dependency to our package.json. You should already have express at this point. Additionally, we're going to setup a .env file to configure a TOKEN_SECRET environment variable. To take advantage of this we will use dotenv.

npm install --save express jsonwebtoken dotenv

In a .env I created a TOKEN_SECRET using the following (you can use any method you like, the below is for the sake of simplicity):

require('crypto').randomBytes(64).toString('hex')

Then paste the result in the .env file

TOKEN_SECRET=569e3cd22e2ff68ef02688c2100204cd29d7ad2520971ad9eea6db1c2be576a666734a4531787448811001a76d63fd5394e1fc8f7083bab7793abead60ba1392

Next, add the following code to make sure we can generate tokens and authenticate them on incoming requests.

var jwt = require('jwt');
var dotenv = require('dotenv');

dotenv.config();

function auth(req, res, next) {
    let token;
    if (req.headers.authorization) {
        token = req.headers.authorization.split(' ')[1];
    } else if (req.query.token) {
        token = req.query.token;
    }
    if (typeof token !== 'string') {
        return res.sendStatus(401);
    }

    jwt.verify(token, process.env.TOKEN_SECRET, (err, user) => {
        if (err) {
            return res.sendStatus(403);
        }
        req.user = user;
        next();
    });
}

app.post('/access', (req, res) => {
    if (!req.body.username) {
        return res.sendStatus(403);
    }
    const user = {
        id: uuid.v4(),
        username: req.body.username
    };

    const token = jwt.sign(user, process.env.TOKEN_SECRET, { expiresIn: '3600s' });
    return res.json(token);
});

Now we have a way to generate auth tokens. In a more realistic scenario, we might decide to see if this authentication method can generate unique tokens according to the logged in user. However, for the time being this is just going to be based on anonymous users. We also have an auth method to verify the incoming token. Let's go ahead and update our /connect function to use our local req.user and make sure it passes through auth function.

app.get('/connect', auth, (req,res) => {
    if (req.headers.accept !== 'text/event-stream') {
        return res.sendStatus(404);
    }

    // write the event stream headers
    res.setHeader('Cache-Control', 'no-cache');
    res.setHeader('Content-Type', 'text/event-stream');
    res.setHeader("Access-Control-Allow-Origin", "*");
    res.flushHeaders();

    // setup a client
    let client = {
        id: req.user.id,
        user: req.user,
        emit: (event, data) => {
            res.write(`id: ${uuid.v4()}`);
            res.write(`event: ${event}`);
            res.write(`data: ${JSON.stringify(data)}`);
        }
    };

    clients[client.id] = client;

    req.on('close', () => {
        disconnected(client);
    });
});

Now all the peer ids will line up with the auth token generated user ids. We'll use this whenever our users actually join a room below.

Joining a room, relaying messages and disconnecting

There are essentially 3 main functions that we care about in this application as far as the server is concerned.

When a user wants to join a room

When we join a room we need to be able to tell all the current clients in that room that a new peer has joined. Additionally, the currently associated client connection needs to communicate with all these existing clients to setup a peer connection by generating an offer.

var channels = {};

app.post('/:roomId/join', auth, (req, res) => {
    let roomId = req.params.roomId;
    if (channels[roomId] && channels[roomId][req.user.id]) {
        return res.sendStatus(200);
    }
    if (!channels[roomId]) {
        channels[roomId] = {};
    }

    for (let peerId in channel) {
        if (clients[peerId] && clients[req.user.id]) {
            clients[peerId].emit('add-peer', { peer: req.user, roomId, offer: false });
            clients[req.user.id].emit('add-peer', { peer: clients[peerId].user, roomId, offer: true });
        }
    }

    channels[roomId][req.user.id] = true;
    return res.sendStatus(200);
});

When a user needs to relay messages to another peer

When a peer-to-peer connection is being established, WebRTC must be able to pass along SDP messages for things like the WebRTC Session, WebRTC Offers, and WebRTC Answers.

This relay information needs to be passed through a signaling server. We're going to simply pass along these messages to whichever intended peer (or peers) the user is requesting to send to.

app.post('/relay/:peerId/:event', auth, (req, res) => {
    let peerId = req.params.peerId;
    if (clients[peerId]) {
        clients[peerId].emit(req.params.event, { peer: req.user, data: req.body });
    }
    return res.sendStatus(200);
});

When a user disconnects entirely from the server

Finally, when a user disconnects from the server we need to clean-up the channels that this user was in. To do this, we're going to update the disconnected function.

function disconnected(client) {
    delete clients[client.id];
    for (let roomId in channels) {
        let channel = channels[roomId];
        if (channel[client.id]) {
            for (let peerId in channel) {
                channel[peerId].emit('remove-peer', { peer: client.user, roomId });
            }
            delete channel[client.id];
        }
        if (Object.keys(channel).length === 0) {
            delete channels[roomId];
        }
    }
}

Setting up the client connection

Now that we have a server that can properly handle communication from our clients, let's go ahead and write our WebRTC library to perform all this communication. In the previous article, we were using simplepeer, however in this article we're going to use the WebRTC api directly instead. This will let us get a bit better handle on what is precisely going on and how you could even do some of this communication manually.

Before we do this, we need to setup a few things like getting the /access token, and setting up the EventSource to stream messages to us.

Add the following to the bottom of our index.html

<script type="text/javascript" src="/static/load.js"></script>

Then, in a new file /static/load.js we need to add the following code to setup the event stream and access token.

var context = {
    username: 'user' + parseInt(Math.random() * 100000),
    roomId: window.location.pathname.substr(1),
    token: null,
    eventSource: null
};

async function getToken() {
    let res = await fetch('/access', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json'
        },
        body: JSON.stringify({
            username: context.username
        })
    });
    let data = await res.json();
    context.token = data.token;
}

async function join() {
    return fetch(`/${context.roomId}/join`, {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
            'Authorization': `Bearer ${context.token}`
        }
    });
}

async function connect() {
    await getToken();
    context.eventSource = new EventSource(`/connect?token=${context.token}`);
    context.eventSource.addEventListener('add-peer', addPeer, false);
    context.eventSource.addEventListener('remove-peer', removePeer, false);
    context.eventSource.addEventListener('session-description', sessionDescription, false);
    context.eventSource.addEventListener('ice-candidate', iceCandidate, false);
    context.eventSource.addEventListener('connected', () => {
        join();
    });
}

function addPeer(data) {}
function removePeer(data) {}
function sessionDescription(data) {}
function iceCandidate(data) {}

connect();

This is nearly all the communication we need to get started! In the above code, we are using the fetch api to make a request to get an access token by providing username in our context.

Once we are done setting up the event source, we can call join which will use the fetch api to POST that we would like to join the current room. If you recall, /:roomId/join will iterate over the clients in a given channel and call add-peer with the newly joined user.id, it will also call add-peer to this client with offer: true in the data.

WebRTC - Setting up a Mesh Network

WebRTC is built using a number of protocols and APIs that work together to achieve the capabilities of capturing and streaming audio/media/data between browsers without an intermediary.

For a full-length guide and overview on how WebRTC works, I suggest looking into WebRTC for the curious as it is actually written by some of the authors of WebRTC

In WebRTC, specifically we are interested in setting up RTCPeerConnection in order to communicate with other members of the network. We will be setting up a peer connection whenever we receive the add-peer message.

const rtcConfig = {
    iceServers: [{
        urls: [
            'stun:stun.l.google.com:19302',
            'stun:global.stun.twilio.com:3478'
        ]
    }]
};

function addPeer(data) {
    let message = JSON.parse(data.data);
    if (context.peers[message.peer.id]) {
        return;
    }

    // setup peer connection
    let peer = new RTCPeerConnection(rtcConfig);
    context.peers[message.peer.id] = peer;

    // handle ice candidate
    peer.onicecandidate = function (event) {
        if (event.candidate) {
            relay(message.peer.id, 'ice-candidate', event.candidate);
        }
    };

    // generate offer if required (on join, this peer will create an offer
    // to every other peer in the network, thus forming a mesh)
    if (message.offer) {
        // create the data channel, map peer updates
        let channel = peer.createDataChannel('updates');
        channel.onmessage = function (event) {
            onPeerData(message.peer.id, event.data);
        };
        context.channels[message.peer.id] = channel;
        createOffer(message.peer.id, peer);
    } else {
        peer.ondatachannel = function (event) {
            context.channels[message.peer.id] = event.channel;
            event.channel.onmessage = function (evt) {
                onPeerData(message.peer.id, evt.data);
            };
        };
    }
}

function broadcast(data) {
    for (let peerId in context.channels) {
        context.channels[peerId].send(data);
    }
}

async function relay(peerId, event, data) {
    await fetch(`/relay/${peerId}/${event}`, {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
            'Authorization': `Bearer ${context.token}`
        },
        body: JSON.stringify(data)
    });
}

async function createOffer(peerId, peer) {
    let offer = await peer.createOffer();
    await peer.setLocalDescription(offer);
    await relay(peerId, 'session-description', offer);
}

This is doing a few things, first we have to actually create an RTCPeerConnection. We are passing along some default ICE/STUN servers to ensure that the ICE/STUN portion of the protocol works successfully as we pass things through the signaling server (our express app). Next, when onicecandidate is called due to the negotiation phase of an offer, it will relay that information along to the peer.

We also are creating the datachannel and subscribing to the messages whenever this happens. We only want to create a new data channel and generate an offer when we are supposed to initiate this portion of the negotiation. Otherwise, we will simply listen for the ondatachannel.

remove-peer, ice-candidate and session-description have to be handled whenever a peer is removed, is initiating candidate or session information. We need to handle ice-candidate and session-description to create answers so that the remote peer can set the localDescription or remoteDescription appropriately.

async function sessionDescription(data) {
    let message = JSON.parse(data.data);
    let peer = context.peers[message.peer.id];

    let remoteDescription = new RTCSessionDescription(message.data);
    await peer.setRemoteDescription(remoteDescription);
    if (remoteDescription.type === 'offer') {
        let answer = await peer.createAnswer();
        await peer.setLocalDescription(answer);
        await relay(message.peer.id, 'session-description', answer);
    }
}

function iceCandidate(data) {
    let message = JSON.parse(data.data);
    let peer = context.peers[message.peer.id];
    peer.addIceCandidate(new RTCIceCandidate(message.data));
}

function removePeer(data) {
    let message = JSON.parse(data.data);
    if (context.peers[message.peer.id]) {
        context.peers[message.peer.id].close();
    }

    delete context.peers[message.peer.id];
}

Notice that in the session-description function we are setting the remoteDescription according to the information provided and we proceed to generate an answer to an offer (if one was provided) before setting our localDescription and relaying that information along. Both the offer and answer provide information in the form of SDP.

Awesome! 🎉 It may not seem like it at first, but we've just create a system for communicating data over UDP using WebRTC Data Channels! If you start up the server with node . and load up the same room id in two different browser windows you should be able to inspect the context.channels.

Realtime Collaborative Drawing

Let's copy over the code from our last article and create a file called /static/draw.js.

const canvas = document.querySelector('canvas');
const ctx = canvas.getContext('2d');

var lastPoint;
var force;

function randomColor() {
    let r = Math.random() * 255;
    let g = Math.random() * 255;
    let b = Math.random() * 255;
    return `rgb(${r}, ${g}, ${b})`;
}

var color = randomColor();
var colorPicker = document.querySelector('[data-color]');
colorPicker.dataset.color = color;
colorPicker.style.color = color;

function resize() {
    canvas.width = window.innerWidth;
    canvas.height = window.innerHeight;
}

function onPeerData(id, data) {
    draw(JSON.parse(data));
}

function draw(data) {
    ctx.beginPath();
    ctx.moveTo(data.lastPoint.x, data.lastPoint.y);
    ctx.lineTo(data.x, data.y);
    ctx.strokeStyle = data.color;
    ctx.lineWidth = Math.pow(data.force || 1, 4) * 2;
    ctx.lineCap = 'round';
    ctx.stroke();
    ctx.closePath();
}

function move(e) {
    if (e.buttons) {
        if (!lastPoint) {
            lastPoint = { x: e.offsetX, y: e.offsetY };
            return;
        }

        draw({
            lastPoint,
            x: e.offsetX,
            y: e.offsetY,
            force: force,
            color: color
        });

        broadcast(JSON.stringify({
            lastPoint,
            x: e.offsetX,
            y: e.offsetY,
            force: force,
            color: color
        }));

        lastPoint = { x: e.offsetX, y: e.offsetY };
    }
}

function up() {
    lastPoint = undefined;
}

function key(e) {
    if (e.key === 'Backspace') {
        ctx.clearRect(0, 0, canvas.width, canvas.height);
    }
}

function forceChanged(e) {
    force = e.webkitForce || 1;
}

window.onresize = resize;
window.onmousemove = move;
window.onmouseup = up;
window.onkeydown = key;

window.onwebkitmouseforcechanged = forceChanged;

resize();

Notice that we are using the onPeerData(id, data) to draw that information to the screen and we are using the /static/load.js function broadcast to broadcast our current drawing information to the room. By the end of this, we now have a fully functioning P2P Mesh Network that uses Server Sent Events as our realtime signaling server.

Conclusion

Phew! We may have done a bit more in this tutorial. We learned about Server Sent Events, we implemented our signaling server and directly used the WebRTC library, we even added support for mesh network peer connections within unique room ids. Now that we have the underlying communication just about squared away, our drawing tool needs a bit of an upgrade in capabilities and tools.

In the next article, we're going to add a few more tools other than the brush and learn a bit about state synchronization. Currently, our drawing state is simply overlayed by executing every operation we receive - but there is nothing that tells us what the state of the drawing surface is when we load the page. We will take a look at how to utilize CRDTs to fix this kind of distributed problem.

In a future article, we will revisit the architecture and add in a PubSub server using Redis. Adding a PubSub server will allow us to create a load balancer and any number of deployed signaling servers with our connections (thus helping us scale).

Cheers! 🍻

If you liked this article, please give me a like and a follow! Also feel free to check out my twitter.

Thanks again! 🏕

CODE

If you're interested in the code for this series, check out my repository on GitHub below:

https://github.com/nyxtom/drawing-webrtc

Thanks again!

Realtime collaborative drawing with canvas and WebRTC (part 1): SimplePeer, WebSockets and Canvas

Tom Holloway 🏕 — Sat, 05 Sep 2020 06:30:07 +0000

This last week I spent some time with my daughter working on a drawing program. I was showing her how computational thinking works by first thinking in terms of breaking down the problem (Problem Decomposition). This makes up one of the four pillars of computational thinking.

Problem Decomposition
Pattern Recognition
Data Representation / Abstractions
Algorithms

Things quickly broke out from there about the kind of fun drawings, emojis, and learning to identify broken behaviors and when to fix them. It's a fun learning exercise if you have any kids, to think of a problem at hand and simply explore it iteratively. You can come up with new ideas on the fly so it makes it quite a playful experience for the little ones.

In any case, I wanted to build on this idea and add in a component for drawing collaboratively using WebRTC. We will be using simplepeer to handle the WebRTC layer as it simplifies the implementation quite a bit. Let's get started!

Setup

First, like all projects, we need to setup to make sure we have a place to draw on the screen as well as have tools to work with. Eventually, we will want the ability to have a tools in a toolbar to select, and be able to select and change properties in a popover. For now, let's setup the boilerplate for the layout.



<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8">
    <title>Map</title>
    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/remixicon@2.5.0/fonts/remixicon.css">
    <link rel="stylesheet" href="index.css">
</head>
<body>
    <div class="flush vstack">
        <div class="menubar hstack">
            <a class="icon-link center">
                <i class="ri-lg ri-landscape-line"></i>
            </a>
            <div class="spacer"></div>
        </div>
        <div class="spacer app">
            <canvas></canvas>
        </div>
    </div>
    <script type="text/javascript" src="draw.js"></script>
</body>
</html>



/** index.css */
:root {
    --root-font-size: 16px;
    --standard-padding: 16px;

    --bg: #fafafa;
    --fg: #666;
    --menubar-bg: #fdfdfd;

    --menubar-shadow: 0 8px 6px -6px #f4f4f4;
}

/** Reset */
html, body, nav, ul, h1, h2, h3, h4, a, canvas {
    margin: 0px;
    padding: 0px;
    color: var(--text-color);
}
html, body {
    font-family: Roboto, -apple-system, BlinkMacSystemFont, 'Segoe UI', Oxygen, Ubuntu, Cantarell, 'Open Sans', 'Helvetica Neue', sans-serif;
    font-size: var(--root-font-size);
    background: var(--bg);
    height: 100%;
    width: 100%;
    overflow: hidden;
}
*, body, button, input, select, textarea, canvas {
    text-rendering: optimizeLegibility;
    -webkit-font-smoothing: antialiased;
    -moz-osx-font-smoothing: grayscale;
    outline: 0;
}

/** Utilities */
.hstack {
    display: flex;
    flex-direction: row;
}
.vstack {
    display: flex;
    flex-direction: column;
}
.center {
    display: flex;
    align-items: center;
}
.spacer {
    flex: 1;
}
.flush {
    height: 100%;
}
.icon-link {
    margin: 0px var(--standard-padding);
    font-size: 1rem;
}

/** Sections */
.menubar {
    padding: var(--standard-padding);
    box-shadow: var(--menubar-shadow);
    background: var(--menubar-bg);
}
.app {
    width: calc(100% - var(--sidebar-width));
}

Note that the above utilities I've added are basic flexbox properties. I just want to be able to lay things out in rows and columns with a simple spacer. I named these hstack, vstack, spacer, and a flush for maximizing height.

Icon Sets with RemixIcon

Additionally, I'm making use of remix icons. It's free / open-source / for commercial and personal use. You can reference it via CDN and the icons themselves are very minimalist while providing some customization on sizing. Very handy!

Drawing Setup

If you took a look at my Drawing interactive graphs with Canvas article, then this code will be very similar to that.



const canvas = document.querySelector('canvas');
const context = canvas.getContext('2d');

var nodes = [];

function resize() {
    canvas.width = window.innerWidth;
    canvas.height = window.innerHeight;
    draw();
}

function draw() {
    context.clearRect(0, 0, canvas.width, canvas.height);
}

window.onresize = resize;
resize();

Great! Our app doesn't do much of anything yet. Let's add some tools that can switch the context around.

Drawing with Shapes

If we're going to draw anything on the screen, we're going to need some kind of brush to do it with. Since we don't have actual paint or pencil particles then we have to make our own "particles" by repeatedly drawing a shape. Let's see what that approach does with the following:



function move(e) {
    if (e.buttons) {
        context.fillStyle = 'green';
        context.beginPath();
        context.arc(e.x, e.y, 
    }
}
window.onmousemove = move;

Here we are creating a new path each time we call beginPath - this will empty the list of subpaths and start a new path in the render context. When we use offsetX and offsetY rather than e.x and e.y due to the fact that our canvas is within an offset element node in the document.

Notice however, that moving the mouse here causes gaps between the mouse events. We actually want a path between these points instead. To do that, we need to keep around the last point and draw a line. Alternatively, we can choose to interpolate the distance between these points and draw many circles in between (this complicates things a bit since now the number of arcs we draw is dependent on the resolution in the steps between points). Instead, let's just use a line approach with a lineCap.



function move(e) {
    if (e.buttons) {
        if (!lastPoint) {
            lastPoint = { x: e.offsetX, y: e.offsetY };
            return;
        }
        context.beginPath();
        context.moveTo(lastPoint.x, lastPoint.y);
        context.lineTo(e.offsetX, e.offsetY);
        context.strokeStyle = 'green';
        context.lineWidth = 5;
        context.lineCap = 'round';
        context.stroke();
        lastPoint = { x: e.offsetX, y: e.offsetY };
    }
}

function key(e) {
    if (e.key === 'Backspace') {
        context.clearRect(0, 0, canvas.width, canvas.height);
    }
}

window.onkeydown = key;

Now we can clear the screen with backspace and the gaps are no longer there because we are drawing paths between the points where the mouse move events occur.

Force / Pressure Sensitivity

I've actually found that you can hook into a Safari only webkitmouseforcechanged event to handle the pressure sensitivity of the mouse. This also works for pointermove. Unfortunate for us, the pressure values and webkitForce are only populated and change to the proper sensitivity on mobile devices and in Safari. In any case, if you open up the app in Safari on desktop and you have a force trackpad you can do this!



var currentForce = 1;

function force(e) {
    currentForce = e.webkitForce || 1;
}

function move(e) {
    if (e.buttons) {
        if (!lastPoint) {
            lastPoint = { x: e.offsetX, y: e.offsetY };
            return;
        }
        context.beginPath();
        context.moveTo(lastPoint.x, lastPoint.y);
        context.lineTo(e.offsetX, e.offsetY);
        context.strokeStyle = 'green';
        context.lineWidth = Math.pow(currentForce, 4) * 2;
        context.lineCap = 'round';
        context.stroke();
        lastPoint = { x: e.offsetX, y: e.offsetY };
    }
}

window.onwebkitmouseforcechanged = force;

Synchronizing State

So far we haven't done much in the way of realtime drawing with other people. As noted in one of my articles on CRDTs, the two approaches to take for synchronization is either:

State based synchronization (with CRDTs)
Op based sychronization (with CRDTs or Operation Transforms)

We're going to instead, stream over each change that is being made through a buffer of changes. On a regular interval we can batch this buffer over the network to the peers in order to update the local state.

If you're interested in learning about how WebRTC works, check out this book, WebRTC for the Curious. It was written by some of the original authors of WebRTC internals! Very helpful for understanding just how much WebRTC does beneath the api.

Setting up a WebSocket Server

In order to negotiate our peers we need to pass along the signals, offers, and connection information through a server. We're going to be using express, http, and ws for the WebSocket library. We want our server to accomplish the following:

Accept incoming connections
Broadcast available connections
Handle RTC handshakes for offers, answers, ice-candidates, hang-ups

First, move the contents of our index.html, draw.js, index.css and related public files to a new folder under /static. Then create a new file called index.js at the root. Run the following command to initialize the node project.



npm init -y

You should see the following output.



{
  "name": "map",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}

Now, you'll need a few dependencies for our project. Run:



npm install --save ws express uuid

That should save to package.json. Now we just need to setup our server to respond to web socket connections and serve our static content out of /static. Update index.js to include the following:



var express = require('express');
var http = require('http');
var ws = require('ws');
var uuid = require('uuid');

const app = express();
app.use(express.static(`${__dirname}/static`));
app.locals.connections = [];

const server = http.createServer(app);
const wss = new ws.Server({ server });

function broadcastConnections() {
    let ids = app.locals.connections.map(c => c._connId);
    app.locals.connections.forEach(c => {
        c.send(JSON.stringify({ type: 'ids', ids }));
    });
}

wss.on('connection', (ws) => {
    app.locals.connections.push(ws);
    ws._connId = `conn-${uuid.v4()}`;

    // send the local id for the connection
    ws.send(JSON.stringify({ type: 'connection', id: ws._connId }));

    // send the list of connection ids
    broadcastConnections();

    ws.on('close', () => {
        let index = app.locals.connections.indexOf(ws);
        app.locals.connections.splice(index, 1);

        // send the list of connection ids
        broadcastConnections();
    });

    ws.on('message', (message) => {
        for (let i = 0; i < app.locals.connections.length; i++) {
            if (app.locals.connections[i] !== ws) {
                app.locals.connections[i].send(message);
            }
        }
    });

});

app.get('/', (req, res) => {
    res.sendFile(path.join(__dirname, 'static/index.html'));
});

server.listen(process.env.PORT || 8081, () => {
    console.log(`Started server on port ${server.address().port}`);
});

In the above code, we want to setup a new http server wrapping the express app. Then we setup a WebSocket server wrapping the http server. When the WebSocket server receives a new connection, we need to push that connection to the local list and assign it a unique id to reference later.

Whenever that connection closes, we need to clean up the connection list and send out the list of available connections to the current list. We send that list of connections to the incoming connection to let them know whomever is connected. Finally, whenever we receive a message we are simply going to just broadcast that message to everyone else. It's not overly complex here, I just wanted to broadcast to make it easier.

You'll also notice the app.get route. I use that to simply make sure to render the default index.html for that route.

Connecting to WebSocket

Now that we have a WebSocket server setup over express, we can connect to that quite quickly with the following code. Add this to a new file called data.js. Add it as a script reference to our index.html at the bottom after data.js.



<script type="text/javascript" src="/data.js"></script>



const wsConnection = new WebSocket('ws:127.0.0.1:8081', 'json');
wsConnection.onopen = (e) => {
    console.log(`wsConnection open to 127.0.0.1:8081`, e);
};
wsConnection.onerror = (e) => {
    console.error(`wsConnection error `, e);
};
wsConnection.onmessage = (e) => {
    console.log(JSON.parse(e.data));
};

Great! Now we've got a list of ids that have connected. You can open up this same thing in another browser window and you should see 2 connection ids. You can easily test whether our WebSocket server is broadcasting every message by typing the following in the console.



wsConnection.send(JSON.stringify({ type: 'test', msg: 'hello world' }));

WebRTC RTCPeerConnection

Now that we've got a mechanism for broadcasting messages over WebSockets, we just need to setup a WebRTC RTCPeerConnection. For this, I've chosen simplepeer. It simplifies the underlying api quite a bit and it even works server-side as well if you want to establish the server as a peer wtrc. Let's update our data.js file to include our peer setup.

Add the following to our index.html to include simplepeer:



<script src="https://unpkg.com/simple-peer@9.7.2/simplepeer.min.js"></script>

We need to store a few local variables for whenever we first connect, the local peer connection ids, and the peer connections themselves. For now, we aren't going to worry about implementing full mesh connectivity and we'll just do a single initiator broadcast.



var localId, peerIds;
var peerConnections = {};
var initiator = false;

wsConnection.onmessage = (e) => {
    let data = JSON.parse(e.data);
    switch (data.type) {
        case 'connection':
            localId = data.id;
            break;
        case 'ids':
            peerIds = data.ids;
            connect();
            break;
        case 'signal':
            signal(data.id, data.data);
            break;
    }
};

function onPeerData(id, data) {
    console.log(`data from ${id}`, data);
}

function connect() {
    // cleanup peer connections not in peer ids
    Object.keys(peerConnections).forEach(id => {
        if (!peerIds.includes(id)) {
            peerConnections[id].destroy();
            delete peerConnections[id];
        }
    });
    if (peerIds.length === 1) {
        initiator = true;
    }
    peerIds.forEach(id => {
        if (id === localId || peerConnections[id]) {
            return;
        }

        let peer = new SimplePeer({
            initiator: initiator
        });
        peer.on('error', console.error);
        peer.on('signal', data => {
            wsConnection.send(JSON.stringify({
                type: 'signal',
                id: localId,
                data
            }));
        });
        peer.on('data', (data) => onPeerData(id, data));
        peerConnections[id] = peer;
    });
}

function signal(id, data) {
    if (peerConnections[id]) {
        peerConnections[id].signal(data);
    }
}

Great! Now we've setup a way for peers to communicate with one another. A lot is going on here underneath the hood with WebRTC, but the gist of it is this:

First User Joins
1. When the page first loads, the WebSocket will connect to the WebSocket Server. The WebSocket Server receives the connection and returns that connection's unique id.
2. Immediately after this the WebSocket Server will broadcast the connection ids to the client.
3. Since peerIds.length === 1, then we are the first to join! This means we will act as the initiator. Finally, since we are the first and only user, no peer connections can be setup yet.
Second User Joins
1. When the next user joins, the same flow will work for this user by getting its connection id. User 1 will receive the list of ids for this new connection as well as this user.
First User Receives Updated Ids
1. A new user has joined and as such we can now initiate a peer connection. When the signal is available (i.e. an RTCPeerConnection Offer can be created), we will transmit the offer over the web socket for the other user to receive it.
Second User Receives Offer
1. The second user will have a local RTCPeerConnection already ready to setup and ready to answer offers once they have been initiated. A web socket message will be received here in the form of signal and simplepeer will go ahead and setRemoteDescription prior to createAnswer.
First User Receives Answer
1. Once the second user has sent over the answer for the initial offer, simplepeer can complete the negotiation and begin transmitting whatever information over the data channel or video/voice if we're using the tracks.

You can test out whether things are working by opening up two separate browser windows after starting up the web server with node ..

Transmitting Draw Information

The only thing left we have to do is transmit our draw data. To do this we simply need to update our move function to additionally broadcast, and the onPeerData function will need to actually draw the result of the message to the canvas. Let's go ahead and do that now.



function broadcast(data) {
    Object.values(peerConnections).forEach(peer => {
        peer.send(data);
    });
}

function onPeerData(id, data) {
    draw(JSON.parse(data));
}

function draw(data) {
    context.beginPath();
    context.moveTo(data.lastPoint.x, data.lastPoint.y);
    context.lineTo(data.x, data.y);
    context.strokeStyle = data.color;
    context.lineWidth = Math.pow(data.force || 1, 4) * 2;
    context.lineCap = 'round';
    context.stroke();
}

function move(e) {
    if (e.buttons) {
        if (!lastPoint) {
            lastPoint = { x: e.offsetX, y: e.offsetY };
            return;
        }

        draw({
            lastPoint,
            x: e.offsetX,
            y: e.offsetY,
            force: force,
            color: color || 'green'
        });

        broadcast(JSON.stringify({
            lastPoint,
            x: e.offsetX,
            y: e.offsetY,
            color: color || 'green',
            force: force
        }));

        lastPoint = { x: e.offsetX, y: e.offsetY };
    }
}

That's it! Let's add a bit of additional flavor by randomizing our color to distinguish between the peers.



function randomColor() {
    let r = Math.random() * 255;
    let g = Math.random() * 255;
    let b = Math.random() * 255;
    return `rgb(${r}, ${g}, ${b})`;
}

var color = randomColor();

Awesome! 🎉

If you liked this article, please give me a follow and a like. Also feel free to check out my twitter if you are so inclined. Thanks again!

Cheers! 🍻

CODE

If you're interested in the code for this series, check out my repository on GitHub below:

https://github.com/nyxtom/drawing-webrtc

Thanks again!

Introduction to CRDTs for Realtime Collaboration

Tom Holloway 🏕 — Wed, 02 Sep 2020 04:36:37 +0000

Cover Photo Credit: @gritzko in Chronofold

Any system that has to maintain some kind of state eventually runs into the problem of data consistency. You might have multiple users trying to edit the same document, from a variety of devices, all of which have varying amounts of latency and connection issues. Data consistency is hard. Resiliency means being able to handle these kind of concurrency and availability issues. Approaches to this usually take the form of ACID guarantees, BASE (basically available, eventually consistent) and in the case of CRDT strong eventual consistency. Let's take the example of a realtime collaborative editor.

In the above photo, there are 3 different sites where changes are sent and received. Elliot denotes Site 1, Kate is Site 2 and Nat is Site 3. Within Elliot's local document, he types some characters. This change is propagated to Kate and Nat. After which they individually make local changes to their document. Before the changes from Kate and Nat are propagated back, Elliot makes additional changes. The resulting state is the merging of all the changes that have been made.

This is where the result converges deterministically to the same result in every site (despite that there is latency, operations can come out of order, and there is no central server that dictates the source of truth). This is the basis of what Conflict-Free Replicated Data Types can provide.

Strong Eventual Consistency

A few definitions:

Site

A network is comprised of sites which include "devices", "peers" operating in parallel, each one producing operations and exchange information with other sites
Operations

Events or actions that mutate data
Causal Relation

An operation is caused by another operation when it directly modifies or otherwise involves the results of that operation. The ordering by causal relationship determines a linear timeline of operations across the network.
Concurrent

If two operations don't have a causal relationship-if they were created simultaneously on different sites without knowledge of each other-they are said to be concurrent
Commutativity

A property of the entire system where changing the order of operations does not change the result. This is a fundamental requirement because due to network latency and other issues (offline/connectivity) an editor may not receive the operations or send them until later. We need to be able to guarantee that we get the same result when this happens. Ex: (3+4 = 4+3).
Strong Eventual Consistency

Eventually consistent services are often classified as providing (Basically Available, Soft state, Eventual Consistency) in contrast to ACID (Atomicity, Consistency, Isolation, Durability) or rather stating that updates will be observed eventually. Strong eventual consistency has a safety guarantee that any two nodes that have received the same set of updates will deterministically be in the same state. Monotonic systems here also will never rollback.
Total Order

State and sequence of operations is guaranteed to be identical on all sites
Partial Order

State and sequence of operations is partially guaranteed to be identical on all sites due to logical timestamping or timestamping except in the case of concurrency and the dependency of in what order operations arrive
Associative

Rearranging parens in an expression does not change the result / order in which operations are performed does not matter as long as the sequence of the operands is not changed.
Idempotent

Operations can be applied multiple times without changing the result beyond the initial application.
Tombstones

A placeholder value that denotes a delete operation for a corresponding piece of data. Typically tombstones are something that come up when using CRDT with realtime editors. Trade-offs are made around absolute mathematically convergence and limiting offline modes to garbage collect old tombstones that have limited concurrency issues in order to reduce document space complexity.

How do we get events to be in the right order?

Getting events/actions/operations to be in the right order means we need some kind of scheme for timing. We have several ways to do this:

Server Timestamp

We can use the server's timestamp in a typical architecture to serve as the authoritative clock but this isn't available in distributed systems. Furthermore, if we tried to use the different timestamps that are on each of the different nodes we also run into the problem of clock drift.

Clock drifting occurs where a clock does not run at exactly the same rate as a reference clock. It gradually drifts apart or desynchronizes from the other clock. If you've ever had an issue with a remote linux server being drastically off on its timestamps, you need to use ntpdate and ntp to measure this clock drift and synchronization to a reference server.

Logical Timestamps

Time, Clocks, and the Ordering of Events in a Distributed System

Leslie Lamport, widely considered to lay the foundation for his work in the theory of distributed systems, published a marvel of work in 1978 on time.

The concept of one event happening before another in a distributed system is examined, and is shown to define a partial ordering of the events. A distributed algorithm is given for synchronizing a system of logical clocks which can be used to totally order the events.

This paper describes what is known as happened before as a way of describing a partial ordering of the events in the system. In that, an event a happens before an event b if a happened at an earlier time than b. Following the fact that real clocks can't be reliable or necessarily used, then happened before has to be defined without using physical clocks.

In the above example, we have 3 processes. P0, P1, P2. We know that we have no global clock in this scenario. Instead all we have are messages that being passed around. A logical clock will increment a local timestamp based on whether it is sending or receiving using the following rules:

Sending: timestamp + 1
Receiving: Max(local timestamp, sender timestamp) + 1

All processes start at 0 as the root timestamp. In this example, P1 will increment to 1 when it sends to P0. P0 receives this message and its local timestamp is still 0. Since it receives a message it takes the max of these values Max(0, 1) + 1 which would be 2.

P1 sends another message to P2 and does so by incrementing its now local timestamp again from 1 to 2. P2 receives this message and must take the Max(0, 2) + 1 which is 3. P2 sends a message to P1 and must increment its local timestamp from 3 to 4. P1 receives this message and takes the Max(2, 4) + 1 which is 5. Doing this kind of ordering helps us get closer to total ordering.

Event Logs

In enterprise circles, this is typically called event sourcing, where every operation is stored as data and playback is used when you need to construct the model.

The fundamental idea of Event Sourcing is that of ensuring every change to the state of an application is captured in an event object, and that these event objects are themselves stored in the sequence they were applied for the same lifetime as the application state itself.

Given that we have a complete timeline of all the operations that ever occurred in the state, we can replay/query/rebuild all or part of the compute history. To give a log total ordering we need to sort operations by time and then by unique origin.

Conflict-Free Replicated Data Types

Given that the focus of our problem is how to prevent inconsistencies between different sites, we need an approach that makes it mathematically always possible to merge and resolve concurrent updates without conflicts. CRDTs provide us with strong eventual consistency and there are two approaches:

State-based CRDTs (CvRDTs)

State based CRDTs send their full local state to other sites. Merging is done by a function which must be commutative, associative, and it must be idempotent. An update also must monotonically increase the internal state, which is to say that the state always grows, even with deletions.

A good example of CvRDTs is in gossip protocols where the state of a replica is propagated and merged with other replicas when topologies change and help reduce network use when these states change.

Operation-based CRDTs (CmRDTs)

Operation based CRDTs, as the name implies, only require sites to exchange the operations themselves. Operations are commutative, but they are not necessarily idempotent. This means that infrastructure must take into account that all operations on a site are delivered without duplication, but can be in any order. Because a deletion is just a placeholder (called a tombstone), an operation can always commute. This means something like the example in the figure where Site 1 deletes MD still translates fine to changes made on other sites since the characters are never really gone.

Unique Identifiers

In almost every implementation of CRDT with respect to text, every character in the document has a unique identifier. Conflicts due to merging are all resolved by this unique identifier plus a logical timestamp per character (usually a Lamport Timestamp).

Questions on implementations and use cases

Having read through much of the literature, it's clear that there are a series of questions that need to be asked in order to properly define what kind of implementation should be selected. These questions might include:

How do you represent the data in the form of CRDTs
What does space and time complexity look like for insert, delete, update operations to those structures
How is state represented, how are operations (in the sense of events/actions as noted above) represented and stored
What guarantee tradeoffs are made based on the implementation chosen?
Do operations like deletions leave behind placeholders for deletions or avoid tombstones in favor of reduced space complexity while trading off for less guarantees around convergence due to interleaving concurrent edits.
What about garbage collection for old versions and edits where tombstones are no longer referenced? In the case of garbage collection, you might decide that after 1 week you might safely say that those edits will not run into concurrency problems due to additional edit/merges. (Some editors resolve this problem entirely by not allowing offline editing)

But the most important question you should be asking:

What are actual real-word performance and usability characteristics based on the intended usage of the application?

Figma

Consider how Figma makes these tradeoffs by combining multiple CRDT-inspired techniques (associativity, commutativity, idempotent) with a last-write-wins (LWW) approach and a central server to coordinate some aspects of conflicts and updates.

Other things like tree re-ordering in the document, undo/redo behaviors, are resolved with domain specific desired implementations. Additionally, Figma side-steps the entire realtime text interleaving/merging issue due to the fact that the text content of a text element is just another property on an object. As such, changes in this system still follow the LWW model they have gone with.

Additionally, they go over the Fractional Indexing approach that looks very much like the approach seen in Logoot/LSEQ.

Given that you have no record of tombstones (like in the case of Logoot/LSEQ) then you run into the problem of interleaving edits on merge. However, Figma can make this trade-off due to the fact that it is a design tool not necessarily centered around realtime text editing.

Known CRDT Types

At the end of the day, what we are ultimately working with is a set of shared data structures. We need a way to represent our model of interaction / application through the structures that help provide us with the mathematical guarantees and trade-offs all mentioned above. Some of the known types include:

G-Set: Grow-Only Set

A set which only allows adds. An element, once added, cannot be removed. Merger of two sets is a union. Operations or complete state merges can utilize this.
2P-Set (Two-Phase Set)

Two G-Sets are combined to create a 2P-Set. With the addition of a remove set (tombstone set), elements can be added and removed. Once removed, an element cannot be re-added; that is, once an element e is in the tombstone set, lookups will not return true for that element. 2P-Set uses remove wins so remove takes precedence over add.
LWW-Element-Set (Last Write Wins Element Set)

Similar to 2P-Set in that it has an add set and remove set, with a timestamp for each element. Elements added are inserted to add set withwh a timestamp, and removes are added to the remove set with a timestamp.
An element is a member if it is in the add set AND (not in the remove set or in the remove set but with an earlier timestamp than the one in the add set).
Merging is a union of the add sets and union of remove sets. When timestamps are equal, bias can be towards either add or remove.
Advantage over 2P-Set is that elements can be reinserted after removal.

Note on LWW Sets + Garbage Collection

There is a trick that Roshi (a distributed LWW-Set cluster built on top of Redis) that you can add which is garbage collecting on inserts. The trick is to garbage collect whatever item is in the existing add set or remove set depending on the incoming timestamp.

When inserting an element E to the logical set S, check if E is already in the add set A or the remove set R. If so, check the existing timestamp. If the existing timestamp is lower than the incoming timestamp, the write succeeds: remove the existing (element, timestamp) tuple from whichever set it was found in, and add the incoming (element, timestamp) tuple to the add set A. If the existing timestamp is higher than the incoming timestamp, the write is a no-op.

OR-Set (Observed-Remove Set) > Similar to LWW-Element-Set except uses tags instead of timestamps. For each element in the set, a list of add-tags and a list of remove-tags are maintained. Insertions are done by generating a unique tag and added to the add-tag list for the element. Elements are removed by having all the tags in the element's add-tag list added to the element's remove tag set (tombstone list). > Merging is done by the union of the add-tag list for each element, and the union of remove-tag lists for each element. > An element is a member if the element's add tag list - remove tag list is nonempty.

Sequence CRDTs

Ordered sequence, list, or even trees are considered part of the family of sequence CRDT implementations. There are a number of algorithms here that are typically used in collaborative editors as an alternative to Operational Transforms. Among the papers include:

WOOT

Seminal work for CRDT beginnings in 2006 assigning every character a unique identifier. WOOT also represents text as a directed acyclic graph of characters - referencing left and right neighbors at insertion. Order of identifiers resolves between concurrent insertions to the same location. Deletions are marked as tombstones. Overhead in metadata made this implementation impractical however. Worth reading about though!
Logoot/LSEQ
TreeDoc

Logoot/LSEQ does not store tombstone deletions and instead takes an approach with fractional / unbounded divisible indexing. This, as discussed earlier, can cause issues with interleaving edits so it becomes impractical to guarantee proper sane convergence with concurrent operations.
RGA

Widely considered to be one of the fastest CRDT implementations. Utilizes a timestamped insertion tree and references hash table lookups for character lookups. Causal trees are a similar approach (named CT but in RGA it's called a timestamped insertion tree). In any case, CT/RGA algorithms have been proven to use the same algorithm
Conclave

Similar non-academic implementation with optimizations and tweaks - based on Logoot/LSEQ.
YATA, YJS

Takes a unique approach by defining rules and behaviors to ensure conflicts are resolved by following the operational constraints laid out in the paper. This includes defining an abstraction of what a type and its operations should look like while building on a simple linked list data structure.
Paper outlines definitions, assumptions, and describing convergence in addition to extensible data types with later examples in visual graphs.
When implemented as in YJS, it outperforms RGA and looks to be the fastest and lower space complexity in a number of benchmarks
Chronofold

Chronofold is a replicated data structure for versioned text. It is designed for use in collaborative editors and revision control systems. Past models of this kind either retrofitted local linear orders to a distributed system (the OT approach) or employed distributed data models locally (the CRDT approach). That caused either extreme fragility in a distributed setting or egregious overheads in local use. Overall, that local/distributed impedance mismatch is cognitively taxing and causes lots of complexity. We solve that by using subjective linear orders locally at each replica, while inter-replica communication uses a distributed model. A separate translation layer insulates local data structures from the distributed environment. We modify the Lamport timestamping scheme to make that translation as trivial as possible. We believe our approach has applications beyond the domain of collaborative editing.

Conclusion

There is a lot of literature out there. Having spent the time to look through these solutions the best one I've come across so far is YJS. I interacted for a bit with the author over twitter and the author has a series of benchmarks designed around actual real-world editing/conflicts and other specific scenarios that are ideal for comparing implementations.

Even still, these solutions are not a catch all and they all have pitfalls and bottlenecks/edge cases that haven't been fully sorted out. Splicing for instance remains a problem in CRDT and there doesn't seem to be a decent solution for it just yet.

In any case, I learned quite a bit and hopefully you did/will to!

Cheers! 🍻

For additional resources on CRDTs:

If you liked this article and are so inclined, give me a like and a follow. Also check out my twitter for similar and other updates!

Thanks again!

Drawing interactive graphs with canvas and javascript

Tom Holloway 🏕 — Mon, 31 Aug 2020 21:15:00 +0000

At my work we monitor network operations and infrastructure through a variety of tools like SNMP, NetFlow, Syslog...etc. One of the ways to help customers figure out what is going on in their networks is to visualize it through graphs! There are a number of great libraries to do this but the main one that I use quite often is d3.js.

But this isn't a post about d3 (that's for another day), it's about utilizing Canvas to draw stuff on the screen. More specifically, we want to draw a series of connected nodes in a graph and be able to drag these nodes around. Let's get started!

Drawing nodes

First thing we will need to do is setup our canvas.



<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8">
    <title>Map</title>
    <link rel="stylesheet" href="index.css">
    <script defer type="text/javascript" src="load.js"></script>
</head>
<body>
    <canvas></canvas>
</body>
</html>



/** index.css */
:root {
    --root-font-size: 12px;
    --bg: #fafafa;
}

/** Reset */
html, body, nav, ul, h1, h2, h3, h4, a, canvas {
    margin: 0px;
    padding: 0px;
    color: var(--text-color);
}
html, body {
    font-family: Roboto, -apple-system, BlinkMacSystemFont, 'Segoe UI', Oxygen, Ubuntu, Cantarell, 'Open Sans', 'Helvetica Neue', sans-serif;
    font-size: var(--root-font-size);
    background: var(--bg);
    height: 100%;
    width: 100%;
    overflow: hidden;
}
*, body, button, input, select, textarea, canvas {
    text-rendering: optimizeLegibility;
    -webkit-font-smoothing: antialiased;
    -moz-osx-font-smoothing: grayscale;
    outline: 0;
}

And now our javascript ⬇️ We're going to start off by keeping around an array of nodes that we want to draw. A node will consist of an x, y, radius, fill, stroke. These properties will correspond to canvas api methods when we go to draw them.



const canvas = document.querySelector('canvas');
const context = canvas.getContext('2d');

var nodes = [];

function resize() {
    canvas.width = window.innerWidth;
    canvas.height = window.innerHeight;
}

window.onresize = resize;
resize();

Let's go ahead and add our drawNode function right now. We're going to use the arc function to draw at a point, radius and angles for the circle. We also manipulate the rendering context for the fill, stroke. Since we are generating a circle with the arc, we want the entire shape to be encapsulated in a Path this is why we are using the beginPath function.



function drawNode(node) {
    context.beginPath();
    context.fillStyle = node.fillStyle;
    context.arc(node.x, node.y, node.radius, 0, Math.PI * 2, true);
    context.strokeStyle = node.strokeStyle;
    context.stroke();
    context.fill();
}

Mouse Functions

Since we want this to be interactive, let's add the ability to track when the user touches or clicks on the canvas and draw the node right there at the cursor position.



function click(e) {
    let node = {
        x: e.x,
        y: e.y,
        radius: 10,
        fillStyle: '#22cccc',
        strokeStyle: '#009999'
    };
    nodes.push(node);
    drawNode(node);
}

window.onclick = click;

Great! Now we have some nodes drawn to the screen but we don't have any way to move them around. Let's take advantage of the target position on the mouseDown function so we can move things around with mouseMove.



var selection = undefined;

function within(x, y) {
    return nodes.find(n => {
        return x > (n.x - n.radius) && 
            y > (n.y - n.radius) &&
            x < (n.x + n.radius) &&
            y < (n.y + n.radius);
    });
}

function move(e) {
    if (selection) {
        selection.x = e.x;
        selection.y = e.y;
        drawNode(selection);
    }
}

function down(e) {
    let target = within(e.x, e.y);
    if (target) {
        selection = target;
    }
}

function up(e) {
    selection = undefined;
}

window.onmousemove = move;
window.onmousedown = down;
window.onmouseup = up;

Bug Fixes

Dragging causes the nodes to be rendered over and over

Uh oh! We need to fix this so we re-render all the nodes whenever this happens. To do this, we just need to add a bit of clearRect to the draw code and instead of drawNode we'll just call it draw.



function click(e) {
    let node = {
        x: e.x,
        y: e.y,
        radius: 10,
        fillStyle: '#22cccc',
        strokeStyle: '#009999'
    };
    nodes.push(node);
    draw();
}

function move(e) {
    if (selection) {
        selection.x = e.x;
        selection.y = e.y;
        draw();
    }
}

function draw() {
    context.clearRect(0, 0, window.innerWidth, window.innerHeight);
    for (let i = 0; i < nodes.length; i++) {
        let node = nodes[i];
        context.beginPath();
        context.fillStyle = node.fillStyle;
        context.arc(node.x, node.y, node.radius, 0, Math.PI * 2, true);
        context.strokeStyle = node.strokeStyle;
        context.fill();
        context.stroke();
    }
}

Clicking and dragging can create a duplicate node

This works pretty well but the problem is if we click too quickly the nodes will appear when we mousedown and then move. Let's instead rely on the move event to clear the state when we want to create a new node.

We will get rid of the window.onclick and click code and instead rely on the mousedown, mouseup, mousemove events to handle selection vs create states. When the mouseup event occurs, if nothing is selected and it hasn't yet been moved then create a new node.



/** remove the onclick code and update move and up code */
function move(e) {
    if (selection) {
        selection.x = e.x;
        selection.y = e.y;
        selection.moving = true;
        draw();
    }
}

function up(e) {
    if (!selection || !selection.moving) {
        let node = {
            x: e.x,
            y: e.y,
            radius: 10,
            fillStyle: '#22cccc',
            strokeStyle: '#009999',
            selectedFill: '#88aaaa'
        };
        nodes.push(node);
        draw();
    }
    if (selection) {
        delete selection.moving;
        delete selection.selected;
    }
    selection = undefined;
    draw();
}

Great! Note, if you update the draw code to key off of the selected state you can change the fill like so:



context.fillStyle = node.selected ? node.selectedFill : node.fillStyle;

Adding Connections

The next thing we are going to do is at some edges to this graph. We want to be able to connect a line from one node to another. To do this, we're going to use a simple line for now and have an edges array defining these connections.

The behavior we want to accomplish is:

mousemove, if there is a selection and the mouse is currently down ➡️ update selection x and y
mousedown, find the node target, if there is a selection clear the selected state, then assign the selection to the target and set its selected state and draw
mouseup, if there is no selection then create a new node and draw, otherwise if the current selection is not selected (because of mouse down) then clear the selection and draw after
additionally mousedown when the selection changes to a new node and we have something already selected we can create an edge



function move(e) {
    if (selection && e.buttons) {
        selection.x = e.x;
        selection.y = e.y;
        draw();
    }
}

function down(e) {
    let target = within(e.x, e.y);
    if (selection && selection.selected) {
        selection.selected = false;
    }
    if (target) {
        selection = target;
        selection.selected = true;
        draw();
    }
}

function up(e) {
    if (!selection) {
        let node = {
            x: e.x,
            y: e.y,
            radius: 10,
            fillStyle: '#22cccc',
            strokeStyle: '#009999',
            selectedFill: '#88aaaa',
            selected: false
        };
        nodes.push(node);
        draw();
    }
    if (selection && !selection.selected) {
        selection = undefined;
    }
    draw();
}

This is nearly the same result as before, except now we can control selection state. What I would like to happen is that we can add an edge such that the current selection and the new selection creates a new edge and line.



var edges = [];

function draw() {
    context.clearRect(0, 0, window.innerWidth, window.innerHeight);

    for (let i = 0; i < edges.length; i++) {
        let fromNode = edges[i].from;
        let toNode = edges[i].to;
        context.beginPath();
        context.strokeStyle = fromNode.strokeStyle;
        context.moveTo(fromNode.x, fromNode.y);
        context.lineTo(toNode.x, toNode.y);
        context.stroke();
    }

    for (let i = 0; i < nodes.length; i++) {
        let node = nodes[i];
        context.beginPath();
        context.fillStyle = node.selected ? node.selectedFill : node.fillStyle;
        context.arc(node.x, node.y, node.radius, 0, Math.PI * 2, true);
        context.strokeStyle = node.strokeStyle;
        context.fill();
        context.stroke();
    }
}

function down(e) {
    let target = within(e.x, e.y);
    if (selection && selection.selected) {
        selection.selected = false;
    }
    if (target) {
        if (selection && selection !== target) {
            edges.push({ from: selection, to: target });
        }
        selection = target;
        selection.selected = true;
        draw();
    }
}

That's it! Now we have some edges between nodes! In a follow-up to this post I will talk about Bezier Curves and how you can create some neat smooth interpolations between those curves what the Canvas api has to offer in terms of functionality here.

Cheers! 🍻

If you liked this article, feel free to give me a follow and a heart/unicorn. Also, if you are so inclined, check out my twitter for similar and other updates!

If you enjoyed this canvas tutorial, check out my other articles on the canvas api below:

Drawing and editing markdown text with canvas

Tokenizing markdown and drawing code blocks

Thanks again! 🏕

10 Awesome Rust Crates and Resources to Learn

Tom Holloway 🏕 — Sun, 30 Aug 2020 17:39:25 +0000

If you're just getting started in Rust, you might wonder what kind of packages you should be exploring and how to get started in the community. Depending on your initial use case, here are a few resources I have found to be immensely helpful in learning rust. Enjoy!

The Definitive Book

Affectionately nicknamed “the book,” The Rust Programming Language will give you an overview of the language from first principles. You’ll build a few projects along the way, and by the end, you’ll have a solid grasp of the language.

https://doc.rust-lang.org/book/

All rustaceans need to make sure they read through this before they go anywhere else. It's the best guide there is on learning the language and it's practically everyone's starting point.

1. Crate: nom

nom is a parser combinators library written in Rust. Its goal is to provide tools to build safe parsers without compromising the speed or memory consumption. To that end, it uses extensively Rust's strong typing and memory safety to produce fast and correct parsers, and provides functions, macros and traits to abstract most of the error prone plumbing.

I was once trying to implement a simple netflow parser and I came across nom. The ergonomics for implementing protocols is straightforward and it includes a number of helpful examples. Here's an example of a color parser!

extern crate nom;
use nom::{
  IResult,
  bytes::complete::{tag, take_while_m_n},
  combinator::map_res,
  sequence::tuple
};

#[derive(Debug,PartialEq)]
pub struct Color {
  pub red:   u8,
  pub green: u8,
  pub blue:  u8,
}

fn from_hex(input: &str) -> Result<u8, std::num::ParseIntError> {
  u8::from_str_radix(input, 16)
}

fn is_hex_digit(c: char) -> bool {
  c.is_digit(16)
}

fn hex_primary(input: &str) -> IResult<&str, u8> {
  map_res(
    take_while_m_n(2, 2, is_hex_digit),
    from_hex
  )(input)
}

fn hex_color(input: &str) -> IResult<&str, Color> {
  let (input, _) = tag("#")(input)?;
  let (input, (red, green, blue)) = tuple((hex_primary, hex_primary, hex_primary))(input)?;

  Ok((input, Color { red, green, blue }))
}

fn main() {}

#[test]
fn parse_color() {
  assert_eq!(hex_color("#2F14DF"), Ok(("", Color {
    red: 47,
    green: 20,
    blue: 223,
  })));
}

In the repository you will find parsers for just about anything from programming languages, network protocols, misc formats like gameboy roms, and text formatting. Definitely worth looking at or keeping as a bookmark.

2. Crate: Bevy Game Engine

Bevy is an amazing game engine crate! It provides a number of fantastic out of the box tooling and features such as:

Cross Platform: Windows, MacOS, and Linux (with planned support for mobile and web)
3D: Lights, meshes, textures, MSAA, and GLTF loading
Sprites: Render individual images as sprites, render from sprite sheets, and dynamically generate new sprite sheets
Assets: An extensible, event driven asset system that loads assets asynchronously in background threads
Scenes: Save ECS Worlds to human-readable scene files and load scene files into ECS Worlds
Plugins: All engine and app features are implemented as modular plugins
Sound: Load audio files as assets and play them from within systems
Multiple Render Backends: Vulkan, DirectX 12, and Metal (with more on the way thanks to wgpu)
Data Driven Shaders: Easily bind ECS components directly to shader uniforms
Hot Asset Reloading: Automatically reload changes to assets at runtime without recompiles or restarts
Events: Efficiently consume and produce Events from within ECS systems
Properties: Dynamically get and set component fields using a string version of their names
Hierarchical Transforms: Create parent-child relationships between entities that propagate Transforms down the hierarchy

The design goals are to keep things very simple for newcomers while providing a lot of flexibility for seasoned developers. The ergonomics and compilation performance is really something to marvel at!

3. Crate: rocket.rs web framework

If you need to write a web site without sacrificing type safety, flexibility, and usability - then take a look at rocket.rs. I've tried using a number of different web frameworks from including actix-web, tide, and warp. There is much debate over the ecosystem here, but I think in the end I would have to put rocket.rs in this list.

#![feature(proc_macro_hygiene, decl_macro)]

#[macro_use] extern crate rocket;

#[get("/hello/<name>/<age>")]
fn hello(name: String, age: u8) -> String {
    format!("Hello, {} year old named {}!", age, name)
}

fn main() {
    rocket::ignite().mount("/", routes![hello]).launch();
}

If you want a better overview and comparison between the different web frameworks and ergonomics check out this post by Luca Palmieri on Choosing a Rust Web Framework 2020 Edition.

4. Crate/Ecosystem: Tokio.rs

Tokio is an async runtime for the Rust programming language. There is some debate at the long term differences between the async-std but in any case, tokio is quite a powerhouse and you can't go very far without running into tokio dependencies in many rust crates.

Here is the starter for the mini redis tutorial:

use mini_redis::{client, Result};

#[tokio::main]
pub async fn main() -> Result<()> {
    // Open a connection to the mini-redis address.
    let mut client = client::connect("127.0.0.1:6379").await?;

    // Set the key "hello" with value "world"
    client.set("hello", "world".into()).await?;

    // Get key "hello"
    let result = client.get("hello").await?;

    println!("got value from the server; result={:?}", result);

    Ok(())
}

Check it out!

5. Crate serde serialization

Serde is a framework for serializing and deserializing Rust data structures into a variety of formats (complimented by things like serde-json

use serde::{Deserialize, Serialize};
use serde_json::Result;

#[derive(Serialize, Deserialize)]
struct Person {
    name: String,
    age: u8,
    phones: Vec<String>,
}

fn typed_example() -> Result<()> {
    // Some JSON input data as a &str. Maybe this comes from the user.
    let data = r#"
        {
            "name": "John Doe",
            "age": 43,
            "phones": [
                "+44 1234567",
                "+44 2345678"
            ]
        }"#;

    // Parse the string of data into a Person object. This is exactly the
    // same function as the one that produced serde_json::Value above, but
    // now we are asking it for a Person as output.
    let p: Person = serde_json::from_str(data)?;

    // Do things just like with any other Rust data structure.
    println!("Please call {} at the number {}", p.name, p.phones[0]);

    Ok(())
}

Here are a number of great examples on their documentation website!

6. Crate: hyper HTTP

hyper is a fast HTTP implementation for Rust

A client for talking to web services
A server for building those web services
Blazing fast thanks to Rust
High concurrency with non-blocking sockets
HTTP/1 and HTTP/2 support

use std::{convert::Infallible, net::SocketAddr};
use hyper::{Body, Request, Response, Server};
use hyper::service::{make_service_fn, service_fn};

async fn handle(_: Request<Body>) -> Result<Response<Body>, Infallible> {
    Ok(Response::new("Hello, World!".into()))
}

#[tokio::main]
async fn main() {
    let addr = SocketAddr::from(([127, 0, 0, 1], 3000));

    let make_svc = make_service_fn(|_conn| async {
        Ok::<_, Infallible>(service_fn(handle))
    });

    let server = Server::bind(&addr).serve(make_svc);

    if let Err(e) = server.await {
        eprintln!("server error: {}", e);
    }
}

This helps if you want to get a bit more lower level than a "web framework" like warp which builds on top of hyper.

7. Crate: textwrap

If you need a small quick library that knows how to wrap text for command line utilities, take a look at this crate.

[dependencies]
textwrap = "0.12"

fn main() {
    let text = "textwrap: a small library for wrapping text.";
    println!("{}", textwrap::fill(text, 18));
}

textwrap: a small
library for
wrapping text.

8. Crate: reqwest HTTP Client

If you need a more "batteries-included" HTTP request client then take a look at reqwest. Some of the includes features:

Plain bodies, JSON, urlencoded, multipart
Customizable redirect policy
HTTP Proxies
HTTPS via system-native TLS (or optionally, rustls)
Cookie Store
WASM

Here's an example below that uses tokio.rs and the async runtime to do a simple GET request.

use std::collections::HashMap;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let resp = reqwest::get("https://httpbin.org/ip")
        .await?
        .json::<HashMap<String, String>>()
        .await?;
    println!("{:#?}", resp);
    Ok(())
}

9. Crate: cargo edit: cargo utility

There are a number of helpful utilities you can add to the built-in cargo command. The one I like to have is cargo-edit which lets you do things like:

cargo add

$ # Add a specific version
$ cargo add regex@0.1.41 --dev
$ # Query the latest version from crates.io and adds it as build dependency
$ cargo add gcc --build
$ # Add a non-crates.io crate
$ cargo add local_experiment --path=lib/trial-and-error/
$ # Add a non-crates.io crate; the crate name will be found automatically
$ cargo add lib/trial-and-error/
$ # Add a crates.io crate with a local development path
$ cargo add my_helper --vers=1.3.1 --path=lib/my-helper/
$ # Add a renamed dependency
$ cargo add thiserror --rename error

cargo rm

$ # Remove a dependency
$ cargo rm regex
$ # Remove a development dependency
$ cargo rm regex --dev
$ # Remove a build dependency
$ cargo rm regex --build

cargo upgrade

# Upgrade all dependencies for the current crate
$ cargo upgrade
# Upgrade docopt (to ~0.9) and serde (to >=0.9,<2.0)
$ cargo upgrade docopt@~0.9 serde@>=0.9,<2.0
# Upgrade regex (to the latest version) across all crates in the workspace
$ cargo upgrade regex --workspace

Super helpful when you don't want to edit Cargo.toml

10. Crate: cargo audit: cargo utility

Cargo-audit will audit Cargo.lock files for crates with security vulnerabilities reported to the RustSec Advisory Database. Very helpful!

Conclusion

These are only a small fraction of the vast Rust community out there. If you're looking to explore more, check out these helpful Awesome Rust repositories for more!

Cheers! 🍻

If you liked this article, please feel free to give me a follow and a like. Also check out my twitter for more updates!

Thanks again!

Resources for learning about CRDT for realtime collaboration

Tom Holloway 🏕 — Sun, 30 Aug 2020 02:32:40 +0000

I've been working on an article to help understand how CRDTs work in realtime collaborative editors. Unfortunately, this knowledge base is quite large and it would be far too much to try to sift down to a small blog post. Instead, I've compiled a few resources I've found to be quite helpful in learning more about CRDT and how to build realtime collaborative editors with this concept. I'm still working on that article and I'm working on making sure to combine that with my series on Writing a Canvas Markdown Editor.

Feel free to share additional resources!

Cheers 🍻

Undo in Git: How to fix common mistakes

Tom Holloway 🏕 — Fri, 28 Aug 2020 17:36:37 +0000

Occasionally, perhaps not all that occasionally, you might make a mistake in your commit workflow. You forgot to fix a lint error, your tests didn't pass so you need to fix that first, or you forgot to add a file, maybe you weren't ready for that commit just yet, maybe you didn't mean to push yet. Whatever the case may be, mistakes happen and you need a way to undo whatever those mistakes were. Here are a few easy workflows you can follow when these mistakes happen.

How to make a change to the previous commit

Let's say you have some code you want to add when you make your first commit. So you type git status and:

nyxtom@enceladus: git status
On branch master

No commits yet

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        index.css
        index.html
        load.js
        load2.js

nothing added to commit but untracked files present (use "git add" to track)

You go ahead and add these files with git add.

nyxtom@enceladus$ git add index.css index.html load.js
~/libvar/map
nyxtom@enceladus$ git status
On branch master

No commits yet

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
        new file:   index.css
        new file:   index.html
        new file:   load.js

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        load2.js

~/libvar/map
nyxtom@enceladus$ git commit -m 'Adding initial project'
[master (root-commit) 9e5fc8f2] Adding initial project
 3 files changed, 187 insertions(+)
 create mode 100644 index.css
 create mode 100644 index.html
 create mode 100644 load.js

But whoops! Turns out there was a problem in your previous commit. You want to change a line of code. So you make your change to the file and use git add -p

nyxtom@enceladus$ git add -p
diff --git a/load.js b/load.js
index 888a7fbd..f0cc06b3 100644
--- a/load.js
+++ b/load.js
@@ -28,6 +28,8 @@ function setup() {
     let text = [''];

     function draw() {
+        context.clearRect(0, 0, window.innerWidth, window.innerHeight);
+
         let offset = 100;
         let tokens = parse(text);
         tokens.forEach(token => {
(1/1) Stage this hunk [y,n,q,a,d,e,?]? y

Instead of making another commit, you can just amend the previous commit! Use git commit --amend.

nyxtom@enceladus$ git commit --amend -m 'Adding initial project'
[master 5923dfe9] Adding initial project
 Date: Fri Aug 28 11:40:45 2020 -0500
 3 files changed, 188 insertions(+)
 create mode 100644 index.css
 create mode 100644 index.html
 create mode 100644 load.js

Great! Now we have fixed that last commit with this change!

How to reset a commit

Now let's say you forgot to add a file, so you go ahead and make another commit like so:

nyxtom@enceladus$ git add load2.js
~/libvar/map (master)+
nyxtom@enceladus$ st
On branch master
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        new file:   load2.js

~/libvar/map (master)+
nyxtom@enceladus$ git commit -m 'Test'
[master a1bf0355] Test
 1 file changed, 147 insertions(+)
 create mode 100644 load2.js

But let's say you don't want that commit yet. Have you run git push yet? No? Okay great. You can unstage the last commit at HEAD with the following: git reset --soft

nyxtom@enceladus$ git reset --soft HEAD^
~/libvar/map (master)+
nyxtom@enceladus$ st
On branch master
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        new file:   load.js

Now the previous commit is still staged. Meaning it's ready to be committed again. If you want to unstage it go ahead and type git restore --staged <file>

How to rebase and squash commits

Let's say you made a few commits like the one below.

nyxtom@enceladus$ git log --oneline
9f44611c (HEAD -> master) Fix tests
2dd82c2b Better comments
5171bfac Fix spacing
ffda820e [#3203] Fixes input validation prior to user api requests

You realize that the last 3 commits actually belong to the original bug fix commit. Fear not! You can use git rebase for this. Let's rebase the last 4 commits.

nyxtom@enceladus$ git rebase -i HEAD~4

You'll notice that when you rebase with -i it is doing so interactively. This means that changes you make in this mode can be done a commit at a time (provided that you are making changes to multiple commits). The editor that pops up will provide you with some options like below:

pick ffda820e [#3203] Fixes input validation prior to user api requests
pick 5171bfac Fix spacing
pick 2dd82c2b Better comments
pick 9f44611c Fix tests

# Rebase 5923dfe9..9f44611c onto 9f44611c (4 commands)
#
# Commands:
# p, pick <commit> = use commit
# r, reword <commit> = use commit, but edit the commit message
# e, edit <commit> = use commit, but stop for amending
# s, squash <commit> = use commit, but meld into previous commit
# f, fixup <commit> = like "squash", but discard this commit's log message
# x, exec <command> = run command (the rest of the line) using shell
# b, break = stop here (continue rebase later with 'git rebase --continue')
# d, drop <commit> = remove commit
# l, label <label> = label current HEAD with a name
# t, reset <label> = reset HEAD to a label
# m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]
# .       create a merge commit using the original merge commit's
# .       message (or the oneline, if no original merge commit was
# .       specified). Use -c <commit> to reword the commit message.
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
# However, if you remove everything, the rebase will be aborted.
#

Since we are concerned with merging the last 3 commits into the first one, we can change all those pick to fixup. This will "squash" the commit into the next one in the history and ignore that log message. There are a number of options you can choose from here so I suggest looking through each in detail.

pick ffda820e [#3203] Fixes input validation prior to user api requests
fixup 5171bfac Fix spacing
fixup 2dd82c2b Better comments
fixup 9f44611c Fix tests

My built-in editor is vim so when I type :wq the rebase will finish and the following is output to the terminal.

Successfully rebased and updated refs/heads/master.

Great! Now type git log --oneline and you should only see a single commit!

nyxtom@enceladus$ git log --oneline
40122c65 (HEAD -> master) [#3203] Fixes input validation prior to user api requests

How to rebase and then push those changes again

Let's say you are working on a branch for a pull request and you already pushed to the remote branch with git push. You made some changes, undid the commit history with git rebase. Unfortunately, now your commit tree is out of sync with the remote commit history. Well you can always git push --force. This will force the remote history to match your current history.

What happens if someone else has made remote changes? You can actually run --force-with-lease to ensure you don't override someone else's changes. It's not very well know, but it's definitely useful!

How to undo a git rebase

Let's say you messed up your rebase and you need to fix what you did and get back to a previous state. Commits are never really lost in git so you can use git reflog to take a look at the commit history prior to the rebase.

nyxtom@enceladus$ git reflog
40122c65 (HEAD -> master, origin/master) HEAD@{0}: checkout: moving from 8115f1e263eeeee4bf8f75e90bbb6cca1a78cfbe to master
8115f1e2 HEAD@{1}: checkout: moving from master to 8115f1e263eeeee4bf8f75e90bbb6cca1a78cfbe
40122c65 (HEAD -> master, origin/master) HEAD@{2}: rebase (finish): returning to refs/heads/master
40122c65 (HEAD -> master, origin/master) HEAD@{3}: rebase (fixup): [#3203] Fixes input validation prior to user api requests
0b378bf2 HEAD@{4}: rebase (fixup): # This is a combination of 3 commits.
e3dc38ab HEAD@{5}: rebase (fixup): # This is a combination of 2 commits.
ffda820e HEAD@{6}: rebase (start): checkout HEAD~4
9f44611c HEAD@{7}: commit: Fix tests
2dd82c2b HEAD@{8}: commit: Better comments
5171bfac HEAD@{9}: commit: Fix spacing
ffda820e HEAD@{10}: commit: [#3203] Fixes input validation prior to user api requests

Great! Let's move back to the commit just right before that rebase happened.

nyxtom@enceladus$ git reset 9f44611c
nyxtom@enceladus$ git log --oneline
9f44611c (HEAD -> master) Fix tests
2dd82c2b Better comments
5171bfac Fix spacing
ffda820e [#3203] Fixes input validation prior to user api requests
5923dfe9 Adding initial project

Great! You're back in business!

Conclusion

If you ever run into trouble, know that all is literally not lost forever. You can even use git fsck --lost-found if you find yourself in a situation where you have orphan commits that aren't actually attached to the commit tree. Be sure to read up on git-scm for more information!

As always, please give me a like and a follow if you liked this article. Also check out my twitter if you're interested!

Cheers! 🍻

If you liked this article, you might also be interested in:

Top 10 git commands everyone should know

25 shell commands to master for system administration

Thanks again!

[META LIST] What twitter lists and people do you follow for developers?

Tom Holloway 🏕 — Thu, 27 Aug 2020 16:34:51 +0000

I’ve been pruning and curating my following on twitter/nyxtom and I’m curious what recommended lists/people follow for things like development, indie hacking, hardware hacking, generative art, mathematics...etc

Here are a few that I've found neat:

Updated from Community:

Design and Development via @vtrpldn

Let me know in the comments below and I'll add you to this meta list! Also, feel free to share your own twitter handle and a bit about what you post if you are interested.

Thanks 🍻

Tokenizing markdown and drawing code blocks in canvas

Tom Holloway 🏕 — Thu, 27 Aug 2020 06:16:21 +0000

If you read my post on "How to write a basic markdown editor with canvas", you should now have a basic way to write some text and headings into a canvas-rendered editor. In this post, we're going to continue our work with the canvas api to add support for embedding code blocks. We'll make use of a few more canvas functions to render some custom shapes and refactor our code to support multiple types of rendering.

Drawing shapes in canvas

Drawing shapes in canvas is pretty straightforward as far as the api is concerned. Simply use the existing canvas rendering context to adjust how you want to draw and follow that with what you want to draw. Think of the various properties on the context as your paintbrush.

Let's say we want to draw a rectangle. To do this we would obtain our rendering context, and call the fillRect and fillStyle calls.

const canvas = document.querySelector('canvas');
const context = canvas.getContext('2d');

context.fillStyle = 'rgb(200, 0, 0)';
context.fillRect(10, 10, 50, 50);

context.fillStyle = 'rgba(0, 0, 200, 0.5)';
context.fillRect(30, 30, 50, 50);

By contrast, if we wanted to draw just the edges of a rectangle we can use the corresponding methods strokeRect and strokeStyle.

const canvas = document.querySelector('canvas');
const context = canvas.getContext('2d');

context.strokeStyle = 'green';
context.strokeRect(20, 10, 160, 100);

The rest of the canvas drawing api typically works in paths and arcs. For instance, to draw a circle we would use the arc and the beginPath with either fill or stroke.

const canvas = document.querySelector('canvas');
const context = canvas.getContext('2d');

context.strokeStyle = 'green';
context.beginPath();
context.arc(100, 75, 50, 0, 2 * Math.PI);
context.stroke();

In addition to arc, we also have the ellipse method:

The ellipse() method creates an elliptical arc centered at (x, y) with the radii radiusX and radiusY. The path starts at startAngle and ends at endAngle, and travels in the direction given by anticlockwise (defaulting to clockwise).

Parsing out the code snippets in markdown

Given that our markdown text contains some other things like headings, we will need a way to find out when we encounter a code snippet. We will use the standard three backticks. Let's write a little snippet to parse out this text.

function parse(lines) {
    let cur = [];
    let tokens = [];
    for (let i = 0; i < lines.length; i++) {
        let line = lines[i];
        let matches = line.match(/^`{3}([a-zA-Z]*)/);
        if (matches) {
           let type = matches[1];
           if (cur.length && cur[0].code) {
               type = cur[0].type;
               tokens.push({ code: cur.slice(1), type });
               cur = [];
           } else {
               cur.push({ line, code: true, type });
           }
           continue;
        } else if (!cur.length && line.match(/^\s*\#/g)) {
            let level = line.match(/^\s*\#/g).length;
            tokens.push({ heading: line, level });
            continue;
        }
        if (!cur.length) {
            tokens.push(line);
        } else {
            cur.push(line);
        }
    }
    if (cur.length) {
        tokens.push(cur[0].line, ...cur.slice(1));
    }
    return tokens;
}

In our snippet above, we're going to go through each line, see if it matches a code block, then depending on the current token state: add the current token, parse out a heading, or append to current until the code block is completed.

You can see the sample output below from parsing some text:

[
  { heading: '# hello', level: 1 },
  '',
  '',
  { code: [ 'A->B', 'B->C', 'B->D' ], type: 'graph' },
  '',
  { heading: '## bleh!', level: 2 },
  '',
  'hi'
]

Rendering tokens of headers and code

Let's go ahead and update our previous draw code and swap things out. We're going to take advantage of the textAlign in the render context so we don't have to worry about measuring the text just yet.

function draw() {
    context.clearRect(0, 0, window.innerWidth, window.innerHeight);

    let offset = 100;
    let tokens = parse(text);
    tokens.forEach(token => {
        if (token.code) {
            offset += renderCode(token, offset);
        } else {
            offset += renderText(token, offset);
        }
    });
}

function renderCode(token, offset) {
    let height = 0;
    token.code.forEach(c => {
        let h = renderText(c, offset);
        height += h;
        offset += h;
    });
    return height;
}

function renderText(token, offset) {
    let lineHeight = 1.5;
    let headingSize = 32;
    let baseSize = 16;
    let height = baseSize * lineHeight;
    if (token.heading) {
        let size = headingSize - (token.level * 4);
        context.font = `bold ${size}px roboto`;
        height = size * lineHeight;
    } else {
        context.font = `${baseSize}px roboto`;
    }

    context.textAlign = 'center';
    context.fillText(token, window.innerWidth / 2, offset);
    return height;
}

The rendering text is mostly the same as before in the previous article, and now I'm simply rendering the code as regular text. Notice also how we can backspace to the code and re-edit what we were working on! This is because the render code is working with the tokens while the input is working with the raw text. Pretty neat!

Drawing the code block

Let's finish up this article by fixing up our renderCode block to actually render something that looks like a block of code. There are a few things that we need to do below:

Find the maximum width of the code block based on measureText
Calculate the height of the code block based on the number of lines, font size and line height
Render an actual rectangle
Adjust the initial offset
Render the lines of code
Adjust the offset after the block

function renderCode(token, offset) {
    let height = 0;
    context.font = '16px roboto';

    let lens = token.code.map(c => c.length);
    let maxLen = Math.max(...lens);
    let maxText = token.code.find(c => c.length === maxLen);
    let maxWidth = Math.max(context.measureText(maxText).width, 300);
    let x = window.innerWidth / 2 - maxWidth / 2;
    let maxHeight = token.code.length * 16 * 1.5;
    context.fillStyle = '#cccccc';
    context.lineWidth = 3;
    context.strokeRect(x, offset, maxWidth, maxHeight);
    context.fillRect(x, offset, maxWidth, maxHeight);

    // before
    offset += 16;
    height += 16;

    token.code.forEach(c => {
        let h = renderText(c, offset);
        height += h;
        offset += h;
    });

    // after
    offset += 16;
    height += 16;

    return height;
}

That's it!

Conclusion

While we haven't reached the stage of formatting our code blocks, we have managed to do a little bit of tokenization and we learned a little bit more about the canvas api. Initially when I wrote this I wanted to demonstrate how to render a graph tree. Unfortunately, layout algorithms for trees are a bit more in depth (😄 pun!) and require some background on tree traversal algorithms. In my next article in this series, I will go over tree traversal and layout algorithms as a setup for rendering an actual graph from markdown.

Stay tuned! 📺 👨🏻‍💻

If you liked this article, please give me a follow and a like. Also feel free to check out my twitter where I post my latest updates and mixed content.

Thanks again!

Cheers! 🍻

Drawing and editing markdown text with canvas and javascript

Tom Holloway 🏕 — Wed, 26 Aug 2020 03:05:19 +0000

This last week I've been mucking about with the canvas api. I've put together some visualizations and went through my old content on p5.js (where I go into length on flow fields and noise algorithms: check it out, I really enjoyed that one).

In my playing around I've been putting together some ideas around graphing tools and decided one of the most basic things users need in a graph tool is the ability to type in a text input. There are a number of ways to do this, including overlaying HTML on top of a canvas drawing surface (or using d3.js). Instead, I chose to just write a simple script that uses the existing canvas api. Like all things, there is more to it than meets the eye, but if you're just trying to get things started - well, here we go.

Setting up our project

To start, you'll need an HTML and a bit of CSS to setup our sample code. It's not much, but obviously it's a starting point.



<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8">
    <title>Map</title>
    <link rel="stylesheet" href="index.css">
    <script type="text/javascript" src="load.js"></script>
</head>
<body>
    <canvas></canvas>
</body>
</html>

In a separate file for css I've setup a few basic reset variables and some root styling. It's not really totally necessary, but I like having these things when I start out.



/** index.css */
:root {
    --root-font-size: 12px;
    --bg: #fafafa;
    --text-color: #333333;
}

/** Reset */
html, body, nav, ul, h1, h2, h3, h4, a, canvas {
    margin: 0px;
    padding: 0px;
    color: var(--text-color);
}
html, body {
    font-family: Roboto, -apple-system, BlinkMacSystemFont, 'Segoe UI', Oxygen, Ubuntu, Cantarell, 'Open Sans', 'Helvetica Neue', sans-serif;
    font-size: var(--root-font-size);
    background: var(--bg);
    height: 100%;
    width: 100%;
    overflow: hidden;
}
*, body, button, input, select, textarea, canvas {
    text-rendering: optimizeLegibility;
    -webkit-font-smoothing: antialiased;
    -moz-osx-font-smoothing: grayscale;
    outline: 0;
}

One of the things I really like about the latest CSS is that you don't really need any build tools for it. You can get the most out of your webapp with just root variables. Often, on small projects like these I don't go much further than that - just some root variables and I'm good.

There's actually a great post on how to do complete turing logic in CSS using these variables. Check it out, the author actually made a full minesweeper game using the "Space Toggle" technique.

Canvas API

The canvas element creates a fixed-size drawing surface that exposes one or more rendering contexts, which are used to create and manipulate the content shown. In this tutorial, we focus on the 2D rendering context. Other contexts may provide different types of rendering; for example, WebGL uses a 3D context based on OpenGL ES.

Create a file load.js with the following



/** load.js */
var canvas, context;
var text = [''];

function setup() {
    canvas = document.querySelector('canvas');
    canvas.width = window.innerWidth;
    canvas.height = window.innerHeight;

    context = canvas.getContext('2d');
    context.font = '18px Roboto';
}

function draw() {
    /* draw code */
}

window.onresize = function () {
    if (canvas) {
        canvas.width = window.innerWidth;
        canvas.height = window.innerHeight;
    }
}

window.onkeypress = function (e) {
}

window.onkeydown = function (e) {
}

window.onload = function () {
    setup();
}

Couple things going on here. First, we're waiting until the window loads via onload meaning we have waited until all resources have been loaded.

Once setup has been called, we grab the canvas and set it to the window height/width. We ensure that the width/height is also set when the window resizes via the onresize event.

Key press / Key Down

Since this is an editor, we want to presumably write something when the keys are pressed. Update the onkeypress and onkeydown code to the following:



window.onkeypress = function (e) {
    if (e.key === 'Enter') {
        text.push('');
    } else {
        text[text.length - 1] += e.key;
    }
    draw();
}

window.onkeydown = function (e) {
    if (e.key === 'Backspace' && text.length && text[0].length) {
        let txt = text[text.length - 1];
        txt = txt.slice(0, txt.length - 1);
        text[text.length - 1] = txt;
        if (!txt.length && text.length > 1) {
            text = text.slice(0, text.length - 1);
        }
    }
    draw();
}

These functions are effectively going to manage our text state. It isn't comprehensive, but for the moment we can do basic things like typing and hitting enter / backspace to make changes to our text array.

Drawing

Let's get to the draw code. Whenever we are in canvas, it is proper to clear the screen first before you make additional draw changes. In visualizations and generative art, you can take advantage of what is already there to create some neat effects. But since we're drawing text on every key stroke and update, then we want to clear the screen and refresh the content as such.



function draw() {
    context.clearRect(0, 0, window.innerWidth, window.innerHeight);

    let offset = 0;
    let totalHeight = 0;
    let height = (18 * 1.5); // font * line height

    let items = text.map(txt => {
        let width = context.measureText(txt).width;
        let item = {
            txt,
            width,
            offset
        };
        offset = offset + height;
        totalHeight += height;
        return item;
    });

    let cY = (window.innerHeight / 2) - (totalHeight / 2);
    items.forEach(item => {
        let x = window.innerWidth / 2 - item.width / 2;
        let y = item.offset + cY;
        context.fillText(item.txt, x, y);
    });
}

In the above code here, we are using the canvas api's measureText. There are alternative methods to measuring text here if we want to be even more precise such as offloading the text into another dom element using the getBoundingBoxClientRect. I've chosen the canvas method for now as we will end up taking advantage of the rendering context below to make additional measurements.

In any case, we have ourselves a minimal text input with support for multiple lines and backspacing. Let's carry on!

Markdown

Since this is supposed to be a markdown editor. Markdown as a spec is fairly minimal, but we aren't going to get to all of it in one post. I'll leave you to expand on this, but for now we will implement just the headings portion of the spec.

To do this, we'll need a few things to parse out our text lines and then swap out our calls to context as appropriate.

Add the following code to parse the text line



function parse(txt) {
    let lineHeight = 1.5;
    let headingSize = 32;
    let baseSize = 16;
    if (txt.trim().startsWith('#')) {
        let level = txt.match(/\s*\#/g).length;
        let size = headingSize - (level * 4);
        return {
            font: `bold ${size}px roboto`,
            height: size * lineHeight,
            txt
        };
    } else {
        return {
            font: `${baseSize}px roboto`,
            height: baseSize * lineHeight,
            txt
        };
    }
}

Then in the draw code update it to call our parse function.



function draw() {
    context.clearRect(0, 0, window.innerWidth, window.innerHeight);

    let offset = 0;
    let totalHeight = 0;

    let items = text.map(txt => {
        let item = parse(txt);
        item.offset = offset;
        offset = offset + item.height;
        totalHeight += item.height;
        return item;
    });

    let centerY = (window.innerHeight / 2) - (totalHeight / 2);
    items.forEach(item => {
        context.font = item.font;
        let width = context.measureText(item.txt).width;
        let x = window.innerWidth / 2 - width / 2;
        let y = item.offset + centerY;
        context.fillText(item.txt, x, y);
    });
}

Notice, that we have moved the measureText code into the code right before we actually attempt to draw it. This is because we have changed the rendering context on the line prior to it with the context.font = item.font. We want to be sure we make the right measurements based on the current rendering context.

Conclusion

There you have it! It's quite basic and minimal, but it's as good a start as any. I'll leave it to you to fill in more of the code to finish the rest of the spec.

As always, if you enjoyed this article, please give me a like and a follow. Any feedback or ideas for future articles are welcomed. Be sure to give me a follow on twitter for similar updates!

Cheers! 🍺

Don’t forget to checkout the next part in the series!