DEV Community: Paul Kinlan

Creating a commit with multiple files to Github with JS on the web

Paul Kinlan — Fri, 24 May 2019 11:10:02 +0000

My site is entirely static. It’s built with Hugo and hosted with Zeit. I’m pretty happy with the setup, I get near instant builds and super fast CDN’d content delivery and I can do all the things that I need to because I don’t have to manage any state.

I’ve created a simple UI for this site and also my podcast creator that enables me to quickly post new content to my statically hosted site.

So. How did I do it?

It’s a combination of Firebase Auth against my Github Repo, EditorJS to create edit the content (it’s neat) and Octokat.js to commit to the repo and then Zeit’s Github integration to do my hugo build. With this set up, I am able to have an entirely self hosted static CMS, similar to how a user might create posts in a database backed CMS like Wordpress.

In this post I am just going to focus on one part of the infrastructure - committing multiple files to Github because it took me a little while to work out.

The entire code can be seen on my repo.

If you are building a Web UI that needs to commit directly to Github, the best library that I have found is Octokat - it works with CORS and it seems to handle the entire API surface of the Github API.

Git can be a complex beast when it comes to understanding how the tree, branches and other pieces work so I took some decisions that made it easier.

I will be only able to push to the master branch known as heads/master.
I will know where certain files will be stored (Hugo forces me to have a specific directory structure)

With that in mind, the general process to creating a commit with multiple files is as follows:

Get a reference to the repo.

Get a reference to the tip of the tree on heads/master branch.
For each file that we want to commit create a blob and then store the references to the sha identifier, path, mode in an array.
Create a new tree that contains all the blobs to add to the reference to the tip of the heads/master tree, and store the new sha pointer to this tree.
Create a commit that points to this new tree and then push to the heads/master branch.

The code pretty much follows that flow. Because I can assume the path structure for certain inputs I don’t need to build any complex UI or management for the files.

const createCommit = async (repositoryUrl, filename, data, images, commitMessage, recording) => {
  try {
    const token = localStorage.getItem('accessToken');
    const github = new Octokat({ 'token': token });
    const [user, repoName] = repositoryUrl.split('/');

    if(user === null || repoName === null) {
      alert('Please specifiy a repo');
      return;
    }

    const markdownPath = `site/content/${filename}.markdown`.toLowerCase();
    let repo = await github.repos(user, repoName).fetch();
    let main = await repo.git.refs('heads/master').fetch();
    let treeItems = [];

    for(let image of images) {
      let imageGit = await repo.git.blobs.create({ content: image.data, encoding: 'base64' });
      let imagePath = `site/static/images/${image.name}`.toLowerCase();
      treeItems.push({
        path: imagePath,
        sha: imageGit.sha,
        mode: "100644",
        type: "blob"
        });
    }

    if (recording) {
      let audioGit = await repo.git.blobs.create({ content: recording.data, encoding: 'base64' });
      let audioPath = `site/static/audio/${recording.name}.${recording.extension}`.toLowerCase();
      treeItems.push({
        path: audioPath,
        sha: audioGit.sha,
        mode: "100644",
        type: "blob"
        });
    }

    let markdownFile = await repo.git.blobs.create({ content: btoa(jsonEncode(data)), encoding: 'base64' });
    treeItems.push({
      path: markdownPath,
      sha: markdownFile.sha,
      mode: "100644",
      type: "blob"
    });

    let tree = await repo.git.trees.create({
      tree: treeItems,
      base_tree: main.object.sha
    });

    let commit = await repo.git.commits.create({
      message: `Created via Web - ${commitMessage}`,
      tree: tree.sha,
      parents: [main.object.sha]});

    main.update({sha: commit.sha})

    logToToast('Posted');
  } catch (err) {
    console.error(err);
    logToToast(err);
  }
}

Let me know if you’ve done anything similar with static hosting. I’m very excited that I can build a modern frontend for what is an entirely server-less hosting infrastructure.

Screen Recorder: recording microphone and the desktop audio at the same time

Paul Kinlan — Mon, 13 May 2019 19:47:24 +0000

I have a goal of building the worlds simplest screen recording software and I’ve been slowly noodling around on the project for the last couple of months (I mean really slowly).

In previous posts I had got the screen recording and a voice overlay by futzing about with the streams from all the input sources. One area of frustration though was that I could not work out how to get the audio from the desktop and overlay the audio from the speaker. I finally worked out how to do it.

Firstly, getDisplayMedia in Chrome now allows audio capture, there seems like an odd oversight in the Spec in that it did not allow you to specify audio: true in the function call, now you can.

const audio = audioToggle.checked || false;
desktopStream = await navigator.mediaDevices.getDisplayMedia({ video:true, audio: audio });

Secondly, I had originally thought that by creating two tracks in the audio stream I would be able to get what I wanted, however I learnt that Chrome’s MediaRecorder API can only output one track, and 2nd, it wouldn’t have worked anyway because tracks are like the DVD mutliple audio tracks in that only one can play at a time.

The solution is probably simple to a lot of people, but it was new to me: Use Web Audio.

It turns out that WebAudio API has createMediaStreamSource and createMediaStreamDestination, both of which are API’s needed to solve the problem. The createMediaStreamSource can take streams from my desktop audio and microphone, and by connecting the two together into the object created by createMediaStreamDestination it gives me the ability to pipe this one stream into the MediaRecorder API.

const mergeAudioStreams = (desktopStream, voiceStream) => {
  const context = new AudioContext();

  // Create a couple of sources
  const source1 = context.createMediaStreamSource(desktopStream);
  const source2 = context.createMediaStreamSource(voiceStream);
  const destination = context.createMediaStreamDestination();

  const desktopGain = context.createGain();
  const voiceGain = context.createGain();

  desktopGain.gain.value = 0.7;
  voiceGain.gain.value = 0.7;

  source1.connect(desktopGain).connect(destination);
  // Connect source2
  source2.connect(voiceGain).connect(destination);

  return destination.stream.getAudioTracks();
};

Simples.

The full code can be found on my glitch, and the demo can be found here: https://screen-record-voice.glitch.me/

Extracting text from an image: Experiments with Shape Detection

Paul Kinlan — Mon, 13 May 2019 12:39:45 +0000

I had a little down time after Google IO and I wanted to scratch a long-term itch I’ve had. I just want to be able to copy text that is held inside images in the browser. That is all. I think it would be a neat feature for everyone.

It’s not easy to add functionality directly into Chrome, but I know I can take advantage of the intent system on Android and I can now do that with the Web (or at least Chrome on Android).

Two new additions to the web platform - Share Target Level 2 (or as I like to call it File Share) and the TextDetector in the Shape Detection API - have allowed me to build a utility that I can Share images to and get the text held inside them.

The basic implementation is relatively straight forwards, you create a Share Target and a handler in the Service Worker, and then once you have the image that the user has shared you run the TextDetector on it.

The Share Target API allows your web application to be part of the native sharing sub-system, and in this case you can now register to handle all image/* types by declaring it inside your Web App Manifest as follows.

"share_target": {
  "action": "/index.html",
  "method": "POST",
  "enctype": "multipart/form-data",
  "params": {
    "files": [
      {
        "name": "file",
        "accept": ["image/*"]
      }
    ]
  }
}

When your PWA is installed then you will see it in all the places where you share images from as follows:

The Share Target API treats sharing files like a form post. When the file is shared to the Web App the service worker is activated the fetch handler is invoked with the file data. The data is now inside the Service Worker but I need it in the current window so that I can process it, the service knows which window invoked the request, so you can easily target the client and send it the data.

self.addEventListener('fetch', event => {
  if (event.request.method === 'POST') {
    event.respondWith(Response.redirect('/index.html'));
    event.waitUntil(async function () {
      const data = await event.request.formData();
      const client = await self.clients.get(event.resultingClientId || event.clientId);
      const file = data.get('file');
      client.postMessage({ file, action: 'load-image' });
    }());

    return;
  }
  ...
  ...
}

Once the image is in the user interface, I then process it with the text detection API.

navigator.serviceWorker.onmessage = (event) => {  
  const file = event.data.file;
  const imgEl = document.getElementById('img');
  const outputEl = document.getElementById('output');
  const objUrl = URL.createObjectURL(file);
  imgEl.src = objUrl;
  imgEl.onload = () => {
    const texts = await textDetector.detect(imgEl);
    texts.forEach(text => {
      const textEl = document.createElement('p');
      textEl.textContent = text.rawValue;
      outputEl.appendChild(textEl);
    });
  };
  ...
};

The biggest issue is that the browser doesn’t naturally rotate the image (as you can see below), and the Shape Detection API needs the text to be in the correct reading orientation.

I used the rather easy to use EXIF-Js library to detect the rotation and then do some basic canvas manipulation to re-orientate the image.

EXIF.getData(imgEl, async function() {
  // http://sylvana.net/jpegcrop/exif_orientation.html
  const orientation = EXIF.getTag(this, 'Orientation');
  const [width, height] = (orientation > 4) 
                  ? [imgEl.naturalWidth, imgEl.naturalHeight]
                  : [imgEl.naturalHeight, imgEl.naturalWidth];

  canvas.width = width;
  canvas.height = height;
  const context = canvas.getContext('2d');
  // We have to get the correct orientation for the image
  // See also https://stackoverflow.com/questions/20600800/js-client-side-exif-orientation-rotate-and-mirror-jpeg-images
  switch(orientation) {
    case 2: context.transform(-1, 0, 0, 1, width, 0); break;
    case 3: context.transform(-1, 0, 0, -1, width, height); break;
    case 4: context.transform(1, 0, 0, -1, 0, height); break;
    case 5: context.transform(0, 1, 1, 0, 0, 0); break;
    case 6: context.transform(0, 1, -1, 0, height, 0); break;
    case 7: context.transform(0, -1, -1, 0, height, width); break;
    case 8: context.transform(0, -1, 1, 0, 0, width); break;
  }
  context.drawImage(imgEl, 0, 0);
}

And Voila, if you share an image to the app it will rotate the image and then analyse it returning the output of the text that it has found.

It was incredibly fun to create this little experiment, and it has been immediately useful for me. It does however, highlight the inconsistency of the web platform. These API’s are not available in all browsers, they are not even available in all version of Chrome - this means that as I write this article Chrome OS, I can’t use the app, but at the same time, when I can use it… OMG, so cool.

Debugging Web Pages on the Nokia 8110 with KaiOS using Chrome OS

Paul Kinlan — Mon, 15 Apr 2019 01:16:30 +0000

This post is a continuation of the post on debugging a KaiOS device with Web IDE, but instead of using macOS, you can now use Chrome OS (m75) with Crostini.

I’m cribbing from the KaiOS Environment Setup which is a good start, but not enough for getting going with Chrome OS and Crostini. Below is the rough guide that I followed.

Make sure that you are using at least Chrome OS m75 (currently dev channel as of April 15th), then:

Ensure that you have Crostini USB support enabled - chrome://flags/#crostini-usb-support
Open up the terminal in crostini
sudo apt-get install usbutils udev - You need to make sure that you have the USB tools installed.
lsusb - You should now see your connected device, if this doesn’t work there might be another issue.
sudo apt-get install --no-install-recommends autoconf2.13 bison bzip2 ccache curl flex gawk gcc g++ g++-multilib git lib32ncurses5-dev lib32z1-dev libgconf2-dev libgl1-mesa-dev libx11-dev make zip lzop libxml2-utils openjdk-8-jdk nodejs unzip python
sudo apt install android-tools-adb android-tools-fastboot
I am not sure I needed it, but I also ran wget -S -O - https://raw.githubusercontent.com/cm-b2g/B2G/1230463/tools/51-android.rules | sudo tee >/dev/null /etc/udev/rules.d/51-android.rules; sudo udevadm control --reload-rules
sudo chmod a+r /etc/udev/rules.d/51-android.rules and then added the device vendor ID to the file.

If all the above is done, you should then be able to adb devices and get your list of connected devices.

Offline fallback page with service worker

Paul Kinlan — Fri, 05 Apr 2019 18:17:22 +0000

Years ago, I did some research into how native applications responded to a lack of network connectivity. Whilst I’ve lost the link to the analysis (I could swear it was on Google+), the overarching narrative was that many native applications are inextricably tied to the internet that they just straight up refuse to function. Sounds like a lot of web apps, the thing that set them apart from the web though is that the experience was still ‘on-brand’, Bart Simpson would tell you that you need to be online (for example), and yet for the vast majority of web experiences you get a ‘Dino’ (see chrome://dino).

We’ve been working on Service Worker for a long time now, and whilst we are seeing more and more sites have pages controlled by a Service Worker, the vast majority of sites don’t even have a basic fallback experience when the network is not available.

I asked my good chum Jake if we have any guindance on how to build a generic fall-back page on the assumption that you don’t want to create an entirely offline-first experience, and within 10 minutes he had created it. Check it out.

For brevity, I have pasted the code in below because it is only about 20 lines long. It caches the offline assets, and then for every fetch that is a ‘navigation’ fetch it will see if it errors (because of the network) and then render the offline page in place of the original content.



addEventListener('install', (event) => ; {
  event.waitUntil(async function() {
    const cache = await caches.open('static-v1');
    await cache.addAll(['offline.html', 'styles.css']);
  }());
});

addEventListener('fetch', (event) => {
  const { request } = event;

  // Always bypass for range requests, due to browser bugs
  if (request.headers.has('range')) return;
  event.respondWith(async function() {
    // Try to get from the cache:
    const cachedResponse = await caches.match(request);
    if (cachedResponse) return cachedResponse;

    try {
      // Otherwise, get from the network
      return await fetch(request);
    } catch (err) {
      // If this was a navigation, show the offline page:
      if (request.mode === 'navigate') {
        return caches.match('offline.html');
      }

      // Otherwise throw
      throw err;
    }
  }());
});

That is all. When the user is online they will see the default experience.

And when the user is offline, they will get the fallback page.

I find this simple script incredibly powerful, and yes, whilst it can still be improved, I do believe that even just a simple change in the way that we speak to our users when there is an issue with the network has the ability to fundamentally improve the perception of the web for users all across the globe.

Quick Logcat - debugging android from the web

Paul Kinlan — Sat, 30 Mar 2019 10:32:48 +0000

I was on the flight to Delhi this last week and I wanted to be able to debug my KaiOS device with Chrome OS - I never quite got to the level that I needed for a number of reasons (port forwarding didn’t work - more on that in another post), but I did get to build a simple tool that really helps me build for the web on Android based devices.

I’ve been using WebADB.js for a couple of side projects, but I thought I would at least release one of the tools I made last week that will help you if you ever need to debug your Android device and you don’t have adb installed or any other Android system tools.

Quick Logcat is just that. It can connect to any Android device that is in developer mode and has USB enabled, is connected to your machine over USB and most importantly you grant access from the web page to connect to the device, and once that is all done it just runs adb shell logcat to create the following output.

Checkout the source over on my github account, specifically the logger class that has the brunt of my logic - note a lot of this code is incredibly similar to the demo over at webadb.github.io, but it should hopefully be relatively clear to follow how I interface with the WebUSB API (which is very cool). The result is the following code that is in my index file: I instantiate a controller, connect to the device which will open up the USB port and then I start the logcat process and well, cat the log, via logcat.

It even uses .mjs files :D

 <script type="module">
    import LogcatController from "/scripts/main.mjs";
    onload = () => {
      const connect = document.getElementById("connect");
      const output = document.getElementById("output");
      let controller = new LogcatController();
      connect.addEventListener("click", async () => {
        await controller.connect();
        controller.logcat((log) => {
          output.innerText += log;
        })
      });
    };
  </script>

ADB is an incredibly powerful protocol, you can read system files, you can write over personal data and you can even easily side-load apps, so if you give access to any external site to your Android device, you need to completely trust the operator of the site.

This demo shows the power and capability of the WebUSB API, we can interface with hardware without any natively installed components, drivers or software and with a pervasive explicit user opt-in model that stops drive-by access to USB components.

I’ve got a couple more ideas up my sleeve, it will totally be possible to do firmware updates via the web if you so choose. One thing we saw a lot of in India was the ability side-load APK’s on to user’s new phones, whilst I am not saying we must do it, a clean web-interface would be more more preferable to the software people use today.

What do you think you could build with Web USB and adb access?

Debugging Web Pages on the Nokia 8110 with KaiOS

Paul Kinlan — Thu, 21 Mar 2019 21:41:53 +0000

We’ve been doing a lot of development on feature phones recently and it’s been hard, but fun. The hardest bit is that on KaiOS we found it impossible to debug web pages, especially on the hardware that we had (The Nokia 8110). The Nokia is a great device, it’s built with KaiOS which we know is based on something akin to Firefox 48, but it’s locked down, there is no traditional developer mode like you get on other Android devices, which means you can connect Firefox’s WebIDE.

Through a combination of reading a couple of blogs, and knowing a bit about adb I worked out how to do it. Note, others might have been able to do it, but it’s not documented in one place cleanly.

(Image above shows the DevTools and also the output of the screenshot tool)

Here are the steps:

Connect a USB cable. Ensure you have adb installed on your main machine.
Download a copy of Firefox 48 (this is the only one I could get to work)
Enable ‘Developer Mode’ by entering *#*#33284#*#* from your phone (note, don’t use the dialer). You will see a little ‘bug’ icon on the top of the screen. [Source]
Attach your USB cable
On your development machine run the following commands
1. adb start-server
2. adb devices to check your phone is connected.
3. adb forward tcp:6000 localfilesystem:/data/local/debugger-socket this sets up a channel from your machine to a socket on the phone. This is what Web IDE uses.
Start Web IDE by opening Firefox, go to Tools and then Web IDE
Web IDE will be open, click ‘Remote Runtime’ and click the open button that has ‘localhost:6000’ in. (this is the tcp forwarding port).
Open a page on the phone, and you should see it on the left.Voila.

Got web performance problems? Just wait...

Paul Kinlan — Sat, 09 Mar 2019 08:10:52 +0000

I saw a tweet by a good chum and colleague, Mariko, about testing on a range of low end devices keeping you really grounded.

The context of the tweet is that we are looking at what Web Development is like when building for users who live daily on these classes of devices.

The team is doing a lot of work now in this space, but I spent a day build a site and it was incredibly hard to make anything work at a even slightly reasonable level of performances - here are some of the problems that I ran into:

Viewport oddities, and mysterious re-introduction of 300ms click-delay (can work around).
Huge repaints of entire screen, and it’s slow.
Network is slow
Memory is constrained, and subsequent GC’s lock the main thread for multiple seconds
Incredibly slow JS execution
DOM manipulation is slow

For many of the pages I was building, even on a fast wifi connection pages took multiple seconds to load, and subsequent interactions were just plain slow. It was hard, but incredibly gratifying at a technical level to see changes in algorithms and logic that I wouldn’t have done for all my traditional web development, yield large improvements in performance.

I am not sure what to do long-term, I suspect a huge swathe of developers that we work with in the more developed markets will have a reaction ‘I am not building sites for users in [insert country x]‘, and at a high-level it’s hard to argue with this statement, but I can’t ignore the fact that 10’s of millions of new users are coming to computing each year and they will be using these devices and we want the web to be the platform of choice for content and apps lest we are happy with the rise of the meta platform.

We’re going to need to keep pushing on performance for a long time to come. We will keep creating tools and guidance to help developers load quickly and have smooth user interfaces :)

Browser Bug Searcher

Paul Kinlan — Sat, 09 Mar 2019 07:49:18 +0000

I was just reflecting on some of the work our team has done and I found a project from 2017 that Robert Nyman and Eric Bidelman created. Browser Bug Searcher!.

It’s incredible that with just a few key presses you have a great overview of your favourite features across all the major browser engines.

Source code available.

This actually highlights one of the issues that I have with crbug and webkit bug trackers, they don’t have a simple way to get feeds of data in formats like RSS. I would love to be able to use my topicdeck aggregator with bug categories etc so I have a dashboard of all the things that I am interested in based on the latest information from each of the bug trackers.

File Web Share Target

Paul Kinlan — Fri, 15 Feb 2019 15:52:03 +0000

I’ve frequently said that for web apps to compete effectively in the world ofapps, they need to be integrated in to all of the places that users expect appsto be. Inter-app communication is one of the major missing pieces of the webplatform, and specifically one of the last major missing features is nativelevel sharing: Web apps need to be able to get data out of theirsilo and into other web sites and apps; they also need tobe able to receive the data from other native apps and sites.

The File Share Target API is a game-changer of an API that is now in ChromeCanary. The API extends the Web Share TargetAPIthat lets apps and sites share simple links and text to web sites by integratingthem into the systems sharing functionality.

This very static file blog utilizes the Web Share Target API so I can quicklyshare links that I find interesting to it from anyAndroid application, and as of last week I enabled the File Share Target API sothat I can upload images to my blog directly from the Camera app onAndroid. This post is all about how Idid it (and stole some code from Jake Archibald — tbf he worked out a lotof the bugs for an integration they are doing in tosquoosh.app.)

The File Share TargetAPIis a very novel API in that it is fully progressive. If your application canhandle Form POST requests then you can integrate easily with this API. Thebasic flow is: when the user chooses your application from the native picker,Chrome will send a Form POST request to your server, it is up to you what youdo with it (handle in a service worker or on the server).

To add support for sharing files into your web app you need to do two things:

Declare support for sharing files via the manifest file,
Handle the Form POST request in your Service Worker.

The manifest declares to the host system how Sharing should be mapped from thehost application to the web app. In the manifest below it essentially says “Whena user shares a file of type ‘image/*’ make a Form POST request to‘/share/image/’ and name the data ‘file’“.

manifest.json

{
  "name": "Blog: Share Image",
  "short_name": "Blog: Share Image",
  "start_url": "/share/image/",
  "theme_color": "#000000",
  "background_color": "#000000",
  "icons": [ {
      "sizes": "192x192",
      "src": "/images/me.png",
      "type": "image/png"
  }],
  "share_target": {
    "action": "/share/image/",
    "method": "POST",
    "enctype": "multipart/form-data",
    "params": {
      "files": [
        {
          "name": "file",
          "accept": ["image/*"]
        }
      ]
    }
  },
  "display": "standalone",
  "scope": "/share/"
}

Once the user shares to your web application, Chrome will make the web requestto your site with the file data as the payload.

It is recommended that you handle the POST request inside your service worker sothat 1) it is fast, 2) resilient to the network not being available. You can dothis as follows:

serviceworker.js - demo

onfetch = async (event) => {
  if (event.request.method !== 'POST') return;
  if (event.request.url.startsWith('https://paul.kinlan.me/share/image/') === false) return;

  /* This is to fix the issue Jake found */
  event.respondWith(Response.redirect('/share/image/'));

  event.waitUntil(async function () {
    const data = await event.request.formData();
    const client = await self.clients.get(event.resultingClientId || event.clientId);
    // Get the data from the named element 'file'
    const file = data.get('file');

    console.log('file', file);
    client.postMessage({ file, action: 'load-image' });
  }());
};

There are a couple of interesting things happening above, which can quicklysummarized as:

Render the UI as the result of the POST request by performing a redirect.
Read the data that is submitted via the form via event.request.formData()
Send the data to the open window (this will be the UI that we redirected theuser to in the first point).

It is entirely up to you what you do with the data that has been posted to yourservice worker, but in the case of my App I needed to show it directly in the UIso I have to find the window the user is using and postMessage the data there.

index.html - demo

navigator.serviceWorker.onmessage = (event) => {
  console.log(event);
  imageBlob = event.data.file;
  // Update the UI with the data that has been shared to it.
  imageShare.src = URL.createObjectURL(imageBlob);
};

And that’s about it. If you already have an API endpoint for your web forms,then this is a simple, yet powerful addition that you can make to your site.

The Web Share Target API incredibly powerful platform primitive that breaks downanother barrier that web apps have had on their host platforms.

Registering as a Share Target with the Web Share Target API

Paul Kinlan — Fri, 07 Dec 2018 05:42:30 +0000

Pete LePage introduces the Web Share Target API and the the availability in Chrome via an origin trial

Until now, only native apps could register as a share target. The Web Share Target API allows installed web apps to register with the underlying OS as a share target to receive shared content from either the Web Share API or system events, like the OS-level share button.

Read full post.

This API is a game changer on the web, it opens the web up to something that was only once available to native apps: Native Sharing. Apps are silos, they suck in all data and make it hard to be accessible across platforms. Share Target starts to level the playing field so that the web can play in the same game.

The Twitter Mobile experience has Share Target already enabled. This post was created using the Share Target I have defined in my sites ‘admin panel’ manifest.json - it works pretty well, and the minute they land file support I will be able to post any image or blob on my device to my blog.

Very exciting times.

Read the linked post to learn more about the time-lines for when this API should go live and how to use the API.

Barcode detection in a Web Worker using Comlink

Paul Kinlan — Tue, 02 Oct 2018 21:05:31 +0000

I’m a big fan of QRCodes, they are very simple and neat way to exchange data between the real world and the digital world. For a few years now I’ve had a little side project called QRSnapper — well it’s had a few names, but this is the one I’ve settled on — that uses the getUserMedia API to take live data from the user’s camera so that it can scan for QR Codes in near real time.

The goal of the app was to maintain 60fps in the UI and near instant detection of the QR Code, this meant that I had to put the detection code in to a Web Worker (pretty standard stuff). In this post I just wanted to quickly share how I used comlink to massively simplify the logic in the Worker.

qrclient.js

import * as Comlink from './comlink.js';

const proxy = Comlink.proxy(new Worker('/scripts/qrworker.js')); 

export const decode = async function (context) {
  try {
    let canvas = context.canvas;
    let width = canvas.width;
    let height = canvas.height;
    let imageData = context.getImageData(0, 0, width, height);
    return await proxy.detectUrl(width, height, imageData);
  } catch (err) {
    console.log(err);
  }
};

qrworker.js (web worker)

import * as Comlink from './comlink.js';
import {qrcode} from './qrcode.js';

// Use the native API's
let nativeDetector = async (width, height, imageData) => {
  try {
    let barcodeDetector = new BarcodeDetector();
    let barcodes = await barcodeDetector.detect(imageData);
    // return the first barcode.
    if (barcodes.length > 0) {
      return barcodes[0].rawValue;
    }
  } catch(err) {
    detector = workerDetector;
  }
};

// Use the polyfil
let workerDetector = async (width, height, imageData) => {
  try {
    return qrcode.decode(width, height, imageData);
  } catch (err) {
    // the library throws an excpetion when there are no qrcodes.
    return;
  }
}

let detectUrl = async (width, height, imageData) => {
  return detector(width, height, imageData);
};

let detector = ('BarcodeDetector' in self) ? nativeDetector : workerDetector;
// Expose the API to the client pages.
Comlink.expose({detectUrl}, self);

I really love Comlink, I think it is a game changer of a library especially when it comes to creating idiomatic JavaScript that works across threads. Finally a neat thing here, is that the native Barcode detection API can be run inside a worker so all the logic is encapsulated away from the UI.

Read full post.