DEV Community: Sanjeet

Speakr - Translate Gestures to Speech

Sanjeet — Tue, 05 Jan 2021 21:30:50 +0000

What I built

Speakr is a web app which allows you to write in air, using your mobile phone as a pen - gesture to speech translation.

Category Submission:

Random Roulette

App Link

Speakr Web App - for mobile phones with an onboard IMU. As of writing, the Sensor API only works on Android.

Screenshots

Description

Speakr is a web app which allows you to write in air, using your mobile as a pen. It utilises the onboard IMU to record movements, before translating them to an image, and then running handwriting recognition on them to determine the written text. This text is then played out loud, via text-to-speech.

Link to Source Code

Speakr App Repo

Permissive License

MIT

Background

I wanted to explore the possibility of using a mobile phone as a pen, with handwriting in air being an interesting opportunity. Almost all phones have an onboard inertial measurement unit so this was definitely possible.

How I built it

Speakr is a React web app served by a simple NodeJS server, hosted on the DigitalOcean App Platform.

Interfacing with the Chrome Sensor API, whilst "draw" is held down, orientation sensor readings are recorded and relative distance is calculated from the initial orientation, with simple trigonometry. This method allows for greater flexibility as letters can be drawn in varying sizes in air, but all are scaled down to the same size. Rendering the letters in an accurate manner was the bulk of the problem.

When "speak" is pressed, the sensor readings are processed through a combination of scaling and offsets before being rendered on to a canvas element.

From the canvas, an image is generated and calls a backend REST API for translation - a NodeJS server running on the Digital Ocean App Platform. Handwriting recognition is done via the Google Vision API, and the returned text is converted to speech with the Google TTS API. This audio is sent to the app and played back on the mobile phone.

Initially, I started off with the accelerometer, however this was too noisy and unreliable. I quickly switched to the orientation fusion sensor.

During the hacking process, I learnt how to use the DigitalOcean App platform, as well as interfacing with Google Cloud Vision APIs for handwriting recognition. I had this concept as an idea for quite some time. To finally dive in and build it feels like a great accomplishment, especially when I got the Google Vision API to recognise the rendered text after a lot of trial and error.

What's next for Speakr

I would like to develop the gesture-to-text concept and apply it to novel applications, for which further research is required. Regarding the app itself, I would like to train custom ML models to recognise shapes/arrows and other special gestures utilising TensorflowJS, and hook it up to integrations such as IFTTT for more flexibility. For example, custom gestures are important for intuitive controls with smart home devices or perhaps signalling an emergency. Perhaps a new mode of communication?

Speakr - How It Works

Sanjeet — Tue, 05 Jan 2021 21:30:38 +0000

Speakr is a web app which allows you to write in air, using your mobile as a pen. It utilises the onboard IMU to record movements, before rendering these paths as letters in an image. Handwriting recognition is then used to determine the written text which is then played out loud, via text-to-speech.

I created Speakr as an experiment for an alternative mode of interaction - this project nicely ties in with the DigitalOcean App Platform Hackathon.

So what is Speakr for? Speakr demonstrates a novel form of communication, which can be applied to many different scenarios. Perhaps, it can be used as part of third-party mobile games for greater interactivity. With this setup, gestures such as arrows and shapes can be recognised with custom ML models, for more intuitive controls with smart home devices. The possibilities presented by gesture to text or speech translation are truly endless.

Here's the code for the Speakr web app, shown in the video. This web app was built with React, NodeJS and Google Cloud APIs, with the server being hosted on the DO App platform. Below, I will walk through key parts of the gesture to image functionality.

How It Works

Below is an initial test of the drawing functionality, rendered from recorded movements on my mobile phone's IMU. Pretty good!

I chose to use the Generic Sensor API for reading sensor data, shown below as a simple callback.

this.sensor = new AbsoluteOrientationSensor({
      frequency: 60,
});

this.sensor.addEventListener("reading", (e) => this.readSensor(e));

Initially, I started off with the accelerometer, however this was too noisy and unreliable. I quickly switched to the orientation fusion sensor.

By combining data from multiple real sensors, new virtual sensors can be implemented that combine and filter the data so that it’s easier to use — these are known as fusion sensors. In this case, data from the onboard magnetometer, accelerometer, and gyroscope are used for the AbsoluteOrientationSensor’s implementation.

After calculating distance moved, the path moved is pushed to the array storing all previous paths for each letter.

readSensor(e) {
    let q = e.target.quaternion;
    let angles = this.toEuler(q);

    if (!this.draw) {
      this.initAngle = angles;
      this.draw = true;
    }

    let pos = angles.map((angle, i) => this.calcDist(angle, i));
    this.text[this.numChar][0].push(pos[0]);
    this.text[this.numChar][1].push(pos[1]);
  }

The Sensor API returned quaternions, so I had to convert to Euler angles for easier manipulation. Here's a basic implementation in Javascript - pitch has been omitted since I am only working with two dimensions.

// Wikipedia Implementation
  toEuler(q) {
    let sinr_cosp = 2 * (q[3] * q[0] + q[1] * q[2]);
    let cosr_cosp = 1 - 2 * (q[0] * q[0] + q[1] * q[1]);
    let roll = Math.atan2(sinr_cosp, cosr_cosp);

    let siny_cosp = 2 * (q[3] * q[2] + q[0] * q[1]);
    let cosy_cosp = 1 - 2 * (q[1] * q[1] + q[2] * q[2]);
    let yaw = Math.atan2(siny_cosp, cosy_cosp);
    return [yaw, roll];
  }

Distance between the projected initial and current angles are calculated using simple trigonometry and angle differences, with the initial angle positions having been set to the starting orientation of the mobile phone, after the 'Draw' button is pressed down.

calcDist(angle, i) {
    angle = (angle - this.initAngle[i]) * (180 / Math.PI);
    angle = angle < 0 ? angle + 360 : angle;
    angle = angle > 180 ? angle - 360 : angle;
    let dist = -1 * Math.tan(angle * (Math.PI / 180));
    return dist;
  }

Finally, the stored movements have to rendered on to a canvas, in order to generate an image. With a combination of scaling and offsets, each 'letter' is resized into a bounded box and combined with all other letters to form an image with the word drawn in the air.

renderText(ctx) {
    ctx.beginPath();

    this.text.forEach((char, i) => {
      let xpos = char[0];
      let ypos = char[1];

      let xmin = Math.min(...xpos);
      let ymin = Math.min(...ypos);

      if (xmin > 0) xmin = 0;
      if (ymin > 0) ymin = 0;

      let xrange = Math.max(...xpos) - xmin;
      let yrange = Math.max(...ypos) - ymin;

      let xmulti = (this.letterSize * this.letterWidth) / xrange;
      let ymulti = (this.letterSize * this.letterHeight) / yrange;
      let multi = Math.min(xmulti, ymulti);

      let xoffset = this.border + (this.letterWidth - xrange * multi) / 2;
      let yoffset = this.border + (this.letterHeight - yrange * multi) / 2;

      let letterOffset = i * this.letterWidth;

      for (let j = 0; j < xpos.length; j++) {
        let x = xoffset + (Math.abs(xmin) + xpos[j]) * multi + letterOffset;
        let y = yoffset + (Math.abs(ymin) + ypos[j]) * multi;

        if (j === 0) {
          ctx.moveTo(x, y);
        } else {
          ctx.lineTo(x, y);
        }
      }
    });

    ctx.stroke();
  }

An image is then generated from the canvas element. Here are some examples of iterations of tweaking various attributes of the render, such as letter size, margins, line width etc. :

This was the bulk of the problem, since it needed to be legible enough for the Google Vision API to work, and essentially do handwriting recognition to determine the written text. This was mainly trial and error, and took quite some time to get to an acceptable level of accuracy.

Finally, this text is converted to speech with the Google TTS API and spoken by the mobile phone.

And that's it - feel free to play around with Speakr!

Setup Remote Development With VS Code in Your Browser

Sanjeet — Wed, 18 Dec 2019 19:26:40 +0000

In this guide, I will be walking through VSCode Server setup with DigitalOcean, as well as tweaks I have used to create a clean, simple workflow.

Code Server runs Visual Studio Code on a remote server, that is accessible through the browser.

I have found the following setup very useful, in terms of flexibility, productivity and cost-efficiency. For work at college, with Github's Student Pack, I can use my DigitalOcean credit to start up my custom development environment, at a click of a button, which is really cool.

I use this setup to develop the SmartGlove dashboard at college. Here is the GitHub repository with more information and steps for testing the web app. This is still work in progress.

Why do I need this?

Consistent environment: With such a setup, you can code on your Chromebook, tablet, and laptop with a consistent development environment, with a feel at home setup.
Multi-Platform: You will have the power and flexibility of VSCode, accessible from any browser, including tablets.
Server-powered: You can take advantage of large cloud servers to speed up tests, compilations, downloads, and more. With this DigitalOcean setup, servers can be scaled up to any size.
Battery Life: You will preserve battery life when you’re on the go since all intensive computation runs on your server.

The Setup

Here's the plan:

1️⃣ Create Droplet from pre-built image. (possibly, from mobile)

2️⃣ Pull from Github for latest update. (startup)

3️⃣ Provide VSCode in browser, with code-server. (startup)

4️⃣ Do stuff.

5️⃣ Push to Github.

6️⃣ Destroy Droplet.

Droplet Creation

For the guide, I have opted to use DigitalOcean due to their simple and intuitive setup process — however other platforms will also work.

Sign Up

To get started, get $100 in free credit with my referral link, if you don’t already have an account on DigitalOcean.

Create Droplet

Now it’s time to create our droplet — the virtual server in which VSCode Server will run, as well as your project.

Image → Ubuntu 18.04 (default)
Plan → $15/mo with 2 GB/2 CPU (most ideal)
Region → New York (default)
Options → (Not required for the tutorial)
Authentication→ Create a new SSH Key and upload it to the dashboard.

Now hit that create button!

Connect with SSH

Personally, I use Termius, however any SSH client works → here’s a neat guide by DigitalOcean.

Code-Server Installation

It’s time to setup Code-Server. Head over to the latest releases page and copy the link to the latest release for Linux.

Execute the following commands in the console:

# Download latest release from Github (insert copied link)
wget https://github.com/cdr/code-server/releases/download/{version}/code-server{version}-linux-x64.tar.gz

# Unpack tarball
tar -xvzf code-server{version}-linux-x64.tar.gz

# Run Code Server  
cd code-server{version}-linux-x64  
./code-server

Retrieve you’re droplet’s public IP address from the DigitalOcean Control Panel and point your browser to http://{PUBLIC IP ADDRESS}:8080/ .

Copy the generated password from the console output, and log in to Code-Server.

You should now have full VSCode functionality, directly from you browser — how cool is that?

Customisation

Authentication

By default password authentication is enabled, with a randomly generated password. You can set the PASSWORD environment variable to use your own:

sudo nano /etc/environment

# and append the following:

PASSWORD= “Your Password”

sudo reboot

Extensions

As the official VSCode Marketplace cannot be used, Coder has created a custom marketplace, which manages open-source extensions.

Fonts

All fonts work as normal, provided you have it installed on your local machine, as it is your browser which renders the text. For example, Fira Code:

Building Image

Start on Boot

For the setup, I added the following cron jobs, in order to have the latest code pulled from Github on boot, and available in Code Server from the get go.

crontab -e

# And add the following (just an example):  

@reboot cd /root/AwesomeProject && git pull  
@reboot /root/code-server\[$VERSION\]-linux-x86\_64/code-server

Take a Snapshot

DigitalOcean provides an easy way to create to perfect images of servers, from which identical Droplets can be built in the future. Rename and take a snapshot. Once finished, you can destroy your current droplet.

Rebuild

During the droplet creation process, click Snapshots and select your newly created image.

Note → Region and base plan settings are restricted to the droplet, from which the snapshot was created.

After creating your droplet, you should be able to access Code Server as before, and carry on where you left off.

Going Further

To conclude, now you can start up and access your favourite development setup at a click of a button, regardless of network or computer, with minimal costs accrued.

Personally, I would love to be able to access my DigitalOcean control panel, on the go from my phone. There are unofficial apps available which use their API, although with limited functionality. [ Android or iOS ]

From testing the Android client, it works well enough for just creating droplets.

Possible Upgrades

Auto-terminate Droplet using the API after 24 hours, after running git push, in order to avoid costs from forgetting to destroy running droplets.
Use Block Storage for programs and development, to preserve exact state since last access, dynamically mounting to new droplets on creation, in order to avoid constant pushing to Github.