DEV Community: hitesh saini

Using Javascript to Improve upon my english language skills!

hitesh saini — Sun, 31 Jan 2021 17:32:42 +0000

This is the story of my journey where I am going to share how I am using my technical skills to improve upon my soft skills.
Please bear with me, As this story becomes from non technical to super technical or vice-versa.
As we know best way to learn a language is to read or write in that language.I was already reading the books (mostly technical books). I also had to Improve upon my writing skills so I found a website called 750words.com where user can write whatever they want in 750 words on daily basic in their private post. So I started to write 750 words on daily basis. I was also looking for ways to practice my spoken English and also so much typing sucks! when it is not code. one day I got to know about speech Recognition API from MDN docs. so I quickly figured out if I can integrate this API in Chrome extension then this will be available on every page in Chrome browser to type with voice, So I started to develop a Chrome extension called speech recognition Toolkit[https://github.com/fxnoob/speech-recognition-toolkit]
core of this extension contains only turning on and off the speech recognition API, excluding this everything is handled as command, which is truly extendable.

So to write a new Command in this chrome extension

new_command.js

// just write a new function with this signature
export default async () => {
  const commandAlias = 'name_of_command' 
  const description = 'description of command'
  return {
    id: 'BAC548F8-69DB-07DE-2AA6-E687AEF889CC_random_id',
    name: commandAlias,
    description,
    match: 'exact',
    exec: async (text, options) => {
      // your logic
      // this will be executed when command is spoken
     // text contains the recognized text 
     // options is an object which contains dom object
     // dom object has useful methods such as simulateWordTyping
    // simulateWordTyping invokes the keyboard events to type words
     const { dom } = options;
     // type anything you want eg. even emojis
      dom.simulateWordTyping('your🌻text');
    },
  };
};

And just Import this new file in https://github.com/fxnoob/speech-recognition-toolkit/blob/master/src/services/commandsService.js to include it in the codebase. isn't that dead simple?

This way I got to make some useful commands such as

typing emojis with voice from list of 1800 emojis.
Doing basic maths with voice (add, subtract, multiply) And couple of other useful stuff.

As need grew, I wanted it to work in my native language too, As I don't know how to type in my native language with available keyboard. Again Speech Recognition API came into rescue. It provides support for multi language speech recognitions with different dialects for free.
So first I had to translate every existing command to all languages which were supported in the Speech Recognition API.
so I maintained json files containing translation for each text/command in every supported language and I created a Service called translation service[https://github.com/fxnoob/speech-recognition-toolkit/blob/master/src/services/translationService.js] to have translation for any text based upon its key and selected language ID.

So to write a new Command for every supported language

new_command.js

// import translation service file
// but first append your translations in json files available in the src/app/_locales directory of project source.
import translationService from '../translationService';
// just write a new function with this signature
export default async (langId) => {
  const commandAlias = await translationService.getMessage(
    langId,
    'command_label'
  );
  const description = await translationService.getMessage(
    langId,
    'command_cmd_name_description'
  );
  return {
    id: 'BAC548F8-69DB-07DE-2AA6-E687AEF889CC_RANDOM-ID',
    name: commandAlias,
    description,
    match: 'exact',
    exec: async (text, options, callback) => {
      // your logic
      // this will be executed when command is spoken
     // text contains the recognized text 
     // options is an object which contains dom object
     // dom object has useful methods such as simulateWordTyping
    // simulateWordTyping invokes the keyboard events to type words
     const { dom } = options;
     // type anything you want eg. even emojis
      dom.simulateWordTyping('your🌻text');
    },
  };
};

That's how, this project became useful for others as well. You can contribute to the project as its open source or just use it if you like.

In next post I'm going to share how I made it work with Different environments other than Chrome pages eg. Windows/Mac/Linux with the help of ElectronJS + chrome extension[https://github.com/fxnoob/voice-typing-for-desktop].
Have a Good day!

A chrome extension with a hand gesture and speech recognition capabilities.

hitesh saini — Tue, 18 Aug 2020 03:00:19 +0000

Here I will be discussing a chrome extension Hand in the Air which I've developed with the help of open-source projects and my little knowledge on browser extensions which manages user scripts and invokes these scripts based upon user interaction (hand gesture and voice input).

I wanted to make a chrome extension which could manage the user scripts(scripts which run on the particular domain) like Greasemonkey but with some user interaction like waving a hand in front of the webcam (hand gesture recognition) or through voice interaction(Speech recognition) so basically a Greasemonkey extension on Steroids.

Before taking on this project I wanted to use reactjs lib and import/export in chrome extension. as create-react-app was not useful in this.
Along the way, I've figured out that I could do it with the help of a transpiler or a next-gen compiler (in this case it was babel) and a bundler which was webpack.

After creating chrome-extension-react-boilerplate setup, I came along this library Gest.js which uses diff algorithm to recognise hand movement(left, right, up, down). I modified it to make it importable in the project as initially it was written in es5. After this main issue was how should I include this library so it asks for camera permission only for the first time. and could operate on any browser tab that user visits. so for this, I put this script on the options page and the background script of the extension. so for first-time browser extension asks camera/audio permissions from options page, then the browser is whitelisted for camera/audio uses then I could easily operate on camera/audio input when the user clicks on the extension icon. on second run extension could get access camera/audio from background script. and recognized gesture direction is sent to the active tab with the help of message passing APIs. After this, I created an options page UI form to save user scripts on a particular domain.

there are two types of user-scripts for hand gesture -

1. default user scripts
2. custom scripts

default scripts are the scripts that are already available in the extension eg. mapping arrow keys with left, right, up, down hand gestures on any webpage(useful in playing games on http://play2048.co).

In custom scripts, the user can type their logic as gesture object was exposed in the custom script APIs.

eg. On tinder web(tinder.com) user can wave hand left to right or vise versa to like or dislike a profile

if (!gesture.error) {
  var el = null;
  if (gesture.direction === "Left") {
    el = document.querySelector('[aria-label="Nope"]');
    el.click();
  } else if (gesture.direction === "Right") {
    el = document.querySelector('[aria-label="Like"]');
    el.click();
  }
}

user can go back and forth in slides on https://www.slideshare.net/

if (!gesture.error) {
  if (gesture.direction === "Left") {
    document.querySelector("#btnNext").click();
  } else if (gesture.direction === "Right") {
    document.querySelector("#btnPrevious").click();
  }
}

`
I defined these terms:-

script handler: a handler that recognizes the gestures eg. gestjs
- script handler callback: a callback which is fired by script handler.

So I thought of replacing gestjs with a generic handler so I thought of integrating voice input feature chrome speech recognition APIs .for I used annyang.js which is built on top of chrome speech recognition API and has a couple of utility classes on top of that. basically it turned out as alexa skills for webpages.

I created 2 same types for voice input -

1. default user scripts
2. custom scripts

default user scripts are the same as described above.
In Custom scripts I exposed a string variable command which contains the recognized word or sentence.

eg. to navigate back and forth in slides on https://www.slideshare.net/ use can say next or previous

if (command === "next") {
    document.querySelector("#btnNext").click();
  } else if (command === "previous") {
    document.querySelector("#btnPrevious").click();
  }

Further Development

I want to add one more handler for eye-movement tracking, cause there is a use case for this lets users say click on search while being on https://google.com. this can be written in the above-defined script but for now, the script has to search on the entire page for visible text or something to match the element and click on that and this brute force search could lead to false results very easily.
So to have eye-tracking I came along this open-source project called webgazer it wasn't ready to use the library so I made a pr there and making couple of more tweaks to make it usable in chrome extension. so that searching could be limited to bounding box of 100px relative to where user is currently looking at the webpage.

thanks for the read and Stay Healthy!