Zoom SDK Powered Claw Machine

Description

Zoom Video SDK allow developers to easily and quickly develop video rich applications to support different scenarios, across multiple platforms (The list below is accurate as of Oct 2022).

• React Native
• iOS
• Android
• Windows
• MacOS
• Web

There are many scenarios which video would plays an important role, example telehealth, remote learning, remote assistance and much more...

Goals

In this multi-part guide you will learn to build a Zoom Video SDK powered Claw Machine, which would be playable over a web browser.

This guide is broken down into :

• Selecting a claw machine / claw game
• Procuring components for this build
• Modifications to electronics
• Introduction to Android on Raspberry Pi
  • Operating Systems
  • GPIO Controls
  • Android Studio
• Programming with Zoom Video SDK for Android
• Programming with Zoom Video SDK for Web
• Finishing Touches

Terminologies

• Transistor semiconductor device used to amplify or switch electrical signals and power
• Protoboard construction base used to build semi-permanent prototypes of electronic circuits
• Raspberry Pi small single-board computers (SBCs) developed in the United Kingdom by the Raspberry Pi Foundation in association with Broadcom
• CSI MIPI Camera Serial Interface (CSI) is a specification of the Mobile Industry Processor Interface (MIPI) Alliance. It defines an interface between a camera and a host processor.
• UVC USB Camera USB video device class (also USB video class or UVC) is a USB device class that describes devices capable of streaming video like webcams, digital camcorders, transcoders, analog video converters and still-image cameras.
• PCB Board medium used in electrical and electronic engineering to connect electronic components to one another in a controlled manner.

Step by Step Guide

Prerequisites

• IDE or Text Editor of your choice (Visual Studio, VS Code, Sublime Text etc..)
  • I'm using Visual Studio Code on macOS for this guide
• Android Studio
• Basic Soldering Skills
• Zoom Video SDK Account refer to this guide https://marketplace.zoom.us/docs/sdk/video/developer-accounts/

Selecting a Claw Machine

For this build, we are looking for a small compact machine. The key requirement which I had in mind are

• USB Powered
• Dedicated catch / claw button
• Ease of modification of PCB board

However ease of modification can only be determined after purchase.

If you would like to use the same build, you might be able to find it here E-commerce site, note that this site is in mandarin

Procuring Components for the build

-Raspberry Pi 3 or Pi 4
-MicroSD Card
-CSI MIPI Camera or UVC USB Camera (Fisheye)

-HDMI LCD Screen

-5V USB Powered LED Strip

-Dupont Header Wires

-Protoboard for Raspberry Pi

-Optional USB Microphone and Microphone

Modifications to electronics

The intention for this section, is to enable our android application to take control of the claw machine. These control are start game, move left, move right, move back, move forward, and catch.

If we disassemble the claw machine to take a look at the PCB board, we quickly discover the movement are triggered by capacitive touch. This is similar technology used by certain build of keyboards. Those marked with yellow stars denotes presence of test points, which we can conveniently solder our logic wires.

1 - 3.3V power source
2 - Used to detect successful catch via IR sensor (unused for this guide)
a - Connect with 1 to move up
b - Connect with 1 to move left
c - Connect with 1 to move down
d - Connect with 1 to move right
e - Connect with 1 to catch
f and g - Connect with each other to start a new game

Connecting or shorting 2 points can be achieve in various manner. You can choose to either use a transistor or an optocoupler.

The Base for each transistor are then directly linked to the GPIO on the Raspberry Pi. Assuming "start" is connected to GPIO Pin 6. Whenever I set Pin 6 (or any other Pins you are using) to Output and High, it will "start" the game on the claw machine.

Introduction to Android on Raspberry Pi

The version of Android which I'm using on Raspberry Pi comes from Lineage OS. The version which runs on Raspberry Pi comes from an author who runs the website https://konstakang.com/

From my own experience, this distribution has pretty good GPIO support, driver supports for USB UVC camera, and driver support for USB Speakers + Microphone.

You want want to refer to https://konstakang.com/ on how to prepare LineageOS on your MicroSD Card.

Zoom Video SDK for Android

Download Zoom Video SDK from https://marketplace.zoom.us

The latest version as of Oct 2022 is v1.5.1 for Android.

Download, extract and open up the mobilertc-android folder on Android Studio

• Add JWT Token to Constants.java Refer to https://marketplace.zoom.us/docs/sdk/video/auth/ on how to generate your JWT token with your SDK Key and Secret Key
• Handle events from Command Channels Look for BaseMeetingActivity.java and search for public void onCommandReceived(ZoomVideoSDKUser sender, String strCmd)

This event handler will receive custom command sent from the web application (which we will create in the next section).

Here's the code snippet

  try {

            switch (strCmd.toLowerCase()) {
                case "start":
                case "s": {

                    startGame();
                }
                break;

                case "catch":
                case "c": {

                    catchToy();
                }
                break;

                case "up":
                case "u": {

                    moveForward();
                }
                break;

                case "down":
                case "d": {
                    moveBackward();
                }
                break;

                case "left":
                case "l": {
                    moveLeft();

                }
                break;

                case "right":
                case "r": {
                    moveRight();
                }



            }
        } catch (Exception ex) {
            ex.toString();

        }

• Call GPIO related commands

I'm using a pre-written GPIO library which can be found below. I've created a class named GPIO.java

https://gist.github.com/lpbas/bca095e23627c7d53addba0a01d6cc09

Within the onCreate() method found in BaseMeetingActivity.java I'll call the below method to initialize the pins. Notice they are set as DIRECTION_OUT

private void initializePins() {

        pin6StartGameOut=new GPIO(6,GPIO.DIRECTION_OUT);
        pin27MoveUpOut=new GPIO(27,GPIO.DIRECTION_OUT);
        pin22MoveDownOut=new GPIO(22,GPIO.DIRECTION_OUT);
        pin23MoveLeftOut=new GPIO(23,GPIO.DIRECTION_OUT);
        pin24MoveRightOut=new GPIO(24,GPIO.DIRECTION_OUT);
        pin12CatchOut=new GPIO(12,GPIO.DIRECTION_OUT);

    }

As startGame(), catchToy(), moveLeft(), moveRight(), moveUp() and moveDown() have similar codebase with slight variations, I've just added one of them below.

GPIO.VALUE_ON to send a high value to the transistor, which then turns on the switch for start game.

   public void startGame(){
        pin6StartGameOut.setValue(GPIO.VALUE_ON);
        try {

            new Handler().postDelayed(new Runnable()
            {
                @Override
                public void run()
                {
                    pin6StartGameOut.setValue(GPIO.VALUE_OFF);

                }
            },125);
        }
        catch (Exception ex){
            ex.toString();

        }


    }

When attempting to run / deploy this on the Raspberry Pi over ADB, you will be prompted for root / SU access. This is normal, as you need root access for GPIO.

Use an arbitrary session name and set a password for your session. We will need this session name when connecting to the same meeting/session from the web.

Zoom Video SDK for Web

My fellow zoomies @jenzushsu has written a web application based off @tommygaessler sample 1 to 1 video chat sample code, which we will be using as a bootstrap.
https://github.com/tommygaessler/vsdktest.
The latest version as of Oct 2022 is v1.5.1 for Web.

Below is the screenshot of the sample 1 to 1 video chat, with responsive web design approach in place to support different form factors to join from the web.

Below is the high level flow of events:
The claw machine starts a video session, as a host, from the Android app

• The participant or “player” will join from a mobile web browser to play the game.

• There is a maximum of 2 participants (including the host) at any single point of time. The sample app will take care of new participants joining when the video session is at capacity.

• In order for the participant to play the claw machine, additional buttons are added on top of the existing buttons.

1 - Start Game / Catch Toy button
2 - Virtual Joystick / Control (nippleJS https://github.com/yoannmoinet/nipplejs)
3 - Game Timer (Animated Countdown Timer https://css-tricks.com/how-to-create-an-animated-countdown-timer-with-html-css-and-javascript/)

You can reference the switch block from the Android to create similar javascript functions to trigger the respective actions (e.g. start game, catch toy). Below is a sample code snippet for startGame()

function startGame() {
  document.querySelector('#stopGame').style.display = 'inline-block'
  document.querySelector('#startGame').style.display = 'none'

  const chat = zmClient.getChatClient();
  chat.sendToAll('s')
  const commandChannel = zmClient.getCommandClient();
  commandChannel.send('s')
}

Using either the Zoom Video SDK Chat Client or Command Channel to send custom commands to the host, the event handler will receive to execute the respective code blocks.

The rest will depend on your own creativity on dynamically having a refreshed timer whenever a game is started or to handle the range of the virtual joystick in moving the claw.

Finishing Touches

Customize it with a little stickers, and USB 5V LED strip, and you are ready to go!