DEV Community: nathant

Build a robot that plays hide and seek (Raspberry Pi + AI)

nathant — Wed, 23 Mar 2022 09:01:32 +0000

Building a robot from scratch can be an intimidating task. We however accepted the challenge to build a robot which you can play hide and seek with. The project has three key elements: a frontend for players, a backend for game logic and the robot itself. In this article we will mostly cover the hardware and software of the robot and how we managed to make it work with the backend and frontend on a high level.

The game.

The idea behind the game is to play hide and seek with a robot. Making use of a webapplication, a game master can start a new game which other players can join. Then, a robot in the area will autonomously join this game too. Next, it’s the robot that will have to find all players in order to win the game. With the help of AI, sensors and a camera, the robot will navigate itself through the room to find the players. If the robot doesn’t succeed in his mission finding all players within a certain time, the players have won from the robot. Cool right? Now let’s dig in to the part where we will explain all the bits and bytes of how we managed to realise this.

Victor the robot.

Please meet Victor, our three-weeled robot which we will explain more about.

Hardware

To build the robot, we used:

“CamJam EduKit 3", a kit that contains most basic parts to build the robot like wheels, motors, etc. Read more about it here.
Raspberry Pi 4B 2GB RAM
Raspberry Pi Camera Module 2

Thanks to the CamJam Edukit building the robot was a pretty easy task. It took us a couple of hours to put all parts together.

We made sure the camera is pointed up so the robot won’t have a hard time detecting and recognizing humans.

Software

Once our robot is put together, we move on to the next step which is writing its software.

We run Python code on our Pi which will do various things like:

Human detection
Facial recognition
Autonomously driving in a space
Communicating with the games’s API service
Orchestrating all the different tasks

Human detection (mobilenet-ssd model)

We struggled for a long time finding a quick and accurate human detection model that works well on our Pi which has limited computing power.

After trying out lots of different models, we decided to use the pre-trained MobileNet-SSD model which is intended for real-time object detection. One reason why we chose this algorithm is because it gives good detection accuracy while being quicker than different models, like for example, YOLO. Especially when attempting to detect humans in real time on low computing devices as in our case.

In the background we also used the open-source library OpenCV which is needed to capture and process camera’s output.

Facial recognition

The robot should be able to recognize faces. To make this possible, we used the well-known face-recognition Python library.

Source: face-recognition documentation

It can recognize and manipulate faces from Python using dlib’s state-of-the-art face recognition built with deep learning. Further it’s also lightweight, which is good for our Pi. At last it achieves very good accuracy scores (99.38% on LFW benchmark). That’s exactly what we were looking for when thinking about a face recognition model.

Autonomous driving (ultrasonic distance sensor)

To make autonomous driving possible, the Python library gpiozero was used. This library contains easy commands to steer the CamJam robot and use the distance sensor.

While driving, the robot avoids possible obstacles by using the ultrasonic distance sensor.

😵‍💫 Ultrasonic distance what?!
An ultrasonic distance sensor sends out pulses of ultrasound and detects the echo that is sent back when the sound bounces off a nearby object. It then uses the speed of sound to calculate the distance from the object.

When a person is detected by the camera, a more precise steering mechanism takes over. This will make the robot drive directly towards the detected person. To make this work, we implemented an algorithm that calculates how much degrees the robot should turn to have the detected person in the center of its sight. Like this, the robot can drive and turn autonoumously through a room.

Here is an example of how we used the ultrasonic distance sensor to drive towards a human:

def is_not_at_human():
    global distance_threshold_human
    distance = sensor.distance * 100
    return distance > distance_threshold_human

def approach_human():
    logging.info('Approaching human.')

    while is_not_at_person():
        robot.forward(speed)
        time.sleep(0.1)

    logging.info('Human reached.')

  robot.stop()

Communication with game API

Communication with the API is important to make sure the robot plays the game correctly, but first the robot needs to connect to an open game.

To make sure the robots can play along, we set up communication between the robot and the backend service with an API. When the robot is switched on, it will start polling. With the use of polling, the robot keeps looking if there is an open game in its vicinity.

💡 Ehm, what’s polling?
The simplest way to get new information from the server is periodic polling. This means sending regular requests to the server: “Hey there, It’s Victor the robot here, do you have anything new for me?”. For example, once every 10 seconds.

When a game is found, the robot keeps polling to retrieve player information and check if the game has started. If that is the case, the robot stops using polling and starts to hunt the players.

When a player is found, the robot sends this information to the API. When all the players are found, or the seeking time is over, the robot disconnects itself from the game and starts looking for another game to join.

Orchestrating all the different tasks with threading

One of the biggest challenges was to orchestrate all the different tasks of the robot in a proper way. The robot’s tasks are:

Driving with distance sensor
Calculating how to follow human
Human detection
Facial recognition

To do this, we used the perks of threading with Python. Each thread will start executing its task once a certain event is fired. For example if a human is detected (event), another thread will execute the code to approach the human. Then once the human is approached (event), another thread will do its actions and so on.

Short overview of the flows:

A user friendly webapp with React.

Players need a web interface to interact with the game. Therefore we built a webapp on which players can start a game, join a game, follow the game progress and so on.

When joining a game, the player has to provide a name and up to six photos. These photos will then be used for the robot’s facial recognition.

The app is build with React and hosted on Firebase. It continously makes use of the backend API to fetch information on the games and players. To achieve a user friendly UI, we chose to work with the well-known React MUI design framework. All this together resulted in an easy-to-use, fast and reliable frontend for players.

Building the API with Java SpringBoot.

The robot and the frontend need to retrieve and manipulate data on the game somehow. To make this possible we made a simple REST API with Java SpringBoot.

The backend’s main responsibility is to store data provided by users and make sure the robot can retrieve it. To do this, the backend makes use of a Firestore database.

Another key thing the backend does is handling incoming events. These events include creating, starting and ending a game and a player being found by the robot.

To make the backend (API) available for the clients, we’ve dropped it in a Docker container and deployed it on Google Cloud Run with CI/CD.

That's about it.

Congrats if you made it until here. While we're already playing hide and seek with Victor, we hope you also managed to build a cute and smart sibling for him.

Credits for the R&D and the article:
Thijs Hoppenbrouwers
Joris Rombauts
Nathan Tetroashvili

This project was commissioned by KdG University College.

Thank you to our mentors at KdG (Geert De Paepe, Toni Mini) for guiding us through this project.

How I built my own simplified React with Chevrotain, Typescript & Webpack

nathant — Sat, 22 Jan 2022 23:18:25 +0000

"Know how to solve every problem that has been solved." - Richard Feynman

Over the past 2 months, I’ve been working on my own very simplified version of React called Syntact. I wouldn’t call it mature yet, but it already has a couple of features working to be usable, such as:

variable declaration
function declaration
components
virtual DOM
dynamic rendering

Besides that, I’ve also built a custom compiler as a replacement for Babel.

I made this project for a course called Advanced Programming which is a part of my bachelor Applied Computer Science. When I started this project, I had no idea what I was doing. But thanks to my coach (s/o to Lars Willemsens) and the almighty internet, I somehow managed to create something cool.

This is not really a tutorial on how to make your own React but it certainly is a good starting point for you if you’d like to do this kind of project yourself. So let’s get started.

1. The Compiler (our own kind of Babel)

Lexing

The first step is to write a ‘lexer’ or a ‘tokenizer’. ‘Lex’ stands for lexical analysis, which basically means splitting your text into tokens. It's being used in creating programming languages but also for text processing and various other things.

Token

A token is a small unit of the code. It is structured as a pair consisting of a token name and a value. Example: the keywords "let" or "const" are tokens.

Lexing with Chevrotain

Writing a lexer is the first and easiest step of the whole process. I chose to use the toolkit Chevrotain to build my lexer.

To use the Chevrotain lexer we first have to define the tokens:

/// Keywords
const Import: chevrotain.ITokenConfig = createToken({ name: "Import", pattern: /import/ });
const From: chevrotain.ITokenConfig = createToken({ name: "From", pattern: /from/ });
const Return: chevrotain.ITokenConfig = createToken({ name: "Return", pattern: /return/ });
const Const: chevrotain.ITokenConfig = createToken({ name: "Const", pattern: /const/, longer_alt: Identifier });
const Let: chevrotain.ITokenConfig = createToken({ name: "Let", pattern: /let/, longer_alt: Identifier });
...

// We then add all the tokens to an array of tokens
let allTokens = [...]

Okay so we defined our tokens and bundled them in an array. Next, we instantiate the lexer by passing the tokens to the constructor and voila. Just like that the Syntact lexer was born.

const syntactLexer: Lexer = new chevrotain.Lexer(allTokens);

Now we can use this lexer to tokenize our input.

Check out Chevrotain’s docs for more info: https://chevrotain.io/docs/tutorial/step1_lexing.html.

Parsing

The second step of the process is parsing. The parser converts a list of tokens into a Concrete Syntax Tree (CST), a fancy term for a tree data structure that represents source code.

To prevent ambiguities, the parser must take into account parenthesis and the order of operations. Parsing itself isn’t very difficult, but as more features get added, parsing can become very complex.

Parsing with Chevrotain

Again, I used Chevrotain to build a parser for Syntact. A Chevrotain parser analyses a token that conforms to some grammar.

Grammar

A grammar is a description of a set of acceptable sentences. Our parser will use this grammar to build its tree. I wrote my grammar with the ANTLR grammar syntax.

Here are some examples from my grammar file:

importStatement 
    :  import SEMICOLON
    ;

binaryExpression
    :   atomicExpression operator atomicExpression 
    ;

In the above example we define how an Identifier should look like, what the escape sequence is and how to recognize an import statement.

But to be honest, when using Chevrotain, it’s not really necessary to write the grammar in such a way in order to have a working parser. On the other side, it will help you to get a better view on how to build your parser.

Writing a parser

Once you've got your grammar mapped out, it's time to start building your parser. As we said before, the parser must transform the output of the lexer into a CST.

First we start by making a Parser class which we will invoke with the array of tokens that we used to define our Lexer.

class SyntactParser extends CstParser {
    constructor() {
        super(allTokens)
        this.performSelfAnalysis()
    }

  // Later on, all grammer rules will come here...

}

Next we write Grammar Rules within our Parser class. Two (shortened) examples:

public importStatement = this.RULE("importStatement", () => {
        this.SUBRULE(this.import)
        this.CONSUME(Semicolon)
    });
});

public function = this.RULE("function", () => {
        this.CONSUME(Function)
        this.CONSUME(Identifier)
        this.CONSUME(OpenRoundBracket)
        this.SUBRULE(this.parameterDeclaration)
        this.CONSUME(CloseRoundBracket)
        this.CONSUME(OpenCurlyBracket)
        this.MANY(() => {
            this.OR([
                { ALT: () => { this.SUBRULE1(this.declareVariableStatement) } },
                { ALT: () => { this.SUBRULE(this.functionStatement) } },
                { ALT: () => { this.SUBRULE(this.functionCall) } }
            ])
        })
        this.OPTION(() => this.SUBRULE(this.returnStatement))
        this.CONSUME(CloseCurlyBracket)
    });

We'll write grammar rules according to the grammar which we've mapped out earlier using the ANTLR grammar syntax.

Once that's done - believe me, it takes a while - we can start parsing the tokens. The output will be a CST that Chevrotain builds for us.

AST

Once we have our CST, we are going to convert it to an Abstract Syntax Tree (AST). An AST is like a CST but it contains information specific to our program which means it doesn't contain unnecessary information like Semicolons or Braces. In order to obtain an AST, we have to ‘visit’ the CST using a CST Visitor or how I like to call it, an Interpreter.

Interpreter

The interpreter will traverse our CST and create nodes for our AST. Thanks to Chevrotain, this is a relatively doable step.

Here is a tiny look at the Syntact interpreter:

class SyntactInterpreter extends SyntactBaseCstVisitor {

    constructor() {
        super();
        this.validateVisitor();
    }

    ...

    declareComponent(ctx: any) {
        const componentName = ctx.Identifier[0].image;
        const parameters = this.visit(ctx.parameterDeclaration);
        const returnStatement = this.visit(ctx.returnStatement);
        const variableStatements = [];

        if (ctx.declareVariableStatement) {
            ctx.declareVariableStatement.forEach((e: any) => {
                variableStatements.push(this.visit(e))
            })
        }

        return {
            type: types.COMPONENT_DECLARATION,
            id: {
                type: types.IDENTIFIER,
                name: componentName
            },
            parameters,
            body: { variableStatements },
            returnStatement
        };
    }

    ...
}

Generator

Get the point of an AST? Cool! Now we can go on and start with the generator. The generator will actually make JS code based on the AST.

I find this one of the hardest parts of the whole parsing process. You’ll have to iterate over all the nodes in the AST and make working JS code from it.

Here is how that might look like:

class SyntactGenerator implements Generator {

    ...

    private convertFunBody(body: any) {
        let returnCode: any[] = [];

        if (body.variableStatements) {
            body.variableStatements.forEach((vS: any) => {
                let datatype = vS.dataType;
                let varName = vS.variableName;
                let value = vS.value;
                returnCode.push(`${datatype.toLowerCase()} ${varName} = ${value};\n`)
            });
        }

        if (body.functionCalls) {
            body.functionCalls.forEach((fC: any) => {
                let params: string[] = [];
                if (fC.params) {
                    fC.params.forEach((p: string) => { params.push(p) })
                }
                returnCode.push(`${fC.function}(${params.join(",")});`)
            });
        }

        return returnCode.join("");
    }

    ...
}

Err, come again, please.

Exhausted and a bit confused after reading all this? I get you. Here’s a recap:

Lexer => responsible for transforming raw text into a stream of tokens.
Parser => transforms the stream of tokens into Concrete Syntax Tree (CST).
CST Visitor/Interpreter => recursively visits each node in CST which results in an Abstract Syntax Tree (AST).
Generator = > actually makes JS code based on the provided AST.

Once we got the above things working, we can start making something I called a “SyntactEngine”.

SyntactEngine

Next, I made a SyntactEngine class. It will make it easier for us to orchestrate the different phases of transpiling our JSX to JS. It holds an entrypoint method called “transpileJsxToJs” which we can later use in our Webpack loader.

class SyntactEngine implements Engine {
    private lexer: Lexer;
    private parser: SyntactParser;
    private interpreter: SyntactInterpreter;
    private generator: Generator;

    constructor() {
        ...
    }

    transpileJsxToJs(input: string): string {
        ...
    }

    tokenizeInput(input: string): ILexingResult {
        ...
    }

    parseInput(lexingResult: ILexingResult): ParseResultType {
        ...
    }

    toAst(parsedInput: ParseResultType) {
        ...
    }

    generateJsFromAst(ast: string): string {
        ... 
    }

}

2. Syntact API

We have a working compiler that can generate JS code from JSX. Now we need to build a Syntact API that can actually do the things that a framework like React can do. Create a virtual DOM, hold states and so on.

I just sticked to a simple virtual DOM for now. For this I made a small recursive algorithm that creates a DOM based on the initial given element (a div for example) and all its members.

Here is a shortened version of the method:

createDom(type: string, props: any, members: any, value: string | null) {
        const element: any = document.createElement(type, null);

        props.forEach((prop: any) => {

            if (prop.type.substring(0, 2) === 'on') {
                /* Check if prop type is a function handler
                 * Note: eval might be a security risk here. */
                element[prop.type.toLowerCase()] = () => {
                    eval(prop.value)
                }
            } else if (prop.type == 'class') {
                element.classList.add(prop.value)
            }
        });

        return element;

    }

3. Webclient + Webpack

Once we’ve got the compiler and the Syntact API, we can start integrating both into our client app using a webpack loader.

The webpack loader will preprocess the Syntact JSX by using the compiler and convert it to JS code. Then, the JS code will use the Syntact API to actually use Syntact’s features.

The End

If you made it this far thanks for reading! I hope this article helps you understand how React and Babel work under the hood.