DEV Community

Cover image for qoin: gRPC API for MediaPipe
hayashikun
hayashikun

Posted on

qoin: gRPC API for MediaPipe

This article is an English-translated of this blog post.
https://blog.hayashikun.com/entry/2020/12/21/183000

I will introduce a project named qoin to make it easy to use mediapipe detection data.
Also, I'll show some application which uses qoin.

What is MediaPipe?

MediaPipe is an open-source, cross-platform application that detects hands, faces, and objects on the video in real-time using machine learning techniques.

google/mediapipe

This project is developed by Google and mainly written in C++.
In the MediaPipe, Tensorflow performs the detection.

MediaPipe is indeed interesting; however, it is difficult to extract, save, and use the detected data in other applications.
The only (as long as I confirmed) way was to rewrite code and build by myself.

So, I developed qoin to use detection data easily.

qoin

qoin is a gRPC server and client application and is written in C++.
It includes the detection part of C++ code from MediaPipe.

qoin-system

Because MediaPipe communicates internally with Protocol Buffers, I used gRPC, neither REST nor GraphQL.

You can run qoin as a gRPC server or client.
The gRPC server returns messages containing detected data as the response, and the service is named PullStream.
Contrary, the gRPC client sent messages as the request, named PushStream.

You can make an application that uses detection data by just implementing a gRPC server or client following the proto definition.

For now, The Face Mesh and Hand Tracking have been implemented.

face-mesh
hand-tracking

The proto definitions of qoin are here.
https://github.com/hayashikun/qoin/tree/master/qoin/proto

The definitions defined in MediaPipe are also needed, so you have to include these definitions explained as follows.

Fun applications

poin

This is a desktop application that a pointer moves in the display following hand moving.
It is written in Rust using conrod as the GUI framework.
There is no official gRPC implementation, but we can use such as tonic.

qoin is running as a server, providing HandTrackingPullStream service.

In the build time, it is looking for the proto definitions of qoin and MediaPipe.
https://github.com/hayashikun/poin/blob/master/build.rs

pyoin

This is a Python application that judges the direction of the head, the shape of the hand, and the number of the fingers.

head-direction
rock-paper-scissors

qoin is running as a server, providing FaceMeshPullStream service for head-direction and HandTrackingPullStream service for rock-paper-scissors.

The proto definitions of qoin and MediaPipe are included here.
https://github.com/hayashikun/pyoin/blob/master/codegen.py

The shape of the hand and the number of the fingers were evaluated using simple neural networks.
https://github.com/hayashikun/pyoin/blob/master/data/janken.ipynb

You can easily save detection data for training.
https://github.com/hayashikun/pyoin/blob/master/app/sampler/hand_tracking.py

qover

The poin and pyoin work on the same PC as the PC qoin running.
qover is an application that exchanges data across the network.

By using qover, you can run poin and pyoin on another machine from the PC qoin running.

qover

As I explained before, qoin runs as both a server and client.
qover receives data from qoin running as a client (PushStream) and sends other applications as a server (PullStrem).

This an example of qover.
qover
qoin is running on the PC as the client, and the data are sent to the qover deployed on ECS.
qover deliver the data to the web page in the smartphone.

The web page was made using Vue, and the detection and display are written like this.
https://github.com/hayashikun/geekten20/blob/master/src/components/HandTransfer.vue

Because of the CORS, you need to use an envoy proxy or something.
https://github.com/hayashikun/qover/blob/master/envoy/envoy.yaml

Bazel

The MediaPipe and many of Google's projects use Bazel as a build tool, and qoin also uses Bazel.

The dependencies are written in the WORKSPACE file.
Most are of MediaPipe.

Some rules such as cc_proto_library are defined in both the @com_google_protobuf and @rules_cc repository, and the build would fail if you load the wrong rules.

Links

Discussion (0)