DEV Community: Z. QIU

Ignition (Gazebo) Math Library

Z. QIU — Thu, 28 Apr 2022 23:16:37 +0000

Gazebo plugin building

Today I tried to compile an Gazebo plugin I wrote in 2014 in latest ROS (noetic) environment, unsurprisingly, there are errors and the compilation failed:

fatal error: gazebo/math/gzmath.hh: No such file or directory
    7 | #include <gazebo/math/gzmath.hh>
      |          ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

I check firstly the version of Gazebo simulator on my PC, it is Gazebo 11:

Then I find this page which presents the migrations required for each new Gazebo verison.

Gazebo vs Ignition

Interesting, I see that the namespace "gazebo" has been replaced by "ignition". But why?

I see answer in its official site:

We branded the new development effort Igntion in order to differentiate it from Gazebo that had come before, which is now referred to as Gazebo Classic. Those changes have gone very well, thanks in no small part to contributions and support from our worldwide community of users and other stakeholders.

In 2022 we encountered a trademark obstacle regarding the use of the name "Ignition". We used this as an opportunity to switch back the widely known name "Gazebo". Going forward, the modern robotics software collection formerly known as Ignition, is now branded Gazebo.

Roadmap of Gazebo project:

Ignition Math folder structure

Below are all the files in repository /usr/include/ignition/math6/ which should provide full math support for robot kinematics.

├── math6
│ └── ignition
│ ├── math
│ │ ├── Angle.hh
│ │ ├── AxisAlignedBox.hh
│ │ ├── Box.hh
│ │ ├── Capsule.hh
│ │ ├── Color.hh
│ │ ├── config.hh
│ │ ├── Cylinder.hh
│ │ ├── detail
│ │ │ ├── Box.hh
│ │ │ ├── Capsule.hh
│ │ │ ├── Cylinder.hh
│ │ │ ├── Ellipsoid.hh
│ │ │ ├── Export.hh
│ │ │ ├── Sphere.hh
│ │ │ └── WellOrderedVector.hh
│ │ ├── DiffDriveOdometry.hh
│ │ ├── Ellipsoid.hh
│ │ ├── Export.hh
│ │ ├── Filter.hh
│ │ ├── Frustum.hh
│ │ ├── GaussMarkovProcess.hh
│ │ ├── graph
│ │ │ ├── Edge.hh
│ │ │ ├── GraphAlgorithms.hh
│ │ │ ├── Graph.hh
│ │ │ └── Vertex.hh
│ │ ├── Helpers.hh
│ │ ├── Inertial.hh
│ │ ├── Kmeans.hh
│ │ ├── Line2.hh
│ │ ├── Line3.hh
│ │ ├── MassMatrix3.hh
│ │ ├── Material.hh
│ │ ├── MaterialType.hh
│ │ ├── Matrix3.hh
│ │ ├── Matrix4.hh
│ │ ├── MovingWindowFilter.hh
│ │ ├── OrientedBox.hh
│ │ ├── PID.hh
│ │ ├── Plane.hh
│ │ ├── Pose3.hh
│ │ ├── Quaternion.hh
│ │ ├── Rand.hh
│ │ ├── RollingMean.hh
│ │ ├── RotationSpline.hh
│ │ ├── SemanticVersion.hh
│ │ ├── SignalStats.hh
│ │ ├── SpeedLimiter.hh
│ │ ├── Sphere.hh
│ │ ├── SphericalCoordinates.hh
│ │ ├── Spline.hh
│ │ ├── Stopwatch.hh
│ │ ├── Temperature.hh
│ │ ├── Triangle3.hh
│ │ ├── Triangle.hh
│ │ ├── Vector2.hh
│ │ ├── Vector3.hh
│ │ ├── Vector3Stats.hh
│ │ └── Vector4.hh
│ └── math.hh

Improve the source code

With help of this documentaiton, I have updated some lines of my former source code. And I finally successfully built the plugin and got the .o file.

#include <ignition/math.hh>
    ...

// Load the controller
void WifibotControllerPlugin::Load(physics::ModelPtr _parent, sdf::ElementPtr _sdf)
{
    ...
    ignition::math::Pose3d pose;
    ignition::math::Vector3d pos;

#if GAZEBO_MAJOR_VERSION >= 8
    pose = this->parent->WorldPose();
#else
    pose = this->parent->GetWorldPose().Ign();
#endif

    robotPose[0] = pose.Pos().X();
    robotPose[1] = pose.Pos().Y();
    tf::Quaternion qt( pose.Rot().X(), pose.Rot().Y(), pose.Rot().Z(), pose.Rot().W() );
    ...
}

    ... 

void WifibotControllerPlugin::write_position_data()
{

    ignition::math::Pose3d orig_pose;


#if GAZEBO_MAJOR_VERSION >= 8
    orig_pose = this->parent->WorldPose();
#else
    orig_pose = this->parent->GetWorldPose().Ign();
#endif


  this->parent->SetWorldPose(ignition::math::Pose3d (robotPose[0], robotPose[1],  0, 0, 0, robotPose[2]  )   );

}

load the plugin

Now, the plugin is working properly and I can control my robot in the simulator:

Netlink Communication between Kernel and User space

Z. QIU — Sun, 09 Jan 2022 15:45:08 +0000

Last year， I have used Netlink in my development work for collecting some events from kernel space. I present in this post some basic practice I have done when I learnt to program with Netlink.

Introduction

Netlink is a Linux kernel interface used for** inter-process communication (IPC)** between both the kernel and userspace processes, and between different userspace processes, in a way similar to the Unix domain sockets. Similarly to the Unix domain sockets, and unlike INET sockets, Netlink communication cannot traverse host boundaries. Netlink provides a standard socket-based interface for userspace processes, and a kernel-side API for internal use by kernel modules. Originally, Netlink used the AF_NETLINK socket family. Netlink is designed to be a more flexible successor to ioctl; RFC 3549 describes the protocol in detail.

My practice

Kernel module

Below is the content of my source file "netlink_kernel.c" which yields a kernel module. There is macro MY_NETLINK 30 which defines my customized netlink protocol. One can also choose available protocols such like NETLINK_ROUTE or NETLINK_INET_DIAG. All available protocols can be seen in this page. Function netlink_kernel_create() creates a netlink socket for user space application to communicate with. More information about netlink programming can found here.

netlink_kernel.c



#include <linux/module.h>
#include <net/sock.h>
#include <linux/netlink.h>
#include <linux/skbuff.h>

/*
  refer to https://elixir.bootlin.com/linux/v5.15.13/source/include/linux/netlink.h
*/

#define MY_NETLINK 30  // cannot be larger than 31, otherwise we shall get "insmod: ERROR: could not insert module netlink_kernel.ko: No child processes"


struct sock *nl_sk = NULL;

static void myNetLink_recv_msg(struct sk_buff *skb)
{
    struct nlmsghdr *nlhead;
    struct sk_buff *skb_out;
    int pid, res, msg_size;
    char *msg = "Hello msg from kernel";


    printk(KERN_INFO "Entering: %s\n", __FUNCTION__);

    msg_size = strlen(msg);

    nlhead = (struct nlmsghdr*)skb->data;    //nlhead message comes from skb's data... (sk_buff: unsigned char *data)

    printk(KERN_INFO "MyNetlink has received: %s\n",(char*)nlmsg_data(nlhead));


    pid = nlhead->nlmsg_pid; // Sending process port ID, will send new message back to the 'user space sender'


    skb_out = nlmsg_new(msg_size, 0);    //nlmsg_new - Allocate a new netlink message: skb_out

    if(!skb_out)
    {
        printk(KERN_ERR "Failed to allocate new skb\n");
        return;
    }

    nlhead = nlmsg_put(skb_out, 0, 0, NLMSG_DONE, msg_size, 0);   // Add a new netlink message to an skb

    NETLINK_CB(skb_out).dst_group = 0;                  


    strncpy(nlmsg_data(nlhead), msg, msg_size); //char *strncpy(char *dest, const char *src, size_t count)

    res = nlmsg_unicast(nl_sk, skb_out, pid); 

    if(res < 0)
        printk(KERN_INFO "Error while sending back to user\n");
}

static int __init myNetLink_init(void)
{
    struct netlink_kernel_cfg cfg = {
        .input = myNetLink_recv_msg,
    };

       /*netlink_kernel_create() returns a pointer, should be checked with == NULL */
    nl_sk = netlink_kernel_create(&init_net, MY_NETLINK, &cfg);
    printk("Entering: %s, protocol family = %d \n",__FUNCTION__, MY_NETLINK);
    if(!nl_sk)
    {
        printk(KERN_ALERT "Error creating socket.\n");
        return -10;
    }

    printk("MyNetLink Init OK!\n");
    return 0;
}

static void __exit myNetLink_exit(void)
{
    printk(KERN_INFO "exiting myNetLink module\n");
    netlink_kernel_release(nl_sk);
}

module_init(myNetLink_init);
module_exit(myNetLink_exit);
MODULE_LICENSE("GPL");

User space application

Below are the content of my file "netlink_client.c". This program shall open a netlink socket with the same protocol and allow user to send/receive msg to/from kernel module in previous section.

"netlink_client.c"



#include <sys/types.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <string.h>
#include <asm/types.h>
#include <linux/netlink.h>
#include <linux/socket.h>
#include <errno.h>

#define NETLINK_USER 30 // same customized protocol as in my kernel module
#define MAX_PAYLOAD 1024 // maximum payload size

struct sockaddr_nl src_addr, dest_addr;
struct nlmsghdr *nlh = NULL;
struct nlmsghdr *nlh2 = NULL;
struct msghdr msg, resp;  // famous struct msghdr, it includes "struct iovec *   msg_iov;"
struct iovec iov, iov2;
int sock_fd;

int main(int args, char *argv[])
{
    //int socket(int domain, int type, int protocol);
    sock_fd = socket(PF_NETLINK, SOCK_RAW, NETLINK_USER); //NETLINK_KOBJECT_UEVENT  

    if(sock_fd < 0)
        return -1;

    memset(&src_addr, 0, sizeof(src_addr));
    src_addr.nl_family = AF_NETLINK;
    src_addr.nl_pid = getpid(); /* self pid */

    //int bind(int sockfd, const struct sockaddr *addr, socklen_t addrlen);
    if(bind(sock_fd, (struct sockaddr*)&src_addr, sizeof(src_addr))){
        perror("bind() error\n");
        close(sock_fd);
        return -1;
    }

    memset(&dest_addr, 0, sizeof(dest_addr));
    dest_addr.nl_family = AF_NETLINK;
    dest_addr.nl_pid = 0;       /* For Linux Kernel */
    dest_addr.nl_groups = 0;    /* unicast */

    //nlh: contains "Hello" msg
    nlh = (struct nlmsghdr *)malloc(NLMSG_SPACE(MAX_PAYLOAD));
    memset(nlh, 0, NLMSG_SPACE(MAX_PAYLOAD));
    nlh->nlmsg_len = NLMSG_SPACE(MAX_PAYLOAD);
    nlh->nlmsg_pid = getpid();  //self pid
    nlh->nlmsg_flags = 0; 

    //nlh2: contains received msg
    nlh2 = (struct nlmsghdr *)malloc(NLMSG_SPACE(MAX_PAYLOAD));
    memset(nlh2, 0, NLMSG_SPACE(MAX_PAYLOAD));
    nlh2->nlmsg_len = NLMSG_SPACE(MAX_PAYLOAD);
    nlh2->nlmsg_pid = getpid();  //self pid
    nlh2->nlmsg_flags = 0; 

    strcpy(NLMSG_DATA(nlh), "Hello this is a msg from userspace");   //put "Hello" msg into nlh

    iov.iov_base = (void *)nlh;         //iov -> nlh
    iov.iov_len = nlh->nlmsg_len;
    msg.msg_name = (void *)&dest_addr;  //msg_name is Socket name: dest
    msg.msg_namelen = sizeof(dest_addr);
    msg.msg_iov = &iov;                 //msg -> iov
    msg.msg_iovlen = 1;

    iov2.iov_base = (void *)nlh2;         //iov -> nlh2
    iov2.iov_len = nlh2->nlmsg_len;
    resp.msg_name = (void *)&dest_addr;  //msg_name is Socket name: dest
    resp.msg_namelen = sizeof(dest_addr);
    resp.msg_iov = &iov2;                 //resp -> iov
    resp.msg_iovlen = 1;



    printf("Sending message to kernel\n");

    int ret = sendmsg(sock_fd, &msg, 0);   
    printf("send ret: %d\n", ret); 

    printf("Waiting for message from kernel\n");

    /* Read message from kernel */
    recvmsg(sock_fd, &resp, 0);  //msg is also receiver for read

    printf("Received message payload: %s\n", (char *) NLMSG_DATA(nlh2));  

    char usermsg[MAX_PAYLOAD];
    while (1) {
    printf("Input your msg for sending to kernel: ");
        scanf("%s", usermsg);

        strcpy(NLMSG_DATA(nlh), usermsg);   //put "Hello" msg into nlh


        printf("Sending message \" %s \" to kernel\n", usermsg);

        ret = sendmsg(sock_fd, &msg, 0);   
        printf("send ret: %d\n", ret); 

        printf("Waiting for message from kernel\n");

        /* Read message from kernel */
    recvmsg(sock_fd, &resp, 0);  //msg is also receiver for read

    printf("Received message payload: %s\n", (char *)NLMSG_DATA(nlh2));   

}
    close(sock_fd);

    return 0;
}

Makefile

Create a Makefile with the following content which will enable us to easily compile the source files.

Makefile



obj-m += netlink_kernel.o

#generate the path
CURRENT_PATH:=$(shell pwd)

#the current kernel version number
LINUX_KERNEL:=$(shell uname -r)

#the absolute path
LINUX_KERNEL_PATH:=/usr/src/linux-headers-$(LINUX_KERNEL)



#complie object
#   extension of "make modules" cmd with -C option and "M=dir" configuration
#   this cmd will switch working directory to the given path followed by the -C option
#   and will search specified source files from the given path configured by "M=" 
#   and compile them to generate ko files

all:
    @echo $(LINUX_KERNEL_PATH)
    make -C $(LINUX_KERNEL_PATH) M=$(CURRENT_PATH) modules

client:
    gcc netlink_client.c -o netlink_client -g

#clean
clean:
    make -C $(LINUX_KERNEL_PATH) M=$(CURRENT_PATH) clean
    rm netlink_client

Test the communication

Make sure that files "netlink_kernel.c", "netlink_client.c" and "Makefile" are in the same directory. In a Linux terminal window, cd into this directory and start the compilation:



make 
make client

Normally kernel module file "netlink_kernel.ko" and user application file "netlink_client" will be generated.

Load the generated kernel module "netlink_kernel.ko" to linux kernel:



sudo insmod netlink_kernel.ko

Execute in a new terminal the following cmd to monitor the kernel messages:



dmesg -Hw

Now start the user application to start the communication:



./netlink_client

Below is the screenshot of my test:

About "Core Dump" in Linux

Z. QIU — Fri, 31 Dec 2021 09:18:40 +0000

What is core dump

A core dump is a file of a computer’s documented memory of when a program or computer crashed. The file consists of the recorded status of the working memory at an explicit time, usually close to when the system crashed or when the program ended atypically. So a core dump file is very important for program developers since we often need it to locate and fix crash issues.

How to activate "core dump"

Firstly one can type ulimit -c in terminal console, if the output is 0，it means that the "core dump" is deactivated by default. In this case, Linux will not generate core dump file when a program crashes.
Then one can activate "core dump" by setting a non-zero size for generated core dump file by executing ulimit -c [kbytes]. For example:

ulimit -c 100：set maximum size of core dump file as 100k
ulimit -c unlimited: no size limit of core dump file

Configuration of core dump

Once the core dump has been activated, we can get core dump file generated in local repository. There is still one issue: the core dump file has always the same name as "core" thus it is always over-written if multiple program crashes occur. What if we want to add pid in generated file name, or if we want this file is generated in some other directory?

core dump filename with process id

    echo 1 > /proc/sys/kernel/core_uses_pid

Then process id will be attached in file name.

One can set the output path and coredump file name pattern by updating sysctl variable "kernel.core_pattern" in file "/etc/sysctl.conf"

Add the following phrase at the end of file sysctl.conf:

     kernel.core_pattern = /var/core/core_%e_%p

Save and exit.

Below are the available options in core_pattern setting:

%c maximum size of the generated file
%e file name
%g group ID of process
%h host name
%p process ID
%s signal which triggered this coredump
%t timestamp of this coredump (number of seconds since 1970-01-01)
%u user ID of the dumped process

Once a modification has been made, one can execute the following command to make it take effect:

sysctl –p /etc/sysctl.conf

Please note: once we have set /proc/sys/kernel/core_uses_pid as 1，the generated core dump file still has pid in its name even we do not set %p in core_pattern.

Practice

In this section I present a small practice of coredump. Firstly I performed the following configurations (root privilege might be required) for coredump file generation:

   ulimit -c 1000000
   ulimit -c
   echo 1 > /proc/sys/kernel/core_uses_pid
   echo "/tmp/corefile/core-%e-%p-%t" > /proc/sys/kernel/core_pattern

Then let's write some code to provoke a "Segmentation fault". I put the code lines below into file "testSegmtFault.c ".

#include <stdio.h>

int main()
{
  int IDs[10] = {0,1,2,3,4,5,6,7,8,9};
  int * id_ptr = 0;

  printf("0x%lx: %d \n", (unsigned long)IDs, *IDs);

  // correct logic
  for  (id_ptr=IDs; id_ptr<=(&IDs[9]); id_ptr++)
  {
      printf("0x%lx: %d \n", (unsigned long)id_ptr, *id_ptr);
  }

  // bad logic
  for  (id_ptr=0; id_ptr<=(&IDs[9]); id_ptr++)
  {
      printf("0x%lx: %d \n", (unsigned long)id_ptr, *id_ptr);
  }
}

Compile it then execute the generated executable file:

gcc -g testSegmtFault.c -o seg
./seg

Execution output is as follows and there is a Segmentation Fault :D:

qiu@qiu-ThinkCentre-E73:~/workspace/c_basics/linux_flux$ ./seg
0x7ffe283297d0: 0 
0x7ffe283297d0: 0 
0x7ffe283297d4: 1 
0x7ffe283297d8: 2 
0x7ffe283297dc: 3 
0x7ffe283297e0: 4 
0x7ffe283297e4: 5 
0x7ffe283297e8: 6 
0x7ffe283297ec: 7 
0x7ffe283297f0: 8 
0x7ffe283297f4: 9 
Segmentation fault

Now as expected, there is a coredump file named "core-seg-20017-1639863524" generated in /tmp/corefile/

Now let's use gdb to debug the crash with help of the executable file and the coredump file:

cd /tmp/corefile
gdb ~/workspace/c_basics/linux_flux/seg core-seg-20017-1639863524

We can see that gdb immediately shows the defective code line number:

Binary search algorithm illustration（2）

Z. QIU — Tue, 23 Feb 2021 09:36:13 +0000

Following previous post, I would like to illustrate the solution of the following problem (figure below) using Binary search algorithm.

As shown in the figure above, we are given a sorted array v = [2, 2, 2, 3, 3, 4, 5, 6, 6, 7] with repeated numbers, and we want to find the index of the first occurrence of target number 3. It's easy for human to find that the answer should be 3 (we suppose index of the first element is 0). How for computers?

Algorithm illustration

As usual, we define three pointers: l, r and mid. They point respectively to the lower (left) bound index, the upper (right) bound index and the middle index of the searching range. We calculate mid as mid = l + (r-l)/2.

Initially, we search the entire array. So l = 0, r = 9 and mid = 4. Here is the first step:

Though mid points already to element whose value is equal to 3, we cannot stop searching since we are not sure if its the first occurrence in the array. So we continue to search by changing r to current value of mid. So we narrow down the searching range to the left half of the array: l = 0, r = 4 and mid is updated to 2:

Now the element pointed by mid is 2 and it is smaller than our target number 3. So we narrow down again the searching range to the right half of current searching range. Now l = 3, r = 4 and mid is updated to 3:

Now the element pointed by mid is 3 and it is equal to the target number 3. We now move r to current mid. So l = r = mid = 3. Solution found!

Overview

I put all the steps of this example in one figure to demonstrate overview of the solution:

C++ Code

#include <vector>
#include <iostream>

using namespace std;



/*
* First occurency search 
*/
int foSearch(vector<int>& array, int target) {

    int l = 0, r = array.size()-1;
    while (l<r)
    {
        int mid = l + (r-l)/2;

        if(array[mid] > target)
        {
            r = mid-1;
        }
        else if(array[mid] == target)
        {
            r = mid;
        }
        else{
            l = mid+1;
        }
    }
    cout << "l=" << l <<  ", r=" << r << endl;
    int res = -1;
    if (array[l] == target)
    {
        res = l;
    }

    return res;  
}


int main()
{   
    vector<int> vec {1, 1, 4, 4,9, 9,12,12, 12,32,43, 43,43, 55, 55, 63, 77, 77, 98, 98, 98, 100};
    int ans = foSearch(vec, 43);
    cout << "ans is : " << ans << endl;
    cout << "value is : " << vec[ans] << endl;  

    return 0;
}

Binary search algorithm illustration

Z. QIU — Mon, 22 Feb 2021 23:17:08 +0000

"A picture is worth a thousand words"

In order to explain to a friend how Binary search algorithm works, I drew several pictures to show solution of a simple problem using this algorithm. Here I would like to share it in this post.

As shown in the figure below, we are given a sorted array v = [1, 2, 3, 4, 5, 7, 9, 10, 12, 17], and we want to find index of the target number 9. One can see easily that the answer should be 6 (suppose index of the first element is 0).

Binary search algorithm illustration

In Binary search algorithm, we define three pointers to elements: l, r and mid. They point respectively to the lower (left) bound index, the upper (right) bound index and the middle index of the range for searching. We calculate mid as mid = l + (r-l)/2.

Initially, we search the entire array. So l = 0, r = 9 and mid = 4. Here is the first step:

The element pointed by mid is 5 and it is smaller than our target number 9. So we narrow down the searching range to the right half of the array: l = mid+1 = 5, r = 9 and mid is updated 7:

Now the element pointed by mid is 10 and it is larger than our target number 9. So we narrow down again the searching range to its left half.

We repeat the same steps until the element pointed by mid is equal to the target number 9. See, we find out the index of number 9 in this array is 6.

Binary search algorithm code

Below is the c++ code for this algorithm:

/*
* Binary search 
*/
int binSearch(vector<int>& array, int target) {

    int l = 0, r = array.size()-1;
    while (l<=r)
    {
        int mid = l + (r-l)/2;
        if(array[mid] == target)
        {
            return mid;
        }
        if(array[mid] > target)
        {
            r = mid-1;
        }
        else{
            l = mid+1;
        }
    }

    return -1;

}

Overview

I put all the steps of this example in one figure to demonstrate overview of the Binary search algorithm:

A similar problem

Here is a similar problem which can also be solved by varying a little the Binary search algorithm: in a sorted array with repeated elements, we would like to find index of the first occurrence of the target number. One can think of the solution. I will present the solution using figures in my next post.

Implementation of Tensorflow Lite model on Android

Z. QIU — Thu, 18 Feb 2021 22:31:59 +0000

Recently in some interview I have been asked about experience of implementing trained tensorflow models in android platform. I have tried one android project cloned from github which embedded a tflite model in it. However, I have not yet tried implementing my own model in an Android application. Thus I did such an exercise today and I successfully made my CNN model work on my Redmi Note 8 pro.

CNN model

Here is the code for training a cnn model with mnist data set. This model then is converted as tflite model and shall be implemented in Android application for recognizing hand-write digits.


import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = "2"
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import datasets, layers, optimizers, Sequential, metrics,models



(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
print(x_train[0,:,:])
## x_train.shape => (60000, 28, 28), y_train.shape => (60000,)
## x_test.shape => (10000, 28, 28), y_test.shape => (60000,)
x_train = tf.expand_dims(x_train, -1)
x_test = tf.expand_dims(x_test, -1)

yt = tf.squeeze(y_train)  

y_train = tf.squeeze(y_train)    
y_test = tf.squeeze(y_test)

print("Dataset info: ", x_train.shape, y_train.shape, x_test.shape, y_test.shape)

batch_size = 128

train_db = tf.data.Dataset.from_tensor_slices((x_train, y_train))
train_db = train_db.shuffle(10000).batch(batch_size)

test_db = tf.data.Dataset.from_tensor_slices((x_test, y_test))
test_db = test_db.batch(batch_size)

train_iter = iter(train_db)
sample = next(train_iter)
print(sample[0].shape, sample[1].shape)  

##  build a standard cnn model
model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))

model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(10))

model.summary()


model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
             metrics=['accuracy'])

train_history = model.fit(train_db, epochs=10,          validation_data=test_db)

## once the model has been trained, convert it to tflite model
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()

with open('qiu_mnist_model.tflite', 'wb') as f:
    f.write(tflite_model)

Implementation in Android app

I refereed to this post for obtaining the original android project. I imported the kotlin version into my Android Studio. However, there were some bugs initially when I loaded my model into it.

My own model is located to asset repository:

The most important thing for this work is the following Gradle setting:

After about 15min of debugging and code modifications, I successfully made my model work.

Check out the video (there is still accuracy issue):

I will upload the android project src code to my github repo once I finish cleaning the code and improve the performance.

reference

My experience of being blackmailed

Z. QIU — Fri, 05 Feb 2021 12:02:11 +0000

Several months ago, one searching functionality of our SaaS under development went down strangely. The server responded error 500 for according web requests. Obviously, the API was still working but there was internal error inside the API function. In the first place, I thought it was the according DB container who went down, however I saw all containers were running normally. After a close investigation, I found that the ES database seemed to be empty. Very weird! So I opened a terminal window, connected to ES and checked all the indices. I saw the information below:

This hacker claimed he had "backed up my database" and demanded 1 bitcoin for retrieving the data. WTF! The price of 1 bitcoin now is about 37K dollars.

This ransom is ridiculously high for me. Fortunately, this db was for test only (that's why it was binded to default port and not secured by password) and the data were backed up beforehand.

I learned some important lessons from this incident though: 1 cypersecurity is very important and one should always pay attention; 2 one should always back up the important data in one or more copies; 3 secure databases and server with strong passwords and use non-default ports

MQTT communication via Cloud broker

Z. QIU — Wed, 03 Feb 2021 10:05:51 +0000

In my startup, we have worked on some AIoT (AI + IoT) projects and we got some MVPs as demo for our client. In some certain project, we have used MQTT protocol for collecting sensors data and transporting them to other smart devices. So in this post, I would like to present a simple demo for building a MQTT communication via a broker on AWS cloud.

MQTT (Message Queuing Telemetry Transport) is an open OASIS and ISO standard (ISO/IEC 20922) lightweight, publish-subscribe network protocol that transports messages between devices. The protocol usually runs over TCP/IP; however, any network protocol that provides ordered, lossless, bi-directional connections can support MQTT. It is designed for connections with remote locations where a "small code footprint" is required or the network bandwidth is limited.

Cloud side

On AWS server, I use docker to run an emqx service which plays a role as "broker".

sudo docker run --rm -ti --name emqx -p 18083:18083 -p 1883:1883 -e EMQX_ADMIN_PASSWORD="myPasswd" emqx/emqx:latest

Output of my putty terminal:

When this service is running, one can access to its administration page by visiting http://3.138.200.179:18083/#/ in web browser:

Subscriber

I have a subcriber.py file which defines a mqtt client who connects to cloud broker with port 1883 and subscribes message named 'qiu_data'.

import paho.mqtt.client as mqtt

## callback upon connection
def on_connect(client, userdata, flags, rc):
    print("Connected, result code: " + str(rc))

## callback upon arrival of msg
def on_message(client, userdata, msg):
    print(msg.topic + " " + str(msg.payload))

client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
# client.on_disconnect = on_disconnect

## I have not yet bindded a domain name for my AWS ECS
client.connect('3.138.200.179', 1883, 600) # 600ms: keepalive interval setting
client.subscribe('qiu_data', qos=0)
client.loop_forever() # keep running

Publisher

I have a publisher.py file which defines another mqtt client who connects to cloud broker with port 1883 and publish some data in 'qiu_data' messages.

import paho.mqtt.client as mqtt
import json
import time

def on_connect(client, userdata, flags, rc):
    print("Connected with result code: " + str(rc))

def on_message(client, userdata, msg):
    print(msg.topic + " " + str(msg.payload))

data = {
    "type": "IoT",
    "timestamp": time.time(),
    "msgId":"8ed7a307-0e82-9738-xxxx",
    "data":{
        "temp":23.5,
        "speed":46.8
    }
}

param = json.dumps(data)

client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.connect('3.138.200.179', 1883, 600) # 600ms for keepalive
client.publish('qiu_data', payload='hello world', qos=0)
time.sleep(1)
client.publish("qiu_data", payload=param, qos=0)

Execution

In a terminal I type python subcriber.py, and in a second terminal I type python publisher.py. See the published topics have been received in the subscriber's terminal:

Breathing Detection Using Computer vision with webcam

Z. QIU — Mon, 18 Jan 2021 13:04:56 +0000

Introduction

When I worked on the development of a healthcare device for detecting baby Apnea (pause of breathing), I have spent some time on studying possible technological solutions. I have tried several different types of sensors including: RGB-camera, 3-axis accelerometer, heat camera, depth camera (kinect), etc. The rgb-camera-based solution is very interesting. Firstly because they are low-cost; moreover, it does not require mounting the sensor on baby's body thus the comfort is guaranteed. That's why I worked for about 3 weeks on this solution.

As experiments, I tried multiple possible algorithms including: feature points tracking, inter-frame difference checking, flux, and color nuance variation tracking.

In this post, I want to present the latter one. This work is inspired by this study. The basic idea is to track the color variation of an ROI on human face. The blood circulation and breathing will cause slight color variation of our skin. This change is so slight that our eyes can not perceive it at all. However, a normal webcam can identify color variation with a resolution as high as 2^16 grades.

ROI on human forehead

I have chosen a small rectangle area as ROI (region of interest) for tracking color variation. Below is the part of code for this purpose.

x1 = 0.4
x2 = 0.6
y1 = 0.1
y2 = 0.25
face_cascade=cv2.CascadeClassifier(cv2.data.haarcascades + "haarcascade_frontalface_default.xml")

def getFaceROI(img):
    gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
    faces = face_cascade.detectMultiScale(gray, 1.2, 5)

    if len(faces)>0:
        img=cv2.rectangle(img, (faces[0][0] + int(x1*faces[0][2]), faces[0][1]+ int(y1*faces[0][3]) ), (faces[0][0]+ int(x2*faces[0][2]), faces[0][1]+ int(y2*faces[0][3]) ), (255, 0, 0), 2)

        return [faces[0][0]+int(x1*faces[0][2]),  faces[0][1]+ int(y1*faces[0][3]),  faces[0][0]+ int(x2*faces[0][2]), faces[0][1]+ int(y2*faces[0][3])]
    else:
        return [0,0,0,0]

Color Calculation

We simply calculate the average value of a given color channel (I used the green channel in my test) of all the pixels in the selected ROI.

def getColorAverage(frame, color_id):
    return frame[:,:,color_id].sum()*1.0 / (frame.shape[0]*frame.shape[1])

Drawbacks

Entire code has been uploaded to github repo below:
https://github.com/QiuZhaopeng/CV2_heartbeat_breathing_detection

This solution suffers from limitations in multiple aspects, e.g. the lighting condition, human should stay still, the camera noisy level, etc. That's why I did not choose this solution for our product (finally we chose the 6-axis sensor solution).

Result

Below is the snapshot of the execution of this program. As one can see, there are low-frequency wave shape mixed with high-frequency sawtooth (small peaks).

I measured my pulses during execution of this program. It can be verified that each small peak corresponds to a pulse beat (heart beat). And the large peak-valley transition corresponds to breathing cycles.

Anyone interested in this small experiment are encouraged to try it and do some further work. Good luck.

MongoDB Compass connection issue

Z. QIU — Thu, 14 Jan 2021 11:26:34 +0000

I am working on migration of our Data Warehouse project from Alibaba Cloud to Huawei Cloud this week. All goes well, except one issue occurs when I tried to connect to MongoDB using MongoDB Compass Community.

As I have presented in my previous post, I have deployed MongoDB service using docker and I have successfully connected to my MongoDB database from IPython console. So I firstly tried again the visit from Ipython today to make sure that the MongoDB service is still running on Huawei ECS. It IS still working well:

Then I open MongoDB Compass Community and fill the login information as below (our domain name has not yet been authorized so I have to use Ip address for hostname):

However, when I click "CONNECT" button, nothing happens.

After a close investigation, I found the reason. After I filled all the fields for connection, Compass yielded a connection URI string as below.

In fact, as explained in official manual of MongoDB, the standard URI syntax for connection is as mongodb://[username:password@]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]. Thus in my case, the Compass software has strangely used ip:port as the Database name to visit. Since there is no database with such name, so it shall do nothing and stays in the connection interface.

Now I correct the URI string manually and fill my database name into this string as below and it works:

Once I click "CONNECT", Compass jumps to display interface of the queried database now:

Native Mobile Application dev using HTML5

Z. QIU — Tue, 12 Jan 2021 17:53:19 +0000

I started learning JavaScript since November 2018. Since then, I fell in love with this powerful language. In this post, I would like to share my experience of app dev using JavaScript and to encourage you guys to develop apps using HTML5 tech.

My first experience with JS started with MUI by using which I would like to develop an mobile application to measure human abdomen movements using smartphone's accelerator. MUI is a H5-based framework for developing "native" applications (can be called Chinese version Reaction Native?). The MUI team has also developed HTML5plus sdk (Chinese-version Cordova) which grants applications the access to host device's capabilities such as sensors, phone status, etc. With these tools, I have easily developed my application within two days. One month later, we have decided to develop an application for communicating with our smart wearable hardware via Bluetooth. Thus along with progress of hardware development, I have developed another application for testing communications with the hardware and for collecting test data. Several months later, this application becomes a very useful tool for our team, which is used to: flash the firmware, testing the stability of BLE connection, adjusting parameters for algorithm and embedded software, collecting debug logs, data collecting during nightly tests, ROB (robustness) issues investigation, etc.

For copyright reason, I cannot share the source code of this application. Below are screenshots of two non-secret parts of my code. The bluetooth dev manual is given here.

This is a video showing how I did the test with my application and the wearable hardware prototype.

Another video of motion data collection on a premature baby using my app:

Putty connection to AWS Linux instance

Z. QIU — Fri, 08 Jan 2021 17:00:34 +0000

I have registered my personal AWS account today (FINALLY). The entire process was easy for me, however, I encountered an issue when I try to connect to AWS Linux instance from ssh within Windows.

I am always using Putty for connection to my Cloud servers. Thus I firstly tried Putty in same way I used to connect to AliCloud. Unfortunately I encountered the following error message.

Then I tried to use ssh -i xxx.pem ... command in CMD console. However, AWS sent back the following complaints:

It seems a permission issue which can easily been solved using chmod in Linux. However, I am using Windows and I'm really not familiar with the complicated "permission" rules in Windows. Some online articles presents solution of this issue using icacls. However, it did not work in my case.

Official doc of AWS has presented a really useful solution. It's quite simple and efficient. Following this tutorial, I executed Putty Key Generator, load my "xxx.pem" file and export an .ppk key file.

Then in Putty interface, I load my .ppk key file (as shown below) and try the ssh connection as usual. And it works.

Reference:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/putty.html