Twilio Hackathon - Owlish

Marcos Jay T. Concon — Fri, 01 May 2020 06:50:34 +0000

THE WHY

I live in a student populated area notoriously known for home burglaries. I've been fortunate enough to have not been targeted, a couple of my friends living close by have lost items worth hundreds of dollars even with guards patrolling the area at certain intervals. Realistically speaking, a lot of these burglaries have happened even with the guards around. I'd like to get out of my room knowing that my belongings are safe and secure. This is why I built...

THE WHAT

Owlish - a camera based application that has its eyes set on efficient monitoring, detection and alerting of anyone that's not supposed to be in your dorm while you're away.

Hardware components

Raspberry Pi 3B+ - where the application is run on
A Pi Camera Module V2 - the eyes of the application
RSL10-SENSE-GEVK dev kit - to activate the application if you're heading out.

Software components

Twilio WhatsApp - to alert via text & media message
TensorFlow lite - to detect intruders using deep learning for object detection
Bluepy - to interface with the RSL10-SENSE
Cloudinary - to host captured image on the cloud

DEMO & CODE

Github Repo
Click the image below to view the demo on Youtube

THE HOW

Hardware setup

The Raspberry Pi is setup with Raspbian OS along with the configured camera module. If properly configured, typing the following command below in the terminal should yield an output as shown.

pi@raspberrypi:~ $ vcgencmd get_camera
supported=1 detected=1

Software setup

i. Configuring TensorFlow lite Object Detection

With Python 3.7 and Git setup, head over to the TensorFlow lite object detection repository to install it by following the README.md documentation step by step. Once everything is set up nicely, change directory to wherever the detect_picamera.py in the cloned repository is and type the command below to start the inference.

pi@raspberrypi:~ $ cd /home/pi/Desktop/twilio_hack/examples/lite/examples/object_detection/raspberry_pi

pi@raspberrypi:~/Desktop/twilio_hack/examples/lite/examples/object_detection/raspberry_pi $ python3 detect_picamera.py   --model /home/pi/Desktop/twilio_hack/detect.tflite   --labels /home/pi/Desktop/twilio_hack/coco_labels.txt

Note: Only Python versions 3.5, 3.6 & 3.7 are the only versions currently supported for the example so make sure you get one of them installed and set as default! Also, make sure to replace the path to the model and labels with your own when typing the command above.

The model is able to detect 90 classes including apples, people, and phones. A tad disappointed it couldn’t pick up the apple logo on my phone though :(

The code snippet below (from detect_picamera.py) is the entry point where we'll capture and locally save the images.

def detect_objects(interpreter, image, threshold):
  """Returns a list of detection results, each a dictionary of object info."""
  set_input_tensor(interpreter, image)
  interpreter.invoke()

  # Get all output details
  boxes = get_output_tensor(interpreter, 0)
  classes = get_output_tensor(interpreter, 1)
  scores = get_output_tensor(interpreter, 2)
  count = int(get_output_tensor(interpreter, 3))

  results = []
  for i in range(count):
    if scores[i] >= threshold:
      result = {
          'bounding_box': boxes[i],
          'class_id': classes[i],
          'score': scores[i]
      }
      image.save('intruder', "JPEG")
      results.append(result)
  return results

ii. Start & stop the application with RSL10-SENSE Bluetooth Low Energy kit

The original mechanism I had in mind to switch the intruder detection application ON when I had left my dorm or switched OFF when I arrived was by checking whether my phone was connected to my dorm’s WiFi network. This is possible but comes at a cost of an additional USB WiFi dongle which I didn’t have at the time of prototyping. If anyone has a workaround without using an additional WiFI adapter dongle, do share!

So instead I used an unused RSL10-SENSE Bluetooth Low Energy device (BLE) I had won from a hardware giveaway a year ago. I figured I could do something useful with it in this hackathon. By default, the firmware in the device is configured to be in Deep sleep mode which starts the BLE advertising for 60 seconds after pressing the RESET button. It then goes back to sleep until the button is pressed again.

According to their specification, in this mode, the replaceable lithium ion cell battery can last over a year without replacement! If you’d like to further customize the firmware, do visit this link for detailed instructions.

To link the BLE device and the Raspberry Pi, we’d need to check the MAC address of the BLE device. Typing the command in the terminal below, shows all the scanned BLE devices in the area. Make sure the ‘RESET’ button on the BLE device is pressed beforehand for it to start advertising!

pi@raspberrypi:~ $ sudo hcitool lescan
LE Scan ...
4A:A1:30:61:7E:47 (unknown)
4A:24:DF:D4:2D:45 (unknown)
18:54:EA:40:A9:70 (unknown)
6C:EA:92:04:AE:49 (unknown)
60:C0:BF:28:65:C0 HB_BLE_Terminal

'HB_BLE_Terminal' is the device name and contains its corresponding mac address. On the PI, I used the Bluepy python module which allows communication with Bluetooth Low Energy devices. Here's the code snippet which scans and checks BLE devices around

def rsl10_activate():
    while True:
        scanner = Scanner().withDelegate(ScanDelegate())
        devices = scanner.scan(3.0) #scan for 3 seconds and repeat
        for dev in devices:
            print ("Device %s (%s), RSSI=%d dB" % (dev.addr, dev.addrType, dev.rssi))
            if dev.addr == "60:c0:bf:28:65:c0":
               print("RSL-10 device found")
               global activate_status
               activate_status = not activate_status
               time.sleep(60)

iii. Send and Receive Alert Message with Twilio WhatsApp in Python

The Twilio documentation is self explanatory and amazingly simple to follow. You’ll need your account_sid and auth_token which can be obtained from the Twilio console.

The only issue here is that sending media with the Twilio WhatsApp API isn't possible with local images, rather an image url with HTTP or HTTPS is expected. That means the captured image of the intruder has to be hosted somewhere to access it via URL.

I used Cloudinary, a cloud platform to easily manage media such as images and videos. Once registered to their generous free plan, their Python SDK for uploading images and retrieving their respective url's was used. Here's a small code snippet.

cloudinary.config(
  cloud_name = "YOUR_CLOUDINARY_NAME",  
  api_key = "YOUR_CLOUDINARY_KEY",  
  api_secret = "YOUR_CLOUDINARY_SECRET"  
)

def upload_cloudinary(image_name):
    #We'll need an identifier to upload different intruders
    image_id = uuid.uuid1().int
    image_id = str(image_id)
    cloudinary.uploader.upload(image_name, public_id = image_id) #upload the captured image "detected"
    return (cloudinary.utils.cloudinary_url(image_id)) #access the uploaded image url via its public_id

Fixing it all up

So once we've got all our individual software components ready, its time to connect them all together. I had to multi-thread my code with the threading and the queue modules as saving, uploading and sending the message over with Cloudinary & Twilio would block & lag the execution of code on camera frame inference.

scan_bluetooth = Thread(target=rsl10_activate)
inference = Thread(target=start_picamera)
send_message = Thread(target=cloudinary_twilio_send)

if __name__ == '__main__':
    scan_bluetooth.start() #thread that scans for bluetooth low energy devices
    inference.start() # thread to start the inference once the RSL-10 BLE dev kit is connected
    send_message.start() #thread to check if an intruder is detected and then send the message over

and finally here's a wild intruder captured by our application!

Future Improvements & Todo's

Using a personal smartphone to activate the system when you're away instead of using the RSL10-SENSE-GEVK dev.
Using Twilio Autopilot for handoff to relevant authorities (guardhouse in my case).
Boost inference performance with Coral USB accelerator

Conclusion & Acknowledgement

Had a ton being able to connect different technologies together to create this application. Special thanks to Twilio for organizing this hackathon enabling us to share our ideas and projects whilst promoting innovation and collaboration. This submission falls under the Exciting X-Factors Category.

If you've got any thoughts about my project, please comment below, I'd like to hear them. Stay safe and thanks for reading!

DEV Community: Marcos Jay T. Concon