DEV Community: Adrian Benavides

GPS Live Tracking With Balena, Neo-6M, And LoRa!

Adrian Benavides — Thu, 10 Oct 2019 17:33:11 +0000

Originally published on my blog here and full code on github.

Project overview
Using the Neo-6M
Using the LoRa-02
Visualization with Balena and Grafana

What's this about? A quick project overview

Whenever I need inspiration on new IoT projects ideas to work on I phone my colleague Ramiro and in less than 10 minutes we have 4 or 5 ideas on the table.

During our last conversation, he told me about an idea he had that caught my attention. "If your bike or motorbike were stolen, wouldn't it be nice to be able to know in that precise instant where it is, or what was its last known position?".

Imagine having a small device that you could use to track any of your most valuable things: your bike, your laptop, your backpack, even your wallet. Or at least, small enough to carry it yourself in your pocket or your jacket.

I spent a couple of days thinking about this concept, about how I could develop it, what technologies to use, what devices, etc. At that moment, the only thing I had clear was that I wanted this device to be autonomous and therefore not dependent on being able to connect to a WiFi network or use a SIM card for data communication.

After looking at several options I ended up deciding that I was going to use a radiofrequency module to transmit my GPS position to a receiver that I would have connected to a Raspberry Pi in my house, where I could process and visualize the data.

And now, let's build it!

Reading GPS data with an ESP8266 and the Neo-6M

The Neo-8M odyssey

This whole project revolves around monitoring my GPS position remotely. It is clear then that the main piece, the only sensor, will be the GPS module.

Well, looking for examples of how to read GPS data using an Arduino board, I saw that almost everyone used the same module: the Ublox Neo-6M. So when I went to Aliexpress to buy a unit I saw that there were two more models available, the Neo-7M and the Neo-8M. My first and only thought was: "The newer the better, I don't care if I haven't seen any examples using the Neo-8M, that's the one I'm going to buy. I'll figure out how to work with it later". WRONG!

After two weeks of waiting, the package finally arrives, I hook it up to my ESP8266... it doesn't work, there is no way to receive data no matter what I do. I change to Windows 10, I connect the Neo-8M to try debugging it with the u-center software and nothing, for some reason it didn't work.

Just considering the idea that I would have to wait another two weeks to receive it, made me rethink whether to discard the project and get on with something else. So I went out for a walk and on my way home I convinced myself to buy the Neo-6M and that, in the meantime, it would be a good idea to work on the visualization. And so I did.

But first, let's check how you can use a Neo-6M with an ESP8266.

Getting GPS data out of the Neo-6M

This module is quite simple to use. Apart from the power pins, it has two more pins to transmit and receive data. You have to be careful with this though, because you have to connect the RX pin of the Neo-6M to the pin of your board that you have configured as TX, and the TX of the Neo-6M to the pin configured as RX of the board.

What I mean is that we are not using the built-in RX/TX pins of the board, but rather we will configure a couple of digital pins (D2 and D3) to interface with the Neo-6M.

It is much easier to express this with an image:

As you can see below, we will use the SoftwareSerial library to read the Neo-6M data and the TinyGPS++ library to parse the readings. You can see a full lastminuteengineers.com tutorial on how Neo-6M works here.

# check out the full code at: https://github.com/adrianbenavides/arduino-helpers
static TinyGPSPlus gpsParser;
static TinyGPSPlus emptyGpsParser;
static SoftwareSerial gpsSerial(NEO6M_RX_PIN, NEO6M_TX_PIN);

void Neo6m::setup()
{
    Serial.print("Setting up Neo-6M GPS module... ");
    gpsSerial.begin(NEO6M_BAUD);
    Serial.println("done");
}

TinyGPSPlus Neo6m::read()
{
    while (gpsSerial.available() > 0)
        if (gpsParser.encode(gpsSerial.read()))
            return gpsParser;

    return emptyGpsParser;
}

void Neo6m::print(TinyGPSPlus gpsParser)
{
    if (gpsParser.location.isValid())
    {
        Serial.print("Latitude: ");
        Serial.println(gpsParser.location.lat(), 6);
        Serial.print("Longitude: ");
        Serial.println(gpsParser.location.lng(), 6);
        Serial.print("Altitude: ");
        Serial.println(gpsParser.altitude.meters());
    }
    else
    {
        Serial.println("Location: Not Available");
    }

    ...
}

Communicating two nodes with radio frequency using the LoRa-02

LoRa-02 wiring with an ESP8266 and the MISO/MOSI nightmare

Now that we have our GPS module working, it is time to work on the transmission of this data for later analysis and visualization.

Remember that we will carry the GPS sensor connected to a battery and that we are working under the limitation of not being able to connect to the internet or use a SIM card to transmit data.

This is where the Ra-02 LoRa (Long-range Radio) module comes into play. Developed by Ai-Thinker, the manufacturer behind the ESP modules, this radio frequency module improves the previous version providing a larger bandwidth, increasing interference resistance, minimizing current consumption, and increasing security. The LoRa-02 mounts an SX1278 IC and works on a 433MHz frequency.

Before jumping into code, I would like to tell you a couple of details about the wiring of this module, something that took me several hours of desperation to resolve... largely because of my ignorance and lack of experience, as you will soon see.

One example after another, I couldn't find a single well-documented one that used an ESP8266 in any of its variants. So I tried to make the simplest example work: setting up the module and send a message... but I just couldn't, something was wrong and I didn't know what it was.

After a considerable period of monkey testing, I had the first good idea of the day: "man, let's take a look at the pinouts of the LoRa and the ESP8266 modules to see if we are missing something". It didn't take me too long to realize the problem: the MISO and MOSI pins of the LoRa module must be connected to the MISO and MOSI pins of the microcontroller. Makes sense right? Believe it or not, I haven't seen ANYONE specifying this "little" detail in any of the examples, implying that the pins used are arbitrary when in fact they are not.

In case you need it, here are the pins that you want to use:

	NodeMcu	UNO, Nano
PIN_SPI_SCK	(14) -> D5	D13
PIN_SPI_MISO	(12) -> D6	D12
PIN_SPI_MOSI	(13) -> D7	D11
PIN_SPI_SS	(15) -> D8	D10

And here is the complete wiring:

Sending and receiving data with the LoRa-02

To work with the SX1278, we have a couple of libraries.

The first one I found was Arduino LoRa, which I didn't get to test, as it doesn't specify what arduino-like boards it supports and I didn't feel like I would be in the mood to find it out by myself.

After searching a little further I found what it seems to be the standard library to work with radio frequency modules: the Radio Head library. It offers a wide set of interfaces for different radio modules, including support for the SX1278 with the class RH_RG95. The only problem is that it's not distributed through GitHub, but don't worry, the guys at Adafruit have us covered with a github'ified version.

Working with the Radio Head library is pretty straightforward: a simple method to set up the module with some configuration options and a couple of methods to receive and send messages.

Again, here are a couple of tricky details that I found interesting to highlight:
1. The maximum message length is 255 characters
2. The message passed to the send() method must be of type uint8_t

As I did in the previous project where I experimented with the MQTT protocol, I've created a wrapper to avoid having the GPS code and LoRa cluttering the main.cpp file, making it easier to read. You can check the complete code here.

# check out the full code at: https://github.com/adrianbenavides/arduino-helpers
RH_RF95 rf95(PIN_SPI_SS, RFM95_INT);
uint8_t rf95_buf[RH_RF95_MAX_MESSAGE_LEN];
uint8_t rf95_buf_len = sizeof(rf95_buf);

void LoRa::setup()
{
    ...

    while (!rf95.init())
    {
        Serial.println(" failed");
        while (1)
            ;
    }

    // Defaults after init are 434.0MHz, modulation GFSK_Rb250Fd250, +13dbM
    if (!rf95.setFrequency(RF95_FREQ))
    {
        Serial.println("setFrequency failed");
        while (1)
            ;
    }

    ...
}

std::string LoRa::receive()
{
    if (!rf95.available())
    {
        delay(100);
        return "";
    }

    if (!rf95.recv(rf95_buf, &rf95_buf_len))
    {
        Serial.println("LoRa - No message received");
        return "";
    }

    ...

    return std::string((char *)rf95_buf);
}

void LoRa::send(std::string &message)
{
    const uint8_t messageLength = message.length() + 1;
    assert(messageLength <= 256);

    rf95.send((uint8_t *)message.c_str(), messageLength);
    rf95.waitPacketSent();

    ...
}

Mounting the GPS receiver!

It turned out to be a bit bigger than I expected but, you know, it's a prototype...

Data aggregation and visualization with Balena and MQTT

Customizing the Balena Sense project to handle any sensor data

In this section, I'm going to detail how to leverage the power of Balena to create an MQTT gateway with a customized data visualization using grafana. The power this ecosystem gives to IoT developers is awesome, especially because it saves you from creating the frontend, database, and visualization engine. And that's a LOT of boilerplate and saved time.

To tell telegraf that it must listen to MQTT messages we need to add the following code to the telegraf.conf file.

[[inputs.mqtt_consumer]]
  ## MQTT broker URLs to be used.
  ## The format should be scheme://host:port, schema can be tcp, ssl, or ws.
  servers = ["tcp://mqtt:1883"]

  ## Topics that will be subscribed to.
  topics = [
    "sensor/#",
  ]

  ## Data format to consume.
  ## Each data format has its own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
  data_format = "influx"

Note that the data_format key is set to "influx", which is the default InfluxDB format, also known as line protocol. An example of a valid message would be: gps lat=41.1234,lng=2.4321, where gps defines the influxdb table and the rest of the message the columns and values to be assigned. Here is the official documentation of this data format.

As for the provisioned dashboards, you can remove the default Balena Sense dashboard, which is defined in the grafana/provisioning/dashboards directory.

Finally, if you need to install any additional grafana plugins, like the trackmap-panel I'm using in this project, you can add them in the balena-mqtt-gateway/grafana/entry.sh file, right before initializing the grafana server.

For further information on how to set up a balena project using a raspberry pi, check out their official documentation here.

GPS data visualization with grafana

At this point, we have every physical piece of the system doing its job, sending, receiving and processing GPS data.

The only thing that is left to do is set up the visualization. To do this we will be using the map plugin I mentioned earlier, and luckily for us, it works out of the box with the provided query:

SELECT median("latitude"), median("longitude")
FROM "location"
WHERE $timeFilter
GROUP BY time($interval)

This is what we'll see in the grafana dashboard if the whole system is working properly:

Summary

It took me a while, but it's working just as I expected! Frankly, I have enjoyed a lot and I have learned a lot of new things working on this project. The satisfaction of seeing a remote device interacting with an RPi that I have in my room is indescribable.

Sometimes, I can't help myself but think about electronics and communication technologies as if it were black magic. Actually, I think is exactly this "childish" feeling that pushes me to keep experimenting and keep looking for new ideas to implement.

I hope you enjoyed this guide, see you next time!

Exploring MQTT With The ESP8266

Adrian Benavides — Thu, 19 Sep 2019 21:40:12 +0000

In any IoT project, we have a number of unique requirements and restrictions. The communication strategy to connect devices and send data between them is critically important.

In this guide, we will explore a powerful, simple and lightweight messaging protocol over the wire: MQTT.

Originally published on my blog here and full code on github.

MQTT Basics
Dev Environment
ESP8266 Filesystem
Communicating Two Devices
Tracking Devices Status

What's The MQTT Protocol?

In the embedded world, we assume that the end-devices are going to work under a variety of processing capacity, energy or communication constraints. And specifically on the subject that concerns us in this post, how we are going to implement the communications of our system is going to have a direct implication on the consumption of resources and energy of the devices.

In this guide we are going to explore MQTT, a lightweight communication protocol over TCP invented in 1999 by Andy Stanford-Clark from IBM and Arlen Nipper from Arcom, initially designed for battery-operated devices to supervise oil pipelines. Later in 2010, IBM released it as a royalty-free protocol. In 2014, OASIS announced that the MQTT v.3.1.1 had become an OASIS standard and a lot of MQTT clients were developed for all programming languages.

The most commonly used broker is the Eclipse's Mosquitto library: an open-source implementation of the MQTT v3.1 and v.3.1.1 standards providing a lightweight method to transport messages, enabling the pub/sub pattern for low power sensors, mobile devices, embedded computers, and microcontrollers. You can install the Mosquitto broker on your PC as well as on any embedded system with Linux support, such as the Raspberry Pi.

Main Features

Data agnostic: you can use it to send sensor data, images or even OTA (over the air) updates
Lightweight and bandwidth efficient: the smallest frame is only 2 bytes long
Per-queue QoS levels
Runs on top of the TCP/IP stack, so you can use it to encrypt the data with TLS/SSL and to have a secure connection between clients
Simple to develop: clients exist for all operating systems and programming languages
Central broker: different devices can communicate with each other without having to worry about compatibility
Session awareness: provides an identity-based approach for subscriptions
Flexible subscription topics using single-level and multi-level wildcards

Basic Terminology

Broker: the software application that receives messages from publishing clients and routes the messages to the appropriate subscribed clients
Client: a device that can publish messages to a topic or subscribe to messages from a topic
Topic: an identifier (a string) used by the broker to filter messages for each connected clients. It is sent by clients to the broker in a subscribe request to express the desire in receiving messages published by other clients. It is sent by the clients when publishing messages to any other client that subscribed on the same topic
Publish: the action of sending a message from a client to the broker
Subscribe: the action of informing the broker about an interest in receiving future messages published by other clients on that topic. A client can subscribe to multiple topics
Unsubscribe: the action of informing the broker that a client doesn't want to receive messages of the specified topic

And now, before jumping into the meat of this guide, let's make sure first that we are on the same page with our development tools.

Set Up For The Experiments

MQTT Broker With Raspberry Pi And BalenaOS

As I said before, Mosquitto can be installed on any PC (Windows, MacOS and Linux), so if you don't have any RPi at hand you can install Mosquitto locally and use the static IP of your computer.

In my case, I am working on a mini home-automation project in which I need to integrate several devices, show data in a webapp and analyze that data to execute certain actions. As you can see, a full development stack is involved here (database, a backend, a frontend) and for this kind of setup there is nothing better than working with containers. Did anyone say Docker?

How can we use Docker on a Raspberry Pi with automated deployments, without having to waste hours configuring Raspbian? This is one of the things you can achieve with Balena. And before you ask: yes, it's free, but "only" for your first 10 devices...

Here's a short video of the process to create a new Balena app (it took me around 15 minutes from start to finish):

Finally, in the upper right corner, you can check out your newly created balena remote git repository. Click on the "?" button to see how to configure your project or check the docs here.

Now, time to set up the MQTT broker:

Create a new github project and add this docker-compose.yml file
Add your balena remote endpoint we saw earlier in the dashboard
Push your changes first to github and then to balena

Done! Within a couple of minutes, you will see in the balena dashboard how the mosquitto container is created and started

ESP8266 Development Environment

I'd recommend you to use Linux because it offers a number of advantages over Mac or Windows that can help you avoid certain frustrations. For example, in a standard Ubuntu installation, I have been able to work with Arduino boards, ESPs and even STMs without having to worry about installing a single driver. And believe me: that it's pure pleasure.

In any case, keep in mind that sometimes it is inevitable to go through Windows since there is some specific vendor development software that is only available for this operating system. So, always check your requirements and make sure you know beforehand what software do you need to work with your devices.

For the experiments conducted in this guide we don't need anything fancy so, luckily, we can stay on Linux.

PlatformIO + VS Code (or Atom)

Forget about using the Arduino IDE. It's anything but functional and practical. It lacks every functionality that makes up a good IDE.

Luckily, we have PlatformIO, an extension for VS Code and Atom that turns these fantastic editors into authentic IDEs for our embed experiments.

From the docs: "It takes care of toolchains, debuggers, frameworks that work on most popular platforms like Windows, Mac and Linux. It supports more than 500 development boards along with more than 30 development platforms and 15 frameworks."

It also allows you to manage your libraries and keep them updated as well as debugging your code easily.

What are you waiting for? Go ahead and install it right now.

Next thing you want to do is download a couple of libraries: PubSubClient and ArduinoJson. Or you could simply clone the repository of this guide.

And now, let's begin!

The ESP8266 Filesystem: Storing WiFi Credentials Into Flash Memory

Since I will be working with several ESP8266 for a while on different projects, I thought it would be a good idea to take some time to preconfigure the modules and forget about hard-coding my wifi network credentials in the scripts and then put dummy text before uploading the project to github. I'm sure I would end up pushing the password and never notice.

Getting this with an ESP8266 is quite simple with the help of the SPIFFS and ArduinoJson libraries. To encapsulate the basic functionalities related to our device (network connection and flash memory management) I have created a small class called NodeMcu.

As you can see, one of the constructors accepts a single argument with which we will pass the stringified Json that we want to persist in memory. In the setup() method it will try to recover the data from the memory or it will fail if it doesn't find the right data to connect to the local network.

Apart from this NodeMcu wrapper, I have also created another one to work with MQTT in a more direct and clean way, with all the external libraries included. Make sure to check out the full code here.

Now that we have our boards properly configured, we can move on to the first experiment.

Connecting Two Devices With MQTT

In this first experiment we are going to focus on the core functionality of the MQTT protocol: publish/listen to messages to/from a topic, and use this to toggle an LED. As a trigger to send the messages I have used 2-way switches, not very suitable for prototypes as you can see in the GIF below, but much cooler the typical flat buttons of four pins that are usually used in most of the tutorials.

The workflow is simple: when we press the button of the device on the left, it will toggle the LED from the right; and vice versa. The only trap is that each device listens to all the messages that are published, so we need to filter out the self-published messages to avoid acting on itself.

Detecting Disconnected Devices With Last Will Messages

One of the most interesting features of the MQTT protocol is that the broker keeps track of the connection between itself and all the connected clients. The broker knows at all times when one of its clients has disconnected. How? Very simple: it continuously pings its clients and when one of them stops responding for a certain period of time (timeout), it assumes that it has been disconnected.

This timeout is defined by the client when it sends the initial connection request to the broker, so it is totally configurable by the client. In this connection request is where we can also include the parameters for the Last Will and Testament (LWT) message which includes: the payload, which topic to publish it, the QoS, and whether the message should be retained or not.

And here appears a new feature of MQTT: retaining a message. This allows us to tell the broker that, whenever we send a "retain" flagged message to a topic, it should store it and keep it safe. Now, when a new client subscribes to that topic, it will receive that retained message. Think it as a welcome note for newcomers.

We can combine these two MQTT functionalities to track the status of our devices, and this is what we are going to do in this experiment. Of course, the PubSubClient library offers us the API to easily work with LWT messages on our Arduino-like boards.

This is the standard algorithm used to manage LWT messages (credits to knolleary from stackoverflow):

When a client connects, it publishes a RETAINED message to a topic unique to it ("clients/clientId/status") with a payload of 1
It also, on connect, sets an LWT message to be published to the same topic, but with a payload of 0. This should also be a RETAINED message
When a client disconnects cleanly, it publishes a RETAINED message to the same topic with a payload of 0
When a client disconnects abruptly (ping fails for the defined timeout), the broker will publish the LWT message as the new RETAINED message

Here you have the code used and here more details about LWT from HiveMQ (parts 8 and 9).

The wiring for this experiment is just the same as before but without the buttons.

Wrapping Up!

In this guide, we have worked with one of the most widely used variants of ESP8266 (NodeMcu) to implement some of the basic concepts of the MQTT communication protocol.

We have also seen how to use the internal storage of our module to store configuration files and speed up the development of our prototypes.

Now you could apply the techniques we have learned here in some real project you may have in mind, implementing an efficient and straightforward communication system to send, for example, data from our sensors to a local server for later analysis.

MQTT is much more than what we have seen here today, but I hope that at the very least these experiments have helped you understand the core concepts of this protocol and how to use them. Keep experimenting and learning!