DEV Community: Jared Wolff

Create an Open Source Powered Air Quality Dashboard

Jared Wolff — Mon, 11 Oct 2021 14:05:54 +0000

This post was originally from jaredwolff.com

Depending on where you live, your air quality can be a big deal. Whether it's pollution from factories or particulates from burning forest fires, they're not good for your lungs or overall health.

While there are plenty of air quality meters out there, it's doubtful they're open source never-mind reasonably priced. In this post, i'll be showing you how to connect an Air Quality Wing, publish data to the web via Golioth and view your data in real time using Grafana.

So first, let's get started by assembling hardware.

Side note: this post is fairly lengthy. You can download a PDF version of it here.

Assembling hardware

In this project, you'll need a few things to get started.

An Air Quality Wing with connected PM2.5 sensor
A Feather compatible processor board. We'll be using the nRF9160 Feather which is NB IoT and LTE Cat-M enabled.
A Feather doubler to connect the two together.
A SIM card with your favorite provider. Some of the options out there include Hologram, Soracom and Twilio.

Assembly is straight forward and only requires you to solder headers if you haven't already. Here's an example of using female headers on the Air Quality Wing.

Every Air Quality Wing comes with a set of male headers. One is 12 pins and the other is 16.

The best way to solder them is size them correctly, place them in a breadboard or already assembled doubler and place the board on top for soldering.

Note: Don't spend too much time holding your iron to the pins as the breadboard is not meant for high heat usage!

You will also have to do a similar solder job if you use a Feather doubler. More on the assembly for that board can be found here.

In the end you should have a setup that looks something like this:

Setting up a device

Currently the Air Quality Wing works with the following boards:

nRF9160 Feather
Particle Xenon
nRF52840 DK

Adding support for new boards is simple thanks to Zephyrs ability to target many hardware targets for the same core source code. In this sample we'll be using the nRF9160 Feather since it can publish directly to Grafana. So here's the instructions to getting the firmware up and running:

Set Up Python Virtual Environment

One of the main dependencies for Zephyr is Python. Instead of installing the required packages directly to your system, I highly recommend use a virtual environment instead.

First lets make a project folder. We can call it aqw:

mkdir -p aqw

If you don't have virtualenv installed already, make sure you install it:

pip3 install virtualenv

Set up a python virtual environment:

virtualenv -p python3 env

Then enable it by running:

source env/bin/activate

Install `west`

west is the command line utility for Zephyr. You'll need it to build the firmware.

pip3 install west

Initialize and update project

Next we'll download the source code and install the dependencies.

west init -m https://github.com/circuitdojo/air-quality-wing-zephyr-demo.git  --manifest-rev main

Then run

west update

This will download all the dependencies. If there are ever fixes to this sample you'll need to pull the changes and then re-run west update:

git pull && west update

Install remaining python dependencies

Finally, there are some important dependencies needed to build your freshly cloned Zephyr repository. Let's install those here.

pip3 install -r zephyr/scripts/requirements.txt

Install toolchain

For Mac run the following: (it does require you install wget. brew is an easy way to do so: brew install wget)
```
 cd ~
 wget "https://developer.arm.com/-/media/Files/downloads/gnu-rm/9-2019q4/gcc-arm-none-eabi-9-2019-q4-major-mac.tar.bz2"
 tar xvfj gcc-arm-none-eabi-9-2019-q4-major-mac.tar.bz2
 rm gcc-arm-none-eabi-9-2019-q4-major-mac.tar.bz2
```
Note for Catalina (and newer) users you will get an error when running these utilities for the first time. You must allow them to be executed in your Security preferences.
For Windows you can download and install the toolchain with this direct link.

That's it for your SDK install! Next, we'll want to generate credentials for your device and add them to the sample code. We'll do that next.

Get credentials

First jump over to Golioth and create an account if you haven't already. Golioth is finally in open beta which means anyone can join!

Once instead you'll be prompted to create a project. So let's do that now. I've named my project Air Quality Wing but you can name it whatever you want. Only unique project names can be used at the moment so you'll need to name yours differently.

Make sure you click Save to continue to the next step.

Once you have a project created, it will open up other options within Golioth including the Devices tab. Click on the Devices tab.

Then click the Create drop down and then Provision with Credentials. You you should get a screen like the one below.

Change the device name to whatever you'd like. Change the identity to match if you'd like. (or leave it be) Finally, add your super secret pre-shared key. You'll need the Identity and PSK for the next step.

Add credentials to sample code

If you've gotten this far, you have your device provisioned! Next is to add the Identity and PSK to the demo code.

You will need to edit demo/golioth/config/golioth.conf with your credentials in order to connect to Golioth's backend. Update CONFIG_GOLIOTH_SYSTEM_CLIENT_PSK_ID and CONFIG_GOLIOTH_SYSTEM_CLIENT_PSK with the credentials you used in the previous step.

# Golioth credentials
CONFIG_GOLIOTH_SYSTEM_CLIENT_PSK_ID="<INSERT CLIENT ID HERE>"
CONFIG_GOLIOTH_SYSTEM_CLIENT_PSK="<INSERT YOUR SUPER SECRET HERE>"

Also, if you're not using the PM2.5 sensor you may need to comment out &hpma_sensor, in static struct aqw_sensor *sensors[] = {.

Build and Deploy

Finally build the application by running the following commands.

cd demo
west build -b circuitdojo_feather_nrf9160_ns -s golioth

Once built you'll get to the end of a bunch of output. The end should look something like this:

[5/16] Linking C executable zephyr/zephyr_prebuilt.elf

[11/16] Linking C executable zephyr/zephyr.elf
Memory region         Used Size  Region Size  %age Used
           FLASH:      189380 B     425472 B     44.51%
            SRAM:       44376 B     178968 B     24.80%
        IDT_LIST:          0 GB         2 KB      0.00%

Then plug in your nRF9160 Feather and either program using newtmgr or nrfjprog. I'll include both here.

For programming directly over USB use newtmgr:

newtmgr -c serial image upload build/zephyr/app_update.bin

If you have a programmer setup, use nrfjprog

nrfjprog --program build/zephyr/merged.hex --chiperase --reset

Then open up a serial console and make sure the board has booted properly:

SPM: NS image at 0x20200
SPM: NS MSP at 0x2001d2e8
SPM: NS reset vector at 0x264b5
SPM: prepare to jump to Non-Secure image.
*** Booting Zephyr OS build v2.6.99-ncs1  ***
[00:00:00.204,681] <err> i2c_nrfx_twim: Error 0x0BAE0002 occurred for message 0
[00:00:00.218,902] <inf> golioth_system: Initializing
[00:00:00.219,268] <inf> aqw_golioth_demo: Connecting..
[00:00:01.979,187] <inf> aqw_golioth_demo: LTE cell changed: Cell ID: 13368336, Tracking area: 1547
[00:00:02.044,860] <inf> aqw_golioth_demo: RRC mode: Connected
[00:00:03.416,809] <inf> aqw_golioth_demo: RRC mode: Idle
[00:00:13.547,454] <inf> aqw_golioth_demo: RRC mode: Connected
[00:00:16.709,899] <inf> aqw_golioth_demo: Network registration status: Connected - roaming
[00:00:16.709,930] <inf> aqw_golioth_demo: Air Quality Wing Golioth Demo
[00:00:16.710,235] <inf> aqw_golioth_demo: PSM parameter update: TAU: -1, Active time: -1
[00:00:18.313,415] <inf> golioth_system: Starting connect
[00:00:18.682,647] <inf> golioth_system: Client connected!
[00:00:20.848,358] <inf> aqw_golioth_demo: Temperature: 26.32333°C
[00:00:20.848,388] <inf> aqw_golioth_demo: Humidity: 60.771179%
[00:00:20.848,419] <inf> aqw_golioth_demo: PM2.5: 4.0 ppm
[00:00:20.848,541] <inf> aqw_golioth_demo: Data size: 42
[00:00:27.561,889] <inf> aqw_golioth_demo: RRC mode: Idle

If you've gotten some sensor measurements within the first 30 seconds, you're golden!

Setting up Grafana

You're almost there! Let's get a Grafana interface set up. If you already have Grafana this should be a very straight forward integration. If you don't I recommend you check out Grafana's hosted service. I've also written about setting up Grafana in a past post.

API keys for Golioth

In order to get the streamed data from the nRF9160 Feather and Air Quality Wing, you'll need to set up API credentials for your Golioth Account.

Navigate to the API Keys section and click Create a Key!

Make sure you have API Key selected and click Save

Finally you'll get a window like the one below. Take note of your X-API-Key value and the project name. We'll need both in a second!

Setting up your data source

Now let's get over to Grafana.

First we'll want to create a new Data Source by going to the gear icon on the left navigation bar and clicking Data Sources

On the next screen click Add data source and use the JSON API option. If you do not see the JSON API option you may need to install it first. Instructions on how are here.

Next you'll want to change the Name to Golioth and also your URL to https://api.golioth.io/v1/projects/{your-project-id} where your {your-project-id} is the same one indicated earlier in the API key generation step.

Note do not include the /devices suffix. Insert the URL above as shown with your project identifier.

Finally go to the bottom and setup a Header under the Custom HTTP Headers section. Set the name to X-API-Key and the value to the API key generated earlier.

Then click Save & test. You should get a nice big Success check mark if things check out!

Creating your dashboard

In this step, we'll import a dashboard that i've already created. You can download it here:

Dashboard JSON Export

Click the + button on the left side menu and click Import

Then click the Upload JSON file button

Then select the Golioth data source that we set up earlier and click Import

The graphs should automatically pull data for your device and it should show some pretty graphs!

You've done it!

Now you can start playing around with the panels and tweak as you need to. More on creating the panel from scratch you can look at Golioth's setup documentation here.

Conclusion

And that's a wrap!

I hope this has been a useful introduction if this is your first time playing with Zephyr. Additionally, now that we have services like Golioth, I'm excited to see more people use it across different hardware platforms and projects.

I'm also excited to announce that the latest version of the nRF9160 Feather are available.

Some of the biggest changes include:

Revamped power supply
USB connector to USB-C

The latest Air Quality Wing, is also finally available!

Some of the big changes include:

Updating the onboard sensors to Sensirion parts
Improved thermal management to prevent contaminating temperature readings
Reduced size to fit Featherwing profile

You can get some right now by heading over to my store. 🏬

Thanks for reading. If you have questions or run into problems the Circuit Dojo community is open and ready to help.

How to Connect the nRF9160 Feather to Mosquitto

Jared Wolff — Mon, 03 Aug 2020 01:59:03 +0000

One thing that's always tripped me up as an IoT developer is figuring the best way to transmit data. There are many different kinds of radios and mediums. On top of that, there are different protocols to boot!

As of this writing, there is one protocol that has reigned supreme in the IoT world:

MQTT

Unlike an HTTP server, a device can connect, publish, and subscribe to topics. These topics then get sent to a broker and distributed to other subscribed devices. It also happens that MQTT on Nordic's nRF9160 is well supported.

In this post, I'll show you how to connect the nRF9160 Feather to a self-hosted Mosquitto instance. You'll learn how to generate your own certificates, and get a hang on how to test your connections.

Ready to play? Let's get to it.

Where to host?

If you want to host Mosquitto, you'll need a server. Since Mosquitto is written in C it's lightweight and can go almost anywhere. Plus it sips resources so you can install it on a budget VPS without much worry. That's where a VPS provider like Digital Ocean or Vultr comes in.

To set up a new server here are some steps:

Login to Digital Ocean. If you don’t have Digital Ocean and would like to support click here to create an account. ($100 credit for 60 days if you do!)
Create a new Droplet
Choose the FreeBSD 12.1 with UFS.
Choose the $5 instance. That's usually more than enough.
Make sure you import your public key. Otherwise, you won't be able to immediately use password-less login.
Hit that green Create Droplet button, and let's get this show on the road.

Important Extra step

For the certs to work with Mosquitto, you'll have to set a domain to point to your VPS IP address. A CNAME or A record works. If you're unsure how to do that, here's a good guide. Note which (sub)domain you've used. We'll need it in a bit...

Install Mosquitto

I run my servers inside of a FreeBSD jail using Bastille. In this tutorial, we'll be skipping the jail part and focus on getting the nRF9160 Feather working.

You should be set with a Digital Ocean instance (or similar) using FreeBSD. If you don’t head back up to the Where to host? section.

Next to install mosquitto one your droplet run pkg install mosquitto. If you're running something other than FreeBSD, this command may differ.
apt-get install mosquitto works on Debian based systems. If you want the most up to date repositories, make sure you run sudo apt-add-repository ppa:mosquitto-dev/mosquitto-ppa beforehand.
Here's what the full output on FreeBSD looks like:

$ pkg install mosquitto
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:12:amd64/quarterly, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
[mosquitto] Installing pkg-1.14.6...
[mosquitto] Extracting pkg-1.14.6: 100%
Updating FreeBSD repository catalogue...
[mosquitto] Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
[mosquitto] Fetching packagesite.txz: 100%    6 MiB   6.6MB/s    00:01
Processing entries: 100%
FreeBSD repository update completed. 31943 packages processed.
All repositories are up to date.
Updating database digests format: 100%
The following 4 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        c-ares: 1.16.1
        ca_root_nss: 3.55
        e2fsprogs-libuuid: 1.45.6
        mosquitto: 1.6.7

Number of packages to be installed: 4

The process will require 2 MiB more space.
682 KiB to be downloaded.

Proceed with this action? [y/N]: y
[mosquitto] [1/4] Fetching mosquitto-1.6.7.txz: 100%  226 KiB 231.1kB/s    00:01
[mosquitto] [2/4] Fetching ca_root_nss-3.55.txz: 100%  285 KiB 291.5kB/s    00:01
[mosquitto] [3/4] Fetching e2fsprogs-libuuid-1.45.6.txz: 100%   34 KiB  34.7kB/s    00:01
[mosquitto] [4/4] Fetching c-ares-1.16.1.txz: 100%  138 KiB 140.9kB/s    00:01
Checking integrity... done (0 conflicting)
[mosquitto] [1/4] Installing ca_root_nss-3.55...
[mosquitto] [1/4] Extracting ca_root_nss-3.55: 100%
[mosquitto] [2/4] Installing e2fsprogs-libuuid-1.45.6...
[mosquitto] [2/4] Extracting e2fsprogs-libuuid-1.45.6: 100%
[mosquitto] [3/4] Installing c-ares-1.16.1...
[mosquitto] [3/4] Extracting c-ares-1.16.1: 100%
[mosquitto] [4/4] Installing mosquitto-1.6.7...
===> Creating users
Using existing user 'nobody'.
[mosquitto] [4/4] Extracting mosquitto-1.6.7: 100%
=====
Message from ca_root_nss-3.55:

--
FreeBSD does not, and can not warrant that the certification authorities
whose certificates are included in this package have in any way been
audited for trustworthiness or RFC 3647 compliance.

Assessment and verification of trust is the complete responsibility of the
system administrator.

This package installs symlinks to support root certificates discovery by
default for software that uses OpenSSL.

This enables SSL Certificate Verification by client software without manual
intervention.

If you prefer to do this manually, replace the following symlinks with
either an empty file or your site-local certificate bundle.

  * /etc/ssl/cert.pem
  * /usr/local/etc/ssl/cert.pem
  * /usr/local/openssl/cert.pem
=====
Message from mosquitto-1.6.7:

--
The mosquitto MQTT Python driver is now provided by net/py-paho-mqtt

All installed package configuration lives at /usr/local/etc/mosquitto/ we'll need to edit mosquitto.conf in that folder to use certificates. Here's what it looks like:

# Daemon configuration
pid_file /var/run/mosquitto.pid
user nobody

# Port to use for the default listener.
port 8885

# At least one of cafile or capath must be defined.
cafile /root/pki/ca.crt

# Path to the PEM encoded server certificate.
certfile /root/pki/issued/mosquitto.crt

# Path to the PEM encoded keyfile.
keyfile /root/pki/private/mosquitto.key

# Path to CRL file
#crlfile /root/pki/crl.pem

# Each client has their own cert
require_certificate true
use_identity_as_username true

# listener port-number [ip address/host name]
listener 1883
protocol mqtt

# listener port-number [ip address/host name]
# listener 8080
# protocol websockets

# =================================================================
# Logging
# =================================================================
log_dest syslog

# Types of messages to log.
log_type all
#log_type warning
# websockets_log_level 127

# -----------------------------------------------------------------
# Default authentication and topic access control
# -----------------------------------------------------------------
# password_file /usr/local/etc/mosquitto/pwfile

Before we can start the server we'll need to provision some RSA certificates. We'll get to that in the next step!

Provision Certs

You can use easy-rsa to generate a CA server and client certs. (These instructions come from this guide.) For production, you should generate your keys and certs on an offline machine. That way your private keys are safe if your server becomes a target.

First, install easy-rsa:

$ pkg install easy-rsa
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        easy-rsa: 3.0.7

Number of packages to be installed: 1

44 KiB to be downloaded.

Proceed with this action? [y/N]: y
[mosquitto] [1/1] Fetching easy-rsa-3.0.7.txz: 100%   44 KiB  44.8kB/s    00:01
Checking integrity... done (0 conflicting)
[mosquitto] [1/1] Installing easy-rsa-3.0.7...
[mosquitto] [1/1] Extracting easy-rsa-3.0.7: 100%

Then lets begin the cert creation process!

$ easyrsa init-pki

Note: using Easy-RSA configuration from: /usr/local/share/easy-rsa/vars

init-pki complete; you may now create a CA or requests.
Your newly created PKI dir is: /root/pki
$
$ easyrsa build-ca

Note: using Easy-RSA configuration from: /usr/local/share/easy-rsa/vars
Using SSL: openssl OpenSSL 1.1.1d-freebsd  10 Sep 2019

Enter New CA Key Passphrase:
Re-Enter New CA Key Passphrase:
Generating RSA private key, 2048 bit long modulus (2 primes)
......................+++++
..................................................................................+++++
e is 65537 (0x010001)
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Common Name (eg: your user, host, or server name) [Easy-RSA CA]:testserver.jaredwolff.com

CA creation complete and you may now import and sign cert requests.
Your new CA certificate file for publishing is at:
/root/pki/ca.crt

Note: You will be prompted for a password at the build-ca step. Make sure you keep this password handy.

Then to generate a server cert use:

$ easyrsa gen-req mosquitto nopass

Note: using Easy-RSA configuration from: /usr/local/share/easy-rsa/vars
Using SSL: openssl OpenSSL 1.1.1d-freebsd  10 Sep 2019
Generating a RSA private key
...............+++++
........................................+++++
writing new private key to '/root/pki/easy-rsa-82720.X2NVQ0/tmp.akOxhO'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Common Name (eg: your user, host, or server name) [mosquitto]:testserver.jaredwolff.com

Keypair and certificate request completed. Your files are:
req: /root/pki/reqs/mosquitto.req
key: /root/pki/private/mosquitto.key
$
$ easyrsa sign-req server mosquitto

Note: using Easy-RSA configuration from: /usr/local/share/easy-rsa/vars
Using SSL: openssl OpenSSL 1.1.1d-freebsd  10 Sep 2019

You are about to sign the following certificate.
Please check over the details shown below for accuracy. Note that this request
has not been cryptographically verified. Please be sure it came from a trusted
source or that you have verified the request checksum with the sender.

Request subject, to be signed as a server certificate for 825 days:

subject=
    commonName                = testserver.jaredwolff.com

Type the word 'yes' to continue, or any other input to abort.
  Confirm request details: yes
Using configuration from /root/pki/easy-rsa-82744.hyuGzt/tmp.lZHLEH
Enter pass phrase for /root/pki/private/ca.key:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
commonName            :ASN.1 12:'testserver.jaredwolff.com'
Certificate is to be certified until Nov  3 01:12:53 2022 GMT (825 days)

Write out database with 1 new entries
Data Base Updated

Certificate created at: /root/pki/issued/mosquitto.crt

You'll be prompted for both the Common Name (i.e. your server name) and the CA cert password in the above step. Important: the Common Name needs to match the domain name of your server! (Remember, we wrote that down earlier?)

To generate the nRF9160 cert use:

$ easyrsa gen-req nrf9160 nopass batch
$ easyrsa sign-req client nrf9160 batch

Follow the same procedure as earlier. The only difference is that we're generating a client cert instead of a server cert.

Once complete, we'll need some files. Here's a full list below:

For your Mosquitto Server

/root/pki/ca.crt
/root/pki/private/mosquitto.key
/root/pki/issued/mosquitto.crt

For your nRF9160 Feather

/root/pki/ca.crt
/root/pki/private/nrf9160.key
/root/pki/issued/nrf9160.crt

If you're using the configuration from above, it's already pointing to your server certificates. All we have to do now is start mosquitto!

$ service mosquitto start
Cannot 'start' mosquitto. Set mosquitto_enable to YES in /etc/rc.conf or use 'onestart' instead of 'start'.

If you get an error about mosquitto_enable simply run:

$ sysrc mosquitto_enable=YES
$ service mosquitto start
Starting mosquitto.

This enables mosquitto to start when your system starts.

Now, check if mosquitto is running by using ps aux


$ ps aux
USER     PID %CPU %MEM   VSZ  RSS TT  STAT STARTED    TIME COMMAND
root   82401  0.0  0.2 11472 2424  -  SsJ  01:02   0:00.00 /usr/sbin/syslogd -ss
root   82457  0.0  0.2 11408 2284  -  IsJ  01:02   0:00.00 /usr/sbin/cron -J 60 -s
nobody 82900  0.0  0.6 16352 6212  -  SsJ  01:17   0:00.02 /usr/local/sbin/mosquitto -c /usr/local/etc/mosquitto/mosquitto.conf -d
root   82488  0.0  0.3 12096 2848  0  IJ   01:02   0:00.01 login [pam] (login)
root   82489  0.0  0.3 13092 3504  0  SJ   01:02   0:00.03 -csh (csh)
root   82902  0.0  0.3 11704 2540  0  R+J  01:17   0:00.00 ps aux

Now that we have a server loaded and running. Let's get the firmware working!

Firmware bits

Dealing with certificates on the nRF9160 Feather is a two-step process. The first step is to load the certificates using the at_client firmware. The second is to load the mqtt_simple library with added TLS support. Let's tackle the certs first!

Program `at_client` first

Change directories to ncs/nrf/samples/nrf9160/at_client/ and start a fresh build:

$ west build -b circuitdojo_feather_nrf9160ns -p

Then flash to your board using:

$ west flash --erase
$ nrfjprog -r

We'll need this sample on your board for the next step.

Add certs to device

To install our new certs, we'll need nRF Connect Desktop installed. You can download it by going here.

You'll also need a custom version of LTE Link Monitor. You can get the modified version on docs.jaredwolff.com.

First, Install nRF Connect Desktop app. Then, copy the LTE Link Monitor .tgz file to %USERPROFILE%\.nrfconnect-apps\local (on Windows) or $HOME/.nrfconnect-apps/local (on Linux/macOS). Here's an example of where it is on Windows:

Close and re-open nRF Connect Desktop (if it's open).

Then, click Open next to the v1.1.1 version of LTE Link Monitor. It will also have local written underneath it.

Next, let's launch it!

Once you've opened the port, hit the reset button. Make sure you turn off Automatic requests.

Then in the command box send AT+CFUN=4. This will shut down your modem so it's ready to upload certs. You can run AT+CFUN? to confirm your modem in that mode.

Open up the Certificate manager.

Make sure you set the security tag. In this case, I'm using 1234. This is an important identifier that you'll need later. Make it whatever you want but I would avoid using 16842753. This is the default tag for NRF Cloud. You don't want to blast away your nRF Cloud certs!

Copy and paste the contents of your ca.crt, nrf9160.crt and nrf9160.key into the boxes. (in that order) You can easily get the certs by using cat on Unix/Linux:

$ cat cat.crt
-----BEGIN CERTIFICATE-----
MIIDdTCCAl2gAwIBAgIUDLkBxLLQO9wosNDtA7E9qvqHOxMwDQYJKoZIhvcNAQEL
BQAwJDEiMCAGA1UEAwwZdGVzdHNlcnZlci5qYXJlZHdvbGZmLmNvbTAeFw0yMDA3
MzEwMTExNDJaFw0zMDA3MjkwMTExNDJaMCQxIjAgBgNVBAMMGXRlc3RzZXJ2ZXIu
amFyZWR3b2xmZi5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC3
de1v8k+FXzY/Im7Z2YKS7wwbBRft5CUxqP1sdYJgMvheS9LhFufk81URZ0lHD9pK
aNPxU1UEmnLvVDTGLJ+YAmMH08xn17FS1R1UVPYzi2ouwqRM2pR9EStsSlP9Zj44
1MsdizABnnlkZndUVLL/gjc4cNsNncMLBSEbsz6b5WzhtAGg3rOpdAxSSblZVSFw
bquCgg5hb2NUzy+JxGtUIsE5d6CxTDdSs4Z3FK/RRYjmCG6qsaya4N5W35yf8h5O
StfKRecl3kq2kCnWa6P+lErG4wuxIBtMkgz2zV+zd1tz4aHXxSdoZTqLz7dTVbFA
zEVnKD+ZReBG+4fwUL6rAgMBAAGjgZ4wgZswHQYDVR0OBBYEFIvdGnjrxRPzvXQi
7XJ70LzpZSOjMF8GA1UdIwRYMFaAFIvdGnjrxRPzvXQi7XJ70LzpZSOjoSikJjAk
MSIwIAYDVQQDDBl0ZXN0c2VydmVyLmphcmVkd29sZmYuY29tghQMuQHEstA73Ciw
0O0DsT2q+oc7EzAMBgNVHRMEBTADAQH/MAsGA1UdDwQEAwIBBjANBgkqhkiG9w0B
AQsFAAOCAQEAIzz1nSSDkPueNPlADRYMDOMFNkxoKA+gRXwDVa7y39As7IZp7Fqr
KAH79U1XtGyDlt6FPKTvDJ7jtd4y8auIGVQO7z3AG9pVU1imIWZHoIqgBUCEhsjp
uMxD23kRCX5kd9dsmF9WOGGxb4kkMv83Rh2rCONQmvnozuI3fJv2ZFX/ORoADGLP
OPSJPl11x+2rxPxiLi+T8RyzDh3DwqnPVsSnbRWV7hosaN0ip/cbnSTaIul9mbCY
ID6qm9leqlY/gha9aZfg+tv1Lm6PT6o8Pzek2VeDoIS5YERBMOwV84hQrZjV3vIE
jT6y663HGsl7KvqVaWdV3fM6Cr7f0QdR5A==
-----END CERTIFICATE-----

You'll need everything from -----BEGIN CERTIFICATE----- to -----END CERTIFICATE-----

Check the Log area for more details. If all went well, it should say your certificates updated.

Using the mqtt_simple example

We'll be using the mqtt_simple sample within the nRF Connect SDK repository. The full path is: ncs/nrf/samples/nrf9160/mqtt_simple. We'll need to make a few edits to add full TLS compatibility. All the files are within the mqtt_simple directory.

First, we'll have to update the proj.conf file. See the highlighted differences:

The # Set the PDP Context section is especially important if you're using a Hologram SIM card. (Included with the nRF9160 Feather) If you are using a SIM that doesn't need it, you do not need this section.

Adopt your CONFIG_MQTT_BROKER_HOSTNAME to your hostname. (configured at the beginning of this guide).

You'll also have to add these lines in KConfig:

config SEC_TAG
    int "Security tag to use for the connection"
    default 1234

config PEER_VERIFY
    int "Peer verify parameter for mqtt_client"
    default 1
    help
            Set to 0 for VERIFY_NONE, 1 for VERIFY_OPTIONAL, and 2 for VERIFY_REQUIRED.

Finally in main add this block to the top of your file:

#if defined(CONFIG_MQTT_LIB_TLS)
static sec_tag_t sec_tag_list[] = { CONFIG_SEC_TAG };
#endif /* defined(CONFIG_MQTT_LIB_TLS) */

Then add this block to client_init under #if defined(CONFIG_MQTT_LIB_TLS)

  struct mqtt_sec_config *tls_config = &client->transport.tls.config;

    client->transport.type = MQTT_TRANSPORT_SECURE;

    tls_config->peer_verify = CONFIG_PEER_VERIFY;
    tls_config->cipher_count = 0;
    tls_config->cipher_list = NULL;
    tls_config->sec_tag_count = ARRAY_SIZE(sec_tag_list);
    tls_config->sec_tag_list = sec_tag_list;
    tls_config->hostname = CONFIG_MQTT_BROKER_HOSTNAME;

The changes should look like this:

Then build with:

$ west build -b circuitdojo_feather_nrf9160ns -p

Finally, flash it using west flash

$ west flash --erase
$ nrfjprog -r

Open your serial terminal and double check that your nRF9160 Feather is connecting. You can also use LTE Link Monitor to view your progress as well. (Example below)

Lots of the information above came from Nordic's post on the subject.

Sending a message

We're almost there! You've configured your nRF9160 Feather to connect to Mosquitto using self-generated certificates. The last part is connecting another device to see if the nRF9160 Feather replies to a message.

I've created a new set of certs for this purpose. I called them test.

$ easyrsa gen-req test nopass batch
$ easyrsa sign-req client test batch

I copied them to my desktop using CyberDuck (a great little visual SFTP client):

You can also use something like scp if your confident in your command line file transfer abilities! Then, open a terminal and run:

mosquitto_sub --cafile ca.crt --cert test.crt --key test.key -q 1 -d -h testserver.jaredwolff.com -p 8885 -t "/my/publish/topic" &
mosquitto_pub --cafile ca.crt --cert test.crt --key test.key -q 1 -d -h testserver.jaredwolff.com -p 8885 -t "/my/subscribe/topic" -m "hello there"

You should see an output like this:

$ mosquitto_sub --cafile ca.crt --cert test.crt --key test.key -q 1 -d -h testserver.jaredwolff.com -p 8885 -t "/my/publish/topic" &
$ mosquitto_pub --cafile ca.crt --cert test.crt --key test.key -q 1 -d -h testserver.jaredwolff.com -p 8885 -t "/my/subscribe/topic" -m "hello there"
Client mosq-CczskQKzMKdtTo4O4s sending CONNECT
Client mosq-CczskQKzMKdtTo4O4s received CONNACK (0)
Client mosq-CczskQKzMKdtTo4O4s sending PUBLISH (d0, q1, r0, m1, '/my/subscribe/topic', ... (11 bytes))
Client mosq-CczskQKzMKdtTo4O4s received PUBACK (Mid: 1, RC:0)
Client mosq-CczskQKzMKdtTo4O4s sending DISCONNECT
MacBook-Pro:Downloads jaredwolff$ Client mosq-qK8tMlJk0Qri4Z7jUo sending PINGREQ
Client mosq-qK8tMlJk0Qri4Z7jUo received PINGRESP
MacBook-Pro:Downloads jaredwolff$ Client mosq-qK8tMlJk0Qri4Z7jUo received PUBLISH (d0, q0, r0, m0, '/my/publish/topic', ... (11 bytes))
hello there

Booyah! You have an active working connection to your very own Mosquitto server!

Conclusion

We've made it to the end! By this point in the post, you should have a Mosquitto server running and an nRF9160 connected. Now you can use your new-found skills to add more devices to your deployments and more!

If you haven't had a chance to play with the nRF9160 you should check out the nRF9160 Feather. It has Nordic Semiconductor's nRF9160 LTE-M, NB IoT + GPS Combo, plus flexible power supply, external flash, and low power shutdown. Oh, and did I mention it's 100% open source? Learn more by checking out the campaign on GroupGets and Hackster Launch. 🎉

Photo credit to the awesome folks at GroupGets!

My Latest Self Hosted Hugo Workflow

Jared Wolff — Fri, 27 Mar 2020 20:01:46 +0000

After hosting with Netlify for a few years, I decided to head back to self hosting. Theres a few reasons for that but the main reasoning was that I had more control over how things worked.

In this post, i'll show you my workflow for deploying my Hugo generated site (www.jaredwolff.com). Instead of using what most people would go for, i'll be doing all of this using a FreeBSD Jails based server. Plus i'll show you some tricks i've learned over the years on bulk image resizing and more.

Let's get to it.

Where to host?

If you want to host your own service, you'll need a server. That's where a VPS provider like Digital Ocean or Vultr comes in. I've been a fan and have used Digital Ocean for a while now.

To set up a new server here are some steps:

Login to Digital Ocean. If you don’t have Digital Ocean and would like to support this blog click here to create an account.
Go to Account Settings -> Security and make sure you have an SSH key setup.
Create a new FreeBSD droplet. Make sure you use the UFS version
Make sure you select the $5 a month plan. For simple installs, this is more than enough!
Make sure your SSH key is selected
Finally click that green Create Droplet button!
SSH in once you’re done: ssh root@<yourserverip>

Setting up your FreeBSD server with Bastille

Up until recently everything was running on a Docker based platform using Exoframe. It was easy and almost brainless. The unfortunate part of this is the fact that Docker takes up wayyyy to many resources. Plus managing files within a Docker container is as much or more work than hosting it natively. Oh and have you checked how much space Docker has been using on your machine lately? On my development machine it's was about 19GB of space. 😬

So what's the alternative?

FreeBSD Jails using Bastille.

I've been playing with Bastille for a few months now. The more I use it, the more it makes 100% sense.

Bastille allows you to create a (now) portable lightweight FreeBSD based jails. These jails are "containers" that have virtually no overhead. There's no daemon (the operating system is the "daemon"!). Plus, Jails are secure compared to the can of worms that Docker is. Yes, you may have to compile and port some utilities. Most though are already supported in FreeBSD's package manager pkg.

In this section you'll learn how to get a jail running with caddy so you can securely host your site.

Let's keep the momentum going!

Once you get the IP address for your server you should login:

ssh root@123.456.789.10

You should get a MOTD message and a sh prompt. Woo!

FreeBSD 12.1-RELEASE-p2 GENERIC

Welcome to FreeBSD!
...

#

Let's install a few important bits using pkg (FreeBSD's package manager)

pkg install restic rsync bastille

We'll be using restic for backups, rsync for transferring files and bastille for jail setup.

You also have to set up some static routes in your pf.conf. Here's an example of mine:

ext_if="vtnet0"

# Caddy related
caddy_addr=10.10.2.20

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo

table <jails> persist
nat on $ext_if from <jails> to any -> $ext_if

# container routes
rdr pass inet proto tcp from any to port 80 -> $caddy_addr port 8880
rdr pass inet proto tcp from any to port 443 -> $caddy_addr port 4443

# Enable dynamic rdr (see below)
rdr-anchor "rdr/*"

block in all
pass out quick modulate state
antispoof for $ext_if inet
pass in inet proto tcp from any to any port ssh flags S/SA keep state

This is a standard pf.conf file for bastille. Make sure you edit caddy_addr to the IP you chose.

Now let's start the firewall. You will get kicked out of your ssh session:

sysrc pf_enable="YES"
service pf start

Then let's get some bastille configuration out of the way:

# set up bastille networking
sysrc cloned_interfaces+=lo1
sysrc ifconfig_lo1_name="bastille0"
service netif cloneup

# bootstrap the base jail and start bastille
bastille bootstrap 12.1-RELEASE update
sysrc bastille_enable="YES"
service bastille start

This will set up your networking, and fetch the latest default base jail you'll use later.

Next, let's set up the jail:

bastille create caddy 12.1-STABLE 10.10.2.20
bastille start caddy

Then install caddy

#install the binary
fetch https://github.com/caddyserver/caddy/releases/download/v1.0.4/caddy_v1.0.4_freebsd_amd64.tar.gz
tar xvf caddy_v1.0.4_freebsd_amd64.tar.gz caddy
bastille cp caddy caddy /usr/local/bin/
rm caddy

#create the caddy user
bastille cmd caddy pw useradd caddy -m -s /usr/sbin/nologin

#install ca root file
bastille pkg caddy install ca_root_nss

When installing ca_root_nss , pkg will have to initialize. Accept the prompts. Once you're done here we'll move on to the next step!

Once install is complete, we should also configure caddy to start on boot. The easiest way to do that is use this rc.d script.

#!/bin/sh

# $FreeBSD: head/net/caddy/files/caddy.in 452063 2017-10-14 12:58:24Z riggs $
#
# PROVIDE: caddy
# REQUIRE: LOGIN
# KEYWORD: shutdown
#
# Add the following lines to /etc/rc.conf.local or /etc/rc.conf
# to enable this service:
#
# caddy_enable (bool):  Set to NO by default.
#               Set it to YES to enable caddy.
# caddy_user (user):        Set user to run caddy.
#               Default is "caddy".
# caddy_group (group):  Set group to run caddy.
#               Default is "caddy".
# caddy_conf (path):        Path to caddy configuration file.
#               Default is /usr/local/etc/caddyfile.conf

. /etc/rc.subr

name=caddy
rcvar=caddy_enable

load_rc_config $name

: ${caddy_enable:="NO"}
: ${caddy_user:="caddy"}
: ${caddy_group:="caddy"}
: ${caddy_conf:="/usr/local/etc/caddyfile.conf"}
: ${caddy_log:="/home/caddy/caddy.log"}
: ${caddy_env:="CADDYPATH=/home/caddy/"}
: ${caddy_https_port:="4443"}
: ${caddy_http_port:="8880"}

pidfile="/var/run/caddy.pid"
procname="/usr/local/bin/caddy"
command="/usr/sbin/daemon"
command_args="-f -p ${pidfile} /usr/bin/env ${caddy_env} ${procname} -agree -http-port ${caddy_http_port}  -https-port ${caddy_https_port} -conf=${caddy_conf} -log=${caddy_log} ${caddy_args}"
extra_commands="reload"

start_precmd=caddy_startprecmd
reload_cmd=caddy_reloadcmd

caddy_startprecmd()
{
      if [ ! -e ${pidfile} ]; then
              install -o ${caddy_user} -g ${caddy_group} /dev/null ${pidfile};
      fi
}

caddy_reloadcmd()
{
      kill -s USR1 $(cat ${pidfile})
}

run_rc_command "$1"

Remove the caddy executable if you haven't already. Then create a new file with vi. This will be your rc.d script!

vi caddy

Then paste the contents of the above script in there, save and exit.

Then make sure the file is executable by using chmod and copy to the Caddy container.

chmod +x caddy
bastille cp caddy caddy /usr/local/etc/rc.d/

Finally, we'll need a Caddyfile. Here's an example of one:

stage.jaredwolff.com {
  tls hello@jaredwolff.com
  log /home/caddy/stage.jaredwolff.com.log
  root /var/www/stage.jaredwolff.com/
  gzip
  log stderr
}

log refers to this site specific access log.

root refers to where the root public folder is on your machine. In my case it's the common /var/www/<name of site>. Set your paths and remember them. We'll need them later!

To have Caddy generate certs for this subdomain, you'll have to set the tls option. An email is all that's needed.

For more on the Caddyfile structure check out the documentation.

Make a file called caddyfile.conf and copy it to /usr/local/etc/ in your Caddy container:

vi caddyfile.conf
# Paste your caddyfile contents and save
bastille cp caddy caddyfile.conf /usr/local/etc/

You should now redirect your DNS to the server IP. That way Caddy can generate/fetch the correct certificates. Then you can start Caddy with:

bastille service caddy caddy start

You can check the log at /usr/home/caddy/caddy.log to make sure that your domain provisioned correctly.

Side note: Getting setup with SSL certs is tough at first. (Especially if you're migrating from another server.) Your site will have to go down for a little bit while you switch your DNS settings and start caddy. (That's if you're using standard caddy 1.0. You can also use the DNS provider plugins here which make things a little easier.)

Now that we have caddy up and running it's time to copy our hugo generated assets over using rsync. We're off to the next step!

Make building and deploying easy

I spend a ton of time programming C code and that means I spend tons of time using Makefiles. For many make (or gmake for GNU make) is the bane of our existences. For building and deploying, make makes it easy to create reusable recipes. That way you know you can deploy with confidence every time.

My Makefile borrows from the one that Victoria Drake had posted not too long ago. I changed it up a bit to match my needs. Let's take a tour and see what's inside:

.POSIX:

HUGO_VERSION := 0.66.0

OPTIMIZED_DIR := optimized
CONTENT_DIR := content
DEST_DIR := public

SERVER := 123.456.789.10
USER := user

The first section contains all the variables that I use to tell the functions later on what to do. It also has a reference to the .POSIX target. This means that the Makefile will be as portable between different versions of make.

Then, I popped in some logic to determine whether i'm deploying to stage or production:

# Set the place where it's deployed to.
ifdef PRODUCTION
$(info Building for production. 🚀)
TARGET := www
else
$(info Building for development. 🚀)
BASEURL := --baseURL "https://stage.jaredwolff.com"
TARGET := stage
endif

By default, recipes below will use the development workflow. To use the production workflow, you can invoke make like this:

PRODUCTION=1 make build

This does add some extra friction to the deploy process. It's a good step though. That way you're sure the deploy is going to the right place!

# Full path
DEPLOY_DIR := /usr/local/bastille/jails/caddy/root/path/to/$(TARGET).jaredwolff.com

Using the TARGET variable above, I then define the path to my server assets. I'm using Bastille to organize my jails so the path is extra lengthly long. (yea, lengthly long) This allows us to use rsync to deploy the files with ease.

Now here come the fun bits. To do a full bulk resize, i'm using the wildcard functionality of the Makefile.

IMAGES := \
$(wildcard $(CONTENT_DIR)/*/images/*.jpg) \
$(wildcard $(CONTENT_DIR)/*/images/*.JPG) \
$(wildcard $(CONTENT_DIR)/*/images/*.jpeg) \
$(wildcard $(CONTENT_DIR)/*/images/*.png) \
$(wildcard $(CONTENT_DIR)/*/*/images/*.jpg) \
$(wildcard $(CONTENT_DIR)/*/*/images/*.jpeg) \
$(wildcard $(CONTENT_DIR)/*/*/images/*.png) \
$(wildcard $(CONTENT_DIR)/*/*/images/*.JPG) \

In this case it will create a huge space delimited list of every image that is within my content directory. The biggest drawback of this method is that it's not space tolerant. An easy fix to this is to make sure that all my photos do not have spaces.

Here's a quick and dirty bash command. You can use to rename files that have spaces and replace them with '_' characters:

for f in *\ *; do mv "$f" "${f// /_}"; done

Next, we rename these entries so the prefix is now the target directory. This will be useful when we want to resize:

OPTIMIZED_IMAGES := \
$(subst $(CONTENT_DIR)/,$(OPTIMIZED_DIR)/,$(IMAGES))

Now check out the optimize recipe:

.PHONY: optimize
optimize: build $(OPTIMIZED_IMAGES)
@echo "🧂 Optimizing images"
rsync -r $(OPTIMIZED_DIR)/ $(DEST_DIR)/
du -sh $(CONTENT_DIR)/
du -sh $(DEST_DIR)/

$(OPTIMIZED_IMAGES):
convert -strip -compress JPEG -resize '730>' $(subst $(OPTIMIZED_DIR)/,$(CONTENT_DIR)/,$@) $@

It first calls the build recipe and then also the $(OPTIMIZED_IMAGES) recipe. The later will optimize the image using the convert command from Imagemagick. In this case i'm only resizing files that are larger than 730px wide. Change your accordingly so you can reap the benefits of an optimized site.

After resizing, the recipe uses rsync to copy the files from the OPTIMIZED_DIR to DEST_DIR.

If we take a look at the build recipe, I first building the assets. Then, I copy the photos from the content dir to optimized dir. The nice thing is that rsync will only move files that have changed. Thus it doesn't have to copy the files over and over and over again every time you build.

Finally, the deploy recipe.

.PHONY: deploy
deploy:
@echo rsync to $(DEPLOY_DIR)
@rsync -r --del public/ $(USER)@$(SERVER):$(DEPLOY_DIR)/
@echo making restic snapshot
@scp scripts/backup.sh $(USER)@$(SERVER):/root/backup.sh
@ssh $(USER)@$(SERVER) sh /root/backup.sh $(DEPLOY_DIR)
@echo "🚀 Site is deployed!"

You can see again that i'm using rsync to sync the contents of public/ to the server. Make sure you set the USER , SERVER and DEPLOY_DIR. In my case DEPLOY_DIR comes out to /usr/local/bastille/jails/caddy/root/var/www/www.jaredwolff.com

When you do finally get a successful deploy you can double check everything is in the correct place. Then once everything looks good you can start up your caddy server using

bastille service caddy caddy start

deploy will also do something extra handy here. It will deploy my restic backup script and run it. I'll talk about this more in the backup section.

All in all, here's the full Makefile

.POSIX:

HUGO_VERSION := 0.66.0

OPTIMIZED_DIR := optimized
CONTENT_DIR := content
DEST_DIR := public

SERVER := 155.138.230.8
USER := root

# Set the place where it's deployed to.
ifdef PRODUCTION
$(info Building for production. 🚀)
TARGET := www
else
$(info Building for development. 🚀)
BASEURL := --baseURL "https://stage.jaredwolff.com"
TARGET := stage
endif

# Full path
DEPLOY_DIR := /usr/local/bastille/jails/caddy/root/var/www/$(TARGET).jaredwolff.com

IMAGES := \
$(wildcard $(CONTENT_DIR)/*/images/*.jpg) \
$(wildcard $(CONTENT_DIR)/*/images/*.JPG) \
$(wildcard $(CONTENT_DIR)/*/images/*.jpeg) \
$(wildcard $(CONTENT_DIR)/*/images/*.png) \
$(wildcard $(CONTENT_DIR)/*/*/images/*.jpg) \
$(wildcard $(CONTENT_DIR)/*/*/images/*.jpeg) \
$(wildcard $(CONTENT_DIR)/*/*/images/*.png) \
$(wildcard $(CONTENT_DIR)/*/*/images/*.JPG) \

OPTIMIZED_IMAGES := \
$(subst $(CONTENT_DIR)/,$(OPTIMIZED_DIR)/,$(IMAGES))

.PHONY: all
all: build optimize

.PHONY: clean
clean:
rm -rf public/
rm -rf optimized/

.PHONY: serve
serve:
@hugo serve -D

.PHONY: ssh
ssh:
@ssh $(USER)@$(SERVER)

.PHONY: build
build:
@echo "🍳 Generating site"
hugo --gc --minify -d $(DEST_DIR) $(BASEURL)
rsync -av --del -f"+ */" -f"- *" $(CONTENT_DIR)/ $(OPTIMIZED_DIR)/

.PHONY: optimize
optimize: build $(OPTIMIZED_IMAGES)
@echo "🧂 Optimizing images"
rsync -r $(OPTIMIZED_DIR)/ $(DEST_DIR)/
du -sh $(CONTENT_DIR)/
du -sh $(DEST_DIR)/

$(OPTIMIZED_IMAGES):
convert -strip -compress JPEG -resize '730>' $(subst $(OPTIMIZED_DIR)/,$(CONTENT_DIR)/,$@) $@

.PHONY: deploy
deploy:
@echo rsync to $(DEPLOY_DIR)
@rsync -r --del public/ $(USER)@$(SERVER):$(DEPLOY_DIR)/
@echo making restic snapshot
@scp scripts/backup.sh $(USER)@$(SERVER):/root/backup.sh
@ssh $(USER)@$(SERVER) sh /root/backup.sh $(DEPLOY_DIR)
@echo "🚀 Site is deployed!"

There are a few other handy nuggets in there you may want to use. clean, serve and ssh have been very helpful when testing and connecting.

In the end you'll have a two step deploy process. The first generates your site with optimized images. The second is deploying to a server for static hosting.

Incremental Backup

After discovering Restic i've been sold on how handy it has been for all my incremental backup needs. In the case of my server, i'm using to back up the root folder of my site. That way, if I need to roll back, I can do so with a few short steps.

Here's how you can set up a local restic repo.

Setting it up

Initializing the repo is simple. The most important part is making sure you don't lose/forget your password!

    # restic init -r /root/backups
    enter password for new repository:
    enter password again:
    created restic repository 32e14c7052 at /root/backups

    Please note that knowledge of your password is required to access
    the repository. Losing your password means that your data is
    irrecoverably lost.

Set the RESTIC_PASSWORD environment variable to avoid entering your password. To make it permanent you'll have to place export RESTIC_PASSWORD="Your password here!" within the .profile file in /root/.

Backing Up

Invoking restic over SSH is tough. So our next best bet?

Transfer a (very brief) shell script to the server and run it after a deploy. Here's the contents of what I'm using today:

#!/bin/sh
export RESTIC_PASSWORD="Your password here!"
restic backup $1 -r /root/backups/

Side note: As I sit here and look at this script, for security reasons you can replace "Your password here!" with $2 which is the second argument to the script. That way you don't need to commit/push the password stored in a static file!

This first sets your backup password. Then it runs restic using the first command line argument as the path. So, to run a backup with this script, it would look something like this:

./backup.sh /path/to/your/public/folder/

Note: you do need to initialize your restic backup before you start backing up. It will barf at you otherwise!

In my case i'm placing the incremental backups on a different folder of my machine. That way they're easily accessible and fast.

Viewing your backups

To view your backups you can run the following command:

# restic snapshots -r /root/backups -g paths -c
enter password for repository:
repository e140b5e4 opened successfully, password is correct
snapshots for (paths [/usr/local/bastille/jails/caddy/root/var/www/www.jaredwolff.com]):
ID        Time                 Host         Tags
--------------------------------------------------
d3328066  2020-03-10 00:30:58  vultr.guest
f3360819  2020-03-10 04:03:03  vultr.guest
231dd134  2020-03-10 04:44:00  vultr.guest
3c1be26a  2020-03-10 04:56:19  vultr.guest
e96c947c  2020-03-10 05:03:00  vultr.guest
34c3682a  2020-03-10 14:01:37  vultr.guest
fbccdb8c  2020-03-10 14:04:26  vultr.guest
9ce11146  2020-03-10 15:38:49  vultr.guest
046b3da3  2020-03-10 15:47:06  vultr.guest
9c28d4bc  2020-03-10 15:48:25  vultr.guest
469dc228  2020-03-10 15:48:54  vultr.guest
6f78af72  2020-03-10 17:00:21  vultr.guest
29ad17b2  2020-03-10 20:18:23  vultr.guest
ed22ce1f  2020-03-10 20:20:24  vultr.guest
9c8c1b03  2020-03-11 13:56:40  vultr.guest
b6cfcfec  2020-03-11 14:08:14  vultr.guest
e8546005  2020-03-11 14:27:22  vultr.guest
49a134fe  2020-03-17 00:47:58  vultr.guest
c0beb283  2020-03-18 20:44:52  vultr.guest
--------------------------------------------------

You can use this list to determine if you need to roll back a deploy.

Restoring

Restoring from a backup, especially in a live environment needs to be quick. After viewing your backups you can restore a specific backup by using its ID.

restic restore d3328066

This will restore the files back to the backup made on 2020-03-10 00:30:58. Awesome. Plus it won't overwrite every single. It will only apply the differences from the current state and the stored state.

Conclusion

We've covered a ton of ground in this post. You've learned how to:

Deploy your own server using Vultr
Use Bastille to create Container-like Jails
Set up Caddy to serve static file assets with TLS
Deploy the files using a fairly simple Makefile and rsync
Back up after every deploy using restic

In the end we have a robust, secure and simple platform for hosting static files and services. Stay tuned as there are more posts like this coming your way soon! In the meantime check out my other posts. Thanks for reading and see you next time! 👍

Meet Pyrinas a New Way to Use Your Xenon

Jared Wolff — Tue, 10 Mar 2020 19:29:49 +0000

This post is a bit lengthy. If you prefer, signup to get the full PDF here. 📖

Talk about how Particle's Mesh deprecation announcement, many have been left to figure out how to deploy their low power sensor networks. There was always the option of using Particle's Built in Bluetooth stack but as it stands today it's not secure.

Previously I had helped form a very simple nRF SDK based hub and spoke IoT deployment. Unfortunately, it was closed source and the company is no longer around.

So what's a guy to do?

Build another one and make it open. (BSD licensed to be exact!) Open and free for anyone to use adopt and improve upon. Plus if you're building a product that's using the code, you don't have to share your improvements or proprietary code with anyone.

In this post i'll be talking about how to get started with Pyrinas. It uses Nordic's time tested SDK as a basis for the kernel of the system. The main concept of Pyrinas is to abstract as much IoT cruft away so you can focus on your application.

So without further ado, let's chat about what Pyrinas is and what it isn't.

What Pyrinas is

Is an embedded "kernel", written in C. It's open and permissive IoT development environment you can use for anything you want. Seriously. It's BSD licensed and can be used in closed source applications.
Using the power of Bluetooth 5.0 Long Range, Pyrinas allows you to communicate with many peripheral devices at one time. Currently Pyrinas has been tested with 3 simultaneous peripheral connections. Theoretically, it can support up to 20 simultaneous connections. (Thanks to Nordic's S140 Softdevice)
Pyrinas transports its data in two ways
- In a familiar string format used by Particle
- A custom Protocol Buffer for raw data transmission.

That way you have a choice of how you want to process and publish your data!

What Pyrinas isn't

Pyrinas is not a RTOS (real time operating system). If you have a need to run multiple threaded applications on embedded, Pyrinas is not for you.
Pyrinas, at this time, does not support Mesh.
An OS for every single kind of Bluetooth SoC on the market. Due to the tight coupling with Nordic's nRF SDK, Pyrinas only works with Nordic's SoCs.
A turnkey solution for IoT. Pyrinas is early on it it's development process. The aim is for it to become a viable option for anyone to develop and publish IoT applications the way they want to. There's no vendor lock in. There's no surprises.

Things you'll need

There are a few things you'll need in order to get going with Pyrinas.

At least 2 Particle Xenons
At least 1 nRF Development board or J-link programmer
Associated USB cables

Getting started with an example

Getting started with Pyrinas involves two repositories.

The OS directory has all the source, SDK dependencies and toolchain you need to use Pyrinas.

The template is where you add all your application code goes. The template provides a starting point for you and your project.

Here's how everything goes together:

Clone the OS directory to some place on your machine:

git clone https://github.com/pyrinas-iot/pyrinas-os.git --recurse-submodules

Once complete, change directories to pyrinas-os and run make setup

cd pyrinas-os
make setup

This will download your toolchain and SDK dependencies.

In order to use OTA DFU, you will need to also generate the DFU key for the process:

make gen_key

The files generated by this process will be used later.

Next, we'll want to use the template to make two new projects. In this example we'll have one "hub" and one "sensor." Simply navigate to the template repository and click the Use this template button.

Then name your new repository. Click the Create repository from template button when you're happy with everything.

Then clone your repository to the same directory as pyrinas-os. Make sure you replace <your username> and <repo name> with your own.

cd ..
git clone https://github.com/<your username>/<repo name>.git hub

After this is done. Go back and create a new repository from the template. We'll be using this one for the sensor node.

Clone this repository once you're done setting it up in the same place as your hub and pyrinas-os repositories.

Now that we have all our repositories, let's start with our sensor node.

Setting up the sensor node repository

Open up the sensor repository using a program like Microsofts VS Code. If you have the command line shortcuts you can use code to open it from the terminal:

code sensor

Before we do anything we'll need to set up the symbolic link to pyrinas-os. Make sure you're in the sensor directory and then run ln -s ../pyrinas-os/ using the terminal.

cd sensor
ln -s ../pyrinas-os/ .

This allows your project to use all the code, SDK and toolchains within the pyrinas-os repository! As an added bonus you can do this as many times as you want. Have multiple Pyrinas projects? No problem.

Alright! Now, let's check out the Makefile. You'll want to customize some of the definitions within the file:

# Start: Your configuration!

# Set this to the directory of pyrinas-os
# If you used a symbolic link this points to
# the `pyrinas-os` folder in this repository
OS_DIR := pyrinas-os

# This should be the serial number of your Jlink programmer.
# To find simply run `jlinkexe`
PROG_SERIAL=1234678

# This is your debugger port for Jlink's RTT. If you
# have mulitple, you will have to change this on each app
# your're using
PROG_PORT=19021

# This is where you set your board type. Here are the supported boards:
# xenon - Particle Xenon
BOARD=xenon

# This is where you can name your app something. Make it specific
APP_FILENAME=pyrinas-template

# This determines whether or not you're using debug mode
# Comment this out or change to false
DEBUG=true

# End: Your Configuration

For example, you may want to setup your programmer serial. This allows you to use multiple programmers at the same time. (Very helpful in debugging both devices at the same time) To get your programmer's serial plug in your development board and run jlinkexe.

jlinkexe
SEGGER J-Link Commander V6.62a (Compiled Jan 31 2020 12:59:22)
DLL version V6.62a, compiled Jan 31 2020 12:59:05

Connecting to J-Link via USB...O.K.
Firmware: J-Link OB-SAM3U128-V2-NordicSemi compiled Jan 21 2020 17:30:48
Hardware version: V1.00
S/N: 581758669
License(s): RDI, FlashBP, FlashDL, JFlash, GDB
VTref=3.300V


Type "connect" to establish a target connection, '?' for help
J-Link>

Find the S/N area. This is your serial number!

Alternatively you can look at the sticker on your development kit. It will contain the serial number for your device.

For the PROG_PORT you want to use different ports for each device you're simultaneously debugging. I've found 19021 and 19020 are perfectly good options for most two-device debugging sessions.

The Makefile also includes the ability to choose a board. In our case there's only one option: xenon. Future revisions of Pyrinas will have multiple options.

APP_FILENAME is the name of your app. We'll rename ours to pyrinas-sensor

Finally, DEBUG is used to either halt execution on an error or restart. For production this should be commented out or set to false. We can leave it as is for now.

The Makefile is also the source for some of the most important commands you'll need during development:

make build - builds your app.
make clean - cleans all remnants of your app.
make debug - opens up the jlinkexe debugger console.
make erase - erases the chip attached to your programmer.
make flash - flashes the current app to your connected device.
make flash_softdevice - flashes the soft_device
make rtt - opens up the debug console.
make ota - generates a zip file used for BLE DFU

Basic sensor node code

Now that we have some of the basics out of the way, let's create a very simple application that publishes on a set interval. If you look at app.c, you'll see some code in the setup() function. Let's delete all the commented out code. (We'll use it later for the hub though)

Your code should now look something like:

#include "app.h"

void setup()
{
  BLE_STACK_PERIPH_DEF(init);

  // Configuration for ble stack
  ble_stack_init(&init);

  // Start advertising
  advertising_start();
}

void loop()
{
}

Now let's create a timer that we'll use to publish on a set interval. Under #include "app.h" create a new timer:

#include "app.h"

timer_define(m_sensor_timer);

We also need to set it up in the setup() function:

// Sensor timer
timer_create(&m_sensor_timer, TIMER_REPEATED, sensor_timer_evt);

You'll notice that timer_create is referring to a event callback called sensor_timer_evt. We'll need to create that guy as well:

static void sensor_timer_evt() {
    // We'll come back in a sec
}

The last thing is to start the timer. Let's do that underneath timer_create:

// Start
timer_start(&m_sensor_timer, 1000);

This will start our repeating timer on a 1 second interval. (timer_start is configured using milliseconds)

Now, inside sensor_timer_evt we'll publish some data. First though we need to make sure that Bluetooth is connected using ble_is_connected.

static void sensor_timer_evt
{
  // Check if we're connected
  if (ble_is_connected())
  {
    // Sends "ping" with the event name of "data"
    ble_publish("data", "ping");
  }
}

Inside the if statement, we'll use ble_publish. The first argument is the name of the event and the second is the value.

Next, in order to receive messages from the other end we'll need to setup a callback:

// Configuration for ble stack
ble_stack_init(&init);

// Setup BLE callback
ble_subscribe("data", ble_evt);

We'll define ble_evt at the top of the file:

static void ble_evt(char *name, char *data)
{
  NRF_LOG_INFO("%s: %s", name, data);
}

In this case we'll use NRF_LOG_INFO to print out the message from the hub.

Finally, in order to get the MAC address easily, we'll have to add a call to print it out in setup().

// Print the address
util_print_device_address();

In the end your file should look something like this:

#include "app.h"

timer_define(m_sensor_timer);

// Catch events sent over Bluetooth
static void ble_evt(char *name, char *data)
{
  NRF_LOG_INFO("%s: %s", name, data);
}

static void sensor_timer_evt()
{
  // Check if we're connected
  if (ble_is_connected())
  {
    // Sends "ping" with the event name of "data"
    ble_publish("data", "ping");
  }
}

void setup()
{
  BLE_STACK_PERIPH_DEF(init);

  // Configuration for ble stack
  ble_stack_init(&init);

  // Setup BLE callback
  ble_subscribe("data", ble_evt);

  // Start advertising
  advertising_start();

  // Sensor sensor timer.
  timer_create(&m_sensor_timer, TIMER_REPEATED, sensor_timer_evt);

  // Start
  timer_start(&m_sensor_timer, 1000);

  // Print the address
  util_print_device_address();
}

void loop()
{
}

Next, we'll program it to some hardware!

Flashing the basic sensor code:

For this step you'll need to have a Xenon handy. You'll also need a programmer, programming cable and two Micro-B USB cables. Here's a picture of everything connected:

Once connected and powered run these commands:

make erase
make flash_softdevice
make flash
make debug

Then in a separate terminal window run

make rtt

make debug and make rtt will create a debugging session. You can issue commands in the make debug terminal to control the device as well. For instance, r followed by Enter will restart the device. (By far my most common use case).

If you've flashed everything successfully, your device should start blinking green. That's a good sign!

Additionally, if you take a look at the make rtt side your output should be similar to this:

###RTT Client: Connecting to J-Link RTT Server via localhost:19021 ...
###RTT Client: Connected.

SEGGER J-Link V6.62a - Real time terminal output
J-Link OB-SAM3U128-V2-NordicSemi compiled Jan 21 2020 17:30:48 V1.0, SN=581758669
Process: JLinkExe
<info> app_timer: RTC: initialized.
<info> app: Boot count: 4
<info> app: Pyrinas started.
<info> app: Address: 11:22:33:44:55:66

Take note of the address displayed above. We'll need that for the hub code!

Setting up the hub repository

If you haven't already, clone your hub repository locally. We'll want to do some of the same steps as we did with the sensor repo like:

Setting up the symbolic link
Updating the Makefile
- Setting your PROG_SERIAL
- Setting PROG_PORT to a port not used by the sensor setup. 19020 in this case is fine.
- Setting APP_FILENAME to pyrinas-hub

If you need a reminder how any of these steps work, go back and review the earlier section.

Next, we'll want to open app.c and uncomment the central/hub based code. Plus you'll want to remove the default un-commented code. Your setup() should look something like this:

void setup()
{
  // Default config for central mode
  BLE_STACK_CENTRAL_DEF(init);

  // Add an addresses to scan for
  ble_gap_addr_t first = {
      .addr_type = BLE_GAP_ADDR_TYPE_RANDOM_STATIC,
      .addr = {0x81, 0x64, 0x4C, 0xAD, 0x7D, 0xC0}};
  init.config.devices[0] = first;

  ble_gap_addr_t second = {
      .addr_type = BLE_GAP_ADDR_TYPE_RANDOM_STATIC,
      .addr = {0x7c, 0x84, 0x9d, 0x32, 0x8d, 0xe4}};
  init.config.devices[1] = second;

  // Increment the device_count
  init.config.device_count = 2;

  // Configuration for ble stack
  ble_stack_init(&init);

  // Start scanning.
  scan_start();
}

You'll notice immediately that there are two clients/devices defined here. Let's remove the second one. Should you, in the future, want to connect more devices this is an example of how to do it.

Reminder: also make sure that you change the init.config.device_count to 1.

Then, you'll want to update the .addr field in ble_gap_addr_t first to match the address we got earlier from make rtt:

// Add an addresses to scan for
ble_gap_addr_t first = {
    .addr_type = BLE_GAP_ADDR_TYPE_RANDOM_STATIC,
    .addr = {0x11,0x22,0x33,0x44,0x55,0x66}};
init.config.devices[0] = first;

The address field uses raw bytes, so it has to be represented that way. Remove the : and place 0x in front of each byte. We end up going from 11:22:33:44:55:66 to {0x11,0x22,0x33,0x44,0x55,0x66}

Now before we flash anything, let's also set up the Bluetooth event handler. As with earlier we'll use ble_subscribe to attach an event handler:

// Setup BLE callback
ble_subscribe("data", ble_evt);

Then place the function at the top of the file:

// Catch events sent over Bluetooth
static void ble_evt(char *name, char *data)
{
  NRF_LOG_INFO("%s: %s", name, data);

  ble_publish("data", "pong");
}

You'll notice we're printing out the message using NRF_LOG_INFO. We're also sending a message back to the sensor in the form of ble_publish("data","pong"); In other-words we're playing a game of ping-pong between the two devices!

In the end your code should look something like this:

#include "app.h"

// Catch events sent over Bluetooth
static void ble_evt(char *name, char *data)
{
  NRF_LOG_INFO("%s: %s", name, data);

  ble_publish("data", "pong");
}

void setup()
{
  // Default config for central mode
  BLE_STACK_CENTRAL_DEF(init);

  // Add an addresses to scan for
  ble_gap_addr_t first = {
      .addr_type = BLE_GAP_ADDR_TYPE_RANDOM_STATIC,
      .addr = {0x11, 0x22, 0x33, 0x44, 0x55, 0x66}};
  init.config.devices[0] = first;

  // Increment the device_count
  init.config.device_count = 1;

  // Configuration for ble stack
  ble_stack_init(&init);

  // Setup BLE callback
  ble_subscribe("data", ble_evt);

  // Start scanning.
  scan_start();
}

void loop()
{
}

Reminder: make sure you set ble_gap_addr_t first or the two devices will not connect!

To program, connect the Xenon to program as you did before. We'll flash it using the same methods as before:

make erase
make flash_softdevice
make flash
make debug

Then in a separate terminal window run

make rtt

Then take a look at each of the make rtt screens. There should be some output! For the hub it should look something like this:

Process: JLinkExe
<info> app: Boot count: 4
<info> app: Pyrinas started.
<info> ble_m_central: Connected to handle 0x0
<info> ble_m_central: Protobuf Service discovered
<info> app: data: ping
<info> app: data: ping

And the sensor side like this:

Process: JLinkExe
<info> app_timer: RTC: initialized.
<info> app: Boot count: 4
<info> app: Pyrinas started.
<info> app: Address: 11:22:33:44:55:66
<info> ble_m_periph: Notifications enabled!
<info> app: data: pong
<info> app: data: pong

The pin and pong messages should continue until you stop them. Awesome! If you get any warnings like this one:

Unable to write. Notifications not enabled!

Use the reset button on either of the devices. This should fix the problem.

Side note: the pairing process for Bluetooth is inherently insecure. Once the pairing process is complete though, the devices are secure. (With the caveat that no one was sniffing the pairing process!) There may be improvements on security in the future.

Congrats! 🎉If you've made it this far, you've deployed your first Pyrinas hub and sensor client! For more information about what Pyrinas can do you should checkout the header files under pyrinas-os/include/. Also, Pyrinas can do anything that you'd normally be able to do with Nordic's SDK. Nordic's Infocenter is a great resource for learning more about what the SDK has to offer.

What does the future hold for Pyrinas?

All future tasks for Pyrinas is shared openly on the Github Repo. Here are some of the high level improvements on the roadmap:

Particle Boron + LTE support - That's right! Cellular will be coming to Pyrinas. As of this writing, the first board to support Pyrinas LTE will be Particle's Boron.
MQTT (over TLS) and HTTPS support - Once we have cellular, we need something to communicate with. That's where MQTT and HTTPS come in. They're some of the most popular protocols for IoT today.
Built in remote OTA support - As it stands today, devices programmed with Pyrinas uses Nordic's Secure Bootloader. That means they can be updated over the air by a computer or cellphone nearby. This isn't sustainable for long term deployments though! Instead, you will be able to push updates to Pyrinas devices over the Cloud. Yup. No reason to get off your couch, you can deploy your updates from anywhere.
Dynamic configuration and management - adding and removing devices from a Pyrinas system currently takes some effort. In future revisions, it will be easier to add and remove devices on the fly. This allows for remote device management with zero headaches.
Support for pre-certified modules and other development boards based on Nordic's nRF52840. Currently the Xenon is the only supported board. Development boards aren't great for full production though. Stay tuned for support for pre-certified modules from vendors like Fanstel and more..
Support for more development environments. Currently Pyrinas supports Mac only.

Star and Watch!

This is only the tip of the iceberg! Stay tuned for more updates and make you you star and watch the repository!

Or, better yet, looking to help out? Contributions are welcome!

How to Develop Particle Iot Apps Using NativeScript

Jared Wolff — Sun, 26 Jan 2020 14:32:20 +0000

If you're developing any type of IoT product, inevitably you'll need some type of mobile app. While there are easy ways, they're not for production use.

In this tutorial, we'll talk about the basics of Particle app development. You'll learn about some of the many app frameworks you can take advantage of. Plus there's libraries, tricks and tools along the way to make your life a ton easier.

App Frameworks

Sometimes it's dang near irritating to program multiple applications natively. You see, Swift (or Objective C 🤮) and Java aren't terrible at first glance (well, maybe except for Obj-C 🤮). But when you're resource constrained, you have to figure out a new game plan. That's where App Frameworks come in.

These frameworks allow an app developer to write, build and test cross platform apps. In some cases, the frameworks convert your app into native code. That means that they run as fast and as well as one written in Swift or Java.

I did the research and as of January 2020, here are some of the most supported frameworks:

The list goes on for days.

I've used a few of these frameworks in the past. I've built a Meteor app which (surprisingly) worked. In the end I had to pick one though. What did I go with?

NativeScript.

For the most part, NativeScript's documentation and on-boarding experience is fantastic. Not only can you preview your app inside an emulator but you can load it directly to your phone too!

One of the cool things about NativeScript is that it supports TypeScript. TypeScript is a superset of Javascript with some extra wiz-bang features. Unlike other languages, Javascript technically has no types. If you've done any Particle development you likely know what a type is. We're talking about int, String, float and more. i.e. they're directives to to make sure your Javascript code stays consistent.

NativeScript is also compatible with most major javascript web frameworks. This includes Vue.Js and Angular.

I've only noticed one major drawback thus far. The mobile preview mode (tns preview command) does not pay well with native libraries. If you have some native platform specific libraries, you'll have to use the emulator or a device (if you have one).

If you're gung-ho and you want to build multiple apps in their respective languages the more power to you. There is an advantage over the above frameworks: tried and true Particle SDKs.

Available Libraries & SDKs

Particle has gone out of their way to make app development a little easier. This is thanks to the massive development work that has gone into their own SDKs. Yup, no more are the days you have to write manual HTTP request handlers.

Here's a link to both the iOS and Android SDKs:

iOS
Android

Though we won't be covering them here, they reflect all the potential calls that you can make using the Cloud API.

Speaking of Cloud API, Particle has also developed a Node.js library as well. As you can imagine, you can use this for your server side code or Javascript based app frameworks. Sadly, it doesn't work with NativeScript. Frameworks that use a WebView should be more compatible.

In the case of this tutorial, we'll be mostly focusing on the Cloud API. This way you have a good understanding of the overall system. It may seem intimidating but if you do it right, you'll get the hang of it real fast.

Making API Calls

In NativeScript you can't use libraries like [request](https://github.com/request/request). (Which happens to be the library Particle's very own DMC used in the CLI — DMC if you're reading this, Hi!) You'll have to use the provided HTTP module. If you scroll all the way to the bottom of that page, you'll see a fully fledged POST example. I'll reproduce it here but with some Particle specific changes:

// Create form post data
var data = new FormData();
data.append("name", "update");
data.append("data", "It's hammer time!");
data.append("private", "true");
data.append("access_token", _token);

// Configure the httpModule
return httpModule
    .request({
        url: `https://api.particle.io/v1/devices/events`,
        method: "POST",
        content: data
    })
    .then(
        response => {
            const result = response.content.toJSON();
            console.log(result);
        },
        e => {
            if (e) console.log(e);
        }
    );

The above is an example of what's equivalent to Particle.publish in DeviceOS. Let's break down the parts.

First of all, one of the main gotchas of Particle's Web API is the data format. I has first expected that they use JSON but I was sorely wrong. After actually reading the documentation I realized that most POST requests were actually application/x-www-form-urlencoded. That means when you submit data, it's the equivalent to hitting the submit button on an HTML form.

Fortunately, there is an easy way to assemble form data in Node/Javascript. We can use the FormData() object. Take a look at the above. There should be some familiar parameter names in the data.append calls.

"name" refers to the name of the event your publishing to.

"data" refers to the string formatted data that you're publishing.

"private" dictates whether or not you want to broadcast this data to the whole Particle world, or just your little corner of it.

"access_token" is a token that you can generate in order to make these API calls. Without a token though, you're dead in the water.

Getting a Token

Where do we get this elusive access_token?

At first I had no idea.

I created an OAuth user and secret in the console. That lead to a dead end. Fiddled around with different API calls and settings. Nothing. Then it hit me like a ton of bricks. There's an access_token attached to the curl request on every device page!

Open up any device, click the little console button near Events. A popup with instructions an a URL will pop up. Copy the text after access_token=. That is your access_token! See below:

You can use this token to make calls to the Particle API. This can be to subscribe, publish, write to a function, read variables and more.

Through the command line

That's nice and everything but how the heck can you programmatically generate one? One way is with the command line.

particle token create is the name of the command you need to know about. When you run it, you'll be prompted to login. (Also enter your Authenticator code if you use one) Then the command line will spit out a shiny new access_token you can use with the API!

Through the API itself

If you couldn't guess, particle token create is a frontend to a raw API call. You can make these API calls directly too. Here's what it looks like in NativeScript.

// Create form post data
var data = new FormData();
data.append("username", "jaredwolff");
data.append("password", "this is not my password");
data.append("grant_type", "password");
data.append("client_name", "user");
data.append("client_secret", "client_secret_here");

// Configure the httpModule
return httpModule
    .request({
        url: `https://api.particle.io/v1/oauth/token`,
        method: "POST",
        content: data
    })
    .then(
        response => {
            const result = response.content.toJSON();
            console.log(result);
        },
        e => {
            if (e) console.log(e);
        }
    );

This call may get more complicated. Mostly in the case if you have two factor authorization setup. It's well worth it when you figure it all out. After all, no one wants to manually create auth tokens if they don't have to!

Now you're ready to write and read from your devices. There's one thing though that may trip you up. Subscribing to events can be troublesome with a regular HTTP client. So much so that if you try to do it with NativeScript's HTTP client, it will lock up and never return. Luckily there is a way to handle these special HTTP calls.

Server Sent What?

Server Sent Events (SSE for short) is an HTTP/S subscription functionality. It allows you to connect to a SSE end point and continuously listen for updates. It's a similar web technology to what companies use for push notifications. It does require some extra functionality under the hood though...

SSE Library

After much head scratching and searching I stumbled upon nativescript-sse. It looked simple enough that I could start using immediately. More problems arose when I tried to use it though.

First, it turns out you can't use the library in tns preview mode. The alternative is to use tns run ios --emulator or use tns run ios with your iPhone connected to your computer. The non-emulator command will automatically deliver your prototype app.

Side note: I had already set up my phone in Xcode. You may have to do this yourself before tns run ios is able to find and deploy to your phone.

Secondly, once I got the library working, I noticed I would get some very nasty errors. The errors seemed to happen whenever a new message from Particle came along. Turns out the underlying Swift library for iOS had fixed this last year. So I took it upon myself to figure out how to upgrade the NativeScript plugin. I'll save you the time to say that it can be a pain and there is a learning curve!

Fortunately after some hacking I got something working. More instructions on how to compile the plugin is in the README. Alternatively, you can download a pre-built one on the Release page of the repository.

Download the .tgz file to wherever you like. Then, you can add it using tns plugin add. The full command looks like this:

tns plugin add path/to/plugin/file.tgz

You can check to make sure the library is installed by running tns plugin list

**jaredwolff$ tns plugin list
Dependencies:
┌─────────────────────┬──────────────────────────────────────────────────────────────────────────────────┐
│ Plugin              │ Version                                                                          │
│ @nativescript/theme │ ~2.2.1                                                                           │
│ nativescript-sse    │ file:../../Downloads/nativescript-sse/publish/package/nativescript-sse-4.0.3.tgz │
│ tns-core-modules    │ ~6.3.0                                                                           │
└─────────────────────┴──────────────────────────────────────────────────────────────────────────────────┘
Dev Dependencies:
┌──────────────────────────┬─────────┐
│ Plugin                   │ Version │
│ nativescript-dev-webpack │ ~1.4.0  │
│ typescript               │ ~3.5.3  │
└──────────────────────────┴─────────┘
NOTE:
If you want to check the dependencies of installed plugin use npm view <pluginName> grep dependencies
If you want to check the dev dependencies of installed plugin use npm view <pluginName> grep devDependencies**

Once installed, invoking the library takes a few steps. Here's an example:

import { SSE } from "nativescript-sse";

sse = new SSE(
            "https://api.particle.io/v1/events/blob?access_token=<your access token>",
            {}

// Add event listener
sse.addEventListener("blob");

// Add callback
sse.events.on("onMessage", data=>{
    // TODO: do stuff with your event data here!
    console.log(data);
});

// Connect if not already
sse.connect();

First you need to import and create an instance of the library. When you create the instance, you will have to enter the URL that you want to use. In this case we'll be doing the equivalent of Particle.subscribe(). It should look something similar to the above: https://api.particle.io/v1/events/<your event name>?access_token=<your access token>. Replace <your event name> and <your access token> with the name of your event and your freshly created token!

Then you set up the library to listen for the event you care about. In this case blob is the event I most care about.

Then make sure you configure a callback! That way you can get access to the data when blob does come along. I've made a TODO note where you can access said data.

Finally, you can connect using the .connect() method. If you don't connect, SSE will not open a session and you'll get no data from Particle.

Placement of the code is up to you but from the examples it looks like within the [constructor() of your model is a good place.](https://github.com/jaredwolff/nativescript-sse/blob/master/demo/app/main-view-model.ts)

Other Examples

If you're curious how to use SSE in other places I have another great example: Particle's CLI.

Particle uses the [request](https://github.com/request/request) library to handle SSE events in the app. Whenever you call particle subscribe blob it invokes a getStreamEvent further inside the code. You can check it out here. The request library has more information on streaming here.

More resources

This is but the tip of the iceberg when it comes to connecting with Particle's API. Particle has some great documentation (as always) you can check out. Here are some important links:

Conclusion

In this post we've talked about app frameworks, NativeScript, NativeScript plugins and Server Sent Events. Plus all the Particle related things so you can connect your NativeScript app to Particle's API. I hope you've found this quick tutorial useful. If you have any questions feel free to leave a comment or send me a message. Also be sure to check out my newly released guide. It has content just like this all about Particle's ecosystem.

Until next time!

Better Battery Life for Delightful Particle Deployments

Jared Wolff — Tue, 24 Sep 2019 21:23:57 +0000

IoT devices (almost) always need batteries. With those batteries some important things to think about:

How long your device will last.
How your device will perform in the environment.
How you'll re-charge or replace the batteries when they die

In this post, you'll learn how I optimized every microamp used by a Particle Xenon based motion sensor. You'll learn about the the roadblocks I encountered, the fixes and how get your designs low power too!

Ready?

The Noodling Begins

After noodling around on the project's direction, I decided to keep it simple. Only one sensor. The sensor? PIR.

PIR stands for Passive Infrared Sensor. Ever walk into a building to see a blinking box in the upper corner? That, my friend, is a PIR sensor. They're used for motion detection for security systems and light control.

What if we can evolve this sensor and give it some more capabilities. Say, connect it to the internet? That opens a few more doors.

If you've read any of my other recent articles, you know that i'm a big fan of Particle. Especially what they're doing with their mesh platform. So much so that I'm writing a guide about it. In this project, we'll use the Particle Xenon to make our motion sensor smart. That way you can have instantaneous motion alerts anywhere.

The final design of the motion sensor board uses a TI TLV8544PWR. It's an ultra-ultra low-power rail-to-rail op amp. Its primary purpose is signal processing of the PIR sensor. In a typical case, it draws about 1.2µA (!!) when all the op-amps are active.

The design also sports a PCF85063ATL/1 which is a standalone low-power real time clock. This device, along with the PIR, can wake up the Particle Mesh device from sleep. For the purposes of this project, it's used to limit the message count when a new PIR event occurs. i.e. a PIR interrupt is only allowed every 2 minutes.

There are other various parts in the mix but the above block diagram is this board at it's essence.

Assembling

The main problem came when I had to actually assemble the device. I was abroad and had zero tools. Luckily there was a FabLab nearby that allowed me to use their tools. (And boy, was I grateful)

Here's a few pictures from the assembly process.

I also had no jig to hold the boards. So, a quick sketch in Fusion360 and some time on the laser cutter yielded some fantastic results!

This board also happened to be the first that I used a metal stencil on. Oh man was it a dream. It's much more consistent compared to Kapton based stencils.

Here's what the boards look like after assembly. I had some issues with bridging around the op-amp but that was easily addressed with some flux and a hot iron.

Here's the front of the design. Nothing but a PIR!

If you do ever find yourself assembling a board like this one here are a few tips:

Most PIR sensors are extremely sensitive to heat. That means you must be careful when applying your soldering iron. The data sheet states that the temperature must be 350°C and you shouldn't apply the tip for more than 3 seconds.
In this design there are some small parts. Normally you can use a microscope to assemble. If you're out of town, like I was when I assembled these boards, you can use your cell phone. Use the camera's pinch-zoom feature to check part orientations and shorts between pins.
In this design, there's a ton of bulk capacitance on the input to the PIR sensor. This takes a while to charge especially if you're using a large 100µf capacitor. In the next design, the motion board will enable monitoring of this voltage. That way the Particle Mesh device will know when the PIR is stable.

Now that we have some hardware, it's time to figure out how to get it all working and as low power as possible. We'll talk about some of the ways we can do that in the next section.

Sleepy Time

Sleep mode allows you to place your device in a low power state. It is often used when your Particle Mesh is idle. The primary goal is to conserve as much battery power as possible.

There are two sleep modes with a few distinct differences. We'll discuss them below:

Stop mode sleep

"Stop mode" is the standard sleep for any Particle Mesh device. It shuts down the network and, when configured, can wake up from an interrupt. Once the device wakes up, it will continue execution where it left off.

You can trigger sleep mode in several different ways. The first way is by setting up a pin interrupt:

    System.sleep(D2, RISING);

This will put the device into stop mode and wait for a rising interrupt on D2. You can connect a sensor IC that toggles D2 when it's done with a sensor reading. Not only will that alert the device a reading is available, but it will also wake the device from sleep!

The Particle Mesh device can use a wakeup timeout when you specify a third argument. Here's an example:

    System.sleep(D2, RISING, 60);

The third argument represents the time, in seconds, the device will remain in sleep. If no pin interrupt occurs, the device will wake back up after that interval. In this case, the device is set to wake up after 60 seconds if no event occurs on D2.

Deep sleep

There is another lower power option:

deep sleep.

Like stop mode, deep sleep mode will disconnect your device, and not retain the RAM. When the device wakes up, start execution from the beginning of your application. Any variables in memory will get reset. Data in non-volatile memory, like EEPROM, is not affected.

It also has a 'wakeup pin' but unlike stop mode, you can't choose the pin. (The default is D8)

You can enter deep sleep by running executing the following function.

    System.sleep(SLEEP_MODE_DEEP);

Measurements

So, what's the actual difference between all these modes? Here's a table of all the Particle Mesh devices and the two modes discussed above.

Li @ 3.4V (mA)	Deep Sleep (uA)	Stop Mode (uA)
Xenon	834	846
Boron	1387	1978
Argon	846	1333

Side note: these measurements are from Agilent 6611C power supply set to 3.4V. The 6611C has a low current measurement mode. In other words, it can measure the current without an ammeter in series.

The results? Not particularly impressive. 0.834mA in deep sleep mode will last about 11.24 days on a 225mAh battery cell. There had to be a better way.

Knowing that the NRF52 Series sips current in sleep mode, the solution was somewhere else. So after perusing the schematic, the culprit became clear. It was all the power management circuitry (highlighted below)! 😬

There is a way around this though. The solution?

Bypass it all.

Bypass Bypass Bypass

The main risk of bypassing was the potential for back powering those circuits. This could lead to unintended operation or even high quiescent currents. In my case though, there was none of that. I saw a noticeable improvement.

When bypassing, the Xenon went from 846µA to 46µA. Thats a factor of 17x! With the motion sense board, it was about 49µA. That means with a 225mAh battery this setup would last about 0.52 years.

Bypassing though came with some extra concerns. Like, "will the board support the voltage of a coin cell which varies from 3.3V down to 2.6V?" It was time to do some research.

Research

So, we're bypassing all the regulators. That means that 3.3V can, over time, turn into 2.6V. Can all the circuitry on the Particle Mesh board handle 3.3V down to 2.6V?

Let's look at the schematic to see what the devices of concern are.

The circled devices are the ones we care about.

This gives us a list of

NRF52840
SKY13351-378LF
MX25L1606E

Now, we can then go and research each of them to understand the power range they can operate in. In this case every device must work at 3.3V and down to 2.6V.

So, where's the best source of how these parts operate? The data sheets.

Here's the one for the flash part:

For Nordic's NRF52840,

And for the antenna switch:

The search took a few minutes and 3 round trips to DuckDuckGo but we've found the information we've needed. All devices can operate in the range we're looking for. (With a slight exception of the flash chip. In this example, it's not used so we can safely ignore it)

So, with that hurdle out of the way, it was time to write some firmware.

Make it Smart

The last step in the process was to get the firmware as optimized as possible. i.e. to keep the device in sleep mode most of the time.

For this first round of testing stop sleep mode was used*.*

    // Sleep differently depending on the situtation
    if( !pirReady ) {
      System.sleep(D2, FALLING, 2000);
    } else if( rtcReadyForMotion ) {
      System.sleep({A5, D2}, {CHANGE, FALLING});
    } else {
      System.sleep(D2, FALLING);
    }

Furthermore, the state of the device, dictated how sleep mode worked. For instance, the device would ignore PIR input if it wasn't ready for motion interrupts. This would prevent the device from waking up until it was time to.

Additionally, the idea was to use Particle.publish() to the cloud. Unfortunately, early complications made that tough. Sometimes the device would connect immediately. Sometimes it would sit there and do nothing. In the end, I changed the Particle.publish() to Mesh.publish().

This, in turn, required another device in the mesh network to subscribe to the message and forward it on to the cloud. An Argon running as an edge router is great for this purpose.

Interestingly, I did notice that when Mesh.connect() is called, Mesh.publish() would work some of the time. I added some delay after Mesh.connect() and the setup has been rock solid ever since. The code looks something like this now:

    // If A5 is high then we have an event.
    if (pirEventOccurred && pirReady) {

      // Reset variable
      pirEventOccurred = false;
      publishInProgress = true;

      // Start connection
      // TODO: fix this. This is a hack. I'm getting unreliable messages *without a delay*. :|
      Mesh.connect();

      // Blink RED led
      digitalWrite(D3,LOW);
      delay(250);
      digitalWrite(D3,HIGH);

      Log.trace("event occurred");
    }

Side note: System.sleep causes the Particle Board to disconnect. So, you need to reconnect to the mesh network when you're ready to publish.

Also, in the sprit of designing low power firmware, using the RGB LED was out. Especially since you can't see when it's inside the enclosure! You can disable it by turning on manual control:

    // Turn off RGB Led
    RGB.control(true);

Once the firmware seemed to be in a good place, it was time to put everything together. Unfortunately the sensor + Particle Mesh combination wasn't ready to cooperate just yet.

Mysterious Restarts

After some testing, I noticed that the device stopped working. I removed it from the plastics and found the RGB LED was blinking white. My best guess? The processor was continuously restarting.

So after some head scratching, I attached the battery to the oscilloscope. I watched as the board started up and attempted to connect to the mesh network. As it did, the battery voltage would promptly drop. This is typical for any type of battery based design.The unexpected part? The Xenon would do a full hardware reset whenever the battery voltage would reach below 2.8V.

What caused these resets? Was it another part? (Like the flash part unable to communicate?) Was it the NRF52?

After searching the answer in data sheets and schematics, it turned out to be the NRF52. More specifically, the power supply supervisor.

In the DeviceOS code, the power supply supervisor is set to restart the NRF52 if it goes below 2.8V. The exact call inside hw_config.c under platform/mcu/nRF52840/src.

And Bingo was his name-o.

The solution was to use the STARTUP macro and run sd_power_pof_threshold_set. Then using one of the many options to set the new threshold:

    /** @brief Power failure comparator thresholds. */
    typedef enum
    {
        NRF_POWER_POFTHR_V21 = POWER_POFCON_THRESHOLD_V21, /**< Set threshold to 2.1&nbsp;V */
        NRF_POWER_POFTHR_V23 = POWER_POFCON_THRESHOLD_V23, /**< Set threshold to 2.3&nbsp;V */
        NRF_POWER_POFTHR_V25 = POWER_POFCON_THRESHOLD_V25, /**< Set threshold to 2.5&nbsp;V */
        NRF_POWER_POFTHR_V27 = POWER_POFCON_THRESHOLD_V27, /**< Set threshold to 2.7&nbsp;V */
    #if defined(POWER_POFCON_THRESHOLD_V17) || defined(__NRFX_DOXYGEN__)
        NRF_POWER_POFTHR_V17 = POWER_POFCON_THRESHOLD_V17, /**< Set threshold to 1.7&nbsp;V */
        NRF_POWER_POFTHR_V18 = POWER_POFCON_THRESHOLD_V18, /**< Set threshold to 1.8&nbsp;V */
        NRF_POWER_POFTHR_V19 = POWER_POFCON_THRESHOLD_V19, /**< Set threshold to 1.9&nbsp;V */
        NRF_POWER_POFTHR_V20 = POWER_POFCON_THRESHOLD_V20, /**< Set threshold to 2.0&nbsp;V */
        NRF_POWER_POFTHR_V22 = POWER_POFCON_THRESHOLD_V22, /**< Set threshold to 2.2&nbsp;V */
        NRF_POWER_POFTHR_V24 = POWER_POFCON_THRESHOLD_V24, /**< Set threshold to 2.4&nbsp;V */
        NRF_POWER_POFTHR_V26 = POWER_POFCON_THRESHOLD_V26, /**< Set threshold to 2.6&nbsp;V */
        NRF_POWER_POFTHR_V28 = POWER_POFCON_THRESHOLD_V28, /**< Set threshold to 2.8&nbsp;V */
    #endif // defined(POWER_POFCON_THRESHOLD_V17) || defined(__NRFX_DOXYGEN__)
    } nrf_power_pof_thr_t;

In this case I set it to NRF_POWER_THRESHOLD_V20

 STARTUP(sd_power_pof_threshold_set(NRF_POWER_THRESHOLD_V20));

It was only after implementing this fix did the full device operate longer than a couple minutes! Now that it's working it's time to pack everything away into a polished enclosure. We'll talk about that step in the next section.

The Enclosure

Instead of sticking raw circuit boards to the wall, the first step was to find or create an enclosure. Unfortunately, I'm no product designer. Designing an enclosure takes a chunk of time that I didn't quite have. So what's the next best bet?

Using a pre-existing enclosure.

Digikey is full of enclosures of all shapes and sizes. Luckily I had a few extra one in my box-of-stuff that seemed to fit the bill.

The box is a Bud Industries CU-1941. It's 83.01mm x 54.00mm with a height of 30.51mm. Which is the perfect size to hold a circuit board plus a Particle Mesh board mushed together.

It also happens to be the same enclosure used to house a modified version of the uCurrent Gold. Here's a picture of the one I built in the not to distant past:

I contacted Bud and asked them for the STEP files. They obliged and while developing the electronics, I was able to mock up the design in Fusion360.

Pro tip: Whenever you're using an off-the-shelf design, you need to do your due diligence. That means checking for fit in all directions. My main concern was making sure that the mated height of the boards was small enough. That way they could fit into the height of the enclosure.

Here's a sectional analysis of the highest part of the motion sense board:

And it fit like a glove! (with some extra space to boot) I could even stick a much much larger battery in there and still have room. Here's what it looks like when everything is mated:

This process takes some time but it's well worth it. It's especially gratifying when you have assembled circuitboards that slide into place. Despite how well everything came together there's always improvements for the next version.

The Finished Product?

Despite some miscellaneous mechanical tweaks (drilling out holes a little big larger, etc), the project came together. Plus, it looks like something you'd by from some spy shop on the internet to boot!

Despite how polished it looks there's one glaring factor.

Power

The device draws 49µA. Replacing a battery every half year though is rough on the wallet and the landfill.

How do we get that current even lower?

As you know by now, the Xenon board takes up 94% of the sleep power. So, why not take it completely out of the picture?

Removing the processor from the mix is as simple as removing the power from the device while in sleep mode. We can do that by gating the power to the Particle Mesh board through the 3.3V pin.

This turns our 49µA of sleep current into 3µA. Thats a difference of 0.5 year and 8 years of standby current!

Conclusion

Creating low power hardware to save battery life can be frustrating at times. As long as you stay creative, you can eke every hour of battery life out of your deployed devices.

Want to learn more? I have more content like this coming out soon. Plus, if you haven't already, you can sign up for updates on my upcoming Ultimate Guide to Particle Mesh. I'm sharing some insider content and list only exclusives you don't want to miss out on!

Additionally, learn how to send push notifications to your phone using this previous tutorial.

Want one of these motion sensor boards?

Join the waiting list here and you'll be the first to know when they're available on my site. This particular version will only work with Xenons.

Finally, thanks for reading this post. It's always great to hear from you guys about what projects you're building. It's and inspiration and keeps me going.

Until next time!

Resources

How to Presence Tracking Using Particle Argon

Jared Wolff — Sun, 08 Sep 2019 13:15:15 +0000

Ever want to add presence or location tracking to a project? Frustrated by the solutions (or lack thereof)?

Do not worry, you're not the only one!

In this post you'll learn how to implement a very basic tracking and notification application. We'll be using a Particle Argon and a Tile Mate.

By the end you'll be able to tell when the Tile is present or not. Plus we'll use Pushover to send push notifications to the devices of your choosing.

Let's get going!

Note before we get started, this post is lengthy. You can download the PDF version so you can download it and view it later.

Initial investigation

The idea of using a Tile wasn't obvious at first glance. Ideally, using a phone seemed to make more sense. Unfortunately, this wasn't as a viable option. It would require some more research and the creation of a Bluetooth iOS app.

So, the idea of using a phone was out.

Then I thought, "What devices do advertise all the time?"

That is what led me down the path of a tracker like Tile.

After it arrived there was some customary testing. First stop, the Tile application.

I was able to connect and use the device. I even made it play a catchy tune. 🎶

Then, I moved on to using one of the Bluetooth scanner apps. I scrolled through all the results and Bingo. There was the Tile!

I even waited a few hours and checked it again. I wanted to make sure it didn't go to sleep after a while. Turns out, it's always advertising. As far as I can tell, about every 8 seconds.

All of this testing lead to one conclusion: it could be easily used for presence detection.

The next step in the process was trying to figure out how to get it working with an Argon.

Advertising

As we had gathered in the previous step, we know that the Tile is advertising about every 8 seconds. That means it should be easily scanned for using any device including an Argon, Zenon or Boron.

For this example I suggest you use an Argon. This is because Bluetooth and Mesh share the same radio. When scanning for the Tile, the Xenon connected to Mesh would often miss the advertising packets. This would lead to false negatives (and frustration!).

Along the same lines, you'll want to make sure your Argon is connected to no mesh network. You can remove it using the CLI. Connect your device to your computer and run the following command:

particle mesh remove <device name/ID>

Make sure that you replace with your device's name or ID.

Alright, back to the good stuff.

Advertising can have a few different purposes in Bluetooth. Typically though, it marks the beginning of the pairing phase. That way other devices know that the advertising device is available.

Additionally, the advertising device will indicate what services it has. We can use this knowledge to filter out devices that don't match.

For example, here's a screenshot of the services available on the Tile device:

When scanning we'll double check that the device we're connecting to has the service UUID of 0xfeed.

Before we get deep into Bluetooth land though, let's set up our app for debugging using the Logger.

Logging

In this tutorial we'll be using the Logger. It allows you to display log messages from your app using particle serial monitor.

One of the cooler features about the logger is the idea of message hierarchy. This allows you, the designer, to selectively mute messages that may not be necessary.

For example, if you have messages used for debugging. You could remove them or comment them out. Or, you could increase the LOG_LEVEL so they're effectively ignored.

Here are the logging levels which are available in logging.h in Particle's device-os repository:

// Log level. Ensure log_level_name() is updated for newly added levels
typedef enum LogLevel {
    LOG_LEVEL_ALL = 1, // Log all messages
    LOG_LEVEL_TRACE = 1,
    LOG_LEVEL_INFO = 30,
    LOG_LEVEL_WARN = 40,
    LOG_LEVEL_ERROR = 50,
    LOG_LEVEL_PANIC = 60,
    LOG_LEVEL_NONE = 70, // Do not log any messages
    // Compatibility levels
    DEFAULT_LEVEL = 0,
    ALL_LEVEL = LOG_LEVEL_ALL,
    TRACE_LEVEL = LOG_LEVEL_TRACE,
    LOG_LEVEL = LOG_LEVEL_TRACE, // Deprecated
    DEBUG_LEVEL = LOG_LEVEL_TRACE, // Deprecated
    INFO_LEVEL = LOG_LEVEL_INFO,
    WARN_LEVEL = LOG_LEVEL_WARN,
    ERROR_LEVEL = LOG_LEVEL_ERROR,
    PANIC_LEVEL = LOG_LEVEL_PANIC,
    NO_LOG_LEVEL = LOG_LEVEL_NONE
} LogLevel;

Cool, log levels. But how do we use them?

We can use them by invoking one of these functions:

Log.trace, Log.info, Log.warn, Log.error.

For example:

Log.trace("This is a TRACE message.");

If we set the log level to LOG_LEVEL_INFO we'll only see messages from Log.info, Log.warn, and Log.error. LOG_LEVEL_WARN? Only Log.warn and Log.error will show up. (Hopefully you get the idea.)

To set it up, we'll set the default level to LOG_LEVEL_ERROR. We'll also set the app specific LOG_LEVEL to LOG_LEVEL_TRACE. The end result should look something like this

// For logging
SerialLogHandler logHandler(115200, LOG_LEVEL_ERROR, {
    { "app", LOG_LEVEL_TRACE }, // enable all app messages
});

This way we don't get spammed with DeviceOS log messages. Plus, we get all the applicable messages from the app itself.

By the way, if you want to set your device to a single LOG_LEVEL you can set it up like this:

SerialLogHandler logHandler(LOG_LEVEL_INFO);

As you continue your journey using Particle's DeviceOS you'll soon realize how handy it can be. Now, let's move on to the good stuff!

Setting it up

First, we'll want to make sure we're using the correct version of DeviceOS. Any version after 1.3 will have Bluetooth. You can get the instructions here.

Next we'll want to start scanning for the Tile. We'll want do do this in the loop() function at a specified interval. We'll use a millis() timer in this case:

// Scan for devices
if( (millis() > lastSeen + TILE_RE_CHECK_MS) ){
    BLE.scan(scanResultCallback, NULL);
}

Make sure you define lastSeen at the top of the file like so:

system_tick_t lastSeen = 0;

We'll use it to track the last time the Tile has been "seen". i.e. when the last time the Argon saw an advertising packet from the Tile.

TILE_RE_CHECK_MS can be defined as

#define TILE_RE_CHECK_MS 7500

This way we're checking, at the very minimum, every 7.5 seconds for advertising packets.

In order to find the Tile device we'll use BLE.scan. When we call it, It will start the scanning process. As devices are found scanResultCallback will fire.

For now, we can define scanResultCallback at the top of the file:

void scanResultCallback(const BleScanResult *scanResult, void *context) {
}

You notice that it includes a BleScanResult. This will contain the address, RSSI and device name (if available) and available service information. This will come in handy later when we're looking for our Tile device!

Remember, that BLE.scan does not return until scanning has been completed. The default timeout for scanning is 5 seconds. You can change that value using BLE.setScanTimeout(). setScanTimeout takes units in 10ms increments. So, for a 500ms timeout would require a value of 50.

For the case of this app, I'd recommend using a value of 8s (8000ms). You can set it like this:

BLE.setScanTimeout(800);

In this case, the device will scan for as long as it takes the Tile to advertise. That way it's less likely to miss an advertising packet.

Handling Scan Results

Now that we have scanResultCallback lets define what's going on inside.

We first want to get the service information inside the advertising data. The best way is to use scanResult->advertisingData.serviceUUID. We'll pass in an array of UUIDs what will be copied for our use.

BleUuid uuids[4];
int uuidsAvail = scanResult->advertisingData.serviceUUID(uuids,sizeof(uuids)/sizeof(BleUuid));

This will populate uuids that way you can iterate over them. uuidsAvail will equal the amount of available UUIDs.

On our case we're looking for a particular UUID. We'll define it a the top of the file:

#define TILE_UUID 0xfeed

Normally UUIDs are much longer. A short UUID like this means it has been reserved or is part of the Bluetooth specification. In either case we'll be checking for it in the same way we would check a 32bit or 128bit version.

For diagnostic reasons we can also print out the device information. In this case the RSSI and the device MAC address is handy:

// Print out mac info
BleAddress addr = scanResult->address;
Log.trace("MAC: %02X:%02X:%02X:%02X:%02X:%02X", addr[0], addr[1], addr[2], addr[3], addr[4], addr[5]);
Log.trace("RSSI: %dBm", scanResult->rssi);

Finally let's set up a loop to see if the device found has the UUID:

// Loop over all available UUIDs
// For tile devices there should only be one
for(int i = 0; i < uuidsAvail; i++){

    // Print out the UUID we're looking for
    if( uuids[i].shorted() == TILE_UUID ) {
        Log.trace("UUID: %x", uuids[i].shorted());

        // Stop scanning
        BLE.stopScanning();

        return;
    }
}

We can easily compare the "shorted" version of the UUID with TILE_UUID. It's a simple integer so no complicated memory compare operations are necessary. So, using if( uuids[i].shorted() == TILE_UUID ) works just fine.

You can also use Log.trace to print out diagnostic information. In this case we're using it to print out the shorted() version of the UUID.

Test It!

Let's test what we have so far!

Program the app to your Argon. Open the terminal and run particle serial monitor to view the debug messages. Heres an example of what you may see:

0000005825 [app] TRACE: MAC: 65:C7:B3:AF:73:5C
0000005827 [app] TRACE: RSSI: -37Bm
0000005954 [app] TRACE: MAC: B3:D9:F1:F0:5D:7E
0000005955 [app] TRACE: RSSI: -62Bm
0000006069 [app] TRACE: MAC: C5:F0:74:3D:13:77
0000006071 [app] TRACE: RSSI: -62Bm
0000006217 [app] TRACE: MAC: 65:C7:B3:AF:73:5C
0000006219 [app] TRACE: RSSI: -39Bm
0000006224 [app] TRACE: MAC: B3:D9:F1:F0:5D:7E
0000006225 [app] TRACE: RSSI: -62Bm
0000006296 [app] TRACE: MAC: D7:E7:FE:0C:A5:C0
0000006298 [app] TRACE: RSSI: -60Bm
0000006299 [app] TRACE: UUID: feed

Notice how the message includes TRACE and also [app]? That means it's a trace message originating from the application code. Handy right?

This code does get spammy quick, especially if you're in an environment with lots of advertising Bluetooth devices. If you're Tile is on and running eventually you'll see a message UUID: feed. That means your Argon found the Tile!

Next we'll use the onboard Mode button to "program" the Tile's address to memory. That way we can filter out all the devices we don't care about.

Add Device On Button Push

First we need to figure out how to monitor the Mode button. The best bet, according to the documentation is to use System.on.

System.on(button_click, eventHandler);

The first argument is the name of the system event. In our case it's button_click. The second argument is an event handler function. We'll call it eventHandler for now.

Now let's create eventHandler

void eventHandler(system_event_t event, int duration, void* )
{

}

Important: you can't use the Log function inside eventHandler. An easy way to test it is to toggle the LED on D7. Let's set it up!

Initialize the LED in setup()

// Set LED pin
pinMode(D7,OUTPUT);

Then we can add this inside eventHandler

if( event == button_click ) {
    if( digitalRead(D7) ) {
        digitalWrite(D7,LOW);
    } else {
        digitalWrite(D7,HIGH);
    }
}

We can then write to D7 (the onboard blue LED). We can even use digitalRead to read what the state of the LED is. It will respond with HIGH or LOW depending on the situation.

Load the firmware onto the device and we'll have nice control over the blue LED!

In the next section, we'll use the Mode button to put the device into a "learning" mode. This will allow us to do a one touch setup with the target Tile device.

Storing Address to EEPROM

In this next step we'll store the address of the Tile into EEPROM. That way when the device is restarted or loses power we'll still be able to identify the Tile later on.

There is one lingering question though. How do we get it to save the address in the first place?

By monitoring the button press, we can put the device into a "learning" mode. The device will scan for a Tile, and save the address if it finds one.

First let's add a conditional within if( uuids[i].shorted() == TILE_UUID ):

// If we're in learning mode. Save to EEPROM
if( isLearningModeOn() ) {
    searchAddress = scanResult->address;
    EEPROM.put(TILE_EEPROM_ADDRESS, searchAddress);
    setLearningModeOff();
}

We'll use the status of D7 as a way of knowing we're in "learning mode". We do this by reading D7 using digitalRead(D7). Let's create a function that makes this more clear:

bool isLearningModeOn() {
    return (digitalRead(D7) == HIGH);
}

We can also replace the digitalWrite(D7,LOW); and digitalWrite(D7,HIGH); with similar functions. That way it's more straight forward what we're doing.

// Set "Learning mode" on
void setLearningModeOn() {
    digitalWrite(D7,HIGH);
}

// Set "Learning mode" off
void setLearningModeOff() {
    digitalWrite(D7,LOW);
}

Then, we assign a global variable searchAddress as the scan result. We setup searchAddress like this at the top of the file:

BleAddress searchAddress;

Next we want to save it to non-volatile memory using EEPROM.put. TILE_EEPROM_ADDRESS is defined as 0xa. You can define TILE_EEPROM_ADDRESS to use whatever memory address tickles your fancy. Here's the full definition placed at the top of the file.

#define TILE_EEPROM_ADDRESS 0xa

Finally, we turn off the LED and "learning mode" using setLearningModeOff()

Every time a device is found we'll use millis() to set lastSeen. Additionally, we can track the last RSSI using lastRSSI. It's a cheap way to to know approximately how close the device is. We'll use scanResult->rssi to get this information and set it to the lastRSSI variable.

Overall, your changes should look something like this:

...

// Print out the UUID we're looking for
if( uuids[i].shorted() == TILE_UUID ) {
    Log.trace("UUID: %x", uuids[i].shorted());

    // If we're in learning mode. Save to EEprom
    if( isLearningModeOn() ) {
        searchAddress = scanResult->address;
        EEPROM.put(TILE_EEPROM_ADDRESS, searchAddress);
        setLearningModeOff();
    }

    // Save info
    lastSeen = millis();
    lastRSSI = scanResult->rssi;

    // Stop scanning
    BLE.stopScanning();

    return;
}

Before this function, we can filter out devices that don't match our searchAddress. Add the following before if( uuids[i].shorted() == TILE_UUID ):

// If device address doesn't match or we're not in "learning mode"
if( !(searchAddress == scanResult->address) && !isLearningModeOn() ) {
    return;
}

This will skip over devices that don't match. It will only proceed if the address matches or we're in "learning mode".

Now, in order for us to load searchAddress on startup, we'll have to load it from flash. Add this line to your setup():

EEPROM.get(TILE_EEPROM_ADDRESS, searchAddress);

Then, check to make sure the address is valid. It won't be valid if all the bytes are 0xFF:

// Warning about address
if( searchAddress == BleAddress("ff:ff:ff:ff:ff:ff") ) {
    Log.warn("Place this board into learning mode");
    Log.warn("and keep your Tile near by.");
}

We should be able to "teach" our Argon the address of our Tile. Let's test it out!

Test it.

Now if we compile and run the app, notice how there's no more log output? We have to "teach" the Tile address to the Particle Device. So, hit the mode button. The blue LED should turn on.

Once your Tile has been found the LED will turn off and you'll see some output on the command line. Similar to what we've seen before:

0000006296 [app] TRACE: MAC: D7:E7:FE:0C:A5:C0
0000006298 [app] TRACE: RSSI: -60Bm
0000006299 [app] TRACE: UUID: feed

The device has been committed to memory!

You can also check if it's still saved after a reset. Hit the reset button and check for the same output as above. If it's showing up, we're still good!

Update the Cloud

Finally let's set up a function called checkTileStateChanged. We'll use it to check for changes to the state of the Tile on a regular interval.

bool checkTileStateChanged( TilePresenceType *presence ) {

}

The main purpose of this function is to compare the lastSeen variable with the "timeout" duration. In our case, our timeout duration is TILE_NOT_HERE_MS which should be set to

#define TILE_NOT_HERE_MS 30000

near the top of your program. There's also two more conditions to look for. One where lastSeen is equal to 0. This is usually because the app hasn't found the Tile yet after startup.

The last case would be if the device has been seen and lastSeen is not 0. So within checkTileStateChanged let's put everything together.

// Check to see if it's here.
if( millis() > lastSeen+TILE_NOT_HERE_MS ) {

} else if ( lastSeen == 0 ) {

} else {

}

return false;

Now we only want this function to return true if the state has changed. So we'll need to take advantage of the TilePresenceType pointer in the agreement.

TilePresenceType is simply an enumeration of all the possible states. You can stick it at the top of your file as well. Here it is:

typedef enum {
    PresenceUnknown,
    Here,
    NotHere
} TilePresenceType;

You'll also need a global variable that we can pass to the function. Set this at the top of your file as well:

// Default status
TilePresenceType present = PresenceUnknown;

Now, we can compare at each stage. Does it meet the criteria? Is the state different than the last one? If so, return true.

Remember, we'll want to set presence to the new updated value. So each condition should update the presence value. For example:

*presence = NotHere;

Here's what the fully flushed out function looks like:

bool checkTileStateChanged( TilePresenceType *presence ) {

    // Check to see if it's here.
    if( millis() > lastSeen+TILE_NOT_HERE_MS ) {
        if( *presence != NotHere ) {
            *presence = NotHere;
            Log.trace("not here!");
            return true;
        }
    // Case if we've just started up
    } else if ( lastSeen == 0 ) {
        if( *presence != PresenceUnknown ) {
            *presence = PresenceUnknown;
            Log.trace("unknown!");
            return true;
        }
    // Case if lastSeen is < TILE_NOT_HERE_MS
    } else {
        if( *presence != Here ) {
            *presence = Here;
            Log.trace("here!");
            return true;
        }
    }

    return false;
}

We can now use this function in the main loop underneath the timer to start Ble.scan(). We can use it to send a JSON payload. In this case we'll include important information like the Bluetooth Address, lastSeen data, lastRSSI data and a message.

// If we have a change
if( checkTileStateChanged(&present) ) {

}

We'll use an array of char to get our address in a string format. You can chain together toString() with toCharArray to get what we need.

// Get the address string
char address[18];
searchAddress.toString().toCharArray(address,sizeof(address));

An example payload string could look something like this:

// Create payload
status = String::format("{\"address\":\"%s\",\"lastSeen\":%d,\"lastRSSI\":%i,\"status\":\"%s\"}",
    address, lastSeen, lastRSSI, messages[present]);

status is simply a String defined at the top of the file:

// The payload going to the cloud
String status;

You notice that there's also a variable called messages. This is a static const array of strings. They're mapped to the values from the TilePresenceType. Here's what it looks like

const char * messages[] {
    "unknown",
    "here",
    "not here"
};

That way PresenceUnknown matches to "unknown", Here matches to "here", etc. It's a cheap easy way to associate a string with an enum.

Finally we'll publish and process. This allows us to send the update immediately.

// Publish the RSSI and Device Info
Particle.publish("status", status, PRIVATE, WITH_ACK);

// Process the publish event immediately
Particle.process();

The overall function should look something like this in the end:

// If we have a change
if( checkTileStateChanged(&present) ) {

    // Get the address string
    char address[18];
    searchAddress.toString().toCharArray(address,sizeof(address));

    // Create payload
    status = String::format("{\"address\":\"%s\",\"lastSeen\":%d,\"lastRSSI\":%i,\"status\":\"%s\"}",
        address, lastSeen, lastRSSI, messages[present]);

    // Publish the RSSI and Device Info
    Particle.publish("status", status, PRIVATE, WITH_ACK);

    // Process the publish event immediately
    Particle.process();

}

Now, let's test it!

Testing it!

We can test to make sure our Publish events are occurring without event leaving Particle Workbench. Open a new terminal by going to View → Terminal. Then use the following command:

particle subscribe --device <device_name> <event_name>

Replace <device_name> with the name or ID of your device.

Replace <event_name> with the name of the event. In our case it's status.

You can then test it all by removing the battery and waiting for the "not here" alert. Plug the battery back in and you should get a "here" alert.

Here's an example of the output

> particle subscribe --device hamster_turkey status

Subscribing to "status" from hamster_turkey's stream

Listening to: /v1/devices/hamster_turkey/events/status

{"name":"status","data":"{\"address\":\"C0:A5:0C:FE:E7:D7\",\"lastSeen\":40154002,\"lastRSSI\":-82,\"status\":\"not here\"}","ttl":60,"published_at":"2019-09-07T02:29:42.232Z","coreid":"e00fce68d36c42ef433428eb"}

{"name":"status","data":"{\"address\":\"C0:A5:0C:FE:E7:D7\",\"lastSeen\":40193547,\"lastRSSI\":-83,\"status\":\"here\"}","ttl":60,"published_at":"2019-09-07T02:29:50.352Z","coreid":"e00fce68d36c42ef433428eb"}

Configuring Webhook

In the last part of this tutorial we'll set up push notifications using a webhook. As mentioned before, we'll use Pushover and their handy API to send push notification(s) to the device(s) of your choice.

Pushover has a fantastically easy API to use. Their application is a Swiss army knife for situations where you don't want to code an app to send push notifications.

The first thing that you'll have to take note is your user key. You can get that by logging into Pushover. Note: you'll need to set up an account first if you haven't already.

It should look something like this:

If you're logged in and don't see this page, click on the Pushover logo and that should bring you back.

Next we'll want to create an application. Click on the Apps & Plugins at the top of the screen.

You should then click Create a New Application. This will allow us to get an API Token that will be needed in the Particle Webhook setup.

Set a name as you see fit. Fill in the description if you want a reminder. Click the box and then click Create Application.

You should go to the next page. Copy and save the API Token/Key we'll need this also in a few steps.

Now, let's setup the Webhook. Jump over to https://console.particle.io and create a new integration.

We'll set the Event Name to status.

The URL to https://api.pushover.net/1/messages.json

Also, if you want to filter by a specific device make sure you select it in the Device dropdown.

Under Advanced Settings we'll finish up by setting a few fields.

Create the following fields: token, user, title, and message. Then set token to the API Token we got earlier. Do the same for the User Key.

The title will show up as the title of your message. Make it whatever makes sense for you.

You can set the message as The Tile is currently {{{status}}}. RSSI: {{{lastRSSI}}}.

We are using mustache templates here. They allow you to use the data in the published payload and reformat it to your liking. In our case, we're using them to "fill in the blanks." The message once processed would look something like this:

The Tile is currently here. RSSI: -77

As a side note, i'll be talking more about these templates in my guide. So stay tuned for that!

Test it

Once your integration is in place, you can test doing what we did in the earlier step. Remove the battery, and wait for the "not here" message. Put it back and wait for the "here" message.

Here's what it would look like on an iPhone:

As you can see, I tested it a bunch! 😬

If you've made it this far and everything is working, great work. You now have a Tile tracker for your house, office or wherever.

The Code

Looking for the finished code for this example? I would be too! It's hosted on Github and is available here.

Conclusion

As you can imagine, the techniques and technologies used in this article can be used in many ways. Let's summarize some of the key take aways:

Using Bluetooth Central to scan for and identify an off-the-shelf Tile device
Storing the Tile identifying information to EEPROM. That way it can be retrieved on startup.
Using our familiar Particle.publish to push updates to the cloud.
Using a Particle Integration Webhook to create push notifications on state change.

Now that you have it all working, expand on it, hack it and make it yours. Oh and don't forget to share! I'd love to hear from you. hello@jaredwolff.com

Like this post? Share it with the world!

Interested in learning more? I'm writing a guide on how to get the most out of the Particle Platform. Learn more about it here.

Ultimate How-to: Bluetooth Swift With Hardware in 20 Minutes

Jared Wolff — Sun, 11 Aug 2019 09:16:41 +0000

In a previous tutorial, you learned how to add Bluetooth to a Particle Xenon application. That way you could control the onboard RGB LED from a test app like nRF Connect or Light Blue Explorer.

In this post, we're going to take it one step further. We're going to develop a Swift app to control a Particle Mesh RGB led. If all goes well, you should have a working app in about 20 minutes!

Let's get started.

Don't have time right now to read the full article?

Download the PDF version here.

Getting set up

Install Xcode. You can download it from the App store here.
You'll also need an Apple login. I use my iCloud email. You can create a new account within Xcode if you don't have one yet.
Install the RGB example code on a Particle Mesh board.

Create the project

Once everything is installed, let's get to the fun stuff!

Open Xcode and go to File → New Project.

Select Single View App.

Then update the Project Name to be to your liking. I've also changed my organization identifier to com.jaredwolff. Modify it as you see fit!

Select a location to save it.

Next find your Info.plist.

Update info.plist by adding Privacy - Bluetooth Peripheral Usage Description

The description I ended up using was App uses Bluetooth to connect to the Particle Xenon RGB Example

This allows you to use Bluetooth in your app if you ever want to release it.

Now, let's get everything minimally functional!

Minimally functional

Next, we'll get a minimally functional app to connect and do a services discovery. Most of the action will happen in the ViewController.swift.

Lets first import CoreBluetooth

    import CoreBluetooth

This allows us to control the Bluetooth Low Energy functionality in iOS. Then let's add both the CBPeripheralDelegate and CBCentralManagerDelegate to the ViewController class.

    class ViewController: UIViewController, CBPeripheralDelegate, CBCentralManagerDelegate {

Let's now create local private variables to store the actual central manager and peripheral. We'll set them up further momentarily.

    // Properties
    private var centralManager: CBCentralManager!
    private var peripheral: CBPeripheral!

In your viewDidLoad function, let's init the centralManager

    centralManager = CBCentralManager(delegate: self, queue: nil)

Setting delegate: self is important. Otherwise the central state never changes on startup.

Before we get further, let's create a separate file and call it ParticlePeripheral.swift. It can be placed anywhere but I placed it in a separate 'group' called Models for later.

Inside we'll create some public variables which contain the UUIDs for our Particle Board. They should look familiar!

    import UIKit
    import CoreBluetooth

    class ParticlePeripheral: NSObject {

        /// MARK: - Particle LED services and charcteristics Identifiers

        public static let particleLEDServiceUUID     = CBUUID.init(string: "b4250400-fb4b-4746-b2b0-93f0e61122c6")
        public static let redLEDCharacteristicUUID   = CBUUID.init(string: "b4250401-fb4b-4746-b2b0-93f0e61122c6")
        public static let greenLEDCharacteristicUUID = CBUUID.init(string: "b4250402-fb4b-4746-b2b0-93f0e61122c6")
        public static let blueLEDCharacteristicUUID  = CBUUID.init(string: "b4250403-fb4b-4746-b2b0-93f0e61122c6")

    }

Back in ViewController.swift let's piece together the Bluetooth bits.

Bluetooth bits

Everything to do with Bluetooth is event based. We'll be defining several functions that handle these events. Here are the important ones:

centralManagerDidUpdateState updates when the Bluetooth Peripheral is switched on or off. It will fire when an app first starts so you know the state of Bluetooth. We also start scanning here.

The centralManager didDiscover event occurs when you receive scan results. We'll use this to start a connection.

The centralManager didConnect event fires once the device is connected. We'll start the device discovery here. Note: Device discovery is the way we determine what services and characteristics are available. This is a good way to confirm what type of device we're connected to.

The peripheral didDiscoverServices event first once all the services have been discovered. Notice that we've switched from centralManager to peripheral now that we're connected. We'll start the characteristic discovery here. We'll be using the RGB service UUID as the target.

The peripheral didDiscoverCharacteristicsFor event will provide all the characteristics using the provided service UUID. This is the last step in the chain of doing a full device discovery. It's hairy but it only has to be done once during the connection phase!

Defining all the Bluetooth functions.

Now that we know what the functions events that get triggered. We'll define them in the logical order that they happen during a connection cycle.

First, we'll define centralManagerDidUpdateState to start scanning for a device with our Particle RGB LED Service. If Bluetooth is not enabled, it will not do anything.

    // If we're powered on, start scanning
        func centralManagerDidUpdateState(_ central: CBCentralManager) {
            print("Central state update")
            if central.state != .poweredOn {
                print("Central is not powered on")
            } else {
                print("Central scanning for", ParticlePeripheral.particleLEDServiceUUID);
                centralManager.scanForPeripherals(withServices: [ParticlePeripheral.particleLEDServiceUUID],
                                                  options: [CBCentralManagerScanOptionAllowDuplicatesKey : true])
            }
        }

Defining the centralManager didDiscover is our next step in the process. We know we've found a device if this event has occurred.

    // Handles the result of the scan
        func centralManager(_ central: CBCentralManager, didDiscover peripheral: CBPeripheral, advertisementData: [String : Any], rssi RSSI: NSNumber) {

            // We've found it so stop scan
            self.centralManager.stopScan()

            // Copy the peripheral instance
            self.peripheral = peripheral
            self.peripheral.delegate = self

            // Connect!
            self.centralManager.connect(self.peripheral, options: nil)

        }

So, we stop scanning using self.centralManager.stopScan(). We set the peripheral so it persists through the app. Then we connect to that device using self.centralManager.connect

Once connected, we need to double check if we're working with the right device.

    // The handler if we do connect succesfully
        func centralManager(_ central: CBCentralManager, didConnect peripheral: CBPeripheral) {
            if peripheral == self.peripheral {
                print("Connected to your Particle Board")
                peripheral.discoverServices([ParticlePeripheral.particleLEDServiceUUID])
            }
        }

By comparing the two peripherals we'll know its the device we found earlier. We'll kick off a services discovery using peripheral.discoverService. We can use ParticlePeripheral.particleLEDServiceUUID as a parameter. That way we don't pick up any services we don't care about.

Once we finish the discovering services, we'll get a didDiscoverServices event. We iterate through all the "available" services. (Though there will only be one!)

    // Handles discovery event
        func peripheral(_ peripheral: CBPeripheral, didDiscoverServices error: Error?) {
            if let services = peripheral.services {
                for service in services {
                    if service.uuid == ParticlePeripheral.particleLEDServiceUUID {
                        print("LED service found")
                        //Now kick off discovery of characteristics
                        peripheral.discoverCharacteristics([ParticlePeripheral.redLEDCharacteristicUUID,
                                                                 ParticlePeripheral.greenLEDCharacteristicUUID,
                                                                 ParticlePeripheral.blueLEDCharacteristicUUID], for: service)
                        return
                    }
                }
            }
        }

By this point this is the third time we're checking to make sure we have the correct service. This becomes more handy later when there are many characteristics and many services.

We call peripheral.discoverCharacteristics with an array of UUIDs for the characteristics we're looking for. They're all the UUIDs that we defined in ParticlePeripheral.swift.

Finally, we handle the didDiscoverCharacteriscsFor event. We iterate through all the available characteristics. As we iterate we compare with the ones we're looking for.

    // Handling discovery of characteristics
        func peripheral(_ peripheral: CBPeripheral, didDiscoverCharacteristicsFor service: CBService, error: Error?) {
            if let characteristics = service.characteristics {
                for characteristic in characteristics {
                    if characteristic.uuid == ParticlePeripheral.redLEDCharacteristicUUID {
                        print("Red LED characteristic found")
                    } else if characteristic.uuid == ParticlePeripheral.greenLEDCharacteristicUUID {
                        print("Green LED characteristic found")
                    } else if characteristic.uuid == ParticlePeripheral.blueLEDCharacteristicUUID {
                        print("Blue LED characteristic found");
                    }
                }
            }
        }

At this point we're ready to do a full device discovery of our Particle Mesh device. In the next section we'll test what we have to make sure things are working ok.

Testing our minimal example

Before we get started, if you run into trouble I've put some troubleshooting steps in the footnotes.

To test, you'll have to have an iPhone with Bluetooth Low Energy. Most modern iPhones have it. The last iPhone not to have it I believe was either the iPhone 4 or 3Gs. (so you're likely good)

First, plug it into your computer.

Go to the top by the play and stop buttons. Select your target device. In my case I chose my phone (Jared's iPhone). You can also use an iPad.

Then you can hit Command + R or hit that Play button to load the app to your phone.

Make sure you have your log tab open. Enable it by clicking the bottom pane button in the top right corner.

Make sure you have a mesh device setup and running the example code. You can go to this post to get it. Remember your Particle Mesh board needs to be running device OS 1.3.0 or greater for Bluetooth to work!

Once both the firmware and app is loaded, let's check the log output.

It should look something like this:

View loaded
Central state update
Central scanning for B4250400-FB4B-4746-B2B0-93F0E61122C6
Connected to your Particle Board
LED service found
Red LED characteristic found
Green LED characteristic found
Blue LED characteristic found

This means that your Phone has connected, found the LED service! The characteristics also being discovered is important here. Without those we wouldn't be able to send data to the mesh device.

Next step is to create some sliders so we can update the RGB values on the fly.

Slide to the left. Slide to the right.

Next we're going to add some elements to our Main.storyboard. Open Main.storyboard and click on the View underneath View Controller.

Then click on the Library button. (It looks like the old art Apple used for the home button)

You'll get a pop-up with all the choices that you can insert into your app.

Drag three Labels and copy three Sliders to your view.

You can double click on the labels and rename them as you go.

If you click and hold, some handy alignment tools will popup. They'll even snap to center!

You can also select them all and move them together. We'll align them vertically and horizontally.

In order for them to stay in the middle, let's remove the autoresizing property. Click the Ruler icon on the top right. Then click each of the red bars. This will ensure that your labels and sliders stay on the screen!

Next let's click the Show Assistant Editor button. (Looks like a Venn diagram)

Note: make sure that ViewController.swift is open in your Assistant Editor.

Then underneath the /properties section, Control-click and drag the Red Slider into your code.

Repeat with all the other ones. Make sure you name them something different. Your code should look like this when you're done:

        // Properties
        private var centralManager: CBCentralManager!
        private var peripheral: CBPeripheral!

        // Sliders
        @IBOutlet weak var redSlider: UISlider!
        @IBOutlet weak var greenSlider: UISlider!
        @IBOutlet weak var blueSlider: UISlider!

This allow us to access the value of the sliders.

Next, let's attach the Value Changed event to each of the sliders. Right click on the Red Slider in the folder view.

It should give you some options for events. Click and drag the Value Changed event to your code. Make sure you name it something that makes sense. I used RedSliderChanged for the Red Slider.

Repeat two more times. Your code should look like this at the end of this step:

        @IBAction func RedSliderChanged(_ sender: Any) {
        }

        @IBAction func GreenSliderChanged(_ sender: Any) {
        }

        @IBAction func BlueSliderChanged(_ sender: Any) {
        }

I've also selected each of the sliders to and un-checked Enabled. That way you can't move them. We'll enable them later on in code.

Also, this is a great time to change the maximum value to 255. Also set the default value from 0.5 to 0.

Back at the top of the file. Let's create some local variables for each of the characteristics. We'll use these so we can write the slider variables to the Particle Mesh board.

        // Characteristics
        private var redChar: CBCharacteristic?
        private var greenChar: CBCharacteristic?
        private var blueChar: CBCharacteristic?

Now, let's tie everything together!

In the didDiscoverCharacteristicsFor callback function. Let's assign those characteristics. For example

    if characteristic.uuid == ParticlePeripheral.redLEDCharacteristicUUID {
        print("Red LED characteristic found")
        redChar = characteristic

As we find each characteristic, we can also enable each of the sliders in the same spot.

            // Unmask red slider
            redSlider.isEnabled = true

In the end your didDiscoverCharacteristicsFor should look like:

    // Handling discovery of characteristics
        func peripheral(_ peripheral: CBPeripheral, didDiscoverCharacteristicsFor service: CBService, error: Error?) {
            if let characteristics = service.characteristics {
                for characteristic in characteristics {
                    if characteristic.uuid == ParticlePeripheral.redLEDCharacteristicUUID {
                        print("Red LED characteristic found")

                        redChar = characteristic
                        redSlider.isEnabled = true
                    } else if characteristic.uuid == ParticlePeripheral.greenLEDCharacteristicUUID {
                        print("Green LED characteristic found")

                        greenChar = characteristic
                        greenSlider.isEnabled = true
                    } else if characteristic.uuid == ParticlePeripheral.blueLEDCharacteristicUUID {
                        print("Blue LED characteristic found");

                        blueChar = characteristic
                        blueSlider.isEnabled = true
                    }
                }
            }
        }

Now, let's update the RedSliderChanged GreenSliderChanged and BlueSliderChanged functions. What we want to do here is update the characteristic associated with the Changed function. I created a separate function called writeLEDValueToChar. We'll pass in the characteristic and the data.

    private func writeLEDValueToChar( withCharacteristic characteristic: CBCharacteristic, withValue value: Data) {

            // Check if it has the write property
            if characteristic.properties.contains(.writeWithoutResponse) && peripheral != nil {

                peripheral.writeValue(value, for: characteristic, type: .withoutResponse)

            }

        }

Now add a call to writeLEDValueToChar to each of the Changed functions. You will have to cast the value to a Uint8. (The Particle Mesh device expects an unsigned 8-bit number.)

            @IBAction func RedSliderChanged(_ sender: Any) {
            print("red:",redSlider.value);
            let slider:UInt8 = UInt8(redSlider.value)
            writeLEDValueToChar( withCharacteristic: redChar!, withValue: Data([slider]))

        }

Repeat this for GreenSliderChanged and BlueSliderChanged. Make sure you changed red to green and blue for each!

Finally, to keep things clean, i've also added a function that handles Bluetooth disconnects.

    func centralManager(_ central: CBCentralManager, didDisconnectPeripheral peripheral: CBPeripheral, error: Error?) {

Inside, we should reset the state of the sliders to 0 and disable them.

            if peripheral == self.peripheral {
                print("Disconnected")

                redSlider.isEnabled = false
                greenSlider.isEnabled = false
                blueSlider.isEnabled = false

                redSlider.value = 0
                greenSlider.value = 0
                blueSlider.value = 0

It's a good idea to reset self.peripheral to nil that way we're not ever trying to write to a phantom device.

                self.peripheral = nil

Finally, because we've disconnected, start scanning again!

                // Start scanning again
                print("Central scanning for", ParticlePeripheral.particleLEDServiceUUID);
                centralManager.scanForPeripherals(withServices: [ParticlePeripheral.particleLEDServiceUUID],
                                                  options: [CBCentralManagerScanOptionAllowDuplicatesKey : true])
            }

Alright! We just about ready to test. Let's move on to the next (and final) step.

Test the sliders.

The hard work is done. Now it's time to play!

The easiest way to test everything is to click the Play button in the top left or the Command + R keyboard shortcut. Xcode will load the app to your phone. You should see a white screen proceeded by a screen with your sliders!

The sliders should stay greyed out until connected to your Particle Mesh board. You can check your log output if the connection has been established.

View loaded
Central state update
Central scanning for B4250400-FB4B-4746-B2B0-93F0E61122C6
Connected to your Particle Board
LED service found
Red LED characteristic found
Green LED characteristic found
Blue LED characteristic found

(Look familiar? We're connected!)

If you followed everything perfectly, you should be able to move the sliders. Better yet, the RGB LED on the Particle Mesh board should change color.

Conclusion

In this article you've learned how to connect your Particle Mesh board and iOS device over Bluetooth. We've learned how to connect to each of the available characteristics. Plus, on top of it all, make a clean interface to do it all in.

As you can imagine, you can go down the rabbit hole with Bluetooth on iOS. There's more coming in my upcoming guide: The Ultimate Guide to Particle Mesh. Subscribers to my list get access to pre-launch content and a discount when it comes out! Click here to get signed up.

Code

The full source code is available on Github. If you find it useful, hit the star button. ⭐️

2 Best Ways to Get Particle Mesh Working With Blynk

Jared Wolff — Sun, 28 Jul 2019 13:22:16 +0000

Writing an app takes time. It takes even more time to write one that works with hardware.

Luckily there's a solution to this problem.

Enter Blynk.

It's an app that connects to your hardware. It has a drag and drop interface with pre built widgets. That means you can build an app in seconds. Then upload your device sensors readings within minutes.

Blynk does work with Argon, Boron or ethernet connected Xenon. Unfortunately it doesn't work over a Particle Mesh network. In this article you'll learn some of the workarounds to get your mesh based projects up an running.

From Particle Cloud to Blynk

Let's start with the most simple use case: getting data from any Particle Device to Blynk.

The Air Quality data from Particle Squared is perfect for this example. So, i'll be using that.

First let's create a new Blynk Project

Grab the Auth Token we'll need that in a bit. You can tap the Auth Token to copy it to your clipboard.

Next, let's add a SuperChart for this example.

Configure the SuperChart to use a Virtual Pin. We don't have access to the actual hardware pins on the device. V0 is a good choice.

To update values in Blynk, we'll have to connect somehow. The best way is to use an Integration in the Particle Console.

In Particle Console, click the icon below the terminal icon. Then click on New Integration.

Look at the example below to see how I filled everything out.

Particle Squared uses the Event Name as **blob. For other projects this may be different. **Remember: your event name is the same as from Particle.publish(eventName, data).

The URL is set to use the blink-cloud.com address. According to their API a sample URL looks like:

I'll also include it here so it's easier to copy

http://blynk-cloud.com/auth_token/update/pin?value=value

Replace auth_token with the Auth Token we got earlier.

Replace pin with the virtual pin we want to modify. In this case V0

Replace the value with the value you want to use.

We'll reference one of the values in the Particle Squared blob. It's organized like this:

{
  "temperature": 28.60,
  "humidity": 45.00,
  "sgp30_tvoc": 18,
  "sgp30_c02": 400,
  "bme680_temp": 27.36,
  "bme680_pres": 1012.43,
  "bme680_hum": 43.80,
  "bme680_iaq": 43.90,
  "bme680_temp_calc": 27.30,
  "bme680_hum_calc": 43.97
}

Particle uses mustache templates. As you can see in the screenshot above, you can set value to {{{temperature}}}.

Note: If you're working on your own project, it's important to publish with JSON. As a reference the Particle.publish command looks like:

// Publish data
Particle.publish("blob", String::format("{\"temperature\":%.2f,\"humidity\":%.2f}",si7021_data.temperature, si7021_data.humidity) , PRIVATE, WITH_ACK);

Click the big blue Save button at the bottom of the screen. Then we can move on to the next step!

Testing

Since creating our Particle Webhook Integration, it's been publishing data to Blynk. Let's go see if it's working.

First, let's go back to the Blynk app. Hit the Play Button in the top Right in Blynk screen.

If your integration has been running for a while, you should see the graph populate with data! In the case you don't see anything, let's check the logs.

Go back to your integration and scroll towards the bottom. We want to see if there are any errors.

Not sure what that looks like? Here's an example of an integration with errors:

You can scroll further down to investigate why the error has occurred.

All the way at the bottom shows the response from the server. Depending on the service, they'll give you information why your API call failed. In my case, I was missing values for two fields.

Particle to Blynk is working!

You now have a basic way of publishing to a virtual pin in Blynk. There are drawbacks though. Most importantly, you'll have to create an integration for every signal virtual pin. If you have eight readings, that means eight integrations.

Bummer.

In the next section, you'll learn a different way to configure Blynk. Let's go!

Local Mesh Using Blynk Library

Unlike the first method, we'll be focusing on changing firmware only.

We’ll use a Argon, Boron or Ethernet Connected Xenon and one regular Xenon. For the rest of this tutorial, we'll call these devices an “edge router”.

The Xenon will run the Particle Squared code. Instead of using Particle.publish we'll be using Mesh.publish. This allows us to publish only to the local mesh network.

Meanwhile the edge router is listening for the message. It collects the values and then uses the Blynk API to publish to the app.

Here are the steps:

Setup our Edge Router

Pull up the menu by pressing Cmd+Shift+P. Type Install Library.

Then enter blynk. The library should download if you haven't already.

Once installed you can include the library at the top of your .ino file like so:

#include <blynk.h>

In our setup() function let's init the Blynk library:

// Put initialization like pinMode and begin functions here.
Blynk.begin(auth);

In our setup() function, subscribe to the temperature event. The connected Xenon will generate this event.

// Subscribe to temperature events
Mesh.subscribe("temperature",tempHandler);

Define tempHandler like this for now:

// Temperature event handler for mesh
void tempHandler(const char *event, const char *data){
}

In the loop() function make sure we have Blynk.run();

// loop() runs over and over again, as quickly as it can execute.
void loop() {
  // The core of your code will likely live here.
  Blynk.run();
}

Finally, for tempHandler we can add a debug print to monitor events. I've used something like this:

Serial.printlnf("event=%s data=%s", event, data ? data : "NULL");

Particle uses this in some of their examples. It's perfect for our purposes as well!

Note: make sure you have Serial.begin() called in your Setup() function!

So now we have tempHandler to receive data from the Xenon. The edge router can now take that data and upload it to Blynk. Let's use the Blynk.virtualWrite function for this:

// Write the data
Blynk.virtualWrite(V0, data);

This will write the temperature value from a Xenon to the V0 pin. If you used something other than V0, be sure to change that value here. (This is the same setup as the previous Particle Cloud to Blynk example)

The final code for the edge router should look something like this. Compile a flash it to your device when you're ready!

/*
 * Project blynk-argon-forwarder
 * Description: Argon Blynk forwarder for Particle Mesh. Forwards data from mesh connected devices to Blynk.
 * Author: Jared Wolff
 * Date: 7/25/2019
 */

#include <blynk.h>

char auth[] = "<ENTER YOUR AUTH KEY>";

// Temperature event handler for mesh
void tempHandler(const char *event, const char *data){
  Serial.printlnf("event=%s data=%s", event, data ? data : "NULL");

  // Write the data
  Blynk.virtualWrite(V0, data);
}

// setup() runs once, when the device is first turned on.
void setup() {

  // Serial for debugging
  Serial.begin();

  // Put initialization like pinMode and begin functions here.
  Blynk.begin(auth);

  // Subscribe to temperature events
  Mesh.subscribe("temperature",tempHandler);

}

// loop() runs over and over again, as quickly as it can execute.
void loop() {
  // The core of your code will likely live here.
  Blynk.run();

}

Remember to set auth using the AUTH TOKEN in the Blynk app!

Setting up a Xenon

Create a new project. This time it will be for the Xenon capturing "temperature data."

Let's add a variable called time_millis to the top of the file. The type is system_tick_t. We'll use it to create a simple delay timer for the temperature readings.

// Global variable to track time (used for temp sensor readings)
system_tick_t time_millis;

For the interval, let's use a preprocessor define

#define INTERVAL_MS 2000

Now let's tie those together in the loop() function. We'll use an if statement to compare our current system time with that of the last event plus offset. If you ever need a simple timer, this is one of the best ways to do it!

// Check if our interval > 2000ms
  if( millis() - time_millis > INTERVAL_MS ) {
  }

Once we're inside, make sure you reset timer_millis:

        //Set time to the 'current time' in millis
    time_millis = millis();

Then, we'll generate the temperature value using the random() function. We'll use the two parameter variant. That way we can set the minimum value and the maximum value:

    // Create a random number
    int rand = random(20,30);

Finally we'll Mesh.publish the value:

    // Publish our "temperature" value
    Mesh.publish("temperature",String::format("%d",rand));

When this example runs, the temperature is broadcast to the mesh network. Then, the edge router receives it and forwards it on to Blynk!

You can flash this firmware whenever you're ready. Here's the full code for the Xenon so you can cross compare:

/*
 * Project blynk-xenon-rgb
 * Description: Recieve RGB level from connected Edge Router. Sends simiulated temperature values via mesh to the Blynk cloud.
 * Author: Jared Wolff
 * Date: 7/25/2019
 */

// How often we update the temperature
#define INTERVAL_MS 2000

// Global variable to track time (used for temp sensor readings)
system_tick_t time_millis;
// setup() runs once, when the device is first turned on.
void setup() {

  // Set time to 0
  time_millis = 0;

}

// loop() runs over and over again, as quickly as it can execute.
void loop() {

  // Check if our interval > 2000ms
  if( millis() - time_millis > INTERVAL_MS ) {
    //Set time to the 'current time' in millis
    time_millis = millis();

    // Create a random number
    int rand = random(20,30);

    // Publish our "temperature" value
    Mesh.publish("temperature",String::format("%d",rand));

  }

}

Give it a test!

Now that we've programmed both devices let's get them talking to each other.

I've already set up the Argon with a mesh network called 8f-9. I'll explain how to get the Xenon connected with the CLI. You can also used the Particle App.

First, let's connect the Xenon to USB and get it into Listening Mode. After connect, hold the Mode button until blinking blue.

Then use the CLI to set up the mesh network. First let's get the device ID:

Jareds-MacBook-Pro:nrfjprog.sh jaredwolff$ particle identify
? Which device did you mean?
  /dev/tty.usbmodem146401 - Argon
❯ /dev/tty.usbmodem146101 - Xenon

If you have multiple devices connect, make sure you select the right one! If prompted, select a device. Your output should look something like:

Your device id is e00fce682d9285fbf4412345
Your system firmware version is 1.3.0-rc.1

We'll need the id for the next step. Now, let's run the particle mesh command.

particle mesh add <xenon id> <id of your argon, boron, etc>

Here's an example below:

particle mesh add e00fce682d9285fbf4412345 hamster_turkey
? Enter the network password [hidden]
▄ Adding the device to the network...

At the end of it you'll see:

Done! The device should now connect to the cloud.

This process is not perfect. During the Adding the device to the network... stage, I had to remove the Xenon using particle mesh remove. Then re-run the particle mesh add command after resetting the Argon.

Now here comes to finale.

Connect the two devices to serial using particle serial monitor --follow

If you have the two devices connected, particle serial monitor will prompt you to select:

Jareds-MacBook-Pro:blynk-xenon-rgb jaredwolff$ particle serial monitor --follow
Polling for available serial device...
? Which device did you mean? /dev/tty.usbmodem146101 - Xenon
Opening serial monitor for com port: "/dev/tty.usbmodem146101"
Serial monitor opened successfully:

Remember: You have to run particle serial monitor for each device you want to connect to.

If all is working, you'll likely see some output from the edge router!

Serial monitor opened successfully:
event=temperature data=21
event=temperature data=28
event=temperature data=21
event=temperature data=27
event=temperature data=28
event=temperature data=26
event=temperature data=23
event=temperature data=26
event=temperature data=21

Looking at the app, the Super Chart should be reacting to this new data.

Compare the last value in the command line to the last on the chart? Do they match? If so, you made it to the end of this example!

Conclusion

In this tutorial you've learned how to forward Particle Cloud data to Blynk. You've also learned how to do the same using a Particle Argon, Boron or ethernet connected Xenon. Awe yea. 😎👍

Now that you have the tools to Blink-ify your Particle Mesh powered projects, it's time to get to work!

Like this post?

This post is an excerpt from my upcoming Ultimate Guide to Particle Mesh. I'll be sharing more exclusive content with my mailing list as it get's closer to launch. You can sign up here for updates.

Still have questions?

7 Effective Ways To Speed Up Your Site

Jared Wolff — Mon, 22 Jul 2019 11:50:23 +0000

This post is originally from my blog on www.jaredwolff.com

Web rankings are dictated by how fast you serve your content.

You need to optimize to get that top spot. As an added benefit your visitors will thank you. (or not even notice — a good thing!)

In this article, I show you how to optimize your site for speed.

Let's get started.

Get the Baseline

First, let's sample your site.

Google's PageSpeed insights tool should be your first stop. After using it you'll know if your site is optimized for both mobile and desktop.

Google bases their speed tests on how fast your page renders content. i.e. They measure how long it takes until your website hits your visitors eyeballs. The longer the wait, the lower the score.

Here's a screenshot from this page on my blog.

This screenshot was taken after optimizing a few big issues. I ditched Disqus, inlined my CSS and Javascript. Performance went from being in the 50/100 range to being in the 90/100 on mobile. Not bad!

One of my main complaints about Pagespeed Insights is how slow it is.

Like.. unbearably slow.

The solution?

Lighthouse.

Using Lighthouse

Lighthouse is an npm package that you can install on your computer. It's basically Pagespeed Insights but on steroids. (Pagespeed Insights is based on Lighthouse)

When you run it locally, you'll keep any information that get's generated from the tests. (Though, i'm sure Chrome phones home about the things you're doing.. 😬)

Here's how to get started:

To install you can run:
```
npm install -g lighthouse
```
Also make sure you have some version of Google Chrome installed. I'm using Chrome Canary.
Then you can run it and get a report like this
```
lighthouse https://www.jaredwolff.com --view
```
--view will cause the report to be opened in your default web browser. Here is a report from the same page as earlier:

It not only includes Performance but Accessibility, Best practices and SEO suggestions.

The downside is you'll have to run this test on a page-by-page basis. I suggest you sample pages that have lots of content. That way you're testing the worst of your site. Once you fix the worst, your whole site will see a performance boost!

Go Static

Remember when websites were 100% HTML & CSS?

Over the past years we went from pure HTML, to Ruby On Rails and slowly we're making our way back.

The reason?

Speed.

Every time you visit a website running Flask, Ruby on Rails or similar it goes something like this:

Make a request for a page
Server pieces your site together
Server minifies and gzip
Sever sends content back to the browser

This itself, doesn't take long. When you multiply that by 1000's or 10's of 1000's you start running into problems.

What if you do all the piecing together once?

That's the advantage of a statically generated website.

Let's take a look at how it works:

How does a static site generator work?

A static site generator is a collection of templates and styles. They can be assembled together to generate different content.

As a backend, static site generators use markdown and (sometimes) JSON.

When it's time to compile, templates are assembled. The Markdown is converted to HTML and then injected into the templates. The result? An output directory of rich "dynamic looking" pages. (Much like the one you're reading right now if you're on my site!)

Personally, my blog is Hugo powered. I've also dabbled with Middleman and Jekyll. No matter what you're looking for, you'll likely find a static site generator that will fit your needs! Netlify has a nice list of Static Site Generators sorted by popularity. Check it out here.

Plugins Galore

Static site generators no longer only compile sites. They optimize, minify and resize images. This built in functionality is another reason why to consider a statically generated site. Tap into these resources, and your site will be notably faster.

For example, originally I was using highlight.min.js for highlighting code. Since discovering the built in syntax highlighting in Hugo, I ditched highlight.min.js. Hugo injects the CSS into the HTML for the code blocks. Leaving nicely formatted (and static) page!

Embedding Javascript and CSS

As I alluded to in the earlier section, you'll take a performance hit when you have to load anything extra.

Recently, Hugo got the ability to copy content from file into the final HTML code. This is fantastic for things like CSS and Javascript. This way, everything is built into your HTML file. There's no extra fetches required!

For example, I place my full stye.css file in the header of the site. That way all the styles get applied immediately.

    <!-- Css -->
    {{- $style := resources.Get "/css/style.css" -}}
    <style>
      {{ ( $style | minify ).Content | safeCSS }}
    </style>

In the footer, i'm exporting the minified lazysizes.min.js into a <script> tag. It's important that this is loaded asap because it dictates how the rest of the site will load.

    <!-- Lazy Load Script -->
    {{- $lazysizes := resources.Get "/js/lazysizes.min.js" -}}
    <script>
      {{ ( $lazysizes | minify ).Content | safeJS }}
    </script>

Side Note: both style.css and lazysizes.min.js are located in my main theme folder under assets. Hugo uses the assets folder to look for these files. If you have a Hugo site and you want to embed your css and javascript I recommend you check out this resource.

Side Note Numero Dos: As you can see above, i'm using Hugo's built in minify function for the embedded items. The javascript is already minified but style.css isn't. There's also another programatic way to minify all your assets. I'll go into more on that in the Optimizing using Gulp section.

Use what's built in

You may be noticing by now that importing javascript can take a toll. Using it for every feature on a website could lead to crap performance!

For example, we could use a javascript library for form validation. But is there a better way?

(yup)

You can use the HTML5 tags like required and use pattern to validate in browser. If you want an example, check out the contact form on my website.

All the validation you see, is done in browser. No extra javascript. No delay in loading. 👌

A great resource for all of this is here: https://css-tricks.com/form-validation-part-1-constraint-validation-html/ Chris also has a detailed step-by-step portion on vanilla javascript validation. You can take it further if you so choose.

Organizing Code

Where you put your javascript and css will affect performance. For instance, I place my javascript and CSS in strategic locations. The main stylesheet in the <head> otherwise the rest are in the footer or inline with the HTML. Similarly, the javascript mostly lives at the bottom of the page. That way, all the important stuff gets loaded first.

Here is an example of how I organize my javascript:

...
</footer>
...

<script src="https://code.jquery.com/jquery-3.4.1.min.js"   integrity="sha256-CSXorXvZcTkaix6Yvo6HppcZGetbYMGWSFlBw8HfCJo="   crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/js-cookie@2/src/js.cookie.min.js"></script>

I encourage you to experiment to see what improves your site performance. If you can go without certain javascript libraries, I highly encourage you nix em!

Lazy Loading

Lazy loading is an effective way to defer loading of assets not in view. This improves, what Google calls, the "time to interactive" experience. An added bonus is if the visitor doesn't scroll down, the images aren't loaded. Thus saving bandwidth and money for high traffic sites.

Lazy loading comes in the form of javascript. There are a few libraries out there:

Lazysizes (⭐️ 11k) - seems to be the most popular in this department. It is larger than some of the alternatives. I've been testing it's usefulness around lazy loading iframes and javascript content that isn't immediately necessary. For instance, I embed an iframe for the Google Docs chart on this post. On the same post I also lazy load the Youtube video towards the end of the article.
Layzr.js (⭐️ 5.5k) - for images only
lozad (⭐️ 5.4k) - does everything that lazysizes does. This library is focused on using the Intersection Observer API. Whereas Lazysizes uses both the Intersection Observer API and
yall (⭐️ 800) - This library is also focused on using the Intersection Observer API.

Setting up Lazy Loading

It's extremely simple to set up. I'll show you how to use lazysizes

Include the file in your HTML
```
<script src="lazysizes.min.js" async=""></script>
```
(Or if you're using Hugo, inject it directly into your HTML like I did in Embedding Javascript and CSS)
Add the lazyload class to whatever you're lazy loading
Change the src tag to data-src

Watch the Magic

When someone visits your site, the lazy loader starts monitoring where the user is. When the visitor scrolls, it loads soon to be viewed content marked with the lazyloadclass.

Here is an example for a Youtube video embedded on this page:

<iframe class="lazyload" width="700px" height="400px" data-src="https://www.youtube.com/embed/IR2W0GmRKk8" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

This prevents the iframe from loading until the user has scrolled close by. Nice!

Optimizing using Gulp

Not everything can be optimized by Jekyll or Hugo. So what can you do?

Enter gulp

gulp is my tool of choice for optimizing my Hugo site. There are a wide array of well supported plugins for gulp. They can do everything from optimize images to minify HTML.

Here are some of my favorites:

gulp-uglify - minifies and compresses javascript. I only have one javascript library i'm using right now which this applies to. If you're project is Javascript heavy though, definitely look into gulp-uglify

gulp-htmlmin - minifies HTML. You can also use it to minify inline Javascript and CSS.

gulp-imagemin - probably the most useful gulp plugin for my purposes. Right now i'm having it resize, convert to jpg and then convert to a progressive jpeg. It's relatively fast and when combined with gulp-cache it only has to be done once! It may seem complicated but the output produces image sizes that Lighthouse loves.

Example

If you're curious, here's a snippet from my gupfile.js for imagemin

gulp.task("resize", function() {
  return gulp
    .src(["content/**/**/*.+(png|jpg|jpeg|gif)"])
    .pipe(cache(imagemin([
      imagemingm.resize({width: 720}),
      imagemingm.convert('jpg'),
      imageminmozjpeg({quality: 80})
    ],{
    verbose: true
    }), {
      fileCache: new Cache({ tmpDir: path.join(process.env.PWD, 'node_modules'), cacheDirName: '.cache' })
    }))
    .pipe(gulp.dest("public/"));
});

Some important things to notice

I've used ** to denote a wildcard value for a directory name. Depending on the depth of your images you may need to add more **/ to your path. This works well for my path which is typically /contents/post-name/images/image-file.jpg
Setting up other plugins that don't come with gulp-imagemin can be confusing. imageminmozjpeg for example, is imported separately. I set it up at the top of my file like so: var imageminmozjpeg = require('imagemin-mozjpeg');
Finally, you can see that i'm using gulp-cache here. Depending on where you're building your site, you may not have to define any options. I use Netlify for my site. The only way for gulp-cache to work is to define tmpDir and cacheDirName to the node_modules folder. That way, when your site is built, your cache is re-loaded. No resizing images if we don't have to!

By using gulp-cache, my build + deploy times went from 10 minutes to ~60 seconds. 🎉 Trading the minor increase in cache size for compute time is definitely worth it. Plus, I'm sure Netlify is happy about that!

Remove the Cruft

When there's convenience, there's always a tradeoff.

That's what I was finding as I continued to use Google Analytics and services like Disqus. Disqus in particular, had a tendency to drive me up the wall. (Ever look at your Javascript Debug console on a site using Disqus? You'll see why..)

I recently wrote a tutorial on how to switch from Disqus to self-hosted Commento. There are two benefits to make the switch:

You get control of your site content. There's no 3rd party service that could go poof tomorrow. (It is up to you to make sure your content is backed up! One advantage of paying for a service instead of hosting).
You'll get a performance boost! I've tested the before and after and the results are in. Disqus was a major drag on network and CPU usage. Commento, is light and resource usage is minimal. Perfect companion for a statically generated site!

As a site owner, the quality and speed is completely dependent on your site features. Remember, there may be better alternatives out there. No need to sell your soul to Google, Disqus, or anyone if you don't need to!

Site Audit

As I wrap up this post, I do want to leave you with one more handy tool.

Serpstat's Site Audit tool.

I used it to crawl my site on a regular interval. It's extremely useful for catching all types of errors. For instance, just recently it caught a few images that were broken. I made a few edits and everything was back up and running!

If you want to keep your site error free, nothing beats a site audit tool like Serpstat's!

You're never done

You read right. You'll never be done.

Improving performance for an actively developed site is never ending. Despite your best efforts you could spend days on the minute of optimizing images and lazy loading. When it comes down to it, how much of it is worth it? That's a question i'm still trying to answer myself!

I hope you've found this article useful. What was your biggest takeaway? Have a optimization tip that I didn't mention? Sound off in the comments below!

How to Use Particle's Powerful Bluetooth Api

Jared Wolff — Mon, 15 Jul 2019 03:33:23 +0000

I was defeated.

I had spent the whole night trying to get a Bluetooth Low Energy project working. It was painful. It was frustrating. I was ready to give up.

That was during the early days of Bluetooth Low Energy. Since then it's gotten easier and easier to develop. The Particle Mesh Bluetooth Library is no exception.

In this walkthrough, i'll show you how to use Particle's Bluetooth API. We'll configure some LEDs and watch them change over all devices in the Mesh network. We'll be using an Argon and Xenon board.

Ready? Let's get started!

P.S. this post is lengthy. If you want something to download, click here for a a beautifully formatted PDF.

Stage 1: Setting Up Bluetooth

Download/Install Particle Workbench
Create a new Project. I picked a suitable location and then named it ble_mesh
Go to your /src/ direcory and open your <your project name>.ino file
Then make sure you change the version of your deviceOS to > 1.3.0

Write the Code

We want to set up a service with 3 characteristics. The characteristics relate to the intensity of the RGB LEDs respectively. Here's how to get your Bluetooth Set Up:

In your Setup() function enable app control of your LED
```
RGB.control(true);
```
Set up your UUIDs at the top of your .ino file
```
const char* serviceUuid = "6E400001-B5A3-F393-E0A9-E50E24DCCA9E";
const char* red         = "6E400002-B5A3-F393-E0A9-E50E24DCCA9E";
const char* green       = "6E400003-B5A3-F393-E0A9-E50E24DCCA9E";
const char* blue        = "6E400004-B5A3-F393-E0A9-E50E24DCCA9E";
```
UUIDs are unique identifiers or addresses. They're used to differentiate different services and characteristics on a device.

The above UUIDs are used in previous Particle examples. If you want to create your own you can use uuidgen on the OSX command line. You can also go to a website like Online GUID Generator.

Use the above settings to get your own UUID. You can then create your service and characteristic UUIDS from this generated one:
```
const char* serviceUuid = "b425040**0**-fb4b-4746-b2b0-93f0e61122c6"; //service
const char* red         = "b4250401-fb4b-4746-b2b0-93f0e61122c6"; //red char
const char* green       = "b4250402-fb4b-4746-b2b0-93f0e61122c6"; //green char
const char* blue        = "b4250403-fb4b-4746-b2b0-93f0e61122c6"; //blue char
```
There's no right or wrong way to do this. But you have to be careful you're not using the UUIDs reserved by the Bluetooth SIG. This is highly unlikely. If you do want to double check you can go here and here.

For now, we'll stick with the first set of UUIDs.
In Setup(), initialize your service.
```
// Set the RGB BLE service
BleUuid rgbService(serviceUuid);
```
This is the first step of "registering' your service. More on this below.
Initialize each characteristic in Setup()
```
BleCharacteristic redCharacteristic("red", BleCharacteristicProperty::WRITE_WO_RSP, red, serviceUuid, onDataReceived, (void*)red);
BleCharacteristic greenCharacteristic("green", BleCharacteristicProperty::WRITE_WO_RSP, green, serviceUuid, onDataReceived, (void*)green);
BleCharacteristic blueCharacteristic("blue", BleCharacteristicProperty::WRITE_WO_RSP, blue, serviceUuid, onDataReceived, (void*)blue);
```
For this setup, we're going to use the WRITE_WO_RSP property. This allows us to write the data and expect no response.
I've referenced the UUIDs as the next two parameters. The first being the characteristic UUID. The second being the service UUID.

The next parameter is the callback function. When data is written to this callback, this function will fire.

Finally the last parameter is the context. What does this mean exactly? We're using the same callback for all three characteristics. The only way we can know which characteristic was written to (in deviceOS at least) is by setting a context. In this case we're going to use the already available UUIDs.

Right after defining the characteristics, let's add them so they show up:

// Add the characteristics
BLE.addCharacteristic(redCharacteristic);
BLE.addCharacteristic(greenCharacteristic);
BLE.addCharacteristic(blueCharacteristic);

Set up the callback function.

// Static function for handling Bluetooth Low Energy callbacks
static void onDataReceived(const uint8_t* data, size_t len, const BlePeerDevice& peer, void* context) {

}

You can do this at the top of the file (above Setup()) We will define this more later.

Finally, in order for your device to be connectable, we have to set up advertising. Place this code at the end of your Setup() function
```
// Advertising data
BleAdvertisingData advData;

// Add the RGB LED service
advData.appendServiceUUID(rgbService);

// Start advertising!
BLE.advertise(&advData);
```
First we create a BleAdvertisingData object. We add the rgbService from Step 3. Finally, we can start advertising so our service and characteristics are discoverable!

Time to test

At this point we have a minimally viable program. Let's compile it and program it to our Particle hardware. This should work with any Mesh enabled device. (Xenon, Argon, Boron)

Before we start testing, temporarily add SYSTEM_MODE(MANUAL); to the top of your file. This will prevent the device connecting to the mesh network. If the device is blinking blue on startup, you'll have to set it up with the Particle App before continuing.
Download the 1.3.0-rc.1 image here. For Xenon, you'll need xenon-system-part1@1.3.0-rc.1.bin. For others look for boron-system-part1@1.3.0-rc.1.bin and argon-system-part1@1.3.0-rc.1.bin. The files are at the bottom of the page under Assets
Put your device into DFU mode. Hold the Mode Button and momentarily click the Reset Button. Continue holding the Mode Button until the LED blinks yellow.
In a command line window, change directories to where you stored the file you downloaded. In my case the command is cd ~/Downloads/
Then run:
```
particle flash --usb xenon-system-part1@1.3.0-rc.1.bin
```
This will install the latest OS to your Xenon. Once it's done it will continue to rapidly blink yellow. Again if you have a different Particle Mesh device, change the filename to match.
In Visual Code, use the Command + Shift + P key combination to pop up the command menu. Select Particle: Compile application (local)
Fix any errors that may pop up.
Then, open the same menu and select Flash application (local)
When programming is complete, pull out your phone. Then, open your favorite Bluetooth Low Energy app. The best ones are NRF Connect and Light Blue Explorer. I'm going to use Light Blue Explorer for this example.
Check if a device named "Xenon-" is advertising. Insert with the unique ID for your device.
Find your device and click the name.
Look at the list of services & characteristics. Does it include the service and characteristic UUID's that we have set so far? For instance, does the service UUID show up as 6E400001-B5A3-F393-E0A9-E50E24DCCA9E?

If everything shows up as you expect, you're in a good place. If not go through the earlier instructions to make sure everything matches.

Stage 2: Handling Data

The next stage of our project is to process write events. We'll be updating our onDataReceived function.

Write the Code

First, let's create a struct that will keep the state of the LEDs. This can be done at the top of the file.

// Variables for keeping state
typedef struct {
  uint8_t red;
  uint8_t green;
  uint8_t blue;
} led_level_t;

The second half of that is to create a static variable using this data type
```
// Static level tracking
static led_level_t m_led_level;
```
The first two steps allows us to use one single variable to represent the three values of the RGB LED.
Next, let's check for basic errors inside the onDataReceive function For instance we want to make sure that we're receiving only one byte.
```
// We're only looking for one byte
  if( len != 1 ) {
    return;
    }
```
Next, we want to see which characteristic this event came from. We can use the context variable to determine this.
```
// Sets the global level
  if( context == red ) {
    m_led_level.red = data[0];
  } else if ( context == green ) {
    m_led_level.green = data[0];
  } else if ( context == blue ) {
    m_led_level.blue = data[0];
  }
```
Remember, in this case context will be equal to the pointer of either the red, green, or blue UUID string. You can also notice we're setting m_led_level. That way we can update the RGB led even if only one value has changed.

Finally, once set, you can write to the RGB object

// Set RGB color
    RGB.color(m_led_level.red, m_led_level.green, m_led_level.blue);

Test the Code

Let's go through the same procedure as before to flash the device.

Put your device into DFU mode. Hold the Mode Button and click the Reset Button. Continue holding the Mode Button until the LED blinks yellow.
Then, open the same menu and select Flash application (local)
Once it's done programming, connect to the device using Light Blue Explorer.
Tap on the characteristic that applies to the red LED.
Write FF. The red LED should turn on.
Write 00. The red LED should turn off.
Do the same for the other two characteristics. We now have full control of the RGB LED over Bluetooth Low Energy!

Stage 3: Sharing Via Mesh

Finally, now that we're successfully receiving BLE message, it's time to forward them on to our mesh network.

Write the Code

First let's remove MANUAL mode. Comment out SYSTEM_MODE(MANUAL);
At the top of the file let's add a variable we'll used to track if we need to publish
```
// Tracks when to publish to Mesh
static bool m_publish;
```

Then initialize it in Setup()

// Set to false at first
m_publish = false;

Then, after setting the RGB led in the onDataReceived callback, let's set it true:

// Set RGB color
RGB.color(m_led_level.red, m_led_level.green, m_led_level.blue);

// Set to publish
m_publish = true;

Let's add a conditional in the loop() function. This will cause us to publish the LED status to the Mesh network:

if( m_publish ) {
    // Reset flag
    m_publish = false;

    // Publish to Mesh
  Mesh.publish("red", String::format("%d", m_led_level.red));
  Mesh.publish("green", String::format("%d", m_led_level.green));
  Mesh.publish("blue", String::format("%d", m_led_level.blue));
}

Mesh.publish requires a string for both inputs. Thus, we're using String::format to create a string with our red, green and blue values.

Then let's subscribe to the same variables in Setup(). That way another device can cause the LED on this device to update as well.
```
Mesh.subscribe("red", meshHandler);
Mesh.subscribe("green", meshHandler);
Mesh.subscribe("blue", meshHandler);
```

Toward the top of the file we want to create meshHandler

// Mesh event handler
static void meshHandler(const char *event, const char *data)
{
}

In this application, we need the event parameter and data parameter. In order use them, we have to change them to a String type. That way we can use the built in conversion and comparison functions. So, inside the meshHandler function add:
```
  // Convert to String for useful conversion and comparison functions
  String eventString = String(event);
  String dataString = String(data);
```

Finally we do some comparisons. We first check if the event name matches. Then we set the value of the dataString to the corresponding led level.

  // Determine which event we recieved
  if( eventString.equals("red") ) {
    m_led_level.red = dataString.toInt();
  } else if ( eventString.equals("green") ) {
    m_led_level.green = dataString.toInt();
  } else if ( eventString.equals("blue") ) {
    m_led_level.blue = dataString.toInt();
  } else {
        return;
    }

  // Set RGB color
  RGB.color(m_led_level.red, m_led_level.green, m_led_level.blue);

Then at the end we set the new RGB color. Notice how I handle an unknown state by adding a return statement in the else section. It's always good to filter out error conditions before they wreak havoc!

Test the Code

Open the Particle App on your phone
Let's set up the Argon first. If it's not blinking blue, hold the mode button until it's blinking blue.

Note: if you've already programmed the app, the LED will be off by default. Hold the mode button for 5 seconds and then continue.
Go through the pairing process. The app walks you though all the steps. Make sure you remember the Admin password for your mesh network.
Program an Argon with the latest firmware (1.3.0) (see Stage 1 - Time to Test - Step 2 for a reminder on how to do this)
Once rapidly blinking yellow, program the Argon with the Tinker app. You can download it at the release page.
Once we have a nice solid Cyan LED (connected to the Particle Cloud) we'll program the app. Use the Cloud Flash option in the drop down menu.

As far as I can tell, Particle forces any device flashed locally into safe mode when connecting to the cloud. It may be my setup. Your mileage may vary here. Best to use Cloud Flash.

Make sure you select the correct deviceOS version (1.3.0-rc1), device type (Argon) and device name (What you named it during setup)
Connect to the Xenon using the phone app
Connect the Xenon to your Mesh network using the phone app
Flash your Xenon using Cloud Flash. Use the name that you gave it during the phone app setup. As long as the device is connected to Particle Cloud or in safe mode (Purple LED), it should program.
Once connected, let's get to the fun part. Open up Light Blue Explorer. Connect to either the Argon or the Xenon.
Select one of the LED characteristics and change the value.

The LED should change on all devices!

Final Code

Here's the final code with all the pieces put together. You can use this to make sure you put them in the right place!!

/*
  * Project ble_mesh
  * Description: Bluetooth Low Energy + Mesh Example
  * Author: Jared Wolff
  * Date: 7/13/2019
  */

//SYSTEM_MODE(MANUAL);

// UUIDs for service + characteristics
const char* serviceUuid = "6E400001-B5A3-F393-E0A9-E50E24DCCA9E";
const char* red         = "6E400002-B5A3-F393-E0A9-E50E24DCCA9E";
const char* green       = "6E400003-B5A3-F393-E0A9-E50E24DCCA9E";
const char* blue        = "6E400004-B5A3-F393-E0A9-E50E24DCCA9E";

// Set the RGB BLE service
BleUuid rgbService(serviceUuid);

// Variables for keeping state
typedef struct {
  uint8_t red;
  uint8_t green;
  uint8_t blue;
} led_level_t;

// Static level tracking
static led_level_t m_led_level;

// Tracks when to publish to Mesh
static bool m_publish;

// Mesh event handler
static void meshHandler(const char *event, const char *data)
{

  // Convert to String for useful conversion and comparison functions
  String eventString = String(event);
  String dataString = String(data);

  // Determine which event we recieved
  if( eventString.equals("red") ) {
    m_led_level.red = dataString.toInt();
  } else if ( eventString.equals("green") ) {
    m_led_level.green = dataString.toInt();
  } else if ( eventString.equals("blue") ) {
    m_led_level.blue = dataString.toInt();
  } else {
    return;
  }

  // Set RGB color
  RGB.color(m_led_level.red, m_led_level.green, m_led_level.blue);

}

// Static function for handling Bluetooth Low Energy callbacks
static void onDataReceived(const uint8_t* data, size_t len, const BlePeerDevice& peer, void* context) {

  // We're only looking for one byte
  if( len != 1 ) {
    return;
  }

  // Sets the global level
  if( context == red ) {
    m_led_level.red = data[0];
  } else if ( context == green ) {
    m_led_level.green = data[0];
  } else if ( context == blue ) {
    m_led_level.blue = data[0];
  }

  // Set RGB color
  RGB.color(m_led_level.red, m_led_level.green, m_led_level.blue);

  // Set to publish
  m_publish = true;

}

// setup() runs once, when the device is first turned on.
void setup() {

  // Enable app control of LED
  RGB.control(true);

  // Init default level
  m_led_level.red = 0;
  m_led_level.green = 0;
  m_led_level.blue = 0;

  // Set to false at first
  m_publish = false;

  // Set the subscription for Mesh updates
  Mesh.subscribe("red",meshHandler);
  Mesh.subscribe("green",meshHandler);
  Mesh.subscribe("blue",meshHandler);

  // Set up characteristics
  BleCharacteristic redCharacteristic("red", BleCharacteristicProperty::WRITE_WO_RSP, red, serviceUuid, onDataReceived, (void*)red);
  BleCharacteristic greenCharacteristic("green", BleCharacteristicProperty::WRITE_WO_RSP, green, serviceUuid, onDataReceived, (void*)green);
  BleCharacteristic blueCharacteristic("blue", BleCharacteristicProperty::WRITE_WO_RSP, blue, serviceUuid, onDataReceived, (void*)blue);

  // Add the characteristics
  BLE.addCharacteristic(redCharacteristic);
  BLE.addCharacteristic(greenCharacteristic);
  BLE.addCharacteristic(blueCharacteristic);

  // Advertising data
  BleAdvertisingData advData;

  // Add the RGB LED service
  advData.appendServiceUUID(rgbService);

  // Start advertising!
  BLE.advertise(&advData);
}

// loop() runs over and over again, as quickly as it can execute.
void loop() {

  // Checks the publish flag,
  // Publishes to a variable called "red" "green" and "blue"
  if( m_publish ) {

    // Reset flag
    m_publish = false;

    // Publish to Mesh
    Mesh.publish("red", String::format("%d", m_led_level.red));
    Mesh.publish("green", String::format("%d", m_led_level.green));
    Mesh.publish("blue", String::format("%d", m_led_level.blue));
  }

}

Conclusion

In this tutorial you learned how to add Bluetooth to a Particle Mesh project. As you can imagine, the possibilities are endless. For instance you can add user/administrative apps into the experience. Now that's awesome. 😎

You can expect more content like this in my upcoming book: The Ultimate Guide to Particle Mesh. Subscribe to my list for updates and insider content. Plus all early subscribers get a discount when it's released! Click here to sign up.

How to Protocol Buffer Bluetooth Low Energy Service Part 3

Jared Wolff — Thu, 11 Jul 2019 15:19:05 +0000

In Part 1 we’ve learned the anatomy of a Protocol Buffer. In Part 2 we’ve learned how a Bluetooth Low Energy Service gets pieced together on the Nordic NRF52 SDK. This final post brings together all the elements in an end-to-end working example.

Let’s get going!

Setting Everything Up

Go look at the Readme.md in each of the example repositories. (You will have to clone the repositories, sign up here to get the code).

I’ve made the setup for the firmware dead simple. For the javascript side, it’s a bit more hairy but’ doable! I’ll also include the instructions here as well:

Firmware Setup (For Mac)

Initialize the full repository (there are submodules!): git submodule update --init
Install protoc using Homebrew: brew install protobuf
Run make sdk. This will download your SDK files.
Run make tools_osx. This will download your ARMGCC toolchain (for Mac). For other environments see below.
Run make gen_key once (and only once)! This will set up your key for DFU.
Run make and this will build your bootloader and main app.

Note: You only have to do steps 1-5 once.

Javascript App Setup (For Mac)

Prerequisite: you will need Xcode command line tools. You can get those here.

Clone this repo to a place on your computer
Make sure you have nvm installed
Run nvm install v8.0.0
Run nvm install v10.15.3
Run nvm use v8.0.0
Run yarn (if you don’t have yarn npm install yarn -g)
Once installed, run nvm use v10.15.3
Then run node index.js to start the example

Using NVM helps mitigate a compile issue with the Bluetooth library. You mileage may vary on this one.

How the heck does this work?

In this project, the Protocol Buffer has two functions. The first as a “command” to the bluetooth device. Secondly, a “response” to that command from the device.

Our example javascript app command uses “This is” as the payload. With the power of some string operations, let's make a full sentence out of it!

The series of events looks something like this:

The test app connects and sends that data using our Bluetooth Low Energy service.
On the firmware decodes the message.
The firmware modifies the original data by adding “ cool.” Resulting in "This is cool"
The firmware encodes the payload and makes the data available for reading.
The app finally reads the characteristic, decodes and displays the result!

Seems complicated but there's some benefits to this:

In Protocol Buffers, data structures are well defined. This means that it has some great error checking abilities. If you're using any type of data structure you may have to code your own data validation.
Encoding and decoding is simple and straight forward. You can encode same data on different platforms and always get the same decoded result.

Want to get the example up and running? Seeing is believing after all. Let's do this.

Is This Thing On?

Once you're done with the setup (above), let's get this firmware programmed! Lucky for you it's only two quick steps!

Plug in your NRF52 DK board
Run make flash_all. (This compiles and flashes all the code for the project.)

Once flashed, the easiest way to know if things are working is when Led 1 is blinking. That means it’s advertising and ready for a connection.

Let’s make sure it’s advertising correctly though. You can grab a tool like NRF Connect for iOS or Android. Or, you can grab LightBlue. (Also available for iOS and Android)

Open up one of the app and scan for advertising devices. You’ll be looking for Nordic_Buttonless.

Now let’s connect and double check our connectable services & characteristics:

As you can see there are two services. One is Nordic’s DFU service. The other is our Protocol Buffer service!

You can compare PROTOBUF_UUID_BASE in ble_protobuf.h with the UUID of the “Unknown Service”. Nordic’s chips are Little Endian while the data here is Big Endian format. (i.e. you’ll have to reverse the bytes to see that they’re the same!)

You can even click in further to see the characteristics in NRF Connect. In the case of LightBlue, the characteristic UUIDs are already shown.

Using the Javascript App

So, once we’re up and advertising, it’s time to run the app.

Simply change to the ble-protobuf-js directory and run node index.js

If you've installed everything correctly, you should start seeing output like:

example started
scanning
peripheral with ID 06f9b62ec5334454875b9f53d2f3fa74 found with name: Nordic_Buttonles

It should then immediately connect and send data to the device. It should receive the response immediately and print it to the screen.

connected to Nordic_Buttonles
Sent: This is
Received: This is cool.

Bingo!

You’ve made it!

🎉 Congrats if you made it this far through the tutorial. You should now be feeling confident enough to start playing with the software code. Maybe you’ll cook up something cool?

Here’s links to some resources that you may find handy.

Protocol Buffer documentation
NanoPB - (the implementation of Protocol Buffers we used for this project)
Nordic SDK Documentation
Again, here are the links to Part 1 and Part 2
If you haven’t already, get all the code for this project

Conclusion

Protocol Buffers are quite versatile and convenient. Using them with Bluetooth Low Energy service is one way that you can take advantage. As the connected world of IoT continues to expand, I have no doubt that we'll be seeing more use cases in the future!

I hope this has inspired you to incorporate Protocol Buffers into your own project(s). Make sure you download the sample code and get started right now.

Has this tutorial been useful for you? Let me know what you think.

DEV Community: Jared Wolff

Create an Open Source Powered Air Quality Dashboard

Assembling hardware

Setting up a device

Set Up Python Virtual Environment

Install west

Initialize and update project

Install remaining python dependencies

Install toolchain

Get credentials

Add credentials to sample code

Build and Deploy

Setting up Grafana

API keys for Golioth

Setting up your data source

Creating your dashboard

Conclusion

How to Connect the nRF9160 Feather to Mosquitto

Where to host?

Important Extra step

Install Mosquitto

Provision Certs

Firmware bits

Program at_client first

Add certs to device

Using the mqtt_simple example

Sending a message

Conclusion

My Latest Self Hosted Hugo Workflow

Where to host?

Setting up your FreeBSD server with Bastille

Make building and deploying easy

Incremental Backup

Setting it up

Backing Up

Viewing your backups

Restoring

Conclusion

Meet Pyrinas a New Way to Use Your Xenon

What Pyrinas is

What Pyrinas isn't

Things you'll need

Getting started with an example

Setting up the sensor node repository

Basic sensor node code

Flashing the basic sensor code:

Setting up the hub repository

What does the future hold for Pyrinas?

Star and Watch!

How to Develop Particle Iot Apps Using NativeScript

App Frameworks

Available Libraries & SDKs

Making API Calls

Getting a Token

Through the command line

Through the API itself

Server Sent What?

SSE Library

Other Examples

More resources

Conclusion

Better Battery Life for Delightful Particle Deployments

The Noodling Begins

Assembling

Sleepy Time

Stop mode sleep

Deep sleep

Measurements

Bypass Bypass Bypass

Research

Make it Smart

Mysterious Restarts

The Enclosure

The Finished Product?

Conclusion

Resources

How to Presence Tracking Using Particle Argon

Initial investigation

Advertising

Logging

Install `west`

Program `at_client` first