Tutorial part 4: key-value store - data streams

Welcome to this part-4 of the key-value database tutorial for codehooks.io serverless Node.js backend. In this part we'll focus on how to work with multiple key-value pairs in a key-value database.

The Codehooks.io key-value database stores all keys as a sorted string index. This is a very useful feature if you need to store and retrieve multiple values together, as a group or as a segment of key-values.

This article is also avaliable at the official codehooks.io website

IoT device - time series data example

To illustrate this feature we'll create an example use-case for capturing data from multiple IoT devices. In this example, each IoT device sends its location, identification and observations to a specific REST API network endpoint.

For example, a temperature sensor device in the "Alpha region West" reports its value each minute. And a wind force sensor in the "Beta region West" reports its values hourly, and so on.

To ensure that we can access and process the observation data sets effectively, we'll create a key string combination that captures all necessary information in one unique key-value pair.

To illustrate this concept lets imagine a series of IoT device observations as shown in the example below.

'/Alpha/West/density/2023-01-09T10:14:00.495Z'  => '4.4191'
'/Alpha/West/humidity/2023-01-09T17:01:00.821Z' => '0.7539'
'/Alpha/West/temp/2023-01-09T21:15:00.450Z'     => '87.100'
'/Beta/West/wind/2023-01-19T21:57:00.316Z'      => '5.9477'
...

Now that we understand the concept for our IoT database, lets begin the development of an example backend API for IoT sensors.

We'll create a serverless REST API endpoint to receive observations from many IoT devices. The API will store each observation with a logic key that effectively creates a time series database with data points that we can explore and use in our application.

Lets get started with creating the first API to receive data observations from a device.

REST API endpoint to POST a single observation

The REST API route takes in 3 parameters for a given device, region, location and device.
And the POST body must contain the actual device observation data point.
For example:

POST /observation/North/Plaza/humid03 

{"observation": "4.02"}

The route parameters are combined into a unique key with a time stamp which gives a correct time series for the data points. The following table shows some example device observations.

IoT devices time series	Data points
/East/BuilingA/temp01/2023-02-22T18:30:14.441Z	21.4
/North/Plaza01/humid3/2023-02-23T05:23:11.306Z	12.5
/East/BuilingB/temp02/2023-02-22T18:32:38.991Z	19.0

The serverless function below implements a REST API endpoint route that can receive data from a device and insert it to the key-value database.

app.post('/observation/:region/:location/:device', async (req, res) => {
    const {region, location, device} = req.params;
    const {observation} = req.body;    
    const key = `/${region}/${location}/${device}/${new Date().toISOString()}`;
    // string key e.g. '/North/Plaza/humid03/2023-02-23T05:23:11.306Z'
    const conn = await Datastore.open();
    const result = await conn.set(key, observation);
    console.log(result)
    res.status(201).end('Data received ok');
})

Test the REST API with curl.

curl -X POST \
  'https://<YOUR_DATABASE_ID>.api.codehooks.io/dev/observation/East/BuilingA/temp01' \
  --header 'x-apikey: <YOUR_API_TOKEN>' \
  --header 'Content-Type: application/json' \
  --data-raw '{
  "observation": "21.4"
}'

Use the CLI command coho info --examples to inspect your project API endpoint and Curl examples.

Inspecting the key-value database with the CLI

The Codehooks CLI can be used to manage the key-value database basic operations, get, set and del. For example, to retrieve all observations for the East locations you can run the following command:

coho get '/East*'

This will list all matching keys in the key-value database as the example output shows below.

{
  key: '/East/BuilingA/temp01/2023-02-23T05:48:19.646Z',
  value: '21.4'
}
{
  key: '/East/BuilingA/temp01/2023-02-23T05:49:06.614Z',
  value: '20.0'
}

2 keys

Endpoint to GET observations subkey

Similiar to the CLI example above, we can use the database API getAll method to retrieve all key-value pairs that match a start segment of a particular key. In this example we are producing CSV data format for easy spreadsheet analysis etc.

app.get('/observations/:prefix', async (req, res) => {
    const conn = await Datastore.open();
    const {prefix} = req.params;
    let count = 0;
    res.set('content-type', 'text/csv');
    res.write('Device, Data\n');
    const stream = conn.getAll(`/${prefix}`);
    stream.on('data', (data) => {
        const outString = `"${data.key}", ${data.val}\n`;
        count++
        res.write(outString);
    }).on('end', () => {
        res.end();        
    })
})

Testing the API with curl again.

curl -X GET \
  'https://<YOUR_DATABASE_ID>.api.codehooks.io/dev/observations/Alpha' \
  --header 'x-apikey: <YOUR_API_TOKEN>' \

Device, Data
"/Alpha/Center/decibel/2023-01-09T11:11:31.218Z", 2.3428
"/Alpha/Center/decibel/2023-01-09T11:12:01.050Z", 6.0632
"/Alpha/Center/decibel/2023-01-09T13:13:30.541Z", 0.7196
"/Alpha/Center/decibel/2023-01-09T15:23:00.589Z", 9.7232
"/Alpha/Center/decibel/2023-01-09T15:34:00.520Z", 5.0089
"/Alpha/Center/decibel/2023-01-09T17:04:00.942Z", 9.1861

Complete source code

The complete source code for our IoT backend application is shown below.

// index.js
import { app, Datastore } from 'codehooks-js' // Standard JS lib for express style code

// list a serie of IoT device observations
app.get('/observations/:prefix', async (req, res) => {
    const conn = await Datastore.open();
    const {prefix} = req.params;
    let count = 0;
    res.set('content-type', 'text/csv');
    res.write('Device, Data\n');
    const stream = conn.getAll(`/${prefix}`);
    stream.on('data', (data) => {
        const outString = `"${data.key}", ${data.val}\n`;
        count++
        res.write(outString);
    }).on('end', () => {
        res.write(`Result: ${count}`);
        res.end();        
    })
})

// register a new IoT device observation
app.post('/observation/:region/:location/:device', async (req, res) => {
    const {region, location, device} = req.params;
    const {observation} = req.body;
    // string key e.g. '/North/Plaza/humid03/2023-02-23T05:23:11.306Z'
    const key = `/${region}/${location}/${device}/${new Date().toISOString()}`;
    const conn = await Datastore.open();
    const result = await conn.set(key, observation);
    console.log(result)
    res.status(201).end('Data received ok');
})

export default app.init(); // Bind functions to the serverless cloud

This part-4 of the key-value store tutorial has shown how you can combine smart keys with the database streaming capabilities to create kind of IoT time series database with codehooks.io.

In our next tutorial [Part-5: Managing data with TTL options(https://codehooks.io/docs/tutorials/key-val-store/part-5-managing-data-with-ttl-options), we'll explore how you can use the time-to-live (TTL) feature to delete old data in the key-value database.