DEV Community: Forgacs Peter

Decoding sensor measurements from hex payloads using Node.js

Forgacs Peter — Sun, 15 Dec 2019 18:45:44 +0000

In recent weeks I was decoding sensor measurements. The measurements were coming from various sensors. The payloads of the sensors were in hex format.

The first step of the process is to collect the sensor's documentation. The documentations usually contain a description of the payload format.

An example payload might look like this:

Pattern:

AABBCCCCDD

Concrete example:

919F003D01

Where:

A - A unique identifier for the sensor model
B - A unique identifier for the message type
C - Temperature measurement
D - Air pressure measurement

Details of each portion:

A - 1 Byte, Unsigned
B - 1 Byte, Unsigned
C - 2 Bytes, Unsigned, Big endian, Celsius
D - 1 Byte, Unsigned, Atm

Some details can change between different portions of the payload.

Size

The payloads are usually in hex format. As a rule of thumb, two characters in the hex format represent 1 Byte, aka. 8 bits.

Signedness

This {signedness} determines what ranges of values can be represented with a certain number of bytes. Usually, if a number is signed its actively mentioned in the documentation, otherwise you can assume it’s unsigned.

Endianness

The [endianness] determines how the bytes should be ordered. Either from left to right or right to left. If not explicitly stated in the documentation it usually means big-endian. If a portion is only 1-byte long the endianness does not matter, since endianness means byte ordering.

Unit of measurement

In the case of measurements, the documentation must specify the unit of measurement it uses.

This is usually not a single unit, instead a portion of an unit. For example: 1 / 16 of a degree Celsius.

This ratio is basically the resolution of the sensor. In this case the sensor can sense temperature difference in 0.0625 increments.

Node.js implementation

There is a great package called binary-parser that can handle binary data elegantly.

It can streamline endianness, signedness and much more. The input of the parser is Buffer so first you must convert your hex string. The output is the parsed object.

const Parser = require('binary-parser').Parser;

const sensorParser = new Parser()
  .uint8("modelId")
  .uint8("messageId")
  .uint16be("temperature")
  .uint8("airPressure")

const buffer = Buffer.from("919F003D01", "hex");
const measurements = sensorParser.parse(buffer);

This produces an object with the following format:

{
    modelId: 145,
    messageId: 159,
    temperature: 61, // in 1 / 16 C
    airPressure: 1   // int Atm
}

Formatters

We can handle unit of measurements with the built in formatters.

In our example the temperature is sent in 1 / 16 degree Celsius but we want to receive values as Celsius.

const temperatureFormatter = (temperature) => {
    return {
        temperature / 16; // Alternative tempearture * 0.0625
    }
}

const sensorParser = new Parser()
  .uint8("modelId")
  .uint8("messageId")
  .uint16be("temperature", { formatter: temperatureFormatter})
  .uint8("airPressure");

This produces:

{   
    modelId: 145,
    messageId: 159,
    temperature: 3.8125, // in C
    airPressure: 1   // int Atm
}

Variable length portions

Some payload formats have variable length internal portions.

AABBBBCC

Where

A : First value we need
B : A variable length portion that has no information for us
C : Second value we need
D : Third value we need

We can handle this situation with an offset and the seek method.

const variableLengthParser = (buffer) =>{
    const parser = new Parser()
    .uint8('first')
    .saveOffset('currentOffset')
    .seek(function() {
        const secondByteLength = 1;
        const thirdByteLength = 1;
        return { buffer.byteLength - this.currentOffset - ( secondByteLength + thirdByteLength )}
    })
    .uint8('second')
    .uint8('third');

    return parser.parse(buffer);
}

In this case, we need an encapsulating function that allows us to reference the buffer itself. After the first argument, the offset is saved. Then inside the seek function, the number of steps are calculated until the end of the variable-length portion.

For that, we need the total buffer length and sizes of portions coming after the variable-length portion.

Skipping bits

Some payloads have bits that represent a certain state of the payload.

As an example, let's say the 1st bit of the 2nd byte is a special signal bit we need.

new Parser
.uint8()
.bit1('specialBit')

One potential problem if we need to get the first and third bit of an 2 Byte portion that is big-endian.

Since big endian has reverse byte order we need to get the bits from the end:

new Parser
.bit13()
.bit1('third')
.bit1()
.bit1('first')

Slicing

Some payload formats contain both hex and ascii portions.

Example

3D01

Where the first two characters are the hex representation of the number 61 and the second two characters literally represent 1.

In these cases splicing the string is the best option we have.

Multiple units of measurement

If we have multiple sensors each sending measurements in different units we need to convert them into a single unit.

We can use the convert-units package and write a utility function to handle this.

const temperatureConverter = ({ unit: currentUnit , value }) => {
    const temperatureUnit = 'c';
    if (convert().from(currentUnit).possibilities().includes(temperatureUnit){
    return convert(value).from(currentUnit).to(temperatureUnit)
  } else {
        return value;
    }
}

const measurements = { temperature { unit: 'K', value: 273.15 }};
const temperatureInCelsius = temperatureConverter(measurements.temperature)

The temperatureConverter takes in a unit as a parameter. Checks if it’s possible to convert it to the selected temperature unit (C). Finally, if it’s possible returns the converted value.

Useful resources

Cryptii

Benchmarking Javascript

Forgacs Peter — Sat, 02 Nov 2019 23:17:39 +0000

In the last few weeks, I've been quite active on coding challenge sites such as Code Wars, Hacker Rank, and Code signal.

After solving a problem, you can check out the most popular solutions.

Seeing how another person solved a particular problem is exciting and allows you to grow as a developer. But if you look through the most upvoted solutions, you can see a pretty concerning pattern.

Usually, the most upvoted programming challenge answers are not the fastest ones but the shortest ones.

Since these sites do not show the performance of your code, I've used Benchmark.js and Jsperf to see the actual performance of some of the top voted solutions and methods that are frequently reoccurring.

Benchmark.js runs each function multiple times and then returns the operations per second value. The bigger the value, the faster the function.

Creating an Array from a String

As the first example, let's look at reversing a string.

Here are 3 different methods of doing the same thing.

// version 1

"hello".split("").reverse();

// version 2

[..."hello"].reverse();

// version 3

Array.from("hello").reverse();

In this example, the string hello is turned into a character array then reversed.

Let's create a benchmark with random length arrays in Node.js.

const { Benchmark } = require("benchmark");

function generateRandomString(length) {
  var result = "";
  var characters =
    "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
  var charactersLength = characters.length;
  for (var i = 0; i < length; i++) {
    result += characters.charAt(Math.floor(Math.random() * charactersLength));
  }
  return result;
}

function generateRandomNumberBetween(min, max){
    return Math.floor(Math.random() * max) + min;
}

// data contains 100 random strings with lneght between 1 and 1000
const data = [];
for (let i = 0; i < 100; i++) {
    data.push(generateRandomString(generateRandomNumberBetween(1, 1000)));
}

const suite = new Benchmark.Suite();
suite.add("string.split()", function() {
    for (const str of data) {
        str.split("");
    }
});
suite.add("Object spread", function () {
    for (const str of data) {
        [...str];
    }
});
suite.add("Array.from()", function() {
    for (const str of data) {
        Array.from(str);
    }
});
suite.on("cycle", function(event) {
    console.log(String(event.target));
});
suite.on("complete", function() {
    console.log("Fastest is " + this.filter("fastest").map("name"));
})
suite.run();

Result:

string.split() x 7,777 ops/sec ±16.99% (89 runs sampled)
Object spread x 284 ops/sec ±2.89% (77 runs sampled)
Array.from() x 138 ops/sec ±1.48% (75 runs sampled)
Fastest is string.split()

As you can see, the simple split method is significantly faster than the Object spread method.

The test ran in Node.js on my laptop, but we can create a Jsperf test to validate the results in different browsers.
You can check out the test here.

Flattening an Array

let arr = [ [1, 2], [3, 4]];

// version 1

arr.reduce((acc, curr) => acc.concat(curr), []);

// version 2

arr.reduce((acc, curr) => [...acc, ...curr], []);

// version 3

[].concat(...arr);

Let's compare them.

const { Benchmark } = require("benchmark");

function generateRandomNumberBetween(min, max){
    return Math.floor(Math.random() * max) + min;
}

function generateTupleArray(length) {
    const tupleArray = [];
    for (let i = 0; i < length; i++) {
        tupleArray.push([generateRandomNumberBetween(1, 1e3), generateRandomNumberBetween(1, 1e3)]);
    }
    return tupleArray;
}

// Contains 100 arrays containing elements between 1 and 1000
const arrays = [];
for (let i = 0; i < 100; i++) {
    arrays.push(generateTupleArray(generateRandomNumberBetween(1, 1e3)))
}


const suite = new Benchmark.Suite();
suite.add("acc.concat(curr)", function() {
    for (const arr of arrays) {
        arr.reduce((acc, curr) => acc.concat(curr), []);
    }
});
suite.add("[...acc, ...curr]", function () {
    for (const arr of arrays) {
        arr.reduce((acc, curr) => [...acc, ...curr], []);
    }
});
suite.add("[].concat(...data)", function() {
    for (const arr of arrays) {
        [].concat(...arr);
    }
});
suite.on("cycle", function(event) {
    console.log(String(event.target));
});
suite.on("complete", function() {
    console.log("Fastest is " + this.filter("fastest").map("name"));
})
suite.run();

Result:

acc.concat(curr) x 11.13 ops/sec ±1.90% (32 runs sampled)
[...acc, ...curr] x 0.48 ops/sec ±9.61% (6 runs sampled)
[].concat(...data) x 442 ops/sec ±1.25% (85 runs sampled)
Fastest is [].concat(...data)

We can see that the fastest method is orders of magnitudes faster than the second one.

Partition problem

As a final example I've created a benchmark for the top 10 most upvoted answers for a particular problem.

You can check out the problem statement here

Result:

#1  x 4,288 ops/sec ±1.15% (87 runs sampled)
#2  x 11,726,715 ops/sec ±0.90% (88 runs sampled)
#3  x 25,793 ops/sec ±1.00% (88 runs sampled)
#4  x 15,749 ops/sec ±1.07% (86 runs sampled)
#5  x 144 ops/sec ±1.09% (79 runs sampled)
#6  x 8,761 ops/sec ±1.26% (86 runs sampled)
#7  x 1,021 ops/sec ±1.16% (84 runs sampled)
#8  x 4,574 ops/sec ±0.95% (88 runs sampled)
#9  x 181,853 ops/sec ±12.23% (81 runs sampled)
#10 x 20,143 ops/sec ±1.03% (83 runs sampled)
Fastest is #2

As you can see, the #1 most popular solution is one of the slowest.

Conclusion

Sometimes seemingly hacky solutions are viable. Other times they have similar or in worse cases, slower performance than standard solutions. Performance can vary between environments.

But one thing is for sure. If you want to make decisions based on data, you must know the tools to benchmark your code.