DEV Community

Jeroen van der Heijden
Jeroen van der Heijden

Posted on

🚀 Optimizing Meta Data Retrieval with ThingsDB 1.7.6

#thingsdb #infrasonar #programming #networktraffic

We at InfraSonar query large volumes of meta data, resulting in many properties that are often empty lists, false booleans, or empty strings.
This redundant data increases payload size and network traffic unnecessarily.

ThingsDB version 1.7.6 introduces new wrap-prefix flags that solve this by allowing us to exclude properties from the wrapped output if they evaluate to a "false" state.

The Problem: Bloated JSON Payloads

Consider our Metric type definition (partial, in reality it looks slightly different):

set_type('Metric', {
    id: '#',                // the metric Id as "id"
    key: 'str<1:>',
    reference: '&Ref?',     // nil for most
    description: 'str',     // optional description
    dataType: 'DataType',
    isRequired: 'bool',
    isFileId: 'bool',       // false for most
    displayFunction: 'int', // mapping to a display function
    derived: '&[Derived]',  // list of derived metrics, empty for most
});
Enter fullscreen mode Exit fullscreen mode

A standard Metric(123456).wrap() call often returns a large object dominated by default or empty values:

(JSON)

{
    "id": 123456,
    "key": "name",
    "reference": null,
    "description": "",
    "dataType": "STRING",
    "isRequired": true,
    "isFileId": false,
    "displayFunction": 0,
    "derived": []
}
Enter fullscreen mode Exit fullscreen mode

The Solution: The ! and ? Wrap-Prefix Flags

The wrap-prefix flags enable a concise way to control data serialization based on truthiness:

  • ! - Excludes the property if its value evaluates to false (e.g., false, 0, "", []).
  • ? - Excludes the property only if its value is explicitly nil.

Applying the Flags

We update the Metric definition by prefixing properties we want to exclude when empty or false:

set_type('Metric', {
    id: '#',
    key: 'str<1:>',
    reference: '?&Ref?',     // Exclude when explicitly `nil`.
    description: '!str',     // Exclude when false (empty string).
    dataType: 'DataType',
    isRequired: '!bool',     // Exclude when false.
    isFileId: '!bool',       // Exclude when false.
    displayFunction: '!int', // Exclude when false (value is 0).
    derived: '!&[Derived]',  // Exclude when false (empty list).
});
Enter fullscreen mode Exit fullscreen mode

The Compact Result

Running the same query, Metric(123456).wrap(), now yields a significantly more compact result:

(JSON)

{
    "id": 123456,
    "key": "name",
    "dataType": "STRING",
    "isRequired": true
}
Enter fullscreen mode Exit fullscreen mode

By leveraging the ! and ? flags, we eliminate unnecessary data transfer, resulting in significantly reduced network traffic and faster query response times when dealing with meta data objects.

Top comments (1)

Collapse
 
sasha_vanderheijden_400 profile image
Sasha Van der Heijden •

This is a fantastic application of the new wrap-prefix flags! Leveraging ! and ? to minimize JSON payload and network traffic for sparse meta data is a brilliant optimization. Great work, InfraSonar and ThingsDB team!