Optimizing ThingsDB: Speeding up with Type Indexes

In the latest release of ThingsDB, we’ve introduced a way to significantly accelerate the type_all() function. By enabling type indexing, you can now loop over all instances of a specific type with near-instant results.

The "Ideal" Way: Hierarchical Access

In ThingsDB, the preferred pattern for accessing data is through your own code logic. At InfraSonar, for example, we use a hierarchical structure for our Container type. Every container can have children, and we retrieve the entire tree starting from a .root container.

new_type('Container');
set_type('Container', {
    name: 'str',
    children: '{Container}',
    get_containers: |this| {
        // Recursively collect all containers in a set
        this.children.reduce(|a, c| a |= c.get_containers(), set(this));
    }
});

// Return a set with all containers via the root:
.root.get_containers();

The Problem: When type_all() is slow

There are scenarios where data isn't easily reachable through a single root, or where you simply need a flat list of every instance of a type. For this, ThingsDB provides type_all().

Historically, type_all() was designed for debugging and one-off migrations, not for high-frequency production code. This is because, internally, ThingsDB had to perform a full scan—looping through every single object in the collection to filter those that matched the requested type.

If you measure the performance, the difference is noticeable:

// Returns an object with `data` (the set) and `time` (execution time)
timeit(type_all('Container'));

Without Index (Full Scan):

{
    "data": [..],
    "time": 0.008419966
}

The Solution: Enabling the Type Index

You can now tell ThingsDB to maintain a dedicated index for a specific type. This moves the effort from the query time to the storage time, making retrievals incredibly fast.

You can enable this on an existing type:

// Enable the 'idx' flag on the Container type
mod_type('Container', 'idx', true);

Or, define it immediately upon creation using the IDX flag:

new_type('Container', IDX);

Performance Impact

Once indexed, running type_all('Container') no longer requires a full scan. ThingsDB simply returns the pre-populated set of instances.

With Type Index enabled:

{
    "data": [..],
    "time": 0.00014754
}

By enabling the index, the execution time dropped from 8.4 milliseconds to just 147 microseconds. This is a 57x speed improvement.

Note: You might need to run the query twice to see the full effect. ThingsDB is "lazy" and typically starts building the index upon the first request after the flag is enabled.

Summary

While traversing your own data structures (like our .root example) remains the most "ThingsDB-native" way to work, the new IDX flag makes type_all() a viable and high-performance tool for your production toolset.