DEV Community: Jeroen van der Heijden

Distributed Locking in Python

Jeroen van der Heijden — Mon, 23 Feb 2026 07:39:08 +0000

When making use of Python's asyncio library, synchronizing code within a single process is a solved problem. A simple asyncio.Lock() ensures that coroutines play nice together. But as soon as your application scales to multiple containers, servers, or microservices, that local lock becomes invisible to the rest of your fleet.

You suddenly face the challenge of Distributed Mutual Exclusion: ensuring that a critical section of code is only executed by one worker at a time, regardless of which machine it is running on.

In this post, we’ll explore a robust distributed lock implementation using ThingsDB. This solution provides the familiar syntax of an asyncio context manager but leverages a globally synchronized backend to manage state across your entire infrastructure.

Why Distributed Locking is a "Hard" Problem

On the surface, it sounds easy: just set a flag in a database. But in a distributed environment, the stakes are higher. If a worker acquires a lock and then crashes or loses network connectivity, that lock could remain "held" in the database forever, creating a permanent deadlock.

ThingsDB solves this by introducing a server-side timeout. If the client holding the lock fails to release it or disappears, ThingsDB automatically releases the lock for the next person in line.

Unlike traditional distributed locks that require "polling" the database, ThingsDB locks are event-driven. They leverage ThingsDB’s emit event system, meaning there is zero wasted overhead. The moment a lock is released, an event triggers the next waiter in line for immediate execution. This ensures maximum throughput while strictly guaranteeing that only one worker is active at a time.

Part 1: The Infrastructure Setup

Before your application can start locking resources, ThingsDB needs to prepare the underlying collection logic. This is an idempotent operation—it only does work the first time it is called.

In a production environment, you should perform this setup during your application's bootstrap phase.

from thingsdb.misc import lock

async def initialize_infrastructure(client):
    # Connect and authenticate
    await client.connect('localhost')
    await client.authenticate('admin', 'pass')

    # This prepares the lock collection in ThingsDB.
    # It’s safe to call every time the service starts.
    await lock.setup(client)

Part 2: The "Initialize Once" Pattern

In a real-world project, you want to avoid creating new lock objects inside your high-frequency loops. Instead, initialize your lock configuration once and reuse that object throughout the lifecycle of your application.

By using functools.partial, we can create a "Lock Factory" that is pre-configured with the correct client, resource name, and timeout settings.

from functools import partial
from thingsdb.misc import lock

# This will hold our pre-configured lock factory
invoice_lock = None

async def setup_application_locks(client):
    global invoice_lock

    # Perform the one-time backend setup
    await lock.setup(client)

    # Initialize the lock object once.
    # If a worker crashes, the lock is freed after 30 seconds.
    invoice_lock = partial(lock.lock,
                           client=client,
                           name='process-invoices',
                           timeout=30)

Note: You can create as many different locks as your application needs. Just ensure that each unique resource or critical section is given its own unique name to avoid unintended interference between different parts of your system.

Part 3: Using the Lock in Your Business Logic

Because the implementation uses the async withsyntax, it integrates seamlessly into any asyncio application. It feels exactly like a local lock, but it’s protecting your data across the globe.

async def process_invoices():
    # Use the pre-configured lock we created during startup
    async with invoice_lock():
        print('Distributed lock acquired! '
              'No other service can enter this block.')

        # Simulate critical work like hitting a payment API
        await asyncio.sleep(5.0)

        print('Work finished. Lock is automatically released.')

Monitoring and Safety

Sometimes you need to know if a resource is busy without actually waiting in line. You can use the locked() function for a non-blocking check:

is_busy = await lock.locked(client, 'process-invoices')
if is_busy:
    print("Another worker is currently processing invoices.")

Key Takeaways

Queue Management: The queue_size and timeout parameters allow you to control exactly how long a request should wait before giving up, preventing "zombie" requests from clogging your system.
Resilience: The server-side timeout ensures that your system is self-healing even in the face of hardware failures or process crashes.
Developer Experience: By mimicking the asyncio.Lock() interface, the learning curve for your team is practically zero.

Conclusion

Distributing your Python application shouldn't mean sacrificing the safety of mutual exclusion. By using ThingsDB to back your locks, you get a high-performance, self-healing synchronization layer that scales with your infrastructure.

Optimizing ThingsDB: Speeding up with Type Indexes

Jeroen van der Heijden — Fri, 20 Feb 2026 10:34:59 +0000

In the latest release of ThingsDB, we’ve introduced a way to significantly accelerate the type_all() function. By enabling type indexing, you can now loop over all instances of a specific type with near-instant results.

The "Ideal" Way: Hierarchical Access

In ThingsDB, the preferred pattern for accessing data is through your own code logic. At InfraSonar, for example, we use a hierarchical structure for our Container type. Every container can have children, and we retrieve the entire tree starting from a .root container.

new_type('Container');
set_type('Container', {
    name: 'str',
    children: '{Container}',
    get_containers: |this| {
        // Recursively collect all containers in a set
        this.children.reduce(|a, c| a |= c.get_containers(), set(this));
    }
});

// Return a set with all containers via the root:
.root.get_containers();

The Problem: When `type_all()` is slow

There are scenarios where data isn't easily reachable through a single root, or where you simply need a flat list of every instance of a type. For this, ThingsDB provides type_all().

Historically, type_all() was designed for debugging and one-off migrations, not for high-frequency production code. This is because, internally, ThingsDB had to perform a full scan—looping through every single object in the collection to filter those that matched the requested type.

If you measure the performance, the difference is noticeable:

// Returns an object with `data` (the set) and `time` (execution time)
timeit(type_all('Container'));

Without Index (Full Scan):

{
    "data": [..],
    "time": 0.008419966
}

The Solution: Enabling the Type Index

You can now tell ThingsDB to maintain a dedicated index for a specific type. This moves the effort from the query time to the storage time, making retrievals incredibly fast.

You can enable this on an existing type:

// Enable the 'idx' flag on the Container type
mod_type('Container', 'idx', true);

Or, define it immediately upon creation using the IDX flag:

new_type('Container', IDX);

Performance Impact

Once indexed, running type_all('Container') no longer requires a full scan. ThingsDB simply returns the pre-populated set of instances.

With Type Index enabled:

{
    "data": [..],
    "time": 0.00014754
}

By enabling the index, the execution time dropped from 8.4 milliseconds to just 147 microseconds. This is a 57x speed improvement.

Note: You might need to run the query twice to see the full effect. ThingsDB is "lazy" and typically starts building the index upon the first request after the flag is enabled.

Summary

While traversing your own data structures (like our .root example) remains the most "ThingsDB-native" way to work, the new IDX flag makes type_all() a viable and high-performance tool for your production toolset.

Understanding the core mechanics of ThingsDB

Jeroen van der Heijden — Mon, 22 Dec 2025 15:46:38 +0000

In my previous post, I promised to dive into the client side. However, to truly appreciate how the client interacts with the system, we first need to pull back the curtain on what happens "under the hood" when you execute a query or run a procedure in ThingsDB.

Step 1: The Gateway and "AWAY" Mode

When you send a query to a ThingsDB node, the first thing it does is check its own status. If the node is currently in "AWAY" mode (a specialized state we’ll explore in a moment), it won't process your request locally. Instead, it acts as a traffic controller, forwarding the query to another active node in the cluster. This ensures that the client never experiences a "blocked" connection.

Step 2: To Compile or Not to Compile?

Once a node accepts the query, it checks its internal cache. If ThingsDB recognizes the exact same query from earlier, it can skip the compilation phase and use a pre-compiled version from the cache.

If the query is new, ThingsDB begins the compilation process. During this phase, the engine determines a critical factor: Does this query require a "change"?

Read-only queries: If no data changes are required, the query executes immediately. No other nodes are involved, making read operations incredibly fast.
Procedures: This is where procedures shine. Unlike raw queries, procedures are always pre-compiled. ThingsDB knows instantly whether they require a change or not, shaving off valuable microseconds.

Step 3: The Race for Consensus (The Change ID)

If a query needs to modify data, ThingsDB initiates a "battle" for the next Change ID. It looks at the current global Change ID and proposes the next increment (Current + 1).

Through a quorum-based consensus, if the majority of nodes agree on this ID, the change is processed. All modifications to the collection are tied to this specific ID and synchronized across the cluster. This mechanism guarantees that every node handles changes in the exact same order, maintaining perfect data consistency.

Step 4: Futures and Non-Blocking Changes

ThingsDB handles complex logic using Futures. If a query contains one or multiple futures that require changes, each future can receive its own Change ID. This is a powerful optimization! It prevents the system from locking up during external module calls and allows you to isolate heavy "change" logic from the rest of your procedure or query.

Strategy Tip: If you have a procedure that is mostly read-only but occasionally needs to write data, wrap the "write" portion in a future. This prevents the entire procedure from generating a global change every time it's called (see this example from the ThingsDB book).

Step 5: Storage and The "Boot" Sequence

ThingsDB uses a dual-mechanism for data persistence:

Full State Dumps: A snapshot of the entire data state at a specific point in time.
Archive Files: A continuous log of every individual change.

When a node boots up, it follows a strict recovery path:

It loads the last Full State Dump.
It identifies the last Change ID from that dump and then replays all subsequent changes from the Archive Files.
It then waits for a peer node to enter Away Mode. The booted node reports its last processed Change ID, and the "Away" node syncs the missing delta.

If a node is too far out of sync (or is a brand-new node being added to the cluster), a Full Synchronization occurs, where the entire state dump is transferred from scratch.

Step 6: The Magic of "AWAY" Mode

"Away" mode is ThingsDB’s secret weapon for maintenance without downtime. While a node is "Away," it performs heavy lifting such as:

Full State Dumps (Snapshots).
Garbage Collection (Memory management).
Scheduled Backups.
Synchronizing peer nodes.

During this time, the node still talks to clients, but it forwards their queries to keep the system responsive. Changes are collected in the background and applied just before the node leaves Away mode and rejoins the cluster. ThingsDB ensures that only one node at a time is in this mode, keeping the rest of the cluster available for primary tasks.

Conclusion

By decoupling blocking maintenance tasks from client requests and using a sophisticated consensus for changes, ThingsDB provides a database experience that is both high-performance and incredibly resilient.

Now that you understand the "brain" of ThingsDB, we are finally ready to look at the Client Side in my next post!

Schema-First Approach with ThingsDB

Jeroen van der Heijden — Fri, 12 Dec 2025 06:45:53 +0000

In many architectures, the separation between database and application code introduces a dangerous gap where data integrity can fail. The application code is responsible for validation, but what if multiple services access the same data?

This post explores a different approach. We'll show you how to leverage ThingsDB's core features—Typed Collections, Procedures, and Rooms—to make the database itself the enforcing authority.

Follow along as we set up a secure, event-driven API for an Order application, demonstrating how to lock down access and ensure every data change is both valid and consistent.

1. Setting the Stage: The Blank Slate

We begin by creating a fresh collection for our application.

(execute the code snippet in the /thingsdb scope)

new_collection('OrderApp');

2. Understanding Typed Collections (The Core Defense)

The first step toward protection is establishing a schema. By default, a new ThingsDB collection starts with an empty, unrestricted object (a basic "thing") at its root. Any user with write access can add anything to it.

We want to lock this down. We create a type, App, to serve as the structured blueprint for the collection's entire existence.

(execute the code snippet in the //OrderApp scope)

// 1. Create a type for the root of my collection
new_type('App');

// 2. Convert the collection root to the 'App' type
.to_type('App');

Why this matters: The collection is now restricted by the App type. No data can be added or modified unless the modification conforms to changes made to the App type definition. This is your primary schema defense.

3. Defining the Data Model

Next, we define the structure of the data we want to store—the Order object—and where to keep it within the App root.

(execute the code snippet in the //OrderApp scope)

// Create an enumerator for consistent order status values
set_enum('OrderStatus', {
    OPEN: 'Open',
    PENDING: 'Pending',
    CLOSED: 'Closed',
});

// Define the Order structure
set_type('Order', {
    id: '#',               // Return ID as `id` when wrapped
    name: 'str',
    price: 'float',
    status: 'OrderStatus', // Enforcing status consistency
    created: 'datetime',
});

// Update the App type: Add a property 'orders' as an array of Order objects
mod_type('App', 'add', 'orders', '[Order]');

The mod_type(..) call automatically creates the orders property as an empty list on the collection root, ready to hold our data.

4. Establishing an Event Channel

For a modern microservice or component-based application, polling for data changes is inefficient. We set up an event channel using a Room to enable real-time communication.

(execute the code snippet in the //OrderApp scope)

// Add a room property to the App type for event handling
mod_type('App', 'add', 'order_events', 'room');

// For easy identification, we give the room a fixed name
.order_events.set_name('api_order_events');

Benefit: Any component can now join this room and instantly react to events like new-order or change-order-status without needing to constantly query the collection.

5. Encapsulating Logic with Procedures (The API Layer)

To guarantee that only valid changes are made to the collection, we expose all actions only through Procedures. We prefix these procedures with api_ for easy security whitelisting later.

(execute the code snippet in the //OrderApp scope)

// 1. Add a new order (with validation on input types)
new_procedure('api_order_add', |name, price| {
    order = Order{
        name:,
        price:,
    };
    .orders.push(order);

    // Emit the 'new-order' event for all listeners
    .order_events.emit('new-order', order.wrap());
    nil;
});

// 2. Change order status
new_procedure('api_order_set_status', |order_id, status| {
    // Input validation: ensures the new status is a valid enum value
    new_status = OrderStatus{||status.upper()};

    // Access and update the specific Order object
    Order(order_id).status = new_status;

    // Emit the 'change-order-status' event
    .order_events.emit('change-order-status', order_id, new_status.value());
    nil;
});

// 3. Retrieve orders by status
new_procedure('api_orders_by_status', |status| {
    order_status = OrderStatus{||status.upper()};
    .orders.filter(|order| order.status == order_status).map_wrap();
});

6. Strict Security: The Locked-Down API User

The final and most crucial step is to create a user that can only execute these specific procedures. This prevents any external service from bypassing your defined business logic.

(execute the code snippet in the /thingsdb scope)

// Create a new user for microservice access
new_user('api');

// Grant access to the collection:
// CHANGE (to modify data), JOIN (to listen to events), RUN (to execute procedures)
grant('//OrderApi', 'api', CHANGE|JOIN|RUN);

// Whitelist: This is the security lock.
// The user can only execute procedures and join rooms starting with 'api_'
whitelist_add('api', 'procedures', /^api_.*/);
whitelist_add('api', 'rooms', /^api_.*/);

// Generate and return the authentication token
new_token('api');

With this setup, the api user is completely restricted. It cannot arbitrarily delete the collection, change its structure, or modify data without going through the validated input and logic defined in your api_ procedures. Your collection is now truly protected by code!

In the next post, we will switch gears to the client side. We will demonstrate how a microservice uses the generated token to connect to the OrderApp collection, how to perform the authenticated api_ queries, and, crucially, how to "listen" to the real-time event changes emitted through the api_order_events room.

No Code in the Database... Right?

Jeroen van der Heijden — Wed, 10 Dec 2025 14:11:39 +0000

"No code in the database" is a mantra instilled in developers from computer science school onward. It's a fundamental principle of traditional database design aimed at separation of concerns and maintainability.

But what if your database IS your code? What if the data store not only saves your state but actively manages your business logic, acts as your API, and functions as your message bus?

The Traditional Problem: Framework Ownership

When you start a project using a traditional SQL database, frameworks like Django, Laravel, or Rails become the central authority.

These frameworks effectively "own" the database. They manage the schema through migrations and use an Object-Relational Mapping (ORM) layer to perform queries. This is why mixing major frameworks that touch the same core data is heavily discouraged—each framework wants to be the single source of truth for the database schema and application logic.

The ThingsDB Shift: The Database as the Hub

With a database like ThingsDB, the model fundamentally changes. Here, the database is your runtime environment, blurring the line between data storage and application logic.

This architecture is especially powerful if your project involves:

Polyglot Microservices: Using many different components or services, possibly written in different programming languages.
Decentralized Access: Needing all components to access the same, unified data source and logic hub.

ThingsDB steps into the role of the Data API and Message Bus, centralizing data management and coordination.

Guaranteeing Data State and Event-Driven Architecture

This integrated approach offers powerful mechanisms to guarantee data integrity and build truly reactive applications:

Strict Data Integrity with Typed Collections and Procedures: By defining procedures with strict input restrictions and leveraging fully typed collections, you guarantee the validity and consistent state of your data. The data cannot be modified in an inconsistent way because the logic lives right alongside it.
No More Polling with Rooms and Events: ThingsDB removes the need for wasteful polling across your services. Instead, you build truly event-driven applications using Rooms and Events. When data changes, all subscribing components are immediately notified, allowing your application to react instantly.

In short, while frameworks like Django and Laravel are perfect for applications where they serve as the central brain, ThingsDB enables a modern, event-driven architecture where the database is a powerful, active participant in your business logic, simplifying communication and guaranteeing data consistency across diverse application components. Choose the tool that best aligns with your project's long-term complexity and inter-service communication needs.

Event-Driven Updates with ThingsDB

Jeroen van der Heijden — Mon, 17 Nov 2025 18:00:34 +0000

In the world of real-time applications, delivering instantaneous updates to users is foundational. From collaborative documents to live chat applications, the ability to instantly reflect changes across all connected clients can make or break the user experience. ThingsDB, with its robust event-driven architecture, offers a powerful solution for achieving this.

Traditionally, implementing event-driven updates in a real-time application, like a chat room, often followed a pattern that could lead to unnecessary complexity. Let's break down the classic approach and then explore how ThingsDB's new "emit to peers only" feature simplifies this process.

Traditional Flow

Imagine a simple chat application. When a user sends a message, the desired flow is that the message appears instantly on the users screen, and via an event for everyone in the chat room.

Here's how that flow typically worked:

A chat client sends a new message to the ThingsDB server.
ThingsDB stores the message and then emits an event to a designated "room" (or topic) that all connected clients are listening to.
All clients subscribed to that room receive the event and display the new message in their respective chat interfaces.

This approach works, but it introduces a subtle challenge: the client that sent the message also receives the same event back.

Here's a visual representation of the traditional flow:

Self-Inflicted Updates

While seemingly harmless, receiving one's own message back as an event can create issues, especially if the client is designed to immediately display the message locally upon sending. This leads to a double-update scenario: the message appears instantly when sent, and then again when the event is received from the server.

To avoid this redundancy, developers often resorted to complex workarounds like implementing intricate code to detect if an incoming event originated from the current client, perhaps by including a unique client ID with each emitted event.

The ThingsDB Solution: Emitting to Peers Only

ThingsDB addresses this challenge elegantly with its new "emit to peers only" option. This powerful feature allows you to broadcast events to all listeners within a room except the one that initiated the event.

Here's how the simplified flow works:

A chat client sends a new message to ThingsDB, explicitly indicating that the event should only be emitted to other peers and displays the message in the chat screen.
ThingsDB stores the message and then emits the event to the designated room. Crucially, it filters out the originating client from the list of recipients.
All other clients subscribed to the room receive the event and display the new message.

Sender Updates Locally: The sending client, having already displayed the message locally, does not receive a redundant event from the server.

Let's visualize this improved flow:

Beyond Chat Rooms

The "emit to peers only" concept extends far beyond chat applications. Consider any scenario where a client initiates an action and immediately reflects that change locally, while also needing to inform other connected clients:

Collaborative Document Editing: When a user types a character, it appears instantly on their screen. Other collaborators see the update via an event, but the typing user doesn't need to process their own keystroke as an incoming event.
Real-time Dashboards: A user performs an action that updates a metric locally. Other users viewing the dashboard receive an event to update their views, but the initiating user's dashboard is already updated.
Gaming: A player moves their character. The local client updates the character's position instantly. Other players receive an event to update the character's position on their screens.

Conclusion

ThingsDB's "emit to peers only" feature is implemented by all supported clients and is addressing a common pain point in real-time application development. For any application built on the promise of instant updates, utilizing this feature means spending less time debugging logic and more time writing clean, purposeful code.

🚀 ThingsDB v1.8.2: Introducing Anonymous Types

Jeroen van der Heijden — Mon, 10 Nov 2025 16:05:51 +0000

Developers using ThingsDB (or other data solutions) often run into a common pattern: you have a defined, complex type, but when you query it, you only need a subset of its fields.

Example: If you have a User type with fields like name, email, and password_hash, you can't just return the entire thing to a client. Before v1.8.2, your options were:

Define a second, wrap-only type (e.g., _User) to exclude the sensitive fields.
Manually construct a map of the desired fields.

ThingsDB v1.8.2 introduces an new solution with Anonymous Types.

⚡ The Solution: Anonymous Type

Anonymous types allow you to define a temporary, ad-hoc type structure precisely where you need it, most often for defining the output of a query or a method call like wrap() or map_wrap().

The syntax is simple and powerful: the ampersand (&) prefix followed by a standard type definition.

Let's look at the classic use case—safely projecting data from a registered type.

// 1. Define the full type
set_type('User', {
    name: 'str',
    email: 'str',
    password_hash: 'str', // SENSITIVE FIELD
});

// 2. For simplicity, we create a list with a single User instance.
.users = [User{
    name: 'Jeroen van der Heijden',
    email: 'jeroen@cesbit.com',
    password_hash: 'SGVsbG8gVGhpbmdzREIhCg==',
}];

// 3. Use an ANONYMOUS TYPE to define the public output
// We only expose 'id', 'name', and 'email', securely omitting the hash.
.users.map_wrap(&{
    id: '#',        // '#' maps to the internal ID of the Thing
    name: 'str',
    email: 'str'
});

Result (JSON):

[
    {
        "id": 123456,
        "name": "Jeroen van der Heijden",
        "email": "jeroen@cesbit.com"
    }
]

No extra type definition needed! The anonymous type ensures type safety and validation on the projected fields without cluttering your global type registry.

📈 Beyond Simplicity: The Optimization Advantage

The &{...} literal syntax isn't just a convenience—it's a performance booster.

Unlike the alternative, ano(...), which constructs the type definition at runtime, the literal &{...} syntax allows the ThingsDB compiler to:

Generate optimal internal code for the data structure.
Bypass runtime overhead for type creation.

Key Takeaway: Always prefer &{...} for static definitions. Only use the ano(...) function when the type structure must be built dynamically based on runtime variables or input.

🔗 Advanced Use Cases: Complex Anonymous Structures

Anonymous types support the full richness of ThingsDB's type system, including arrays and nested structures.

Example:

// Nested output structure
&{
    request_id: '#',
    audit_trail: [{   // An array of audit check objects
        user_id: '#',
        timestamp: 'int',
        status: '*enum', // Match an existing Enum definition
    }],
    total_records: |this| this.audit_trail.len(),  // Computed
}

This single block creates a temporary, highly-structured schema for an output object.

Upgrade to ThingsDB v1.8.2 today and start enjoying the freedom and speed of on-the-fly type definition!

🚀 What if your Data Solution Contains Migration History?

Jeroen van der Heijden — Thu, 06 Nov 2025 19:27:06 +0000

ThingsDB 1.8.0 introduces significant features designed to help teams manage collection structure changes with confidence and clarity. The headline feature is a robust Commit History system, complemented by improvements to structure-only exports.

💾 Feature 1: Version Control for Your Collection Structure

When you work with ThingsDB, you typically have a clear separation between two types of collection operations:

Routine Operations: Queries used by your application at runtime (e.g., adding an item, updating a record). These are the "normal" changes.
Structural Operations: Queries used for setup or migrations (e.g., adding or changing types, updating procedures). These represent major changes to your application's data schema.

The new Commit History feature is specifically designed to version-control these Structural Operations.

Enabling and Using Commit History

When you choose to enable the commit history on a collection using the set_history() function, ThingsDB enforces a strict rule: any structural change must be explicitly committed.

This is why functions that modify a collection structure (like new_type() or set_type()) now require an explicit call to commit() before they can be executed. This ensures every critical change is logged with an explanatory message.

Example: Making a Committed Migration

This example shows the required structure for a migration script when commit history is enabled. The commit() call stages the message first, and the commit is finalized upon successful execution of the entire query.

// 1. Stage the commit message and enable structural changes
commit('Adds an initial user type for authentication');

// 2. Perform the actual structural change
new_type('User');
set_type('User', {
    username: "str",
    password_hash: "str",
});

The resulting commit object permanently stores the commit message, the user who ran the query, and the entire block of code that was executed.

🔍 Feature 2: Searching and Auditing Commit History

The new history() function is your primary tool for exploring this permanent archive of structural changes. You can search, filter, and audit every critical modification made to your collections.

Key Filtering Options

The history() function accepts several powerful filters:

Option	Description
id	Retrieves a specific, unique commit object instead of a list.
first/last	Limits the search to the X oldest (`first`) or X newest (`last`) commits.
before/after	Filters commits that occurred prior to or after a specified datetime object.
match	Uses a regular expression to search across the commit's user name, message, and code.
has_err	Filters based on execution status: `true` for failed commits, `false` for successful ones.

Example: Finding Failed Migrations

This code snippet demonstrates how to search for all commits that failed (i.e., resulted in an error) within the last year, allowing you to quickly debug structural issues:

history({
    scope: '//app_data',
    detail: true,
    has_err: true,
    after: datetime().move('years', -1),
});

🧹 Feature 3: Pruning Old History

History is valuable, but it shouldn't be infinite. The del_history() function allows you to prune old, unnecessary records using the same powerful filters:

Example: Removing Old History

This code removes all commits that were created more than six months ago in the //stuff collection, helping to keep your history archive clean:

del_history({
    scope: '//stuff',
    before: datetime().move('months', -6),
});

⚠️ Warning: Disabling commit history via set_history(<scope>, false) will permanently delete all existing historical records for that scope. This action cannot be undone.

🎁 Bonus Feature: Improved Structure Exports

Finally, the export function has been improved. A structure-only export (export({dump: false})) now includes complex enumerators as well, ensuring you get the complete "framework" or schema of a collection without any data, making it easier to manage and deploy database templates.

Conclusion: ThingsDB 1.8.0 brings professional-grade schema versioning to your data solution. Start using Commit History today to ensure every structural change is recorded, audit-able, and traceable!

Documentation links:

🚀 Optimizing Meta Data Retrieval with ThingsDB 1.7.6

Jeroen van der Heijden — Sat, 25 Oct 2025 10:49:26 +0000

We at InfraSonar query large volumes of meta data, resulting in many properties that are often empty lists, false booleans, or empty strings.
This redundant data increases payload size and network traffic unnecessarily.

ThingsDB version 1.7.6 introduces new wrap-prefix flags that solve this by allowing us to exclude properties from the wrapped output if they evaluate to a "false" state.

The Problem: Bloated JSON Payloads

Consider our Metric type definition (partial, in reality it looks slightly different):

set_type('Metric', {
    id: '#',                // the metric Id as "id"
    key: 'str<1:>',
    reference: '&Ref?',     // nil for most
    description: 'str',     // optional description
    dataType: 'DataType',
    isRequired: 'bool',
    isFileId: 'bool',       // false for most
    displayFunction: 'int', // mapping to a display function
    derived: '&[Derived]',  // list of derived metrics, empty for most
});

A standard Metric(123456).wrap() call often returns a large object dominated by default or empty values:

(JSON)

{
    "id": 123456,
    "key": "name",
    "reference": null,
    "description": "",
    "dataType": "STRING",
    "isRequired": true,
    "isFileId": false,
    "displayFunction": 0,
    "derived": []
}

The Solution: The ! and ? Wrap-Prefix Flags

The wrap-prefix flags enable a concise way to control data serialization based on truthiness:

! - Excludes the property if its value evaluates to false (e.g., false, 0, "", []).
? - Excludes the property only if its value is explicitly nil.

Applying the Flags

We update the Metric definition by prefixing properties we want to exclude when empty or false:

set_type('Metric', {
    id: '#',
    key: 'str<1:>',
    reference: '?&Ref?',     // Exclude when explicitly `nil`.
    description: '!str',     // Exclude when false (empty string).
    dataType: 'DataType',
    isRequired: '!bool',     // Exclude when false.
    isFileId: '!bool',       // Exclude when false.
    displayFunction: '!int', // Exclude when false (value is 0).
    derived: '!&[Derived]',  // Exclude when false (empty list).
});

The Compact Result

Running the same query, Metric(123456).wrap(), now yields a significantly more compact result:

(JSON)

{
    "id": 123456,
    "key": "name",
    "dataType": "STRING",
    "isRequired": true
}

By leveraging the ! and ? flags, we eliminate unnecessary data transfer, resulting in significantly reduced network traffic and faster query response times when dealing with meta data objects.

Great article about using ThingsDB as a complete back-end solution, Thanks Michal!

Jeroen van der Heijden — Sat, 07 Dec 2024 13:05:19 +0000

Let me explain a ThingsDB Todo app demo

Michal Štefaňák ・ Aug 5 '24

#webdev #database #showdev #javascript

DEV Community: Jeroen van der Heijden

Distributed Locking in Python

Why Distributed Locking is a "Hard" Problem

Part 1: The Infrastructure Setup

Part 2: The "Initialize Once" Pattern

Part 3: Using the Lock in Your Business Logic

Monitoring and Safety

Key Takeaways

Conclusion

Optimizing ThingsDB: Speeding up with Type Indexes

The "Ideal" Way: Hierarchical Access

The Problem: When type_all() is slow

The Solution: Enabling the Type Index

Performance Impact

Summary

Understanding the core mechanics of ThingsDB

Step 1: The Gateway and "AWAY" Mode

Step 2: To Compile or Not to Compile?

Step 3: The Race for Consensus (The Change ID)

Step 4: Futures and Non-Blocking Changes

Step 5: Storage and The "Boot" Sequence

Step 6: The Magic of "AWAY" Mode

Conclusion

Schema-First Approach with ThingsDB

1. Setting the Stage: The Blank Slate

2. Understanding Typed Collections (The Core Defense)

3. Defining the Data Model

4. Establishing an Event Channel

5. Encapsulating Logic with Procedures (The API Layer)

6. Strict Security: The Locked-Down API User

No Code in the Database... Right?

The Traditional Problem: Framework Ownership

The ThingsDB Shift: The Database as the Hub

Guaranteeing Data State and Event-Driven Architecture

Event-Driven Updates with ThingsDB

Traditional Flow

Self-Inflicted Updates

The ThingsDB Solution: Emitting to Peers Only

Beyond Chat Rooms

Conclusion

🚀 ThingsDB v1.8.2: Introducing Anonymous Types

⚡ The Solution: Anonymous Type

📈 Beyond Simplicity: The Optimization Advantage

🔗 Advanced Use Cases: Complex Anonymous Structures

🚀 What if your Data Solution Contains Migration History?

💾 Feature 1: Version Control for Your Collection Structure

Enabling and Using Commit History

Example: Making a Committed Migration

🔍 Feature 2: Searching and Auditing Commit History

Key Filtering Options

Example: Finding Failed Migrations

🧹 Feature 3: Pruning Old History

Example: Removing Old History

🎁 Bonus Feature: Improved Structure Exports

🚀 Optimizing Meta Data Retrieval with ThingsDB 1.7.6

The Problem: Bloated JSON Payloads

The Solution: The ! and ? Wrap-Prefix Flags

Applying the Flags

The Compact Result

Great article about using ThingsDB as a complete back-end solution, Thanks Michal!

Let me explain a ThingsDB Todo app demo

Michal Štefaňák ・ Aug 5 '24

The Problem: When `type_all()` is slow