DEV Community: Alexey Timin

ReductStore v1.19: Open Data Backbone for Robotics and ROS

Alexey Timin — Wed, 08 Apr 2026 00:00:00 +0000

ReductStore 1.19.0 is now available. This release extends the storage model for robotics and telemetry workloads and introduces new integration points for ROS and Zenoh.

To download the latest release, visit the Download Page.

What's new in 1.19.0?

The first major change in v1.19 is licensing. ReductStore Core is now open source under Apache 2.0, which makes the core database easier to evaluate, integrate, and extend in production systems.

The second major change is the data model. ReductStore now supports hierarchical entry names, similar to ROS topics, and adds entry attachments for schemas and metadata. This makes it possible to represent structured robotics data without flattening topic hierarchies or moving context into external systems.

The release also introduces a native Zenoh API for direct ingestion and querying over Zenoh, and ReductBridge for ROS1 and ROS2 integration.

Nested Data Model with Attachments

The new hierarchical data model lets you organize data in a path-based structure, similar to ROS topics, Zenoh key expressions, or MQTT topics. Instead of relying on a flat namespace, ReductStore can now store data in a form that matches the structure used by upstream systems.

Each entry can also include attachments for schemas and metadata. These attachments preserve the context required by downstream tooling without changing the record payload itself. For example, the ROS Extension can use them to decode serialized ROS messages and export them to MCAP files.

Native Zenoh API

Zenoh is increasingly used in robotics and edge environments as a low-overhead protocol for distributed data exchange. With the native Zenoh API, ReductStore can participate directly in Zenoh-based systems without requiring an additional bridge or adapter.

You can start ReductStore with the Zenoh API enabled using a minimal Docker configuration:

docker pull reduct/store:v1.19.0docker run 
 --env "RS_ZENOH_ENABLED=ON" 
 --env "RS_ZENOH_CONFIG={}" 
 --env "RS_ZENOH_SUB_KEYEXPRS=**" 
 -p 8383:8383 -p 36597:36597 -p 7446:7446 reduct/store:v1.19.0

Once enabled, the API allows you to write data to ReductStore directly over Zenoh. If a sample includes a JSON attachment, ReductStore stores it as record labels:

import json
import zenoh

KEY = "factory/line1/camera"
PAYLOAD = b"<binary payload>"
LABELS = {"robot": "alpha", "status": "ok"}

with zenoh.open(zenoh.Config()) as session:
    session.put(
        KEY,
        PAYLOAD,
        attachment=json.dumps(LABELS).encode(),
    )

You can also query data through Zenoh. For conditional queries, pass a when expression in the query attachment. If you want all matching records returned individually, use zenoh.ConsolidationMode.NONE:

import json
import zenoh

KEY = "factory/line1/when-query"
CONSOLIDATION = zenoh.ConsolidationMode.NONE
attachment = json.dumps({"when": {"&status": {"$eq": "ok"}}}).encode()

with zenoh.open(zenoh.Config()) as session:
    replies = [
        reply
        for reply in session.get(
            KEY,
            timeout=5.0,
            attachment=attachment,
            consolidation=CONSOLIDATION,
        )
        if reply.ok
    ]

    for reply in replies:
        print(reply.ok.payload.to_bytes())

Data ingested through Zenoh is stored in ReductStore like data written through the HTTP API, so it remains available for querying, replication, and downstream tools and extensions.

ReductBridge for ROS Integration

This release also introduces ReductBridge, a new project for integrating ReductStore with ROS1 and ROS2. ReductBridge automatically labels ROS messages and stores related schemas and metadata as attachments.

This makes raw ROS payloads usable after ingestion: they can be decoded later with the ROS Extension or exported to MCAP files. ROS support is the first target, but the same pattern can be extended to operating system metrics, logs, and other telemetry to build a unified storage layer with consistent labeling and metadata.

What’s Next

The next area of work is data compression and storage efficiency, especially for robotics workloads where payload sizes and formats vary significantly. We plan to introduce backend compression optimized for ReductStore's storage model, where many records are packed into a single block.

We also plan to store metadata in Parquet format. This should improve query efficiency, make metadata accessible without scanning the underlying data blocks, and simplify integration with analytics and data lake tooling.

Compatibility and Migration

Starting with v1.19, ReductStore Docker images no longer run as root for security reasons. If you deploy with Docker, make sure the mounted data directory is writable by UID/GID 10001:10001 before upgrading.

If you have questions or feedback, join the ReductStore Community forum.

Thanks for using ReductStore!

ReductStore Core Adopts Apache 2.0 License

Alexey Timin — Tue, 17 Mar 2026 00:00:00 +0000

Hello, everyone!

Starting with ReductStore v1.19 , we are changing how we license and package the project. From this version onward, ReductStore is split into two editions: ReductStore Core and ReductStore Pro.

Previously, ReductStore shipped as a single edition under BUSL-1.1 (Business Source License). With v1.19, the core database is open source under Apache 2.0, while Pro continues under commercial terms.

What's changing

With this release, ReductStore Core becomes the open-source foundation of the project and is now available under the Apache License 2.0. It includes the database server and the core functionality that most users rely on for edge deployments and everyday data workflows.

Alongside Core, we will continue offering ReductStore Pro , which is distributed under a commercial license. The Pro edition provides additional capabilities and support options for teams with more advanced requirements. See Pricing for a clear comparison of what is included in each edition.

Why we're doing this

We want to better support open source communities and make it easier to contribute to, integrate with, and build on top of ReductStore Core. Just as importantly, we want users to be able to run ReductStore on the edge without licensing restrictions for the core database use cases.

At the same time, we want to keep a sustainable commercial model for companies that build more complex cloud setups or need advanced support and functionality around robotics and IIoT data formats. That is the role of ReductStore Pro.

What this means for you

If you use the core database functionality, you can freely adopt, integrate, and distribute ReductStore Core under the Apache 2.0 license. Existing workflows and upgrades remain straightforward.

For teams that rely on advanced functionality or require commercial support, ReductStore Pro remains available as the supported commercial edition. We will clearly document which features belong to Core and which are part of Pro in the documentation and release notes.

If you have questions or want to discuss which edition fits your use case, please reach out on the ReductStore Community forum or via our contact page.

ReductStore v1.18.0 Released with Resilient Deployments and the Multi-entry API

Alexey Timin — Thu, 05 Feb 2026 00:00:00 +0000

We are pleased to announce the release of the latest minor version of ReductStore, 1.18.0. ReductStore is a high-performance storage and streaming solution designed for storing and managing large volumes of historical data.

To download the latest released version, please visit our Download Page.

What's new in 1.18.0?

In this release, we have added support for resilient deployments to build a more robust, fault-tolerant, and highly available ReductStore cluster. Now, you can implement hot-standby configurations, automatic failover, and seamless recovery to ensure uninterrupted service even in the face of hardware failures or network issues. You can also elastically scale read-only nodes to handle increased read workloads without impacting the performance of the primary nodes.

Additionally, we have introduced a new Multi-entry API that allows you to efficiently manage and query multiple entries in a single request. This API is designed to optimize performance and reduce latency when working with large datasets, making it easier to retrieve and manipulate data in bulk.

Resilient Deployments

In ReductStore v1.18.0, resilient deployments are now a first-class feature. Using the RS_INSTANCE_ROLE setting, you can build topologies that keep your ingestion endpoint available during node failures and scale reads independently from writes.

Hot standby (active-passive) for write availability

Run two nodes against the same backend (a shared filesystem or the same remote backend). Only one node is active at a time: the active node holds a lock file and refreshes it, while the standby waits and takes over when the lock becomes stale.

Set RS_INSTANCE_ROLE=PRIMARY for the active node and RS_INSTANCE_ROLE=SECONDARY for the standby.
Put both nodes behind a single virtual endpoint (load balancer / reverse proxy).
Route traffic only to the node that returns 200 OK on GET /api/v1/ready (the inactive node returns 503).
Tune failover behavior with RS_LOCK_FILE_TTL (how long the standby waits) and RS_LOCK_FILE_TIMEOUT (how long a node waits to acquire the lock).

To avoid split-brain writes, don’t run both nodes in STANDALONE mode against the same dataset.

Read-only replicas for read scaling

Add one or more REPLICA nodes to serve queries from the same dataset. Replicas never write and periodically refresh bucket metadata and indexes from the backend, so newly written data may appear with a small delay.

Route writes to a dedicated ingestion node (or the active node in a hot-standby pair).
Route reads to replicas to scale query workloads horizontally.
Tune staleness with RS_ENGINE_REPLICA_UPDATE_INTERVAL.

For an end-to-end walkthrough (including S3-based standalone, active-passive, and replicas), see the S3 Backend tutorial. For architecture options and operational notes, see the Disaster Recovery guide.

Multi-entry API

The new Multi-entry API makes it possible to work with multiple entries in a single request. In practice, this is most useful for querying : instead of running one query per sensor/stream and merging results on the client, you can request all the entries you need at once and process a single result stream (each returned record includes its entry name).

Here is a Python example using reduct-py to query multiple entries in one call:

import asynciofrom reduct import Clientasync def main() -> None: async with Client("http://localhost:8383", api_token="my-token") as client: bucket = await client.get_bucket("my-bucket") # Query multiple entries in a single request (since ReductStore v1.18). # You can mix exact names and wildcards. entries = ["sensor-*", "camera"] async for record in bucket.query( entries, start="2026-02-05T10:00:00Z", stop="2026-02-05T10:05:00Z", when={"&score": {"$gte": 10}}, ): payload = await record.read_all() print(record.entry, record.timestamp, len(payload))if __name__ == " __main__": asyncio.run(main())

What’s Next

We’re already working on the next improvements to make ReductStore easier to integrate into real-world data pipelines:

Native Zenoh API

Zenoh is becoming a common choice for data exchange in distributed, edge-first systems (robotics, industrial IoT, and telemetry). In upcoming releases, we plan to add a native Zenoh API so ReductStore can join Zenoh networks seamlessly.

This will make it easier to ingest and serve data directly through Zenoh—without custom bridges—so your storage layer fits naturally into existing Zenoh-based deployments.

Entry attachments (metadata)

Today, labels work well for filtering and replication, but many projects also need structured metadata tied to an entry itself: data format, schema version, units, encoding, calibration details, and other context.

We plan to introduce attachments for entries , allowing you to store and retrieve this kind of metadata alongside your data streams, making datasets more self-describing and easier to consume across teams and tools.

I hope you find those new features useful. If you have any questions or feedback, don’t hesitate to use the ReductStore Community forum.

Thanks for using ReductStore!

ReductStore v1.17.0 Released with Query Links and S3 Storage Backend Support

Alexey Timin — Tue, 21 Oct 2025 00:00:00 +0000

We are pleased to announce the release of the latest minor version of ReductStore, 1.17.0. ReductStore is a high-performance storage and streaming solution designed for storing and managing large volumes of historical data.

To download the latest released version, please visit our Download Page.

What's new in 1.17.0?

This release includes several new features and enhancements. First, there are query links for simplified data access. Second, there is support for S3-compatible storage backends.

These new features enhance the usability and flexibility of ReductStore for various use cases in the cloud and on-premises environments and make it easier to share and access data stored in the database.

🔗 Query Links for Data Access

ReductStore now supports query links, enabling users to generate temporary, public URLs for specific data records — without requiring authentication. This makes it easier to share datasets with external collaborators , embed links into dashboards, or integrate with third-party systems that need read-only access to specific data.

You can create query links directly from the Web Console (or any SDKs):

Open the Data Browser page and select a record you want to share.
Click the “Share record” icon in the action panel.
Configure an expiration time to automatically revoke access after a defined period.

Once generated, anyone with the link can access the selected record via a simple HTTP(S) request — no access token required. The link only has access to the specific query for which it was created, along with the creator's permissions. This provides a secure and convenient way to expose selected data for collaboration and analysis.

☁️ S3-Compatible Storage Backend

ReductStore now supports S3-compatible storage backends , allowing you to use object storage instead of a local file system for your underlying data. This update brings greater flexibility and scalability for managing large datasets in the cloud.

Previously, ReductStore supported only local disk storage, and users had to mount S3 buckets as local disks via FUSE drivers. With this release, ReductStore can now natively integrate with S3-compatible backends — no additional software or mounting is required.

This feature is designed with performance and cost optimization in mind. ReductStore uses a local disk cache layer to speed up read and write operations, while batching multiple records into a single data block to reduce storage and retrieval costs. This approach works especially well with cost-efficient AWS S3 storage classes such as S3 Standard-IA or S3 Glacier.

To run ReductStore with an S3-compatible backend, use the following environment variables:

docker run -p 8383:8383 \
 -e RS_REMOTE_BACKEND_TYPE=s3 \
 -e RS_REMOTE_BUCKET=<YOUR_S3_BUCKET_NAME> \
 -e RS_REMOTE_REGION=<YOUR_S3_REGION> \
 -e RS_REMOTE_ACCESS_KEY=<YOUR_S3_ACCESS_KEY_ID> \
 -e RS_REMOTE_SECRET_KEY=<YOUR_S3_SECRET_ACCESS_KEY> \ 
 -e RS_REMOTE_CACHE_PATH=/data/cache \
 -e RS_LICENSE_PATH=<PATH_TO_YOUR_LICENSE_FILE> \ 
 -v ${PWD}/data:/data/cache \
 reduct/store:latest

Read more about configuring S3-compatible storage backend in the documentation

info

This feature requires a commercial license. Please see the Pricing page for more details.

What’s Next

We’re continuing to develop new features to make ReductStore even more powerful and user-friendly. Here’s a preview of what’s coming in the next releases:

📦 Multiple Entries in a Single Request

Currently, each write or query request must target a single entry. This can be limiting when dealing with multiple entries or dynamic lists of entries in your applications.

In upcoming versions, ReductStore will support batch operations across multiple entries within a single API request. This improvement will simplify integrations and reduce overhead for large-scale data ingestion and querying workflows.

🔒 Read-Only Mode for ReductStore

Like most databases, ReductStore currently requires exclusive access to its data directory while running. As a result, running multiple instances on the same dataset—for load balancing or high availability—is not yet possible.

To address this, we’re introducing a read-only mode that will allow one writer instance* and multiple reader instances to access the same dataset concurrently. This approach will enable scalable read operations and improved availability without adding the complexity of clustering or replication mechanisms.

I hope you find those new features useful. If you have any questions or feedback, don’t hesitate to use the ReductStore Community forum.

Thanks for using ReductStore!

Building a Resilient ReductStore Deployment with NGINX

Alexey Timin — Sat, 13 Sep 2025 00:00:00 +0000

If you’re collecting high-rate sensor or video data at the edge and need zero-downtime ingestion and fault-tolerant querying, an active–active ReductStore setup fronted by NGINX is a clean, practical pattern.

This tutorial walks you through the reference implementation, explains the architecture, and shows production-grade NGINX snippets you can adapt.

What We’ll Build

We’ll set up a ReductStore cluster with NGINX as a reverse proxy, separating the ingress and egress layers. This architecture allows for independent scaling of write and read workloads, ensuring high availability and performance.

Ingress layer

The ingress layer handles all writes and replicates data to the egress layer. Its nodes may have limited storage capacity, while they need only to handle writes and replicate data to the egress nodes. It can use high-rate storage like NVMe SSDs or even RAM disks, depending on your data volume.

Egress layer

The egress layer handles all reads and serves data to clients. Its nodes are optimized for read performance and can use larger, slower storage like HDDs or cloud object storage. Each egress node holds a complete copy of the dataset, allowing for high availability and load balancing.

NGINX Load Balancer

The NGINX load balancer sits in front of both layers, exposing two stable endpoints:

http://<host>/ingress → load balances writes across ingress nodes
http://<host>/egress → load balances reads across egress nodes

This separation allows you to scale each layer independently and ensures that writes and reads are handled optimally.

It is also important to note that NGINX must maintain session affinity (stickiness) for both ingress and egress requests to ensure that queries remain consistent and throughput is maximized.

Quick Start

Clone the example and bring it up:

git clone https://github.com/reductstore/nginx-resilient-setup
cd nginx-resilient-setupdocker 
compose up -d

This will start two ingress nodes and two egress nodes with NGINX in front, all configured to replicate data between them. Check the docker compose file for details on how the nodes are set up.

Now we need to write some data and verify that we can read it back.Install the reduct-cli tool if you haven't already, then run the following commands to set up aliases for the ingress and egress endpoints:

reduct-cli alias add ingress -L http://localhost:80/ingress --token secret
reduct-cli alias add egress -L http://localhost:80/egress --token secret

Then copy some data from our Demo Server to the ingress layer and read it back from the egress layer:

# Add demo server alias to the CLI
reduct-cli alias add play -L https://play.reduct.store --token reductstore
# Copy data from the demo server to ingress
reduct-cli cp play/datasets ingress/bucket-1 --limit 1000
# Read/export via egress
reduct-cli cp egress/bucket-1 ./export_folder --limit 1000

NGINX Configuration

Below is a distilled config you can adapt for open-source NGINX:

# Upstreams
# Separate pools for ingress (writes) and egress (reads)
upstream reduct_ingress {
    ip_hash;   # stickiness for writes
    server ingress-1:8383 max_fails=3 fail_timeout=10s;
    server ingress-2:8383 max_fails=3 fail_timeout=10s;
    keepalive 64;
}

upstream reduct_egress {
    ip_hash;   # stickiness for queries
    server egress-1:8383 max_fails=3 fail_timeout=10s;
    server egress-2:8383 max_fails=3 fail_timeout=10s;
    keepalive 64;
}

server {
    listen 80;
    server_name _;

    client_max_body_size 512m;
    proxy_read_timeout 600s;
    proxy_send_timeout 600s;

    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $remote_addr;

    location /ingress/ {
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_pass http://reduct_ingress/;
    }

    location /egress/ {
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_pass http://reduct_egress/;
    }
}

In the config above, we define two upstream blocks: reduct_ingress for handling write requests and reduct_egress for handling read requests. Each block uses ip_hash to ensure session affinity, which is crucial for maintaining consistent writes and reads.

ReductStore Configuration Notes

The configuration between nodes of each layer is identical. To reach the desired architecture, you need to provision buckets and replication tasks for ingress nodes and buckets only for egress nodes. See the configuration files in the example repo for details.

Failure Drills

When the setup is running, you can simulate failures to see how it behaves:

Kill an ingress node → writes continue via other ingress nodes.
Kill an egress node → reads continue via other egress nodes; replication resyncs when it’s back.
Simulate total ingress outage → analysis continues on egress; for true ingestion continuity, pair with a pilot-light instance in another location.

Runbook

Here’s a high-level runbook for deploying this architecture in production:

Provision ingress + egress ReductStore nodes
Create buckets and replication tasks
Expose /ingress and /egress via NGINX with ip_hash
Test with demo dataset
Validate reads from egress
Run failure drills

References

I hope you find this article interesting and useful. If you have any questions or feedback, don’t hesitate to use the ReductStore Community forum.

ReductStore v1.16.0 Released With New Extensions and Context Replication

Alexey Timin — Sat, 30 Aug 2025 00:00:00 +0000

We are pleased to announce the release of the latest minor version of ReductStore, 1.16.0. ReductStore is a high-performance storage and streaming solution designed for storing and managing large volumes of historical data.

To download the latest released version, please visit our Download Page.

What's new in 1.16.0?

The v1.16.0 release introduces two new extensions designed to enhance data workflows for robotics and columnar data, along with support for replicating context records during queries.

Querying and Replicating Data with Context

We’ve extended the conditional query syntax with directives that allow users to modify global query behavior. The first directives introduced are #ctx_before and #ctx_after, which enable the inclusion of context records that occur before or after each matching record in a query.

This feature is particularly useful when analyzing specific events or conditions in your data, as it helps provide a clearer picture of the surrounding context. For instance, you can use these directives to include records from a few seconds before or after an anomaly or incident, aiding in root cause analysis or pattern recognition.

Here’s an example of how to use the #ctx_before directive in a query:

{ "#ctx_before": "5s", "&anomaly_score": { "$gt": 0.8 }}

This query returns all records with an anomaly score greater than 0.8, along with the context records that occurred within 5 seconds before each matching entry.

New ReductSelect Extension

ReductStore is fundamentally a blob storage system and does not allow direct manipulation of stored data. However, with its extension mechanism, we can introduce new capabilities while keeping the core system simple.

The new ReductSelect extension enables users to query and transform data stored in CSV or JSON formats, making it easier to build flexible and efficient data processing workflows.

For example, the following query uses ReductSelect to extract specific columns from CSV data and filter rows using the same conditional syntax available in ReductStore's native query language:

{
  "ext": {
    "select": {
      "csv": {
        "has_headers": true
      },
      "columns": [
        { "name": "temperature", "as_labels": "temp" },
        { "name": "humidity" }
      ]
    }
  },
  "when": {
    "&temperature": { "$gt": 30 }
  }
}

This query selects the temperature and humidity columns from a CSV file, renames temperature to temp, and filters rows where the temperature is greater than 30°C.

These simple transformations enable you to ingest structured data very quickly and retrieve only subsets of it for further processing and analysis.

New ReductROS Extension

Another exciting addition is the ReductROS extension, which provides tools for extracting and transforming data stored in ReductStore into formats compatible with the Robot Operating System (ROS).

With this extension, you can extract data from MCAP files containing ROS 2 messages and convert it into JSON format, making it easier to analyze and visualize. It also supports transforming raw binary data—such as images—into more accessible formats like JPEG or base64 strings.

For example, the following query extracts data from a ROS 2 topic and encodes the image payload as a JPEG:

{
  "ext": {
    "ros": {
      "extract": {
        "topic": "/camera/image",
        "encode": { "data": "jpeg" }
      }
    }
  }
}

ReductROS is still in active development, and we plan to expand its capabilities with support for additional ROS message types and more flexible extraction options in future releases. Stay tuned for updates!

What next?

We are constantly working on improving ReductStore and adding new features to provide the best experience for our users. In the next release we plan to add new features and improvements, including:

Shareable Query Links

We are developing a feature that allows users to generate and share links to specific queries in ReductStore.

This will simplify collaboration by enabling team members to access query results without needing direct access to the ReductStore instance. It will also allow users to download results directly via a link and support integration with external tools and platforms such as Foxglove.

Integration with Grafana

We are also working on a Grafana plugin that enables users to visualize and analyze data stored in ReductStore directly within Grafana dashboards.

This integration will provide a seamless experience with Grafana’s powerful visualization tools, allowing you to:

Build custom dashboards using data from ReductStore.
Monitor your data streams and historical records in real time.
Visualize labels and data output in JSON or CSV formats.

Stay tuned for the first release—coming soon!

I hope you find those new features useful. If you have any questions or feedback, don’t hesitate to use the ReductStore Community forum.

Thanks for using ReductStore!

ReductStore v1.15.0 Released With Extension API and Improved Web Console

Alexey Timin — Wed, 07 May 2025 00:00:00 +0000

We are pleased to announce the release of the latest minor version of ReductStore, 1.15.0. ReductStore is a high-performance storage and streaming solution designed for storing and managing large volumes of historical data.

To download the latest released version, please visit our Download Page.

What's new in 1.15.0?

This release includes several new features and enhancements. These are the extension API, the improved Web Console and the new conditional query operators.

Extension API

ReductStore is blob storage. It doesn't know anything about the data it stores. We are determined to maintain this because it enables us to ingest and query data in any format with optimal performance. We know that you sometimes need to perform processing and special queries based on the original data format.

For example, if you ingest data in the JSON format, you should be able to query only some fields of the JSON object or use it in the query condition for filtering. The new extension API makes it possible.

The extension API is experimental and not yet documented. We are developing extensions for columnar data, CSV and MCAP formats. Once we have enough experience, we will document the API and publish the extensions so that you can build your own extensions for your data formats.

For most curious users, a demo extension that scales JPEG images on the fly can be found on GitHub: https://github.com/reductstore/img-ext

Improved Web Console

In the v1.14.0 release, we introduced the ability to browse data in the Web Console. This release includes two new features: the ability to upload files to the database and update labels in the Web Console.

The upload feature can be useful when you store some artifacts e.g. AI models or configuration files in the storage and want to update it occasionally.

New Conditional Query Operators

We have expanded the set of conditional query operators with new ones that allow you to filter and aggregate data more effectively:

$each_n - keeps only every N-th record in the result set.
$each_t - keeps only one record within given time period in seconds.
$limit - limits the number of records in the result set.
$timestamp - allows you to filter records by timestamp.

The $timestamp operator can be particularly useful if you store timestamps and metadata in another database and want to retrieve blobs from ReductStore:

client = Client("http://localhost:8080")
bucket = await client.get("my_bucket")
async for record in bucket.query(
  start=1231231081,
  end=1231231085, 
  when ={ "$in": ["$timestamp", 1231231081, 1231231082, 1231231083, 1231231084,] },): 
  content = await record.read_all()

You can read more about the new operators in the Conditional Query documentation.

What next?

We are constantly working on improving ReductStore and adding new features to provide the best experience for our users. In the next few releases we plan to add new features and improvements, including:

Integration with ROS

ReductStore is a great solution for storing and managing large amounts of data in robotic applications. Currently, we are working on integrating ReductStore with ROS to provide a seamless experience for storing and retrieving data in ROS applications. We have started a new ROS2 Agent that allows you to store and retrieve data in ReductStore from ROS2 applications. The agent is designed to be easy to use and integrate with existing ROS2 applications. We are also going to add support for the mcap format with the new Extension API, which will allow you to retrieve data in the original format from mcap files, filter topics and many other features.

Golang SDK

Our big goal is to integrate Grafana by the end of 2025. This month we started work on the Golang SDK, which is the first step towards achieving this goal. The project is still in the early stages of development, but you can already check it out on GitHub: ReductStore Golang SDK.

I hope you find those new features useful. If you have any questions or feedback, don’t hesitate to use the ReductStore Community forum.

Thanks for using ReductStore!

Data Acquisition System for Manufacturing: Shop Floor to Cloud

Alexey Timin — Tue, 22 Apr 2025 00:00:00 +0000

As modern manufacturing becomes increasingly data-driven, the need for efficient data acquisition systems is more critical than ever. In my previous article, Building a Data Acquisition System for Manufacturing, we discussed the challenges of data acquisition in manufacturing and how ReductStore can help solve them. Here we will learn how to use ReductStore at the edge of the shop floor and stream data to the cloud.

DAQ Edge Device

The shop floor is the place where data is generated, and it is essential to have a reliable data acquisition (DAQ) device that can collect and process this data. The DAQ device can process data locally or act as a FIFO buffer, sending data to long-term storage on premises or in the cloud.

The DAQ device can be a Raspberry Pi with a USB stick or a powerful industrial computer with a 1TB or larger NVMe SSD. Despite the differences in hardware, your DAQ device should have the following capabilities:

Data ingestion : The DAQ device must be able to ingest data from various sources, such as sensors, PLCs, and other devices, with minimal latency and high throughput.
Data storage : The DAQ device must be able to store data locally and provide data retention policies to manage the amount of data.
Data replication : The DAQ device must be able to replicate data to the cloud or local storage for long-term retention and analysis.

These features don't look like much, but the devil is in the detail. If you have very intensive data acquisition, you can't store and replicate everything. You need an intelligent data collection system that can filter and process data before sending it to long-term storage. You may also face connectivity issues, and your DAQ device should be able to handle these issues gracefully without losing data.

Let's take a look at ReductStore and how it can help you build a modern data acquisition system for manufacturing.

Demo Setup

For demonstration purposes, we will build a simple data acquisition system that includes:

Data Source : A simple data source that generates random data and writes it to ReductStore.
ReductStore : A ReductStore instance running on the DAQ device that replicates data to a cloud instance.
Cloud Storage : A cloud instance of ReductStore at play.reduct.store (API token: reductstore) that stores the replicated data.

Provisioning ReductStore

For this demo, we need to create a bucket where we will ingest data and a replication task that will replicate data to the cloud. You can do this using the Web console, CLI or SDKs, but let's provision the resources we need using environment variables. Here is a simple docker-compose.yml file that creates a ReductStore instance with a bucket and a replication task:

services:
  reductstore:
    image: reduct/store:latest
    container_name: reductstore
    environment:
      RS_DATA_PATH: /data

      RS_BUCKET_1_NAME: data
      RS_BUCKET_1_QUOTA_TYPE: FIFO
      RS_BUCKET_1_QUOTA_SIZE: 1GB

      RS_REPLICATION_1_NAME: repl-task
      RS_REPLICATION_1_SRC_BUCKET: data
      RS_REPLICATION_1_DST_HOST: https://play.reduct.store
      RS_REPLICATION_1_DST_TOKEN: reductstore
      RS_REPLICATION_1_DST_BUCKET: demo
      RS_REPLICATION_1_WHEN: '{"&score": {"$$gt": 0.8}}' # we need $$ to escape $ in YAML

    volumes:
      - ./data:/data
    ports:
      - 8383:8383

In this example we will create a bucket called data with a FIFO quota of 1GB and a replication task called repl-taskwhich will replicate data from the data bucket to the demo bucket on our demo server. The RS_REPLICATION_1_WHEN environment variable contains a filter that allows you to replicate only data with a score greater than 0.8.

This is a simple example, but you can use more complex filters to replicate data based on your needs.

Now you can start ReductStore with Docker Compose:

docker-compose up

You can open the Web Console at http://localhost:8383 and check the status of the bucket and replication task.

Data Ingestion

Now we need to create a data source that will generate random data and write it to ReductStore.

We provide several SDKs for different programming languages, but for this example we will use the Python SDK. Let's install it:

pip install reduct-py

Now we can create a simple data source. Create a file called data_source.py and add the following code:

from asyncio import sleep
from random import randbytes, randint

from reduct import Client

BLOB = randbytes(100 * 1024)  # 100 KB of random bytes

async def send_data():
    async with Client("http://127.0.0.1:8383") as client:
        # Get bucket
        bucket = await client.get_bucket("data")

        while True:
            # Generate score
            score = randint(1, 100) / 100.0

            # Write record
            await bucket.write(
                "scored_data",
                BLOB,
                labels={
                    "score": score,
                },
            )

            print(f"Record with score {score} written")
            await sleep(1)


if __name__ == "__main__":
    import asyncio
    loop = asyncio.new_event_loop()
    loop.run_until_complete(send_data())

As you can see, it's very simple. You can read working with the Python SDK in the Quick Start With Python guide.

Let's run the data source:

python data_source.py

You can check if the data is ingested by opening the Web Console and checking the data bucket.

Data Replication

Now we need to check that the data is replicated to the demo server. You can do this by opening the Web Console at https://play.reduct.store then logging in with the reductstore token and checking the demo bucket. You should find the scored_data entry with the replicated data:

You should see the records with a score greater than 0.8, as we specified in the replication task.

Let's simulate a network outage and see how ReductStore handles it. If you run the demo setup locally on your computer, you can simply disconnect from the internet for a while and see in the web console that the replication task has pending records. The replication task will store any pending records in a log to be sent later. When you re-enable the Internet connection, the replication task will automatically send the pending records to the demo server.

Now you don't need to worry about network outages or data loss. ReductStore will take care of that for you.

Best Practices

The example we have provided is very simple and does not cover all aspects of building a modern data collection system for manufacturing. Here are some best practices to consider when building your own system based on ReductStore:

Security: Use secure connections (HTTPS) and authentication tokens to protect your data. Read more about access control in the Access Control guide.
Data retention: Use the FIFO quota to limit the amount of data stored in ReductStore. This will help you avoid running out of disk space and keep your data fresh.
Store data in the cloud: Use the ReductStore Cloud to store your data in the cloud. It uses a cloud object storage as a backend and reduces the cost of storing data.

Conclusion

Building a modern and robust data acquisition system for manufacturing can be a challenging task. In this article, we have shown you how ReductStore can help you build a data acquisition system that can ingest, store, and replicate data from the shop floor to the cloud with minimal effort and maximum performance.

Resources

We hope this article has provided you with valuable insights into building a modern data acquisition system for manufacturing. If you have any questions or comments, feel free to use the ReductStore Community Forum.

Building a Data Acquisition System for Manufacturing

Alexey Timin — Mon, 17 Mar 2025 00:00:00 +0000

Large manufacturing plants generate vast amounts of data from machines and sensors. This data is valuable for monitoring machine health, predicting failures, and optimizing production. It also serves as a foundation for building industrial AI models for predictive maintenance, quality control, and process optimization.

A Data Acquisition (DAQ) system collects this data, processes it, and stores it for further analysis. It typically consists of edge devices that gather real-time data, central servers or cloud storage for retention, and software that enables analytics and AI-driven insights.

An example of a 3 tier DAQ system based on ReductStore.

Traditional automation solutions like SCADA and historians are complex, expensive, and not optimized for modern cloud-based AI applications. They often limit access to data, making it difficult for engineers and data scientists to develop machine learning models and gain actionable insights.

In this article, we’ll explore the challenges of building a modern DAQ system for manufacturing and how ReductStore can simplify the process and support ELT (Extract, Load, Transform) workflows for advanced analytics and AI applications.

The Role of DAQ Systems in Manufacturing

In the 21st century, manufacturing is becoming increasingly data-driven. While many factories are already equipped with automation systems, SCADA systems, and historians that collect data from machines and sensors, these solutions are limited when it comes to today's data needs.

Traditional systems struggle with data-intensive sources such as vibration analysis and video surveillance, which generate large volumes of unstructured data. They don't have interfaces to efficiently access the history of the raw data , which is essential for building advanced analytics and AI models. In addition, they often lack the flexibility to integrate with modern cloud-based storage and analytics platforms.

This is where a dedicated data acquisition (DAQ) system comes in. A DAQ system efficiently collects, processes, and stores manufacturing data, providing real-time access for optimization and predictive analytics. They are more flexible and scalable than traditional automation solutions and can handle multiple data sources, from sensors and PLCs to cameras and vision systems.

Importantly, implementing a DAQ system does not mean replacing the existing automation infrastructure. In many cases, DAQ systems are deployed alongside automation systems, complementing rather than replacing them.

Challenges of Building a DAQ System

DAQ systems are often perceived as simpler and more cost-effective than full-scale automation solutions. However, designing a robust DAQ system for manufacturing comes with its own set of challenges:

Handling Massive Data Volume and Velocity – Manufacturing plants generate vast amounts of high-frequency data from machines and sensors. Efficiently collecting, processing, and storing this data in real-time requires a scalable architecture.
Ensuring Reliable Connectivity with Low Latency – Edge devices must transmit data to central servers or cloud storage with minimal latency and high reliability, even in industrial environments with network constraints.
Balancing Cost and Scalability – As data volume grows, the DAQ system must scale efficiently without excessive costs, requiring optimized storage, compression techniques, and cloud integration.

With these challenges in mind, let’s explore how ReductStore can simplify the process of building a DAQ system for manufacturing.

A DAQ system can range from a single edge device near a machine to a complex infrastructure spanning multiple factories and cloud instances. Below is a high-level overview of its structure.

Shop Floor

The shop floor is where the manufacturing process takes place, and it is the closest point to the machines and sensors that generate data. Due to the real-time nature of manufacturing data, it is crucial to have edge devices positioned near the machines to collect and transmit data efficiently.

Edge devices can either process data locally or act as FIFO buffers , temporarily storing data before sending it to the next level of the system. Their role depends on system requirements, but their primary function is to ensure seamless data acquisition without overloading the network or losing critical information.

An example of a DAQ edge device based on ReductStore.

Data Ingestion

Industrial data comes in a variety of formats and frequencies. A DAQ system must support acquisition from multiple sources such as sensors, PLCs, and cameras. As manufacturing environments rely on various industrial protocols such as OPCUA, IO-Link, and GenICam, the DAQ system must be able to collect data from these sources. For this purpose, edge devices are equipped with data connectors that collect data from different sources and convert it into standardized formats e.g. JSON, CSV, WAV and JPEG.

By using standard protocols, a DAQ system remains flexible and scalable, allowing integration with a variety of industrial devices. However, the raw data collected from these sources is often unstructured and storage-intensive , requiring effective processing and storage strategies.

ReductStore enables the storage of a history of unstructured data and provides an HTTP API and SDKs for multiple programming languages, allowing data connectors to write data in various formats directly to storage.

Data Storage

Data storage in a DAQ system must handle diverse data types and support real-time ingestion. Additionally, edge devices have limited storage , meaning the system cannot afford to stop data collection when storage is full.

ReductStore solves these challenges by providing a unified data storage solution optimized for manufacturing environments:

High-Speed Ingestion – Designed for fast, real-time data collection from multiple sources.
FIFO Quotas – Automatically deletes old data when storage is full, ensuring continuous operation without manual intervention.
Data Labeling – Enables structured organization by tagging data based on source, type, or timestamp, making retrieval and analysis more efficient.

Data labeling is particularly useful for structuring information. For instance, video data can be labeled with machine states (e.g., running, stopped, fault), making it easy to filter and retrieve condition-specific records.

Data Replication

In most cases, edge devices serve as temporary storage before data is transferred to a central server or cloud storage for long-term retention and further analysis.

However, manufacturing environments present several connectivity challenges that must be addressed. One of the most common issues is network reliability. Edge devices may operate in remote locations with unstable or slow networks, making it difficult to maintain a continuous connection to the central server. In addition, limited bandwidth and firewall restrictions can further complicate data transfer. Security concerns often restrict direct access to edge devices, requiring a more sophisticated replication mechanism.

info

Although IoT protocols such as MQTT can address many of connectivity issues, using a database for streaming data ensures persistence and reliability - data is stored on disk and can be replicated even after prolonged loss of connectivity.

ReductStore is designed to meet these industrial constraints with robust replication capabilities:

Push Replication - Edge devices push data to the central server/cloud only when a connection is available, eliminating the need for a permanent connection and direct access to the edge device.
HTTP/HTTPS Support - Ensures compatibility with firewalls and industrial network configurations.
Batch Replication - Sends multiple records in a single request, optimizing bandwidth usage and reducing network overhead.
Conditional Replication - Replicates only necessary data based on labels, timestamps, or other conditions, optimizing network traffic and storage costs.

With ReductStore, setting up replication requires no custom logic - simply label your data, define replication conditions, and the system does the rest. This approach ensures efficient, cost-effective, and scalable data collection for manufacturing environments.

Factory Data Store

The factory storage system is the next level of the DAQ architecture, responsible for aggregating data from multiple edge devices and providing structured access for data scientists and engineers.

Unlike edge devices, which have limited processing power and storage , the factory storage system can leverage more powerful hardware and larger storage capacity, allowing for longer data retention and more advanced processing.

An example of a DAQ factory storage based on ReductStore.

Massive Data Volumes

The factory storage can be a dedicated server , NAS , or SAN optimized for high-throughput data storage. But independent of the implementation, the storage system must be capable to store TBs or even PBs of data because of the high-frequency data generated by manufacturing processes.

info

Usually, large amount of unstructured data is stored just in a file system or object storage, which is not optimized for historical data access.

ReductStore provides a scalable and efficient storage solution for manufacturing data, designed to handle massive amounts of data. and ensure fast access to manufacturing data:

Indexing by time - ReductStore indexes data by time, enabling fast retrieval of data over time intervals.
Separate storage for metadata - ReductStore stores and manages metadata separately from data, allowing it to scale and handle more data.
Batching - ReductStore batches data at the communication and file system level, optimizing data storage and access.

Data Management

When we talked about the need for factory storage to handle massive amounts of data, we meant not only storing the data, but also providing efficient data management. with access control, data retention policies, and data labeling.

The data engineers and data scientists need to be able to manage the data for the following tasks:

Retrieve data - Efficiently search and retrieve data based on timestamps, labels. This is important for building analytics and AI models.
Data Labeling - Label data based on source, type, or timestamps, enabling efficient search and conditional data retrieval.
Data Access Control - Restrict access to sensitive data based on user roles and privileges.

With ReductStore, you can manage data efficiently and securely, ensuring that only authorized users have access to sensitive data. ReductStore provides a RESTful API and SDKs for multiple programming languages, enabling data engineers and data scientists to programmatically interact with the data store.

Cloud Storage and Processing

Factory storage can be further extended to a cloud-based data acquisition system , providing additional scalability, redundancy, and advanced analytics capabilities. It allows data to be accessed from anywhere, enabling remote monitoring, predictive maintenance, and AI-driven insights.

Building a cloud component of the DAQ system that can handle data from multiple plants requires a robust architecture that can scale horizontally and vertically. There are several key components to consider when designing a cloud-based data acquisition system:

Data Ingestion - The cloud component must be able to ingest data from multiple factories and edge devices, handling different data types and formats.
Cost-Effective Storage - Cloud storage costs can escalate quickly with large volumes of data. The system must be optimized for cost-effective storage and retrieval.
Data Transformation - Data from multiple sources must be transformed and aggregated for analytics, visualization, and AI applications.
Automated workflows - The cloud component should support automated workflows to efficiently scale and manage the cloud infrastructure.

Let's discuss these parts in more detail.

An example of a cloud-based DAQ storage based on ReductStore.

Data Ingestion

Many cloud platforms offer services for data ingestion, with many of them leveraging the Pub/Sub pattern using message brokers such as Kafka, MQTT, or AMQP. The general concept is to have a message broker that receives data from edge devices and then transmits it to the cloud storage. Subsequently, data is transformed with AWS Lambda or Google Cloud Functions, and the aggregated data is stored in a database. This approach is referred to as ETL (Extract, Transform, Load).

While this approach is widely used and highly scalable, it has some drawbacks:

Loss of raw data - The raw data is transformed during the ETL process, resulting in the loss of the source of truth.
Complexity - Establishing and maintaining a message broker, data transformation, and cloud storage can be complex and time-consuming.
Costs - cloud services can be costly, particularly when dealing with substantial data volumes.

warning

It is important to understand that during the ETL process, the raw data is transformed, making it irrecoverable. This limitation precludes the ability to revert to the raw data or apply a different transformation to historical data.

ReductStore uses the ELT (Extract, Load, Transform) approach, where data is extracted from edge devices and loaded directly into cloud storage. This approach greatly simplifies the ingestion process by leveraging ReductStore's replication mechanism.

Scalable Storage

The most common cloud storage solutions are object storage such as Amazon S3, Google Cloud Storage, and Azure Blob Storage. It is many times cheaper than persistent disk and provides a scalable and durable storage solution for large volumes of data. Unfortunately, as mentioned earlier, object storage is not optimized for historical data access.

ReductStore can use object storage as backend storage using FUSE drivers. This allows you to store data in a scalable and cost-effective manner while maintaining fast access to historical data.

Data Transformation

Data transformation is a critical step in the cloud data acquisition system. It involves aggregating data from multiple sources, cleaning and normalizing data, and preparing it for analytics, visualization, and AI applications. Cloud platforms provide services such as AWS Lambda, Google Cloud Functions, and Azure Functions for data transformation; they are elastically scalable, but can be expensive.

ReductStore provides a subscription mechanism that allows you to subscribe to data changes and apply transformations on the fly , With the ReductStore SDKs, you can write your own transformation functions in Python, JavaScript, or any other language.

Automated Workflows

The cloud data acquisition system must support automated workflows to efficiently manage the cloud infrastructure. This includes scaling resources on demand, monitoring system health, and handling failures gracefully.

ReductStore provides a SaaS solution that takes care of the storage infrastructure management, monitoring, and scaling so you can focus on building analytics and AI applications.

Conclusion

In this article, we have explored the challenges of building a modern data acquisition system for manufacturing. Every DAQ system is unique and depends on the specific requirements of the manufacturing environment. It may include all of the components described or only a subset of them. The data transformation can be done on the edge devices, factory storage, or cloud storage, depending on the use case. What can change the scale and complexity of the system. However, the key principles of efficient data acquisition, processing, and storage remain the same, and your DAQ system should be designed to handle massive amounts of data and ensure reliable connectivity.

With ReductStore, you can adopt the ELT approach , simplify the process of building a manufacturing DAQ system, replicate data from edge devices to central servers or cloud storage, and provide efficient data storage and management.

ReductStore v1.14.0 Released With Many Improvements

Alexey Timin — Tue, 25 Feb 2025 00:00:00 +0000

We are pleased to announce the release of the latest minor version of ReductStore, 1.14.0. ReductStore is a time series database designed for storing and managing large amounts of blob data.

To download the latest released version, please visit our Download Page.

What's new in 1.14.0?

This release introduces several new features and enhancements, including new conditional query operators, I/O and replication settings, and data browsing in the Web console.

New Conditional Query Operators

In version 1.13 we introduced support for conditional queries, allowing you to filter data using labels in complex conditions. Now we have added several new conditional query operators that allow you to filter data more effectively.

Arithmetic Operators perform arithmetic operations on labels in a query like arithmetic addition, subtraction etc. This allows you to perform complex calculations on the fly when querying data and use the results to filter records. For example, you can use arithmetic operators to filter records where the deviation of score is greater than 10 of its mean:

{ "$gt": [{ "$abs": [{ "&score": { "$sub": "&mean_score" } }] }, 10 ]}

String Operators perform string operations on values in a query like checking if a string contains a substring, starts with, or ends with. You can use string operators to filter records based on the string values of their labels. For example, you can filter records where the name label contains the substring bottle:

{ "&name": { "$contains": "bottle" }}

Miscellaneous Operators provide additional functionality like checking if a label exists in a record, casting a label value to a different type, or referencing a label value in a record. These can be useful if you want to filter records based on the presence of a label, explicitly cast a label value to another type, or explicitly reference a label value in a record.

All these new operators are available in the latest version of ReductStore and its client SDKs. You can start using them right away to improve your data retrieval experience.

I/O and Replication Settings

Since version 1.14.0, ReductStore gives you more control over I/O and replication settings which you can apply using environment variables.

The following I/O settings are available:

Name	Default	Description
`RS_IO_BATCH_MAX_SIZE`	8MB	Maximum size of a batch of records sent to the client.
`RS_IO_BATCH_MAX_RECORDS`	85	Maximum number of records in a batch sent and received from the client.
`RS_IO_BATCH_MAX_METADATA_SIZE`	8KB	Maximum size of metadata in a batch of records sent and received from the client.
`RS_IO_BATCH_TIMEOUT`	5s	Maximum time for a batch of records to be prepared and sent to the client. If the batch is not full, it will be sent after the timeout.
`RS_IO_BATCH_RECORD_TIMEOUT`	1s	Maximum time to wait for a record to be added to a batch. If the record is not added, the unfinished batch will be sent to the client.

The I/O settings allow you to control the size and number of records in a batch, the size of metadata, and the timeouts for sending batches to the client. This can be useful if you want to optimise the performance of your ReductStore instance and tailor it to your specific use case and network conditions. In addition, the RS_IO_BATCH_MAX_METADATA_SIZE and RS_IO_BATCH_MAX_RECORDS settings specify the size of HTTP/1 headers, which is important if your instance is behind a reverse proxy or load balancer.

In addition, you can now configure replication settings using the following environment variables:

Name	Default	Description
`RS_REPLICATION_TIMEOUT`	5s	Timeout for attempts to reconnect to the target server in seconds.
`RS_REPLICATION_LOG_SIZE`	1000000	Maximum number of pending records in the replication log. The oldest records are overwritten when the limit is reached.

The replication settings allow you to control the timeout for attempts to reconnect to the target server and the maximum number of pending records in the replication log.

Read more about all available settings in the Configuration Reference.

Data Browsing in Web Console

ReductStore has an embedded Web Console that allows you to manage your buckets, monitor data ingestion, and manage replication settings. Since version 1.14.0, the Web Console has a new feature that allows you to browse the data stored in your buckets.

Now you can easily view the records stored in your buckets, filter them by labels, and download the contents of the records. This feature can be useful if you want to quickly check the data stored in your buckets or debug problems related to data ingestion.

You can try it right now with our demo server (API token: reductstore ) or with our SaaS offer.

What next?

We have very exciting plans for the future of ReductStore, including new features, enhancements and integrations.

Extensions

Firstly, we are working on an extension system that will allow you to extend the functionality of ReductStore with custom plugins and integrations written in Rust. One of the first extensions we plan to release is a plugin for structured data such as JSON, CSV and Parquet, which will allow you to store structured data in ReductStore and query it with SQL-like queries.

The goal is to make ReductStore more versatile and suitable for a wider range of use cases, and not just blob storage, but an advanced way to store and query data with different structures.

Web Console Improvements

We are also planning to improve the Web Console by adding more features for data management, diagnostics and monitoring. One of the features we are working on will allow data and updated records to be uploaded directly from the Web Console without using the HTTP API.

I hope you find those new features useful. If you have any questions or feedback, don’t hesitate to use the ReductStore Community forum.

Thanks for using ReductStore!

[Boost]

Alexey Timin — Fri, 21 Feb 2025 16:29:50 +0000

ReductStore vs. MongoDB: Which One is Right for Your Data?

AnthonyCvn for ReductStore ・ Feb 21

#database #comparison

ReductStore v1.13.0 Released With New Conditional Query API

Alexey Timin — Thu, 05 Dec 2024 00:00:00 +0000

We are pleased to announce the release of the latest minor version of ReductStore, 1.13.0. ReductStore is a time series database designed for storing and managing large amounts of blob data.

To download the latest released version, please visit our Download Page.

What's new in 1.13.0?

This release introduces a new conditional query API that should significantly improve your experience when querying or removing records. The new conditional queries allow you to use logical and comparison operators to filter records by labels. Previously, you could only filter records by labels using the include and exclude options, which were limited to exact matches. This means that you had to classify your records in advance at the ingest stage to be able to filter them later. Now, all you have to do is label your records with metric labels and then use the conditional queries to filter them by any condition you want.

Conditional Query Syntax

The conditional query syntax is inspired by the MongoDB query language and is based on the JSON format. The query consists of a set of conditions that are combined using logical operators. It can be written in simple object notation:

{ "&label_name": { "$gt": 10 }}

Or in array notation:

{ 
  "$any_of": [
    { "&label_name": { "$gt": 10 } }, 
    { "&label_name": { "$lt": 20 } }, 
    { "&label_name": { "$eq": 15 } }
  ]
}

Here we refer to labels with the & symbol followed by the label name. All operators start with the $ symbol followed by the operator name. Constant values are just JSON values. And that's it! You don't have to learn a new query language to start using conditional queries.

The current version of the Conditional Query API supports all the logical and comparison operators you need to filter records by labels. For a complete list of supported operators, see the Comparison Operators and Logical Operators sections of the documentation.

Real-world Example

But why do you need to use conditional queries? Let's look at a real-world example. Suppose you collect raw vibration data from a machine and store it in ReductStore. You can calculate the vibration metrics, such as RMS and crest factor, and store them as labels. Now you can use the conditional queries to filter the records by the vibration metrics. For example, we know that when the machine is working, the RMS must be greater than 10, and some problems with the engine happen when the crest factor is greater than 2.5. You can write the following conditional query to get the raw data for detailed analysis:

{ "&rms": { "$gt": 10 }, "&crest_factor": { "$lt": 2.5 }}

With our Python SDK, the conditional query can be used like this:


import time
import asyncio
from reduct import Client, Bucket

async def main():
    # Create a client instance, then get or create a bucket
    async with Client("http://127.0.0.1:8383", api_token="my-token") as client:
        bucket: Bucket = await client.get_bucket("vibration")

        # Query vibration data with RMS > 10 and crest factor < 2.5
        async for record in bucket.query(
                "sensor-1",
                when={
                    "&rms": {"$gt": 10},
                    "&crest_factor": {"$lt": 2.5}
                },
        ):

            _raw_data = await record.read_all()


loop = asyncio.get_event_loop()
loop.run_until_complete(main())

We have updated all of our official SDKs to support the new Conditional Query API for querying and removing data. Read our Guides to learn more about conditional queries and how to use them in your applications.

What next?

This release introduces only the basic set of logical and comparison operators. We plan to add more operators and functionality to the Conditional Query API in future releases. It will also be available in the replication engine to filter records during the replication process.

I hope you find those new features useful. If you have any questions or feedback, don’t hesitate to use the ReductStore Community forum.

Thanks for using ReductStore!

DEV Community: Alexey Timin

ReductStore v1.19: Open Data Backbone for Robotics and ROS

What's new in 1.19.0?​

Nested Data Model with Attachments​

Native Zenoh API​

ReductBridge for ROS Integration​

What’s Next​

Compatibility and Migration​

ReductStore Core Adopts Apache 2.0 License

What's changing​

Why we're doing this​

What this means for you​

ReductStore v1.18.0 Released with Resilient Deployments and the Multi-entry API

What's new in 1.18.0?​

Resilient Deployments​

Hot standby (active-passive) for write availability​

Read-only replicas for read scaling​

Multi-entry API​

What’s Next​

Native Zenoh API​

Entry attachments (metadata)​

ReductStore v1.17.0 Released with Query Links and S3 Storage Backend Support

What's new in 1.17.0?​

🔗 Query Links for Data Access​

☁️ S3-Compatible Storage Backend​

What’s Next​

📦 Multiple Entries in a Single Request​

🔒 Read-Only Mode for ReductStore​

Building a Resilient ReductStore Deployment with NGINX

What We’ll Build​

Ingress layer​

Egress layer​

NGINX Load Balancer​

Quick Start​

NGINX Configuration​

ReductStore Configuration Notes​

Failure Drills​

Runbook​

References​

ReductStore v1.16.0 Released With New Extensions and Context Replication

What's new in 1.16.0?​

Querying and Replicating Data with Context​

New ReductSelect Extension​

New ReductROS Extension​

What next?​

Shareable Query Links​

Integration with Grafana​

ReductStore v1.15.0 Released With Extension API and Improved Web Console

What's new in 1.15.0?​

Extension API​

Improved Web Console​

New Conditional Query Operators​

What next?​

Integration with ROS​

Golang SDK​

Data Acquisition System for Manufacturing: Shop Floor to Cloud

DAQ Edge Device​

Demo Setup​

Provisioning ReductStore​

Data Ingestion​

Data Replication​

Best Practices​

Conclusion​

Resources​

Building a Data Acquisition System for Manufacturing

The Role of DAQ Systems in Manufacturing​

Challenges of Building a DAQ System​

Shop Floor​

Data Ingestion​

Data Storage​

Data Replication​

Factory Data Store​

Massive Data Volumes​

Data Management​

Cloud Storage and Processing​

Data Ingestion​

Scalable Storage​

Data Transformation​

Automated Workflows​

Conclusion​

What's new in 1.19.0?

Nested Data Model with Attachments

Native Zenoh API

ReductBridge for ROS Integration

What’s Next

Compatibility and Migration

What's changing

Why we're doing this

What this means for you

What's new in 1.18.0?

Resilient Deployments

Hot standby (active-passive) for write availability

Read-only replicas for read scaling

Multi-entry API

What’s Next

Native Zenoh API

Entry attachments (metadata)

What's new in 1.17.0?

🔗 Query Links for Data Access

☁️ S3-Compatible Storage Backend

What’s Next

📦 Multiple Entries in a Single Request

🔒 Read-Only Mode for ReductStore

What We’ll Build

Ingress layer

Egress layer

NGINX Load Balancer

Quick Start

NGINX Configuration

ReductStore Configuration Notes

Failure Drills

Runbook

References

What's new in 1.16.0?

Querying and Replicating Data with Context

New ReductSelect Extension

New ReductROS Extension

What next?

Shareable Query Links

Integration with Grafana

What's new in 1.15.0?

Extension API

Improved Web Console

New Conditional Query Operators

What next?

Integration with ROS

Golang SDK

DAQ Edge Device

Demo Setup

Provisioning ReductStore

Data Ingestion

Data Replication

Best Practices

Conclusion

Resources

The Role of DAQ Systems in Manufacturing

Challenges of Building a DAQ System

Shop Floor

Data Ingestion

Data Storage

Data Replication

Factory Data Store

Massive Data Volumes

Data Management

Cloud Storage and Processing

Data Ingestion

Scalable Storage

Data Transformation

Automated Workflows

Conclusion

What's new in 1.14.0?

New Conditional Query Operators

I/O and Replication Settings

Data Browsing in Web Console

What next?

Extensions

Web Console Improvements

What's new in 1.13.0?

Conditional Query Syntax

Real-world Example