DEV Community: Rebecca Tao

InfluxDB 3 Core vs. 1.8: Query Performance and Compatibility Tested

Rebecca Tao — Wed, 12 Mar 2025 08:14:15 +0000

Highlights

TSBS testing demonstrates that for the vast majority of queries, InfluxDB 3 Core returns results more slowly than InfluxDB 1.8, up to 48x slower in the DevOps use case and 30x slower in the IoT use case.

In the IoT use case, six of the twelve InfluxQL test queries could not be executed in Influx 3 Core due to incompatibility. These queries ran normally on InfluxDB 1.8, indicating that InfluxDB 3 Core’s support for InfluxQL is still limited.

When testing the larger TSBS dataset (4,000 devices with 10 metrics each), InfluxDB 3 Core could not perform at high concurrency and required reducing the number of workers to three or fewer.

Following our data ingestion testing of InfluxDB 3 Core published last week, we decided to compare the query performance of InfluxDB 3 Core with InfluxDB 1.8. Once again, we used the Time Series Benchmark Suite (TSBS) to ensure a level playing field for all database platforms. Because InfluxDB 3 Core supports InfluxQL, the TSBS query set for InfluxDB 1.8 should be able to run in the latest alpha without modification. However, during our testing, we found that InfluxQL compatibility is not complete.

InfluxData has listed in their documentation a number of InfluxQL features that are not yet supported in the latest alpha. Even taking these into consideration, we were unable to obtain results for several TSBS queries due to errors, whereas all queries return results normally in InfluxDB 1.8. Additionally, the queries that did return in InfluxDB 3 Core showed significantly lower performance compared with InfluxDB 1.8.

DevOps Use Case

As an initial performance comparison, we took the TSBS DevOps data from the smallest scenario of 100 devices and ran each query 4,000 times with 8 workers, obtaining the average response time.

We can see that InfluxDB 3 Core outperformed InfluxDB 1.8 in only two categories, double-groupby-all and high-cpu-all. For the other queries, it ranged from 1.6x slower in double-groupby-5 to 48x slower in lastpoint.

Considering the cardinality improvements already evident in InfluxDB 3 Core, we increased the scale of the comparison from 100 devices to 4,000 devices to see whether the new version would perform better in a larger-scale scenario. However, at this point we began to encounter an error: InfluxDB began to reject HTTP requests, though it did not crash, and nothing was written to its logs. The error is shown as follows:

To resolve this issue, we had to reduce the number of workers to three or even two. Most of the more complex queries were affected: double-groupby-1, double-groupby-5, double-groupby-all, high-cpu-all, groupby-orderby-limit, and lastpoint.

IoT Use Case

We then moved to the TSBS IoT use case, which is more complex but more representative of the time-series data that TDengine users typically manage. Unfortunately, six of the twelve queries failed to run in InfluxDB 3 Core.

Note: Entries listed as — indicate that the query failed or returned no results.

InfluxDB 3 Core outperformed InfluxDB 1.8 in only one category, breakdown-frequency. For the other categories that returned results, it ranged from 1.1x slower in avg-load to 30x slower in last-loc.

The six queries that failed were stationary-trucks, long-driving-sessions, long-daily-sessions, avg-daily-driving-duration, avg-daily-driving-session, and daily-activity. Of these, the latter three queries were not compatible with the InfluxQL implementation in InfluxDB 3 Core. As an example, the avg-daily-driving-duration query returned the following error.

Similar errors occurred for avg-daily-driving-session and daily-activity. As an experiment, we modified the InfluxQL queries and were able to get them to run in InfluxDB 3 Core with some effort, demonstrating that the InfluxQL API is not fully compatible with InfluxDB 1.8.

In addition, the stationary-trucks, long-driving-sessions, and long-daily-sessions queries returned empty result sets. The queries all returned very quickly but did not contain any results. At this time, we have not been able to update these queries for InfluxDB 3 Core.

Furthermore, when we increased the size of the query dataset to 4,000 devices, we also had to reduce the number of workers as described in the DevOps scenario. The affected queries were last-loc, high-load, avg-vs-projected-fuel-consumption, avg-daily-driving-session, avg-load, and breakdown-frequency.

Try for Yourself

To automate the testing process and enable our users to compare databases using TSBS, we have developed a script that you can run on your own machine. On an Ubuntu 22 machine, clone the enh/add-influxdb3.0 branch of our TSBS fork to the /usr/local/src directory. Then open the scripts/tsdbComp directory and run the tsbs_test.sh --help command as the root user.

sudo -s
git clone https://github.com/taosdata/tsbs
cd tsbs
git checkout enh/add-influxdb3.0
cd scripts/tsdbComp
./tsbs_test.sh --help

This describes the scenarios that you can test and the configuration options available to you. Note that for performance reasons, a machine with 4 cores and 8 GB of RAM is required to use this script.

Conclusion

TSBS testing shows that InfluxDB 3 Core is still inferior to InfluxDB 1.8 in terms of query performance. More importantly, InfluxQL compatibility is still an issue for InfluxDB 3 Core. In addition, as queries in InfluxDB 3 Core are limited to 72 hours, many of the typical time-series use cases, such as those in TSBS, will be difficult or impossible to implement. As InfluxDB 3 Core is still in the public alpha phase, we are sure that its performance and compatibility will be improved in later releases. However, we advise open-source users to be wary of upgrading for the time being.

Building Real-Time Dashboards with Grafana

Rebecca Tao — Mon, 10 Mar 2025 17:22:21 +0000

Time-series data is everywhere, from monitoring industrial sensors and tracking financial markets to logging system performance and analyzing user behavior. However, raw data alone isn’t always useful. Effective visualization helps uncover patterns, detect anomalies, and drive better decisions. This is where Grafana, an open-source analytics and monitoring tool, shines. In this article, we’ll explore how to use Grafana to visualize time-series data and make the most of its powerful features.

Why Grafana for Time-Series Data?

Grafana is one of the most popular visualization tools for time-series data, thanks to its versatility, real-time monitoring capabilities, and compatibility with various databases. Here’s why it stands out:

Supports Multiple Data Sources: Works seamlessly with time-series databases like TDengine, InfluxDB, Prometheus, and TimescaleDB, as well as relational databases like PostgreSQL and MySQL.
Customizable Dashboards: Provides a flexible dashboard interface with multiple visualization options, such as line charts, heatmaps, and histograms.
Real-Time and Historical Data Analysis: Allows users to explore both real-time streaming data and historical trends for deeper insights.
Alerting System: Enables setting up alerts based on specific conditions, ensuring timely responses to critical events.
Plugins and Extensibility: Offers a wide range of plugins for additional features and integrations.

Real-World Use Cases

Industrial IoT Monitoring

Companies use Grafana to visualize sensor data from thousands of IoT devices, tracking metrics like temperature, pressure, and power consumption.

Financial Market Data

Traders and analysts use Grafana to track stock prices, market trends, and portfolio performance with real-time updates.

IT Infrastructure Observability

Organizations monitor server uptime, CPU utilization, and database performance to prevent downtime and optimize resources.

Conclusion

Grafana is a powerful tool for visualizing and monitoring time-series data, enabling businesses and engineers to gain deep insights into trends and anomalies. Whether you’re tracking system performance, analyzing IoT data, or monitoring financial markets, Grafana provides the flexibility and real-time capabilities needed for effective decision-making.

If you’re working with time-series data, integrating Grafana with TDengine can further enhance performance, providing high-speed queries and optimized storage tailored for massive time-series datasets.

Ready to take your data visualization to the next level? Set up Grafana today and start exploring the hidden insights in your time-series data!

TDengine Achieves 10x Compression vs. Elasticsearch for Smart Vehicle Solution Provider

Rebecca Tao — Fri, 07 Mar 2025 08:59:43 +0000

A smart vehicle solutions provider had developed a system for their tire products. As the scale of data within the system continued to grow, the overall business response speed and stability of the system became pain points for their internal and external customers. To improve performance with large-scale data, the project team responsible for the system decided to replace Elasticsearch with TDengine. This article provides a comprehensive analysis of how the system was transformed by the introduction of TDengine.

Project Background

Over the past two years, the system had been integrated with a smart rental system as well as the fleet management system of a major domestic logistics firm. With this expansion, Elasticsearch was no longer able to meet users’ data demands. The team began researching alternative data management platforms and in the end chose to deploy a specialized time-series database.

After conducting various evaluations, TDengine was ultimately selected as the time-series database for the smart tire system. The main reasons mentioned are listed as follows:

Business Suitability: TDengine is designed and optimized for industrial use cases like IoT and connected cars.
Operational Costs: TDengine Enterprise includes 24/7 technical support for operations and maintenance, significantly reducing operational costs.
Data Compression: TDengine’s columnar storage architecture enables a high data compression ratio. Compared with Elasticsearch, real-world tests showed that TDengine achieved a compression ratio of over 1:10, cutting disk storage costs.
High Read/Write Performance: TDengine has excellent write performance for time-series data. Tests with a three-node high-performance cloud disk setup achieved a write speed of 300,000 data points per second. Its read performance is also outstanding, with query speeds 10 times faster than Elasticsearch for the same volume of time-series data.
Low Resource Consumption & High Availability: TDengine has low server resource usage and offers high availability and stability. It supports user-defined functions (UDFs) for algorithm integration and various built-in mathematical functions.
Cold & Hot Data Separation: TDengine supports tiered storage, automatically migrating historical data to lower-cost storage media, further reducing enterprise data storage expenses.

System Architecture & Implementation Insights

The current project uses TDengine Enterprise 3.1.1.11 in a three-node cluster, with each node configured with a 16-core CPU, 64 GB of RAM, and 3 TB of high-performance cloud storage. Below is a simplified data flow architecture:

Edge gateways collect metrics such as tire temperature, pressure, and leakage status at set intervals (in seconds). After initial data integration and threshold comparison at the edge, the edge gateway sends data to the cloud gateway at three-minute intervals (or in seconds during alert conditions) via TCP and a proprietary protocol.

After passing through the cloud gateway, real-time streaming data undergoes decoding, real-time analysis, alert pushing, and data integration before being batch-written into TDengine. Data is presorted before ingestion to ensure sequential writes and reduce data fragmentation.

TDengine’s “one table per device” model was adopted for data storage, and the database for vehicle tire temperature and pressure data was configured as follows:

Data retention period: 180 days
File partitioning: 1-day intervals
Data differentiation: Timestamp-based distinction for multiple tire positions on the same vehicle

Additionally, by leveraging TDengine’s built-in functions and user-defined functions, real-time stream processing was implemented for time-series algorithms (e.g., real-time temperature situational awareness) to improve warning efficiency, reduce false alarms, and enhance responsiveness.

Performance & Transformation Results

Since going live, the project has been running stably with a steady resource consumption rate. Compared with Elasticsearch, TDengine has demonstrated significant advantages in storing time-series data:

Disk Usage:

Elasticsearch: Uses Lucene for JSON-based storage, leading to low compression and high disk usage.
TDengine: Columnar storage ensures high compression, achieving a 10x reduction in disk usage compared to Elasticsearch.

Memory Usage (64 GB per node):

Elasticsearch: Each of the 4 nodes used more than 54 GB of memory.

TDengine: Each of the 3 nodes used only about 4 GB of memory.

Other advantages of TDengine include:

High Read/Write Performance: Extremely fast sequential write speeds and highly efficient time-range queries.
Complex Time-Series Computation: The UDF feature allows real-time stream computing for tire temperature trends and other complex time-series analytics without affecting overall system stability or latency.
Data Subscription Services: Using TDengine’s data subscription feature, data is seamlessly transferred to a cloud compute platform for near real-time visualization and modeling. Additionally, third-party platforms can leverage TDengine’s data subscription feature for seamless data integration, facilitating data exchange with logistics companies, vehicle manufacturers, and public transportation providers.

Visualization & Future Prospects

In the field of vehicle tire temperature and pressure monitoring, the completeness, real-time performance, interactivity, and flexibility of displayed information are crucial. TDengine’s efficient query capabilities and easy-to-use SQL syntax enable seamless implementation of these requirements. TDengine has increased performance and reduced costs at this company, and they have also used its integrations to built a dashboard displaying various vehicle alert events, supporting fleet operators in making data-driven decisions.

InfluxDB 3.0 vs 1.8: A Surprising Decline in Ingestion Speed

Rebecca Tao — Thu, 06 Mar 2025 01:25:57 +0000

Highlights

TSBS testing demonstrates that for datasets with fewer than 1 million devices, InfluxDB OSS 3.0 actually ingests data more slowly than InfluxDB OSS 1.8.

For datasets of 1 million or more devices, InfluxDB OSS 3.0’s data ingestion performance is faster than InfluxDB OSS 1.8.

InfluxDB OSS 3.0’s data ingestion interfaces are not consistent with InfluxDB 1.8, meaning that users cannot migrate smoothly.

When InfluxDB 3.0 was released to enterprise customers back in 2023, it was claimed to be a major performance improvement over previous versions. At the time, InfluxData published a benchmark report comparing the new enterprise version with InfluxDB OSS 1.8, which showed great results across the board for InfluxDB 3.0.

We wanted to try it for ourselves, but we’re not enterprise customers — so like the rest of you, we had to wait for a year and a half till InfluxDB OSS 3.0 finally came out this January. And although the current release is admittedly just a “public alpha,” we still had high hopes for its performance. Considering that InfluxData rearchitected their entire product for the second time, causing great frustration to existing users hoping for a smooth upgrade process, and even deprecating their Flux language, they must have been able to produce amazing results. Otherwise, it would hardly be worth upgrading, right?

Putting InfluxDB to the Test

To prove how much better InfluxDB 3.0 performs, we decided to compare it against InfluxDB 1.8 using the Time Series Benchmark Suite (TSBS) originally developed by InfluxData and now maintained, to some extent, by Timescale. Because InfluxDB 3.0 supports InfluxQL and InfluxDB’s line protocol, in theory the test suite for 1.8 should be able to run on the new public alpha. In fact, we did run into several compatibility issues during the testing process and had to find workarounds; this is discussed in the Methodology section below.

We wanted to use TSBS because it would give more comprehensive results than the somewhat simplistic benchmarks run by QuestDB and also be a fair comparison on an open and well-established framework. The results, however, were surprising.

TSBS includes two use cases, a DevOps scenario involving CPU monitoring, and an IoT scenario involving vehicle tracking. For this comparison, we used TSBS to generate the data for both use cases and ingest it into InfluxDB 3 0.1.0 revision v2.5.0-14345, InfluxDB OSS 1.8, and TDengine OSS 3.3.5.8. The following tables show the data ingestion performance of each system in metrics per second.

TSBS DevOps Use Case

TSBS IoT Use Case

In the largest-scale datasets, ingestion performance increased by 5.4x in DevOps and 4.9x in IoT — a far cry from the “45x better write throughput” that InfluxData’s benchmark report claims for InfluxDB 3.0 Enterprise. More alarmingly, in the three datasets with 100,000 devices or fewer, InfluxDB 1.8 actually outperformed InfluxDB 3.0. We can see that for data ingestion, the claimed performance improvements in InfluxDB 3.0 are either limited to enterprise customers or do not hold up against independent testing.

InfluxData’s claims from their website

Although InfluxDB 3.0 has been rebuilt from the ground up, its ingestion performance still cannot match a modern time-series database like TDengine. The test results show that TDengine‘s ingestion was between 4.4x and 11.3x faster than InfluxDB 3.0, and between 3.1x and 22.8x faster than InfluxDB 1.8.

Methodology

All testing was performed on a 40-core machine with 256 GB of RAM. While this is slightly smaller than the machine used in InfluxDB’s benchmark report and larger than the machine used by QuestDB, we do not believe that upgrading or downgrading hardware would affect the overall trends in performance.

Because TSBS has not been updated to support InfluxDB 3.0, we were forced to make certain modifications to enable testing. In the interest of fairness, we tried to limit our modifications as much as possible, but the following issues needed to be addressed:

1.The commands to show, create, and delete databases used in TSBS for InfluxDB 1.8 were not operational in InfluxDB 3.0. Instead, we used the InfluxDB v3 API as follows:

GET /api/v3/configure/database to show databases
POST /api/v3/configure/database to create databases
DELETE /api/v3/configure/database to delete databases

2.With more than one worker, InfluxDB 3.0 would experience failures to write data and throw the error Invalid write response (status 409): catalog update error: table already exists. To avoid this, we modified TSBS to retry writes that failed due to this error, instead of exiting. We used the /api/v3/write_lp interface to write data to InfluxDB 3.0.

You can review all modifications on our fork of TSBS and also run the test suite yourself using our scripts.

Conclusion

While the newly redesigned InfluxDB 3.0 promises much, when it comes to ingestion, at least for open-source users, it fails to deliver. Comprehensive testing demonstrates that its ingestion performance is only somewhat superior to InfluxDB 1.8 for the largest datasets that most users will never encounter, and even falls behind for sites that include fewer than a million devices. Although the currently released version is a public alpha not ready for production, after more than a year, we have to ask: is this the best that InfluxData can offer to the open-source community?

With the difficulty of upgrading existing deployments to InfluxDB 3.0, open-source users need to decide whether it’s worth the trouble. If ingestion performance has been a bottleneck for you, it’s clear that, at least for now, upgrading to 3.0 is not a solution. Instead, consider a time-series database like TDengine that respects its community by consistently releasing high-performance, full-featured software that is available to everyone.

InfluxDB 3.0 vs 1.8: A Surprising Decline in Ingestion Speed

Rebecca Tao — Thu, 06 Mar 2025 01:25:57 +0000

Highlights

TSBS testing demonstrates that for datasets with fewer than 1 million devices, InfluxDB OSS 3.0 actually ingests data more slowly than InfluxDB OSS 1.8.

For datasets of 1 million or more devices, InfluxDB OSS 3.0’s data ingestion performance is faster than InfluxDB OSS 1.8.

InfluxDB OSS 3.0’s data ingestion interfaces are not consistent with InfluxDB 1.8, meaning that users cannot migrate smoothly.

Putting InfluxDB to the Test

TSBS DevOps Use Case

TSBS IoT Use Case

InfluxData’s claims from their website

Methodology

1.The commands to show, create, and delete databases used in TSBS for InfluxDB 1.8 were not operational in InfluxDB 3.0. Instead, we used the InfluxDB v3 API as follows:

GET /api/v3/configure/database to show databases
POST /api/v3/configure/database to create databases
DELETE /api/v3/configure/database to delete databases

You can review all modifications on our fork of TSBS and also run the test suite yourself using our scripts.