Introduction
Data lakes have become the backbone of modern analytics, allowing organizations to store petabytes of structured and unstructured data in cloud object storage services such as Amazon S3, Azure Blob Storage, and Google Cloud Storage. Open table formats like Apache Iceberg, Delta Lake, Apache Hudi, and Parquet have further accelerated this trend by enabling multiple analytics engines to access the same datasets without vendor lock-in or unnecessary data duplication.
While querying data directly from cloud object storage offers tremendous flexibility, it also introduces performance challenges. Network latency, metadata lookups, and large Parquet file scans can significantly slow query execution compared to locally stored data.
ClickHouse® 26.3 LTS addresses these challenges with several enhancements designed specifically for data lake workloads. These include improved parallel object storage reads, a built-in Parquet metadata cache, and asynchronous Iceberg metadata prefetching. Together, these optimizations reduce query latency and make interactive analytics over cloud-hosted datasets significantly faster.
In this article, we'll explore how ClickHouse® queries data lakes, the challenges involved, the new optimizations introduced in version 26.3 LTS, and practical examples of how these improvements benefit real-world analytical workloads.
Why Data Lake Query Optimization Matters
Modern analytics architectures increasingly separate storage from compute.
Instead of copying data into multiple databases, organizations store analytical datasets in cloud object storage and allow various query engines to access the same files directly.
This architecture provides several benefits:
- Lower storage costs
- Vendor-neutral open formats
- Simplified data management
- Multiple analytics engines sharing the same datasets
However, it also introduces several performance challenges:
- Remote storage latency
- Metadata lookups
- Large file scans
- Network overhead
ClickHouse® 26.3 introduces several optimizations that specifically target these bottlenecks.
Understanding Data Lake Queries
A data lake is a centralized repository that stores both raw and processed data using open formats, allowing multiple analytics engines to query the same datasets without duplication.
Unlike traditional data warehouses, where data is imported into proprietary storage, data lakes keep information inside cloud object storage.
Popular storage platforms include:
- Amazon S3
- Azure Blob Storage
- Google Cloud Storage
Common open table formats include:
- Apache Iceberg
- Delta Lake
- Apache Hudi
- Parquet
Instead of importing data into MergeTree tables, ClickHouse® can query these datasets directly.
During query execution, ClickHouse:
- Reads metadata
- Identifies the required files
- Reads only the required columns
- Applies predicate pushdown where possible
- Returns results without copying data into native storage
Cloud Object Storage
│
▼
Parquet / Iceberg / Delta / Hudi
│
▼
ClickHouse
│
▼
SQL Analytics
This architecture eliminates duplicate storage while allowing ClickHouse® to serve as a high-performance analytical engine over existing data lakes.
Challenges of Querying Data Lakes
Although cloud object storage is highly scalable and cost-effective, querying remote datasets introduces several challenges.
Remote Storage Latency
Unlike local disks, every query must retrieve data across the network, increasing response times.
Metadata Overhead
Open table formats maintain metadata describing snapshots, manifests, partitions, and data files.
Before reading any actual data, ClickHouse® must first retrieve and process this metadata.
Large File Scans
Poor partitioning or inefficient pruning may require scanning significantly more data than necessary.
Repeated Metadata Reads
Interactive dashboards often execute the same queries repeatedly.
Without caching, ClickHouse® must repeatedly download identical metadata from remote storage.
How ClickHouse® Queries Data Lakes
ClickHouse® supports querying data directly from cloud object storage without requiring ingestion into MergeTree tables.
Supported technologies include:
- Apache Iceberg
- Delta Lake
- Apache Hudi
- Parquet files
- Amazon S3 and compatible object storage
During query execution, ClickHouse® reads only the required files and columns, minimizing unnecessary I/O and enabling efficient analytics over remote datasets.
Data Lake Enhancements in ClickHouse® 26.3 LTS
Version 26.3 introduces several improvements that significantly reduce latency when querying remote object storage.
1. Faster Parallel Reading
One of the biggest improvements is enhanced parallel reading of remote data.
When a query accesses a relatively small number of large files, ClickHouse® now distributes work more efficiently across available CPU cores.
This optimization applies to:
- Apache Iceberg
- Delta Lake
- Apache Hudi
- Object storage reads
Benefits include:
- Better CPU utilization
- Faster remote file processing
- Lower overall query execution time
For many workloads, queries become several times faster on multi-core systems.
2. Built-in Parquet Metadata Cache
Reading Parquet files requires accessing the file footer to obtain schema and metadata information.
Before ClickHouse® 26.3, repeated queries frequently reread this metadata from remote storage.
The new Parquet Metadata Cache stores footer information in memory.
Benefits include:
- Reduced metadata reads
- Lower remote I/O
- Faster repeated queries
- Improved dashboard responsiveness
The cache is enabled by default and automatically tracks object changes using file ETags to maintain consistency.
3. Asynchronous Iceberg Metadata Prefetching
Apache Iceberg maintains metadata describing snapshots, manifests, partitions, and data files.
Earlier versions often fetched this metadata during query execution, increasing planning time.
ClickHouse® 26.3 introduces asynchronous metadata prefetching.
Instead of waiting during query execution, ClickHouse® refreshes Iceberg metadata in the background and serves queries from the cache whenever possible.
Benefits include:
- Reduced planning latency
- Faster repeated queries
- Improved dashboard responsiveness
Architecture Overview
Data Lake
│
▼
Cloud Object Storage
(Amazon S3 • Azure Blob • GCS)
│
▼
Apache Iceberg • Delta Lake
Apache Hudi • Parquet
│
▼
ClickHouse® 26.3 LTS
• Parallel File Reads
• Parquet Metadata Cache
• Iceberg Metadata Prefetch
│
▼
Fast SQL Analytics & Dashboards
Example 1: Querying a Parquet Dataset
SELECT
country,
count() AS total_events
FROM s3(
'https://my-bucket.s3.amazonaws.com/events/*.parquet'
)
WHERE event_date >= today() - 7
GROUP BY country;
In this query, ClickHouse® reads Parquet files directly from cloud storage.
Because only the required columns are read and filtering is applied early, significantly less data must be processed.
Example 2: Querying an Iceberg Table
SELECT
product_category,
SUM(revenue) AS total_sales
FROM iceberg('analytics.sales')
WHERE order_date >= today() - 30
GROUP BY product_category
ORDER BY total_sales DESC;
Here, ClickHouse® uses Iceberg metadata to identify the necessary files.
The metadata cache and asynchronous prefetching reduce planning overhead while enabling efficient execution.
Real-World Example
Imagine an e-commerce company storing several years of clickstream data in Amazon S3 using Apache Iceberg.
Instead of copying terabytes of historical data into ClickHouse®, analysts query the Iceberg tables directly.
With ClickHouse® 26.3:
- Enhanced parallel reading speeds up remote file processing.
- The Parquet metadata cache avoids repeatedly reading footer metadata.
- Iceberg metadata prefetching reduces planning latency.
As a result, dashboards become significantly faster without requiring data duplication or complex ETL pipelines.
Performance Benefits
| Feature | Benefit | Why It Matters |
|---|---|---|
| Enhanced Parallel Reading | Faster query execution | Better CPU utilization when reading remote files |
| Parquet Metadata Cache | Lower query latency | Eliminates repeated metadata reads |
| Iceberg Metadata Prefetching | Faster query planning | Metadata is available before execution |
Together, these optimizations improve:
- Interactive dashboards
- BI workloads
- Exploratory analytics
- Recurring analytical queries
Best Practices
To maximize performance when querying data lakes with ClickHouse®:
- Store analytical datasets in Parquet format.
- Partition datasets using commonly filtered columns.
- Select only the columns required by your queries.
- Apply filters as early as possible.
- Take advantage of the built-in Parquet metadata cache.
- Enable asynchronous Iceberg metadata prefetching for frequently queried datasets.
- Keep ClickHouse® updated to benefit from ongoing data lake optimizations.
When Should You Use These Features?
ClickHouse® 26.3 is particularly valuable for organizations that:
- Store analytical datasets in cloud object storage
- Query Apache Iceberg tables
- Query Delta Lake datasets
- Analyze Apache Hudi data
- Perform ad hoc SQL analysis over Parquet files
- Build interactive BI dashboards
- Operate large-scale cloud-native analytics platforms
Key Takeaways
- Query cloud-hosted datasets without duplicating data.
- Parallel reading accelerates remote file processing.
- The Parquet metadata cache reduces repeated metadata access.
- Iceberg metadata prefetching lowers query planning latency.
- ClickHouse® 26.3 makes interactive analytics over modern data lakes significantly faster and more efficient.
Conclusion
As organizations continue adopting cloud-native architectures and open table formats, efficiently querying data lakes has become essential for modern analytics. ClickHouse® already enables users to query Apache Iceberg, Delta Lake, Apache Hudi, and Parquet datasets directly from cloud object storage without duplicating data or building complex ETL pipelines.
ClickHouse® 26.3 LTS further strengthens these capabilities with enhanced parallel object storage reads, a built-in Parquet metadata cache, and asynchronous Iceberg metadata prefetching. Together, these improvements reduce metadata overhead, minimize remote I/O, improve CPU utilization, and significantly accelerate query execution.
Whether you're building business intelligence dashboards, exploring petabyte-scale datasets, or developing cloud-native analytics platforms, these enhancements make ClickHouse® an even more compelling engine for high-performance analytics directly on modern data lakes.
Top comments (0)