DEV Community: Altinity

Join OSA CON 2025: Two Days of Open‑Source Analytics and AI (Nov. 4–5)

Altinity — Mon, 20 Oct 2025 10:31:10 +0000

The Open Source Analytics Conference (OSA CON) brings together data engineers, analysts, product managers, and builders shaping the future of open-source analytics.

It’s all about accelerating AI while celebrating the tools and communities that make modern data stacks possible. Expect candid talks from open‑source innovators rather than product pitches.

When & Where

OSA CON 2025 takes place online on November 4–5. It’s completely free to attend. Sessions run in your local timezone on Airmeet, and full replays will be available right after the event.

Who It’s For

If you work with data, whether as a data scientist, engineer, analyst, designer, or product manager, OSA CON is for you.

It’s also a great space for open‑source maintainers, contributors, and decision‑makers to connect.

What to Expect

This year’s line‑up covers AI, databases, streaming, data lakehouses, ingestion, orchestration, governance, visualization, and more. Here are some of the standout sessions:

[Keynote] Open Source’s Massive Unfair Advantage in the AI Era: Maxime Beauchemin (Apache Superset & Airflow Creator) explains why open source is positioned to dominate AI and why communities, not corporations, are the new centers of gravity.
[Keynote] Don’t Fire Your Developers and Other Lessons for the AI Revolution: Robert Hodges (Altinity CEO) explores how AI will change developer workflows and why humans still matter.
[Keynote] How Open Source Businesses Will Thrive in the Age of AI: Heather Meeker (Chinstrap Community Co-Founder) dives into commercial open source (COSS) models and how generative AI will reshape them.
Smarter Analytics: AI‑Driven Intelligence in Modern Databases: Peter Zaitsev (Percona & Coroot Founder) shows how AI can turn databases into context‑aware engines, from schema optimization to self‑driving databases and multimodal analytics.
What the Spec?!: New Features in Apache Iceberg™ Table Format V3: Danica Fine (Snowflake Lead Developer Advocate for open source) highlights the leap from Iceberg V2 to V3, walking through user‑facing improvements such as variant types for semi‑structured data and row‑lineage to enhance CDC.
Streaming Analytics in Action: Real-World Case Studies from Uber, Razorpay, and Stripe: Jayesh Asrani (StarTree Principal Solutions Architect) shares how three high‑growth companies use next‑gen streaming architectures for real‑time decision‑making and discover reference designs you can apply.
Agnostic is the Only Constant: Embracing the Lakehouse Paradigm Without Lock‑In: Viktor Kessler (Vakamo Co‑Founder & CEO) introduces Lakekeeper and shows how Iceberg‑based lakehouses remain portable across clouds and compute engines.
ClickHouse® Chronicles: Real‑World War Rooms with Human and AI Agents: Hear battle‑tested stories of running ClickHouse at scale and solving production incidents from Shivji Kumar Jha (Nutanix Staff Engineer) and Anurag Pandey (Incerto Co-Founder). If you’re into ClickHouse internals, this is a must‑attend.
Why Your FAQ Search Sucks and How I Fixed It: Chris Dabatos (TiDB Developer Advocate) shows how he built a lightning‑fast semantic FAQ search in ~300 lines of Python using TiDB and Amazon Bedrock embeddings.
Open Source Healthcare Analytics: 20 % ICU Transfer Reduction at Scale: Rakshit Khare (AWS Sr. Global Specialist Solutions Architect) shares an open‑source platform that reduced ICU transfers by 20 %, built with Kafka, Spark, and TimescaleDB, and discusses privacy‑preserving analytics and federated learning.

Backed by the Community

OSA CON is powered by the teams who build and maintain the tools developers rely on daily.

Altinity, the second largest contributor to the ClickHouse project, and Preset (the SaaS version of Apache Superset) headline as Platinum Sponsors.

VELODB, DataKitchen, Fiveonefour are sponsors, and Open Source Observability Day, Packt, Clowder Space, Airflow Summit, The Untitled Data Company, Abracadata, and Beam Summit are community partners.

Ready to Join?

Registration is open! Mark your calendar for November 4–5, 2025, and get ready to learn, connect, and contribute to the open-source analytics movement.

Understanding Detached Parts in ClickHouse®

Altinity — Tue, 16 Sep 2025 11:28:35 +0000

If you’ve ever worked with ClickHouse, you’ve probably encountered its unique way of handling data. One of the more intriguing concepts in ClickHouse is detached parts. While they might sound like a minor technical detail, understanding detached parts is essential. ClickHouse is designed to fully utilize available system resources. However, when the system becomes overloaded and resources such as memory, CPU, and I/O are maxed out, there is a high risk of out-of-memory (OOM) errors or unresponsive pods being restarted by the kubelet due to failed liveness probes. These abrupt or unexpected restarts can result in the appearance of detached parts.

In this blog post, we’ll break down what detached parts are, why they exist, and how to manage them effectively.

What Are Detached Parts?

Let’s first go to the basics. In ClickHouse, data is stored in parts. Each part is an immutable chunk of data, organized in a columnar format. These parts are the building blocks of ClickHouse tables, and they allow the database to achieve its famous performance for analytical queries.

ClickHouse merges consolidate multiple smaller data parts within a table into fewer, larger parts to improve query performance and reduce storage overhead. When data is inserted into a MergeTree table, it’s initially written as small, immutable parts. Over time, the system automatically merges these parts based on certain rules and heuristics (like size and age) to keep the number of parts manageable and maintain optimal read efficiency. While merges improve performance in the long run, they consume CPU, disk I/O, and memory, so their behavior can be fine-tuned or throttled for low-resource environments.

Figure 1: Merging smaller parts into a bigger one

But what happens when data changes? Since, ClickHouse doesn’t modify existing parts directly, (they are immutable, remember?), merges also handle data cleanup for deleted or updated rows. These type of merges are called mutations. If a part needs to be mutated, ClickHouse will create a new part with the changes, mark the old one as inactive and the new one as active, so the cleaning background process can delete those and reclaim space.

Inactive parts are essentially data segments that are no longer part of the active dataset but still exist on disk. They’re like old snapshots of your data, waiting to be cleaned up.

Ok, so what is the difference between inactive parts and detached parts? Think of detached parts as data chunks that are still physically present on your disk but are intentionally excluded from the table’s active data set. They can be explicitly detached by a ClickHouse command or can be automatically detached by Clickhouse, which is the most common case.

In contrast to detached parts, inactive parts are not explicitly put aside by an administrator. Instead, they represent an internal, transient state of data parts that are no longer considered the “current” version and are slated for eventual removal.

In the next image you can see the building blocks of a part and how it is named. The naming is important, and I will explain it in a later section.

Figure 2: The basic building blocks of a part

Why Do I Get Detached Parts?

Detached parts are a byproduct of ClickHouse’s design philosophy, which prioritizes immutability and performance. Here are some common scenarios where detached parts come into play:

1. Failed Inserts/Merges/Mutations

ClickHouse doesn’t modify the existing data. Instead, it creates new parts with the updated data and after mutation is finished, simply marks the old ones as inactive. This ensures that queries can continue running without interruption while the merge/mutation is in progress.

If a data ingestion or merge/mutation operation fails because of:

Unexpected/hard restart
A bug in ClickHouse leading to damage of mutated / merged data
Misconfiguration of macros / volumes or similar user mistakes
Partial merge due to lack of fsyncs

ClickHouse might leave behind detached parts as a safety measure. This prevents data corruption and allows you to recover from the failure if possible.

For example, a merge/mutation failed because of a hard restart and we have the source parts, which were merged, but the merge was half-flushed at the moment of the hard restart. ClickHouse will detach the source parts as ‘covered’, and the new part which has some columns unreadable will produce some errors during queries. So what can we do? We can recover from old covered parts in the detached directory and drop the corrupted half-merged part. We will talk about this in the section Reattaching detached parts.

2. Manual Detachment

You can manually detach parts using the ALTER TABLE ... DETACH PART command. This is useful for troubleshooting or temporarily removing data without deleting it permanently. This procedure will help in the above example. We would detach the corrupted part and attach the older parts.

3. Replication and Merges

In replicated tables, detached parts can occur during merge operations or when there are inconsistencies between replicas. ClickHouse detaches parts to ensure data consistency across the cluster.

Where Are Detached Parts Stored?

Detached parts live in a special subdirectory called detached within the table’s data directory. For example, if your table is located at:

/var/lib/clickhouse/data/database_name/table_name/

You’ll find the detached parts in:

/var/lib/clickhouse/data/database_name/table_name/detached/

Each detached part is stored in its own directory, named after the part’s identifier. This makes it easy to identify and manage them.

Prefixes for Internally Detached Parts

When ClickHouse moves a part to the detached folder itself, it’s typically because it has identified an issue with the part or it’s no longer needed in the active set. These prefixes provide valuable context for troubleshooting. You’ll also see these in the reason column of the system.detached_parts table. While some prefixed detached parts (like ignored_ and cloned_) are often safe to remove after a while, it’s crucial to understand the reason before deleting any detached part, especially those with broken_, unexpected_, or consistency-related prefixes.

Here are the most common prefixes and their meanings:

ignored_

Meaning: This is one of the most common and generally “safe” prefixes. It means that the part was found during a system check (e.g., at startup or during an ATTACH operation for a replicated table) and was determined to be redundant. This usually happens when a larger, merged part already covers the data blocks contained in this ignored_ part. In essence, the data in the ignored_ part has been successfully incorporated into a newer, more efficient part.
Action: ignored_ parts are typically safe to delete (ALTER TABLE ... DROP DETACHED PART 'ignored_part_name'). They indicate successful data processing.

covered_by_broken_

Meaning: This prefix indicates that the part’s data is covered by a broken part (see broken_ below). This means that a part that should have superseded this one (e.g., a merged part) was found to be corrupted or invalid.
Action: This is a more serious indicator than ignored_. You should investigate why the covering part was broken_. If you’re confident that the data is not lost (e.g., it exists on another replica in a replicated setup or the covered parts still exist), you might eventually delete these, but thorough investigation is recommended first. For example part 202501_12_78_3 is broken but covering parts 202501_12_34_1, 202501_35_67_2, and 202501_68_78_1 are ok, so we can drop safely the broken part and reattach the covering ones.

broken_ / broken-on-start_

Meaning: This is a critical prefix. It tell us that ClickHouse encountered a problem trying to load or read data from this part. This could be due to:
- Corrupted data: Files within the part are damaged (e.g., bad checksums, incomplete writes).
- Missing files: Some files necessary for the part are absent.
- Hardware issues: Disk errors.
broken-on-start_ specifically refers to parts detected as broken during the ClickHouse server startup process.
Action: Parts with this prefix indicate potential data loss or integrity issues. Do not blindly delete these parts. You need to investigate the ClickHouse server logs (clickhouse-server.log) for more details about why the part was marked as broken. If you can confirm the data is available elsewhere or is not critical, you can then consider deleting them.

unexpected_

Meaning: This prefix is used for parts that ClickHouse finds on disk but are not registered in ZooKeeper (for ReplicatedMergeTree tables) or are otherwise not expected by the table’s metadata. This can happen if an insert operation was interrupted, a replica got out of sync, or during manual file system operations.
Action: For replicated tables, these often indicate inconsistencies with ZooKeeper. You might be able to resolve them by running SYSTEM SYNC REPLICA or SYSTEM RESTART REPLICA. For non-replicated tables, it could mean an incomplete write. Investigate the logs. Deleting these without understanding the cause can lead to data loss.

cloned_

Meaning: This prefix appears when ClickHouse performs a “clone” operation, typically during replica repair or data movement within a replicated cluster. If ClickHouse has some parts locally while repairing a lost replica, existing parts might be renamed with this prefix and put in the detached directory to avoid conflicts with the incoming parts from other replicas.
Action: cloned_ parts are generally safe to delete once you’re sure the replica has successfully synced and the data is consistent across the cluster. They are essentially superseded copies.

noquorum_

Meaning: In a replicated setup, this prefix might appear if parts were written but the write could not achieve the required quorum of replicas (when using insert_quorum feature), or if a part was supposed to be committed but the commit failed for some reason.
Action: Investigation is needed. It implies that a write might not have been fully consistent across all replicas.

merge-not-byte-identical_ / mutate-not-byte-identical_

Meaning: These are more specific to replicated environments. They indicate that a merge or mutation operation produced a result part that was not byte-identical across all replicas. This is a severe consistency issue and should be investigated immediately, as it suggests a problem with the data or the replication mechanism.
Action: High priority investigation is required. This often points to deeper issues in your cluster or data. This could happen during a rolling upgrade. Imagine that you move from an intel based instance to an arm based one. If the cluster is upgraded in a rolling fashion, there can be times when 1 replica is using arm based cpu and the other is still using intel. This can cause replication problems and multiple parts could be detached.

How to Manage Detached Parts in ClickHouse

While detached parts are a normal part of ClickHouse’s operation, they can accumulate over time and consume disk space. Here’s how you can manage them effectively.

1. Investigating Detached Parts

Let’s explain what you might’ve seen while exploring ClickHouse files: the first number in a part’s name it’s the table’s partition—think of it like the shelf where that chunk of data lives. For most tables, it might look like a date in YYYYMMDD format, which makes scanning part names surprisingly intuitive. For example, the part 20240601_10_20_2 tells us this part holds data in the 20240601 partition (say, “June 1, 2024”), spans block numbers 10 through 20, and sits at merge level 2. (If you’d like to know more, the Altinity Knowledge Base has a great article explaining part names and multiversion concurrency control.)

As you see, this naming pattern is more than just pretty labeling—it’s ClickHouse’s way of quickly checking if data overlaps, is duplicate, or is safely covered by already active parts in the table. When dealing with detached parts, ClickHouse reads this naming structure to know whether the data they’re holding is already there. Next time you’re poking around the /detached folder, remember: read those part names from left to right—they tell you the full story, starting with the partition, not just where the data begins.

The knowledge base also has an article with a query that lets you see if detached parts are covered or not. For example, you can run the query and get something like this:

┌─database───────┬─table────────────┬─partition_id─┬─name────────────────────┐
│ test           │ mt4              │ all          │ ignored_all_0_3407_746  │
│ test           │ mt4              │ all          │ ignored_all_3408_3408_0 │
└────────────────┴──────────────────┴──────────────┴─────────────────────────┘

Then because the query will check min_block_number and max_block_number and check if those blocks are covered by active healthy parts, then you know that those ignored parts could be dropped. If the query does not return any row but you see detached parts in system.detached_parts table then most probably those parts are not covered and you need to investigate.

2. Reattaching Detached Parts

Attaching a problematic or overlapping part could cause unexpected issues. A much safer workflow is to create a clone of your original table’s structure, and then play with the detached parts in isolation. Here’s how you can do that:

Create a new table with the same structure as the original: CREATE TABLE visits_test AS SELECT * FROM visits
Copy the detached part(s) from the original table’s detached/ directory to the same directory in the new table. You’ll find the detached directory at a path like /var/lib/clickhouse/data/[db]/[table]/detached/. Copy the relevant part folders directly (be sure ClickHouse isn’t running critical operations while you do this!).
Attach the part in your new table: ALTER TABLE visits_test ATTACH PART '20240601_10_20_2';

Now, you can freely query and inspect this part’s data in the visits_test table, without any risk to your production data or consistency. If something goes wrong, your original table remains untouched. This method is excellent for data forensics, debugging, or simply making sure the content of a detached part is what you expect before taking further action.

In short: using a scratch/test table is the safest way to handle and explore detached parts in ClickHouse. It’s quick, it’s easy, and it gives you peace of mind when working around lower-level storage operations.

3. Deleting Detached Parts

To free up disk space, you can permanently delete detached parts. Use the following command:

ALTER TABLE table_name DROP DETACHED PART 'part_name';

Alternatively, you can manually delete the directories from the detached folder on disk. However, this approach is riskier and should only be used if you’re confident about what you’re doing.

4. Monitoring Detached Parts

This table is your first stop for understanding why parts are in the detached directory. To get a list of all detached parts for a table, you can query the system.detached_parts table:

SELECT * FROM system.detached_parts WHERE table = 'table_name';

This query provides detailed information about each detached part, including its name, size, and creation time.

Best Practices for Handling Detached Parts

Schedule Regular Cleanups: Detached parts can accumulate over time, so it’s a good idea to periodically check for and remove unnecessary parts.
Backup Before Deletion: Before deleting detached parts, make sure you have backups or are certain the data is no longer needed.
Optimize Mutations: Use mutations sparingly. Consider using table engines like ReplacingMergeTree or CollapsingMergeTree to avoid frequent updates and deletes.
Monitor Disk Usage: Keep an eye on disk usage, especially if you’re working with large datasets or performing frequent mutations.
Provision adequately your clusters: If you have an underprovisioned cluster that is hit with tons of unoptimized queries, then there is a high chance of hard restarts and getting detached parts.

Final Thoughts

Detached parts are a fascinating aspect of ClickHouse’s architecture, reflecting its commitment to immutability and performance. While they might seem like a nuisance at times, they play a crucial role in ensuring data consistency and reliability.

By following the best practices outlined in this post, you can keep your ClickHouse cluster in top shape and avoid the pitfalls of unmanaged detached parts. Whether you’re a database administrator, a data engineer, or just a curious tech enthusiast, understanding detached parts is a small but important step in mastering ClickHouse.

Have you encountered detached parts in your ClickHouse journey? Share your experiences, tips, or questions in the comments below! Let’s keep the conversation going.

Announcing Project Antalya – Infinitely Scalable ClickHouse® Query on 10x Cheaper Iceberg Storage

Altinity — Thu, 04 Sep 2025 13:44:50 +0000

Originally published at altinity.com on April 16, 2025.

Exploding data volume is blowing up database cost and stability. We’re bringing real-time analytics to the next level by extending ClickHouse with Iceberg shared storage. Cheap and real-time: what’s not to like?

Is your data volume blowing up your analytics budget? You’re not alone. Today’s observability and AI applications generate petabytes daily, pushing traditional analytics architectures to their breaking point. We help ClickHouse users work around cost and management of large clusters on a daily basis. It’s time for a permanent fix.

Meet Project Antalya – our game-changing solution that combines ClickHouse’s lighting-fast queries with Iceberg’s cost-efficient storage. This isn’t just another feature update–it’s a fundamental rethinking of analytics economics that delivers:

90% reduction in storage costs by replacing replicated MergeTree with Iceberg/Parquet
10 to 100x faster queries through revolutionary swarm cluster technology
Zero application changes required – a drop-in upgrade for open source ClickHouse deployments
Universal data access that eliminates silos between real-time, AI, and batch workloads

Starting today, we’re inviting you to try out swarm clusters, our breakthrough technology that delivers sub-second reads on Iceberg/Parquet tables. Install an open source Project Antalya build and scale up to your price/performance point. Stop worrying about costs. It’s time to start building systems for the next generation of analytic systems. Welcome to Project Antalya.

Show me the code already!

We know many of you have been waiting — If you already know about Project Antalya and just want to try it out, jump to our GitHub antalya-examples repo.

So what’s the problem?

Rising costs are crushing analytics ROI

ClickHouse appeared as an open source project on GitHub in 2016. Its compressed columnar storage and lightning fast performance drastically reduced the cost of real-time analytics. It was helped by the fact that ClickHouse installations rarely exceeded a few tens of terabytes in total, which kept overall bills in check for most users.

The high performance of ClickHouse encouraged users to apply it to use cases with vastly larger data sizes, like observability and LLM monitoring. Altinity now operates or supports applications that load multiple petabytes per day. This represents an increase of up to three orders of magnitude.

ClickHouse is of course as fast as ever, but the enormous growth in data has led to a number of practical problems related to cost. The two most important are:

High storage costs – The traditional ClickHouse table design uses MergeTree with block storage with copies replicated across up to 3 servers. This design is around 10x more expensive than keeping the same data on S3-compatible object storage.
Excessive compute costs – Open source ClickHouse does not have a simple way to scale compute temporarily for a large batch job or split the resources used for data loading from resources for queries. Users instead provision for peak combined load, which needlessly inflates compute costs.

If data and compute costs were the only problem, we could solve them within ClickHouse. That would ignore another important development in the marketplace for analytics.

Silos of proprietary batch and real-time data are disappearing

Workloads for real-time analytics, AI, data science, and batch reporting often operate on exactly the same source data, originally stored in separate silos for each application. The separation has become complex and costly as data size has increased. Instead, users increasingly operate workloads on a single copy of source data stored on data lakes consisting of object storage, Parquet, and open table formats, of which Iceberg is most popular.

Keeping a separate copy of real-time data stored in ClickHouse is a common but costly design. Besides extra storage costs, separate silos mean separate data pipelines, which is yet another source of complexity and expense. There’s an obvious argument to reduce cost and complexity by sharing storage with other workloads. But there’s upside as well. ClickHouse has an adaptable, toolkit-like design and runs anywhere. Data lakes need real-time ingest, query, and data compaction. ClickHouse has those capabilities, but there’s no easy way to apply them.

Reduce storage costs and share data. Better data lakes are a freebie

An ideal solution to both problems would be a savvy implementation of compute/storage separation that combines the fast query and optimized native data structures of ClickHouse with shared storage on Iceberg. This preserves the strengths of ClickHouse but adds the ability to operate on very large tables shared with other workloads. Existing ClickHouse applications could extend seamlessly onto data lakes. At the same time, ClickHouse’s adaptable features would make data lakes better.

Embracing Iceberg requires rethinking low-level design assumptions of ClickHouse. However, the pay-off is huge. That’s exactly what we’re doing with Project Antalya.

Project Antalya – Transforming ClickHouse for Cost-Efficient Data Lake Analytics

Project Antalya adapts ClickHouse to use Iceberg data lakes as primary storage. It leverages the current strengths of ClickHouse: ability to run anywhere, open source licensing, and a powerful toolkit of existing features like parallel query, SQL, role-based access control, and tiered storage. The architecture has three main goals:

Store table data as single copies in a globally accessible Iceberg data lake. This reduces storage cost by 10x and enables sharing with other workloads like AI.
Allow applications to see all data, regardless of location, from a single ClickHouse SQL connection. This keeps access simple and enables existing ClickHouse applications to take advantage of data lake storage without changes.
Enable fast, independent scaling of ingest, merge, transformation, and query. This reduces costly over-provisioning of compute and opens the door to use of cheap resources like spot instances.

Most importantly, Project Antalya introduces additions to ClickHouse but does not break current features. Existing applications can seamlessly extend storage onto Iceberg. Developers build groundbreaking applications by combining real-time ClickHouse query and native storage with Iceberg data. Here’s the full architecture.

Project Antalya is more than just a “better ClickHouse.” It’s a powerful new way to build high-performance, low-cost analytic systems that can meet real-time SLAs on shared data lake tables. We are focusing on three main areas of innovation.

Iceberg, Parquet, and S3 performance – Low-level ClickHouse changes to speed up reads and (in future) writes. This includes reading Bloom filters, PREWHERE processing, Parquet metadata caching, and the like. We’re committing these features to upstream ClickHouse wherever possible.
Data access improvements – Introduction of stateless compute swarms, distributed caching, and tiered storage onto Iceberg storage. These are the key underpinnings to cut cost while keeping query at real-time speeds.
Cloud Native Runtime – ClickHouse and of course Project Antalya builds run anywhere. That said, we’re focused on delivering templates for cloud native operation that enable fast swarm auto-scaling, cache deployment, monitoring, as well as templates for deploying Kubernetes itself.

We are implementing the architecture in phases. The first phase introduces swarm clusters, which offer real-time compute on Iceberg tables using cheap, stateless ClickHouse servers. This is available today. Future phases will introduce tiered storage on Iceberg as well as the ability to build materialized views on data lakes. We’ll reveal yet more great features in time. Join our Slack Workspace if you have questions or watch for regular roadmap updates on this blog.

Let’s walk through a couple of use cases that show the versatility of the Antalya Architecture. There’s a lot more, but this should help you see where we are going.

Real-time query on Iceberg tables with cheap swarm clusters

Iceberg tables with Parquet reduce storage costs by 10x over replicated MergeTree. But there’s a down side. In today’s open source Clickhouse, they make queries painfully slow and expensive to scale. As query load goes up, users resort to costly over-provisioning to maintain performance. The only alternative is proprietary storage engines like SharedMergeTree.

The first step, of course, is to make Parquet reads as quick as possible. The second step is scaling compute to meet demand. Swarm clusters solve that problem. Swarm clusters are scalable pools of self-registering, stateless ClickHouse servers. ClickHouse clusters delegate reads on Parquet files to the swarm. This speeds up queries while minimizing load on existing ClickHouse applications.

Here’s how it works.

Swarms have a number of important features that make them both fast and cheap at reading Parquet.

Caching – Caches can reduce latency by 90% or more. There are three main types.
- ClickHouse initiator nodes cache Iceberg metadata. This reduces the cost of query planning.
- Swarm clusters cache S3 file blocks and parquet file metadata. They reduce the cost of reading files on specific swarm nodes.
- External HTTP caches save API requests to S3, including slow S3 ListObjectsV2 calls. They reduce the number of API calls (hence your cloud bill) and and also reduce latency across the entire swarm.
Auto-scaling – Swarm nodes combine easily with Kubernetes cluster autoscalers. They register themselves when they come up and deregister when they go away.
Ephemeral – Swarm nodes can run on spot instances, which typically reduce compute costs by somewhat over 50%. They use minimal storage, which further reduces costs.

Swarm clusters are not limited to Iceberg tables. They can read any Parquet data stored on S3. That includes Hive tables as well as Parquet files using wildcards.

To see swarm clusters in action for yourself, check out our antalya-examples repo on GitHub.

Extend large ClickHouse tables seamlessly to Iceberg storage

Tiered storage is a popular ClickHouse feature that automatically moves table data to cheaper storage as it ages. As a user you just see a single table regardless of where the data is actually stored. We’re extending tiered storage to move data into Iceberg, which reduces cost for older data by around 10x. You’ll still see a single table but older data will be 90% less expensive.

Here’s how it works. Queries that hit tiered tables are automatically dispatched to local storage and Iceberg storage in parallel. The swarm provides additional compute capacity for scanning cold data. That helps to keep clusters that manage hot data small. Applications just see a single set of results.

Existing ClickHouse features like Merge Table Engine help with this implementation. The design includes features to avoid gaps or duplicates in data when selecting. We expect to release the implementation around mid-year.

The best part about tiered storage is that you will be able to turn it on with a simple configuration setting without application changes. We expect large ClickHouse installations will be able to reduce storage cost by 80% or more once tiering is fully enabled.

Some Frequently Asked Questions

Shared Iceberg storage for ClickHouse clusters is a big jump forward. We expect you have a few questions at this point.

Is Project Antalya open source?

Yes. Code is available in the Antalya branch in the Altinity ClickHouse repo on GitHub. It’s the same repo we use for Altinity Stable Builds.

Is Project Antalya a fork of ClickHouse?

No. We’re submitting as many changes as practical to upstream ClickHouse. We’re balancing getting working code out quickly against maintaining compatibility with upstream.

Wait! What about Delta Lake Tables?

We love Delta Lake. It’s just that we need to pick a place to start and Iceberg at this point is more popular among users at large. We’ll turn to Delta Lake once we have Iceberg in hand. Upstream ClickHouse is already implementing support for Delta Kernel, which will improve support.

Where’s the performance test?

We’ll publish detailed performance results shortly and also show you how to reproduce them. The in-house results are good and getting better. If you can’t wait, check out our launch webinar for Project Antalya. It includes a nice demo showing 35x scaling.

What is your roadmap?

We have a well-defined roadmap that we’ll be opening up shortly. For now the main priorities are:

Swarm clusters (what we’re releasing today)
Hosted Iceberg catalogs and tooling
Iceberg Tiers for MergeTree tables
Swarm cluster auto-scaling

There’s a lot more on the way.

How can I get started?

Look at the antalya-examples repo. It shows how to set up Project Antalya on Docker and Kubernetes. Also, read our upcoming blog articles. We’ll be publishing a Project Antalya Tutorial and also show how to get the best performance from queries on Iceberg tables.

How can I contribute?

We welcome contributors. If you want to submit a PR, check out the antalya-examples README.md for more information.

Does this work in Altinity.Cloud?

Yes! We’ve done a lot of the internal development using Altinity.Cloud. It’s our favorite test bed because it can run any ClickHouse build version. Come back in May and we’ll let you try it yourself. We’ll have built-in support for swarm clusters by then. There is a follow-up webinar coming on May 21st that will go deep on production use of swarm clusters, including deployment in Altinity.Cloud.

In conclusion…

Project Antalya is more than just a new release — it’s a vision for the future of analytics. Many people will see Project Antalya as a “better” ClickHouse — and they’re not wrong. But this isn’t just an evolution. It’s a reimagined architecture that combines the capabilities of ClickHouse, cloud native operation, and Iceberg/Parquet to deliver analytic systems ready for a decade of growth. Get the performance you want, at the price you can afford, anywhere you choose to run. That’s the vision of Project Antalya.

We’re building this together — in the open — with your insights, challenges, and goals shaping the roadmap. This release, and all that follow, are dedicated to the community and customers who made it possible.

Try it out. We can’t wait to see what you’ll create.

ETL vs ELT Cage Fight: Combining Rudderstack & ClickHouse to Build Real-Time Data Pipelines

Altinity — Mon, 27 Nov 2023 16:22:02 +0000

In the realm of data engineering, the choice between ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) has been a topic of heated debate for years. In this article, we'll delve into the nuances of these approaches and explore how the combination of Rudderstack and ClickHouse can empower developers and data engineers to build robust real-time data pipelines.

Understanding the ETL and ELT Paradigms

ETL: Extract, Transform, Load

The ETL approach has been a staple in data pipeline architecture for decades. It involves extracting data from source systems, applying transformations to cleanse and structure it, and then loading the processed data into a data warehouse for further analysis. This method is particularly suitable for structured data that requires extensive transformations before reaching the data warehouse.

ELT: Extract, Load, Transform

In contrast, the ELT approach flips the order of operations (well, sort of, except the extract phase is still the same). Data is first extracted from source systems and loaded directly into the data warehouse. Transformations are then performed within the data warehouse, taking advantage of its powerful processing capabilities. ELT is advantageous for scenarios involving semi-structured or unstructured data and situations where maintaining an original copy of the data is important.

Introducing Rudderstack and ClickHouse

Rudderstack: A Customer Data Platform

Rudderstack is a powerful customer data platform designed to streamline data collection and integration processes. It provides standardized event schemas and offers software development kits (SDKs) for web, mobile, and server-side applications. With over 200 out-of-the-box integrations, Rudderstack enables seamless data delivery to various downstream tools, including marketing automation and analytics platforms.

ClickHouse: A Versatile Data Warehouse

ClickHouse is an open-source enterprise-grade data store designed for analytics and data warehousing. It offers unique deployment flexibility, allowing you to run ClickHouse clusters on-premises, in the cloud such as the Altinity.Cloud platform, or even on Kubernetes. ClickHouse excels at efficient data storage and enables rapid query performance, making it ideal for data analytics. Altinity is the leading service and software provider for ClickHouse for managing and analyzing extremely large volumes of data. Altinity helps companies build and maintain systems extracting insight from data at an unmatched performance.

Combining Forces: ETL and ELT for Real-Time Data Pipelines

Now, back to the cagefight. ETL vs ETL, which one should you be using? We say both. Although each has it’s own use cases, we like to think ETL and ELT are not competing paradigms but complementary strategies that can work together harmoniously to optimize data pipelines. Here's how this combination can empower developers and data engineers:

1. Flexible Transformations

Rudderstack provides real-time event streaming, ETL pipelines, and reverse ETL pipelines, catering to a variety of transformation needs.

ClickHouse's powerful querying capabilities within the data warehouse allow developers to perform complex transformations on the data, unlocking new insights.

2. Standardized Event Collection

Rudderstack's standardized event schemas streamline data collection, making it easier to integrate with downstream tools and data warehouses.

3. Efficient Data Storage and Analytics

ClickHouse's cloud-native and on-premises deployment options ensure versatile and efficient data storage, while its performance ensures rapid analytics.

4. Original Data Copy

ELT's emphasis on loading raw data into the data warehouse preserves the original data, enabling developers to experiment with various transformations without losing valuable information.

5. Scenario-Based Optimization

ETL is well-suited for structured data that requires substantial transformation before loading.
ELT shines when dealing with semi-structured or unstructured data and situations where data preservation is crucial.

Real-World Example: Funnel Analysis

To put these concepts into context, let's consider a real-world scenario: funnel analysis for an e-commerce website. Using a combination of ETL and ELT, developers can collect user behavior data using Rudderstack's SDKs, standardize the event schema, and load it into ClickHouse. From there, ClickHouse's powerful capabilities allow developers to perform transformations that enable insightful funnel analysis, tracking users' journeys from website visits to conversions.

If you're interested in learning how to efficiently combine RudderStack and ClickHouse to build effective data pipelines, we have an excellent hands-on demo for you straight from the experts. A webinar, previously hosted by Altinity and Rudderstack, delved into defining standard transformations, and demonstrated how to implement them in both RudderStack and ClickHouse. The recording of the webinar is available on-demand and you can access it here: Altinity and Rudderstack Joint Webinar Recording or watch more webinars like that on our YouTube channel!

Conclusion

The ETL vs ELT debate need not be a divisive one; rather, it's an opportunity to harness the strengths of both approaches. The combination of Rudderstack's streamlined data collection and integration with ClickHouse's versatile data warehousing capabilities empowers developers and data engineers to create efficient, real-time data pipelines that drive actionable insights. Whether you're dealing with structured, semi-structured, or unstructured data, the harmony of ETL and ELT can pave the way for powerful data analytics and informed decision-making.

How to Save Money and Deliver Real Time Analytics with ClickHouse

Altinity — Wed, 26 Jul 2023 21:13:03 +0000

Everyone knows ClickHouse is fast, but what about cheap? In this blog, we dive into some secrets that ClickHouse champion cheapskates use to reduce costs.

Before we dive in, it is helpful to understand the drivers of costs in an analytic database. The drivers can be grouped into five main categories:

Proprietary licenses - these license fees often require users to pay for each instance of the database, each user access it, or each feature that is enabled.
Storage - large data sets requires a lot of storage space, and also fast access and retrieval of data
Compute - processing and analyzing data demands a lot of computing power, especially for complex queries and operations
Networking - clusters that distribute data across different locations or data centers can incur significant networking costs, depending on the distance and bandwidth
Human labor - managing and maintaining analytic databases can require skilled and experienced personnel.

As human labor is a complex topic, we will focus on the other 4 categories.

Develop on a laptop with 100% open source

A fantastic way to reduce costs when you start with ClickHouse is to use your own laptop for development.

There are three open source dev patterns that work on a laptop

Install ClickHouse packages directly
Run ClickHouse using docker
Run complete ClickHouse app with docker compose

We will dig into the third pattern as it is the most relevant pattern that we see broadly across development. Best of all it works on practically any laptop, regardless of your operating system.

Get ready to go by installing docker followed by docker compose. The steps are easy and complete in a few minutes.

If you have an application, it’s easy to turn it into a Docker image. We are going to build a Docker image that consists of Ubuntu plus the curl program, a small but invaluable program to send HTTP requests. Curl is the “application.”

Let’s generate the image.

cat > Dockerfile << END
# Simple analytic client with curl installed.
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y curl
CMD ["sleep", "infinity"]
END

docker build -t myclient:latest - < Dockerfile

The next step is to create a Docker compose file for ClickHouse and the app:

cat > docker-compose.yml << END
version: '3'
services:
  clickhouse_server:
    image: altinity/clickhouse-server:22.8.15.25.altinitystable
      ports:
       - "8123:8123"
       - "9000:9000"
      volumes:
       - ./clickhouse_database:/var/lib/clickhouse
  ubuntu_client:
    image: myclient:latest
END

The final step to start your application, ensure it’s running, and then start using your application. ‘Docker exec’ allows us to connect to the application container and send commands.

$ docker compose up -d
...
$ docker ps
CONTAINER ID ... NAMES
dff28a725b38 altinity/clickh... cheapskate-clickhouse_server-1
23a641654ac2 myclient:latest... cheapskate-ubuntu_client-1

$ docker exec -it 23a6 bash
root@23a641654ac2:/# curl http://cheapskate-clickhouse_server-1:8123
Ok.

ClickHouse answered back. And that’s it! You can now use ClickHouse on your laptop and save 100% on license fees.

Tune apps to limit resource usage

Another wonderful way to develop cost efficient applications is to tune them to limit resource usage. The same factors that make your ClickHouse application fast also make it cheap, because it runs faster with fewer resources for a shorter time. Optimizing apps to use less resources can result in huge cost savings - sometimes up to 90%!

Start with schema design. ClickHouse does not automatically create an efficient schema for you. You have to make choices. For example, you want to optimize your data types to make them small - use codecs and an effective compression algorithm (like ZSTD). You can use alias columns to save space. You also want to pay attention to your ORDER BY because it affects compression and query speed. Here’s an example of a fully optimized table design.

Below is an illustration of the speed between different levels of schema optimization. We can see an 80% reduction in the data with optimized data types and ZSTD compression.

As you are tuning your application, we suggest using ClickHouse system tables that contain useful information about your compression ratio, query time, and resource consumption. Here’s a query on table system.tables that calculates the data shown above.

SELECT name, total_rows AS rows, 
  formatReadableSize(total_bytes) AS size,
  total_bytes / rows AS bytes_per_row
FROM system.tables 
WHERE database = 'default'
ORDER BY name

In addition, check your query performance. The following example selects from system.query_log to find the details of recent queries (such as how long they took, how much memory they used, how many rows and bytes they read). This will help you to analyze your queries and make them more efficient.

SELECT
     event_time,
     type,
     query_duration_ms / 1000 AS duration,
     read_rows,
     read_bytes,
     result_rows,
     formatReadableSize(memory_usage) AS memory,
     query
FROM system.query_log
WHERE (user = 'test') AND (type = 'QueryFinish')
ORDER BY event_time DESC
LIMIT 50

Using TTLs to limit growth

TTLs are another good cost-saving tip. TTLs (time to live) can time out rows, which reduces your cost because you’re not storing endless data as your time series data keep coming. TTLs can also move, aggregate, and recompress data.

Let’s look at an example. Below, we can see the TTL deletes rows at 12 months. We use ZSTD(1) to recompress rows at one month, and we also apply a higher level of ZSTD to recompress the data after six months.

CREATE TABLE default.web_events_with_ttl_2 (
           `time` DateTime,
           . . .
           `float_value` Float32
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(time)
ORDER BY (user_id, toStartOfDay(time), session_id, time)
TTL time + INTERVAL 1 MONTH RECOMPRESS CODEC (ZSTD(1)),
    time + INTERVAL 6 MONTH RECOMPRESS CODEC (ZSTD(10)),
    time + INTERVAL 12 MONTH DELETE

When we look at the impact of the TTL, in this particular dataset, we can see that the oldest part (2208_3_3_1) has a much smaller number at 264,000 bytes compared to the newest data which is over 600,000 bytes.

SELECT partition, name, rows,
     data_compressed_bytes AS compressed,
     data_uncompressed_bytes AS uncompressed
FROM system.parts
WHERE (table = 'web_events_with_ttl_2') AND active
ORDER BY name DESC

┌─partition─┬─name─────────┬──rows─┬─compressed─┬─uncompressed─┐
│ 202304    │ 202304_1_1_0 │ 50000 │     613930 │      1388890 │
│ 202302    │ 202302_2_2_1 │ 50000 │     327461 │      1388890 │
│ 202208    │ 202208_3_3_1 │ 50000 │     264054 │      1388890 │
└───────────┴──────────────┴───────┴────────────┴──────────────┘

While TTLs can help you reduce storage costs by limiting data growth and/or compressing the data, the amount of savings will vary. If you don’t want to lose any source data, you may want to use TTL to compress data instead of deleting it. Using tiered storage for older data can yield even higher cost savings. Check out our recorded webinar for a full explanation.

Scale compute capacity down when not needed

Re-scaling compute lowers cloud costs dramatically. You can resize your VMs and connect it to the same block storage. A simple way to do this is with the Altinity ClickHouse Kubernetes Operator.

Below is a picture that illustrates what happens inside a Kubernetes cluster that also includes the Altinity Operator.

For each server in a ClickHouse cluster, we will create a Pod, which is a running container. The Pod has one or more persistent volume claims (storage requests) that connect to data on actual AWS EBS volumes.

A simple setting in Kubernetes will make the pod reschedule on a different VM. The trick is to have a provisioner (like Karpenter) that watches Kubernetes and its pod changes. It will automatically make new VMs as needed to match the pod requirements. You don’t have to do anything. It’s all done for you.

How do we make that change? Here’s an example of defining a ClickHouse installation using the Altinity ClickHouse Operator. First, we use pod templates to specify replica properties. We set one node, one shard and one replica. We also have a pod template that specifies how our pods should be arranged.

apiVersion: "clickhouse.altinity.com/v1"
kind: "ClickHouseInstallation"metadata:
     name: "prod"
spec:
     configuration:
     clusters:
          - name: "ch"
          layout:
               shardsCount: 1
               replicasCount: 1
          templates:
               podTemplate: clickhouse-zone-2a

Next, we change the name of the instance type. In this example, we are telling the provisioner that we want to run in an m5xlarge VM in availability zone us-west-2.

 podTemplates:
  - name: clickhouse-zone-2a
    spec:
      containers:
      - name: clickhouse
        image: altinity/clickhouse-server:22.8.15.25.altinitystable
      nodeSelector:
        node.kubernetes.io/instance-type: m5.xlarge
    zone:
      key: topology.kubernetes.io/zone
      values:
      - us-west-2a

You can use kubectl to apply the above definition. The provisioner will make an m5.xlarge VM if it’s not available. Best of all, if you want a different VM you can just change the name in the definition and reapply using kubectl. Kubernetes will drop the old VM and put your pod on a new VM of the right size.

Resizing can be useful for different scenarios. For example, if you have a cyclical pattern in ClickHouse, you can reduce your capacity when the demand is less. You can also have different levels of compute, and use less compute for older data.

There are other tricks as well. You can even turn off unused pods by setting the number of replicas in the definition to 0. Kubernetes will deallocate the VM completely. In these cases, your storage will remain the same, but you save on compute. Cost savings vary but may reach 50% in the best cases.

Conclusion

In this article, we give 4 ways that any developer can reduce costs in ClickHouse. If you enjoyed this blog, make sure to check out our webinar Run ClickHouse like a Cheapskate - 6 Ways to Save Money While Delivering Real-Time Analytics for two other great cost-savings hacks in ClickHouse.

DEV Community: Altinity

Join OSA CON 2025: Two Days of Open‑Source Analytics and AI (Nov. 4–5)

When & Where

Who It’s For

What to Expect

Backed by the Community

Ready to Join?

Understanding Detached Parts in ClickHouse®

What Are Detached Parts?

Why Do I Get Detached Parts?

1. Failed Inserts/Merges/Mutations

2. Manual Detachment

3. Replication and Merges

Where Are Detached Parts Stored?

Prefixes for Internally Detached Parts

ignored_

covered_by_broken_

broken_ / broken-on-start_

unexpected_

cloned_

noquorum_

merge-not-byte-identical_ / mutate-not-byte-identical_

How to Manage Detached Parts in ClickHouse

1. Investigating Detached Parts

2. Reattaching Detached Parts

3. Deleting Detached Parts

4. Monitoring Detached Parts

Best Practices for Handling Detached Parts

Final Thoughts

Announcing Project Antalya – Infinitely Scalable ClickHouse® Query on 10x Cheaper Iceberg Storage

Show me the code already!

So what’s the problem?

Rising costs are crushing analytics ROI

Silos of proprietary batch and real-time data are disappearing

Reduce storage costs and share data. Better data lakes are a freebie

Project Antalya – Transforming ClickHouse for Cost-Efficient Data Lake Analytics

Real-time query on Iceberg tables with cheap swarm clusters

Extend large ClickHouse tables seamlessly to Iceberg storage

Some Frequently Asked Questions

Is Project Antalya open source?

Is Project Antalya a fork of ClickHouse?

Wait! What about Delta Lake Tables?

Where’s the performance test?

What is your roadmap?

How can I get started?

How can I contribute?

Does this work in Altinity.Cloud?

In conclusion…

ETL vs ELT Cage Fight: Combining Rudderstack & ClickHouse to Build Real-Time Data Pipelines

Understanding the ETL and ELT Paradigms

ETL: Extract, Transform, Load

ELT: Extract, Load, Transform

Introducing Rudderstack and ClickHouse

Rudderstack: A Customer Data Platform

ClickHouse: A Versatile Data Warehouse

Combining Forces: ETL and ELT for Real-Time Data Pipelines

1. Flexible Transformations

2. Standardized Event Collection

3. Efficient Data Storage and Analytics

4. Original Data Copy

5. Scenario-Based Optimization

Real-World Example: Funnel Analysis

Conclusion

How to Save Money and Deliver Real Time Analytics with ClickHouse

Develop on a laptop with 100% open source

Tune apps to limit resource usage

Using TTLs to limit growth

Scale compute capacity down when not needed

Conclusion

Join OSA CON 2025: Two Days of Open‑Source Analytics and AI (Nov. 4–5)