Day 66/100 - ClickHouse Keeper vs ZooKeeper: Migration and Benchmarks

Introduction

Distributed ClickHouse® clusters rely on a coordination service to manage replication, distributed DDL execution, leader election, and metadata consistency across nodes. Without a reliable coordination layer, replicas can become inconsistent, distributed queries may fail, and cluster management becomes significantly more complex.

For many years, Apache ZooKeeper has served as the default coordination service for ClickHouse®. While ZooKeeper is proven and widely adopted across the big data ecosystem, it introduces additional infrastructure, operational complexity, and resource consumption because it must be deployed and maintained as a separate cluster.

To simplify distributed deployments, ClickHouse® introduced ClickHouse Keeper—a native coordination service built specifically for ClickHouse workloads. It implements the ZooKeeper protocol, allowing it to serve as a drop-in replacement while offering lower operational overhead and improved performance for ClickHouse clusters.

In this article, we'll compare ZooKeeper and ClickHouse Keeper, understand their architectural differences, walk through a migration process, and examine why Keeper has become the recommended choice for modern ClickHouse deployments.

---

What Is Apache ZooKeeper?

Apache ZooKeeper is an open-source distributed coordination service used by many large-scale distributed systems, including Kafka, Hadoop, HBase, and ClickHouse®.

Within ClickHouse®, ZooKeeper is responsible for coordinating operations across cluster nodes.

Its primary responsibilities include:

• Tracking replicated table metadata
• Coordinating replication between replicas
• Executing distributed DDL queries
• Maintaining cluster topology information
• Supporting leader election
• Synchronizing metadata changes across nodes

ZooKeeper provides reliable coordination, but it was designed as a general-purpose coordination system rather than one optimized specifically for ClickHouse.

---

Limitations of ZooKeeper

Although ZooKeeper has been the standard coordination service for years, it comes with several operational challenges.

Common limitations include:

• Requires a dedicated cluster of at least three nodes for fault tolerance
• Introduces additional infrastructure to manage
• Consumes extra CPU and memory resources
• Requires separate monitoring and maintenance
• Adds operational complexity during upgrades
• Can become a bottleneck in heavily replicated ClickHouse clusters
• Requires administrators to manage another distributed system alongside ClickHouse®

As clusters grow larger, these operational costs become increasingly significant.

---

What Is ClickHouse Keeper?

ClickHouse Keeper is ClickHouse®'s native coordination service.

Rather than relying on an external ZooKeeper installation, Keeper is integrated directly into ClickHouse and is designed specifically for ClickHouse replication workloads.

Most importantly, Keeper implements the ZooKeeper protocol, making it compatible with existing ClickHouse replication mechanisms without requiring application changes.

Key characteristics include:

• Native integration with ClickHouse®
• ZooKeeper protocol compatibility
• Uses the Raft consensus algorithm
• Can run embedded inside ClickHouse servers
• Can also run as a standalone service
• Lower latency for ClickHouse coordination workloads
• Reduced resource consumption
• Simplified deployment and operations

For most new ClickHouse deployments, Keeper is now the recommended coordination service.

---

ZooKeeper vs ClickHouse Keeper

Feature	ZooKeeper	ClickHouse Keeper
Built into ClickHouse	❌	✅
Consensus Algorithm	ZAB	Raft
ZooKeeper Protocol Compatibility	Native	Compatible
Separate Installation Required	✅	❌
Operational Complexity	Higher	Lower
Resource Usage	Higher	Lower
Performance	Good	Optimized for ClickHouse
Recommended for New Deployments	❌	✅

---

Why Migrate to ClickHouse Keeper?

Migrating to Keeper provides several operational and performance benefits.

These include:

• Simplified architecture
• Lower infrastructure costs
• Fewer servers to maintain
• Reduced operational overhead
• Faster startup and recovery
• Lower CPU and memory usage
• Native ClickHouse integration
• Easier configuration management

Organizations deploying new ClickHouse clusters typically benefit from using Keeper unless they already depend heavily on an existing ZooKeeper ecosystem.

---

Typical Cluster Architecture

ZooKeeper-Based Deployment

ClickHouse Node 1 ──┐
ClickHouse Node 2 ──┤──── ZooKeeper Cluster
ClickHouse Node 3 ──┘

                    ├── ZooKeeper Node 1
                    ├── ZooKeeper Node 2
                    └── ZooKeeper Node 3

A ZooKeeper deployment requires an entirely separate coordination cluster that must be monitored, upgraded, and maintained independently.

---

ClickHouse Keeper Deployment

ClickHouse Node 1 (+ Keeper) ──┐
ClickHouse Node 2 (+ Keeper) ──┤── Keeper Raft Cluster
ClickHouse Node 3 (+ Keeper) ──┘

No external ZooKeeper installation is required.

The coordination service becomes part of the ClickHouse infrastructure itself.

---

Deploying ClickHouse Keeper

Keeper supports two deployment models.

---

Mode 1: Embedded Deployment

In embedded mode, Keeper runs inside the ClickHouse server process.

Example configuration:

<keeper_server>
    <tcp_port>9181</tcp_port>
    <server_id>1</server_id>

    <log_storage_path>
        /var/lib/clickhouse/coordination/log
    </log_storage_path>

    <snapshot_storage_path>
        /var/lib/clickhouse/coordination/snapshots
    </snapshot_storage_path>

    <coordination_settings>
        <operation_timeout_ms>10000</operation_timeout_ms>
        <session_timeout_ms>30000</session_timeout_ms>
        <raft_logs_level>warning</raft_logs_level>
    </coordination_settings>

    <raft_configuration>
        <server>
            <id>1</id>
            <hostname>clickhouse-node-1</hostname>
            <port>9234</port>
        </server>

        <server>
            <id>2</id>
            <hostname>clickhouse-node-2</hostname>
            <port>9234</port>
        </server>

        <server>
            <id>3</id>
            <hostname>clickhouse-node-3</hostname>
            <port>9234</port>
        </server>
    </raft_configuration>

</keeper_server>

This deployment is ideal for most production clusters because it eliminates the need for additional coordination servers.

---

Mode 2: Standalone Deployment

Keeper can also run as an independent service.

clickhouse-keeper --config /etc/clickhouse-keeper/config.xml

Standalone mode is often preferred in larger enterprise deployments where coordination services are managed independently from database nodes.

---

Migrating from ZooKeeper to ClickHouse Keeper

Migration can usually be completed with minimal disruption by following a structured process.

---

Step 1: Verify Replication Health

Before making any changes, ensure all replicas are synchronized.

SELECT
    database,
    table,
    is_readonly,
    queue_size
FROM system.replicas;

A queue_size of 0 indicates that replication is fully caught up.

---

Step 2: Configure Keeper

Add the Keeper configuration to config.xml on every ClickHouse node.

Each node must have:

• A unique server_id
• Correct Raft configuration
• Shared cluster membership information

---

Step 3: Update Cluster Configuration

Replace ZooKeeper endpoints with Keeper endpoints.

Old configuration:

<zookeeper>
    <node>
        <host>zk1</host>
        <port>2181</port>
    </node>
</zookeeper>

New configuration:

<zookeeper>
    <node>
        <host>keeper1</host>
        <port>9181</port>
    </node>
</zookeeper>

Because Keeper implements the ZooKeeper protocol, most replication settings remain unchanged.

---

Step 4: Restart ClickHouse

Restart each ClickHouse server individually.

sudo systemctl restart clickhouse-server

Restarting one node at a time minimizes disruption and keeps the cluster available throughout the migration.

---

Step 5: Validate the Migration

Verify that replication remains healthy.

SELECT
    database,
    table,
    queue_size,
    is_session_expired
FROM system.replicas;

Healthy output should show:

• queue_size = 0
• is_session_expired = 0

This confirms successful migration.

---

Performance Comparison

The following table summarizes typical operational characteristics.

Metric	ZooKeeper	ClickHouse Keeper
Startup Time	Higher	Lower
Memory Usage	Higher	Lower
CPU Usage	Higher	Lower
Coordination Latency	Good	Lower
Deployment Complexity	High	Low
Maintenance Effort	High	Low
ClickHouse Integration	External	Native

In many production environments, Keeper also demonstrates:

• Faster write coordination
• Lower memory consumption
• More consistent latency under heavy workloads
• Faster replica synchronization
• Faster recovery after failures

Actual performance varies depending on workload, hardware, cluster size, and configuration.

---

Common Migration Challenges

Administrators may encounter several issues during migration.

Common examples include:

• Duplicate or incorrect server_id values
• Missing Raft configuration
• Firewall rules blocking ports 9181 or 9234
• Unsynchronized replicas before migration
• Version incompatibilities
• Incorrect hostname resolution

Always verify replication health before redirecting production workloads to Keeper.

---

Best Practices

To ensure a smooth migration:

• Schedule maintenance windows whenever possible.
• Back up ZooKeeper metadata.
• Verify replica synchronization before migration.
• Restart one node at a time.
• Monitor system.replicas continuously.
• Validate distributed DDL execution.
• Ensure compatible ClickHouse versions across the cluster.
• Test failover after migration completes.

---

Choosing Between ZooKeeper and Keeper

Choose ZooKeeper if:

• You already maintain a mature ZooKeeper infrastructure.
• Multiple applications depend on ZooKeeper.
• Operational processes are already built around ZooKeeper clusters.

Choose ClickHouse Keeper if:

• You're deploying a new ClickHouse cluster.
• You want a simpler architecture.
• You want fewer infrastructure components.
• Your coordination workload is primarily ClickHouse replication.
• You want lower operational overhead.

For most modern ClickHouse deployments, Keeper is the preferred option.

---

Conclusion

ClickHouse Keeper simplifies distributed ClickHouse® deployments by replacing the need for a dedicated ZooKeeper cluster with a lightweight, native coordination service.

By implementing the ZooKeeper protocol while optimizing specifically for ClickHouse workloads, Keeper reduces infrastructure complexity, lowers operational costs, improves resource efficiency, and simplifies cluster administration.

Migrating from ZooKeeper is generally straightforward because existing replication configurations require minimal changes. By validating replication health, carefully updating cluster configurations, and following a phased migration process, organizations can transition safely with minimal downtime.

As ClickHouse continues to evolve, Keeper has become the recommended coordination service for most new deployments, offering a simpler, more efficient, and highly reliable foundation for managing distributed ClickHouse clusters.

---

Key Takeaways

• ClickHouse® clusters require a coordination service for replication and distributed operations.
• Apache ZooKeeper has traditionally provided this functionality but requires separate infrastructure.
• ClickHouse Keeper is a native coordination service optimized specifically for ClickHouse.
• Keeper implements the ZooKeeper protocol, making migration straightforward.
• Embedded deployment simplifies infrastructure by eliminating external coordination servers.
• Migration typically involves verifying replication, updating configuration, restarting nodes, and validating cluster health.
• Keeper generally offers lower latency, lower resource usage, and reduced operational complexity.
• For most new ClickHouse® deployments, ClickHouse Keeper is the recommended coordination service.