No leader, many replicas, one question: is replication healthy? Read-only tables, stuck queues and replica delay in one screen
Most databases have a leader. You find it, you check the replicas behind it, done.
ClickHouse doesn't work like that. Every node accepts writes, and replicated tables converge through ZooKeeper or ClickHouse Keeper. So "who's the leader?" is the wrong question. The right one is: is replication healthy on every node?
Answering it usually means running a handful of queries against system.replicas and system.replication_queue on every node and comparing the results by eye. That's what I wanted Ivory to do for me.
The problems
- Read-only tables. When a node loses its session with ZooKeeper, its replicated tables go read-only. The node is up, it answers queries, and inserts fail.
-
Stuck replication queues. A fetch or merge task fails, retries forever, and the queue grows. The actual error is buried in
system.replication_queue. -
Replica delay. A node falls behind, and once it passes
max_replica_delay_for_distributed_queries, distributed queries stop using it. -
Cluster config drift. One node's
<remote_servers>names a different set of replicas than the others, and nothing complains. - Every check is one query per node. Three nodes, five queries: fifteen copy-pastes.
How Ivory solves them
A cluster that matches your config
Ivory reads system.clusters for the cluster named exactly like your cluster in Ivory. So the one rule to remember: name your Ivory cluster after its <remote_servers> entry.
Why by name? Because a ClickHouse node can belong to several clusters at once — including the test_shard_localhost ones a stock build ships with. Asking "who is in this cluster?" has one answer; asking "which cluster are you in?" doesn't. If a node declares no cluster with that name, Ivory warns and lists the clusters it does declare, so you see the mistake immediately.
It works in both directions, too: a replica ClickHouse lists that you haven't added to Ivory gets a warning, and so does a node you added that ClickHouse doesn't list. Config drift stops being silent.
Every node is a replica — and that's correct
ClickHouse has no leader, so Ivory shows every member as a replica. That's not a fault, it's how ClickHouse works.
-
Lag — the largest
absolute_delayacross the node's replicated tables. Ivory warns past 30 seconds. - State — whether the node is actually running. A node that serves reads stays running even if its replication is in trouble. That trouble is shown as a warning instead.
Warnings for what actually breaks
Ivory warns about:
- replicated tables gone read-only — usually a lost session with the coordination store,
- replicas without an active coordination session,
- stuck tasks in the replication queue, with the error that blocks them,
- delay past ClickHouse's own
max_replica_delay_for_distributed_queries.
That stuck-queue warning is my favourite. Instead of digging through system.replication_queue on every node, you see the failing task and its error right on the overview.
Operations
With no leader, there's nothing to switch over or fail over. What you can do:
-
Reload the configuration (
SYSTEM RELOAD CONFIG), -
View the settings (
system.settings).
SQL console and dashboard
Each node has a SQL console, browsing of databases and tables, and a dashboard with databases, connections, sizes and uptime.
Saved queries cover the things you usually check by hand: active queries, the replication queue, the coordination connection, detached parts, disks and more.
One limitation, to be honest: running queries can't be cancelled from Ivory. ClickHouse identifies a running query by a string query_id, not a process id, and Ivory's cancel works with process ids. Rather than pretend, Ivory doesn't offer the button.
The machine behind each node
ClickHouse loves disks and memory. Give Ivory an SSH key, and each node shows its host's CPU, memory, network, processes and any log file — including ClickHouse's own server log. If it runs in Docker, the Container tab has its logs, inspect, start, stop and restart.
Need a new ClickHouse cluster?
Ivory deploys one over plain SSH from a template. The templates run clickhouse/clickhouse-server:24, write the cluster into a config.d file before starting, set the {cluster} and {replica} macros to the Ivory cluster and node names, and create the database user named on the deploy screen.
ClickHouse needs a ZooKeeper (or ClickHouse Keeper). The deploy screen asks for its address once, for the whole cluster.
-
ClickHouse (Multi Host) — three nodes
clickhouse1–clickhouse3, one per machine, on port9000. Replace the example addresses in the replica list with your hosts. -
ClickHouse (Single Host) — all three on one machine on ports
9000–9002. Its coordination store defaults to Ivory's ZooKeeper single-host template, so deploy that first on the same machine.
Two things good to know:
- The Ivory cluster name is written into ClickHouse's XML config, so it must be a valid XML element name.
- Under Docker's bridge networking,
CREATE … ON CLUSTERcan hang. Create replicated tables on each node instead.
Ask your AI agent
Connect Claude, Cursor or Codex to Ivory's built-in MCP server and ask:
"Is replication healthy on my ClickHouse cluster?"
"Why are tables read-only on clickhouse2?"
"Show me stuck tasks in the replication queue and explain the errors."
"Which tables use the most disk across the cluster?"
It works through Ivory, with your permissions and nothing more. See the MCP docs.
Try it
docker run -p 80:80 --restart always veegres/ivory
Open http://localhost, finish the setup wizard and add your ClickHouse cluster — remember to name it after <remote_servers>. ClickHouse support is in beta, so feedback and issues are very welcome.
No leader, no problem — as long as every replica keeps up. 😉
Useful links:
Top comments (0)