I was testing out CH-Ops - an admin GUI for self-hosted ClickHouse - on a simple setup: 1 shard, 2 replicas. Stumbled onto the replication queue almost by accident.
Here's what I did: I stopped one of the nodes (let's call it Node B), then inserted some data through the other one (Node A). Just wanted to see what would happen.
Then, while Node B was still down, I checked it in CH-Ops. It had stuff sitting in its replication queue.
My first assumption was: okay, this must be showing what's left to replicate across the cluster - the total pending replication work.
So I switched over and checked Node A, the one that was actually up and had just received the insert.
Its queue was empty.
That didn't match what I expected at all. If the queue was a cluster-wide "here's what still needs to replicate" view, Node A should've shown something too - it was the one that had the fresh data now waiting to reach Node B. Instead it was Node B, the down one, sitting there with pending tasks.
That mismatch is what sent me digging. Turns out the queue isn't cluster-wide at all - it's specific to each ClickHouse instance. Once I brought Node B back up, its queue drained in seconds and the data showed up.
That whole experiment is basically the entire post in miniature. Here's the mental model I ended up with.
A Queue Belongs to a Replica, Not to the Table
This is the first thing to get straight.
With a ReplicatedMergeTree table, you can have multiple replicas holding copies of the same data. It's tempting to think of replication as one shared pipe between them.
It isn't.
Each replica keeps its own local replication queue.
So if you see:
Replica 1 → queue_size = 0
Replica 2 → queue_size = 25
that doesn't mean 25 operations are waiting somewhere in the middle for both replicas to pick up.
It means Replica 2, specifically, has 25 tasks it hasn't finished yet.
Once that clicked for me, the rest of the system made a lot more sense.
So Where Do These Tasks Come From?
Replication in ClickHouse isn't a direct copy from one replica to another. There's a coordination layer in between.
ClickHouse Keeper holds the replication log - the record of operations that happened on the table. When something replicated happens (a part gets inserted, a merge runs, a mutation is applied), that operation gets logged in Keeper.
Each replica watches that log and figures out: what do I need to do to catch up to this?
That "what do I need to do" becomes a task, and the task lands in that replica's queue.
Depending on what happened, a task might be:
- fetch a part
- merge parts
- apply a mutation
- drop a range
- run another replicated operation
So the queue is really just a to-do list, generated from the shared log, but executed locally.
Where You Actually See This: system.replicas
CH-Ops was showing me a friendlier view of exactly this, but under the hood, this is the table doing the work. It's usually the first place people look, and for good reason - it gives you a quick health snapshot per replica.
SELECT
database,
table,
replica_name,
is_leader,
is_readonly,
queue_size,
inserts_in_queue,
merges_in_queue,
part_mutations_in_queue,
absolute_delay
FROM system.replicas;
Example output:
┌─replica_name─┬─queue_size─┬─inserts_in_queue─┬─merges_in_queue─┬─absolute_delay─┐
│ replica_1 │ 0 │ 0 │ 0 │ 0 │
│ replica_2 │ 12 │ 3 │ 9 │ 4 │
└──────────────┴────────────┴──────────────────┴─────────────────┴────────────────┘
queue_size gives you the total. inserts_in_queue, merges_in_queue, and part_mutations_in_queue break that total down by task type, which already tells you more than the single number does.
But it still doesn't tell you why those 12 tasks are sitting there. For that, you need the next table.
The Detail View: system.replication_queue
If system.replicas tells you how much work is waiting, system.replication_queue tells you what that work actually is.
SELECT
database,
table,
replica_name,
type,
create_time,
num_tries,
last_exception
FROM system.replication_queue
ORDER BY create_time;
Instead of a single number, you now get the actual list:
GET_PART
MERGE_PARTS
GET_PART
MUTATE_PART
GET_PART
...
You can see the task type, how long it's been sitting there (create_time), how many times it's been retried (num_tries), and - this is the important one - last_exception.
That single column is often the difference between "the queue is 12" and "I know exactly why the queue is 12."
Two Tables, Two Different Questions
Once I had both queries in my toolkit, it became clear they answer different questions:
| Table | Question it answers |
|---|---|
system.replicas |
How much work is a replica behind on, right now? |
system.replication_queue |
What is that work, specifically, and is it stuck? |
Neither one alone gives you the full picture. system.replicas is a summary. system.replication_queue is the detail behind that summary.
Queue Size Is Not the Same as Replication Lag
This one tripped me up early on, so it's worth calling out explicitly.
A replication queue measures pending work. Replication lag (absolute_delay) measures how far behind a replica is.
They're related, but they're not interchangeable.
A replica can have:
queue_size = 50
and be processing those 50 tasks fast enough that lag barely moves. A few minutes later, the queue is empty and nothing was ever "behind" in any meaningful sense.
A replica sitting at:
queue_size = 5
for hours, on the other hand, is a very different story - even though 5 sounds small next to 50.
The number by itself doesn't tell you which situation you're in. That's the whole point of this post: the queue is a data source, not a verdict.
The Mental Model, Put Simply
Here's the picture:
The replica is the participant - it holds data and does work.
The replication queue is that participant's pending work, generated from what Keeper says has happened on the table.
The queue exists because the replica has catching up to do. That's it. It's not a fault indicator on its own - it's a reflection of activity and progress.
What This Sets You Up For
Understanding the queue as "a to-do list, not a fault code" is step one.
Step two - the part where you actually decide when to worry - comes down to trend, task type, and last_exception, and that's a big enough topic to deserve its own post.
For now, the takeaway is simpler:
A non-zero replication queue isn't a problem statement. It's a starting point for one.
Know what the queue is before you try to diagnose what it means.
Top comments (0)