Repair at 1,000 nodes: why we stopped running nodetool repair by hand

At small scale, repair is a cron job. At 1,000 nodes it is a scheduling problem, and if you treat a scheduling problem like a cron job, the cluster will teach you the difference during business hours.

I spent years operating Cassandra estates past the thousand node mark. Here is what repair actually does, why it is not optional, and how the operational model has to change with scale.

Why repair is a deadline, not a chore

Cassandra replicas drift. Hinted handoff covers short outages, but hints have a window, three hours by default. A replica down longer than that has permanently missed writes, and only anti entropy repair reconciles it.

The part that turns repair from hygiene into a deadline is tombstones. Deleted data is protected from resurrection by grave markers that compaction may purge after gc_grace_seconds, ten days by default. The safety of that purge rests on one assumption. Every replica heard about the delete before the marker vanished. Repair is how they hear.

So the rule is absolute. Every node completes repair at least once per gc_grace_seconds. Miss it, and a replica that slept through a delete will happily hand the missing row back to the cluster. Zombie data, silent, and by the time anyone notices, days of writes have been made against resurrected garbage.

Repair is not cleanup. It is the second half of every delete you have ever issued.

What a repair costs

Mechanically, nodes build Merkle trees over their data ranges, exchange them, and stream any ranges whose hashes disagree. Three costs follow.

1. Building trees reads data. Sequential I/O competing with your workload.
2. Streaming mismatches saturates network and creates new SSTables, which means the third cost.
3. Compaction debt. A big repair is followed by a compaction hangover, and nodetool compactionstats pending tasks tell the story.

Merkle tree resolution is finite, so one tree over a huge range means one tiny mismatch streams a disproportionately large chunk. This is overstreaming, and it is why repairing a giant range in one shot is both slow and wasteful.

Full, incremental, subrange, and what breaks at scale

Full repair, whole range, everything rehashed every time. Correct, brutal, does not scale. At 1,000 nodes, naive nodetool repair across the fleet means the cluster is effectively always repairing, and repair sessions colliding on shared ranges fail in tedious ways.

Incremental repair marks SSTables repaired so they are skipped next time. On paper, the fix. In practice it drags anticompaction behind it, rewriting SSTables to segregate repaired from unrepaired data, and its operational history, particularly before Cassandra 4.0's fixes, was rocky enough that many large operators simply did not trust it. We did not.

Subrange repair breaks the token range into small segments repaired one at a time. Small Merkle trees, precise comparisons, minimal overstreaming, and each segment is a small retryable unit of work. This is the primitive that actually scales.

But subrange repair across 1,000 nodes is thousands upon thousands of segments needing ordering, throttling, retries and collision avoidance. Congratulations, it is a scheduling problem.

Reaper: repair as a system, not a command

We ran Cassandra Reaper, open source, originally Spotify's and now community maintained, as the scheduler.

• Splits every table's range into segments and runs them steadily with concurrency caps
• An intensity knob throttles repair pressure so p99s do not feel it. We tuned it to stay invisible.
• Failed segments retry individually, so a node blip costs one segment, not the whole run
• A web UI that answers the only question leadership asks, which is whether we are inside the gc_grace window, yes or no

The operating policy that came out of years of this:

- every table repaired well inside gc_grace, target a complete cycle in ~7 days against a 10 day grace
- repair pressure tuned to be invisible in p99 read latency
- alert not on "repair failed" but on "time since last completed repair per table"
- pause repairs during topology changes, resume, never skip

That last alert framing matters. Failures are noise. Staleness is the actual risk. Measure the deadline, not the activity.

The small scale corollary

Under about 50 nodes you do not need Reaper's ceremony, but you do need the same two invariants. Every node repaired inside gc_grace_seconds, and repairs that do not collide. A boring, monitored schedule beats heroics.

Repair is the tax on eventual consistency. You can pay it on a schedule you chose, or all at once with interest, with zombie data as the collection notice.