Large partitions don't fail loudly. They fail everywhere at once.

Most Cassandra problems announce themselves. A timeout, an exception, a graph going vertical.

Large partitions do not. They degrade reads, compaction, repair and GC simultaneously and gradually, which is why the incident channel fills with four unrelated looking symptoms and no suspect.

If you have been following this series, you have already met large partitions twice without the name. They are the humongous allocations in the heap dump post, and they are the overstreaming amplifier in the repair post. This post is about the disease itself.

What large means and why it hurts

A partition is the unit Cassandra reads, compacts, streams and materializes. The rule of thumb that has served me well is to aim under 100MB per partition and treat anything past that as debt. Old timers remember the hard 2GB era limits. Modern Cassandra handles big partitions better, which mostly means they hurt you more gradually.

Where the pain lands:

• Reads. A partition read materializes structures proportional to what is scanned. In G1 terms, a multi hundred MB partition read is a parade of humongous allocations. GC pause spikes on whichever node holds it, the cluster marks it slow, and speculative retries fan the load elsewhere.
• Compaction. Partitions are compacted as units. Giant partitions make giant, slow compaction tasks that starve neighbours, and pending tasks climb.
• Repair. One mismatched cell in a giant partition can stream the whole thing. The overstreaming problem, concentrated.
• Hotspotting. Big partitions are usually hot partitions, the same key taking disproportionate traffic. Size and heat compound.

How partitions get big, always the same story

The partition key models an unbounded thing.

-- looks innocent in the design review
CREATE TABLE events_by_device (
  device_id uuid,
  ts timestamp,
  payload blob,
  PRIMARY KEY (device_id, ts)
);

A partition key of device_id means every event that device ever produces, forever, in one partition. A chatty device, a popular customer, a busy trading day. Growth has no ceiling because the model gave it none.

I have seen the one big customer partition at multiple companies. It is practically a genre.

Finding them

nodetool tablehistograms ks events_by_device
# Percentile      Partition Size
# 50%             61,214 bytes
# 99%             386,857,368 bytes     <- there it is
# Max             2,395,318,855 bytes

The p50 to p99 gap is the signature. A healthy median hiding a monster tail.

To name the offenders on disk:

# per SSTable, list partitions over a threshold
sstablepartitions -t 100 /var/lib/cassandra/data/ks/events_by_device-*/

And Cassandra logs them at write time. Grep system.log for Writing large partition. Those log lines are the cheapest early warning system you will ever ignore.

The fix: give the partition a ceiling

You cannot cap a device's lifetime events. You can cap a partition, by putting time into the key.

CREATE TABLE events_by_device_v2 (
  device_id uuid,
  day date,                       -- the bucket
  ts timestamp,
  payload blob,
  PRIMARY KEY ((device_id, day), ts)
);

Now a partition is one device, one day. Bounded by physics instead of hope. Two design decisions follow.

Bucket size is a calculation, not a vibe. Estimate rows per day times row size, then pick the bucket, hour, day or week, that lands typical partitions in single digit MB and your worst realistic case under about 100MB. Do the arithmetic for your loudest tenant, not your average one. Averages are how the enormous customer surprises you.

Multi bucket reads are now yours to orchestrate. Last 6 hours spanning midnight means two partitions. Drivers make querying a known list of buckets cheap. Just design the access pattern alongside the schema, not after it. This pairs well with TWCS from the compaction post, because bucketed, TTL'd time series is the workload TWCS was born for.

Migration reality. Existing giants must be rewritten into v2. Dual write new data, backfill old in throttled batches, the same discipline as the 2TB MongoDB delete earlier in this series, chunk, sleep, watch the cluster. Then cut reads over and drop v1. Weeks, calmly, not a weekend, heroically.

The review question that prevents all of it

Every Cassandra schema review I run ends with one question. What bounds this partition?

If the answer is a business quantity, customers will not have many orders, then it is unbounded, because business quantities grow. That is their job. The only acceptable answers have units of time, or a hard modulo in the key.

Large partitions are never an ops problem. They are a design review that ended one question too early.