When a Cassandra node OOMs, or GC pauses start stretching into seconds, the reflex is to blame Cassandra, or its favourite scapegoat, the JVM.
I have read a lot of Cassandra heap dumps, and here is the uncomfortable pattern. The heap usually contains exactly what the data model asked for. The dump does not convict the database. It convicts a partition.
This post is the workflow I actually use, in order, because the order saves hours.
Step 0: GC logs before heap dumps
A heap dump is a biopsy. GC logs are the patient history, and they are already on disk. Two things I look for first:
# gc.log (G1)
Pause Young (Normal) ... 180ms
Pause Young (Normal) ... 210ms
Pause Full (Allocation Failure) ... 8.4s <- the incident
...
Humongous Allocation ... 18MB
Humongous Allocation ... 22MB
Full GC pauses. G1 doing an emergency stop the world collection because normal cycles could not keep up. On a Cassandra node, an 8 second pause does not just slow queries. The node misses gossip, gets marked down by peers, and hints start piling up elsewhere. One node's GC problem becomes the cluster's problem.
Humongous allocations. G1 divides the heap into regions, typically 4 to 32MB. Any single allocation larger than half a region is humongous, allocated awkwardly across contiguous regions and collected inefficiently.
And what does Cassandra allocate in one giant contiguous chunk? A large partition being materialized for a read. Humongous allocation warnings in a Cassandra gc.log are large partitions announcing themselves. You can often skip the heap dump entirely at this point and go straight to nodetool tablehistograms to find the table with a monster p99 partition size.
Step 1: take the dump without causing an incident
jmap on a live heap stops the world for the duration of the dump. On a 20GB heap, that is long enough for the cluster to declare the node dead. Never dump a node that is still serving.
# isolate first, node stays alive but stops serving
nodetool disablebinary && nodetool disablegossip
jmap -dump:live,format=b,file=/data/dumps/cass_$(hostname)_$(date +%s).hprof <pid>
# afterwards: rejoin or just restart the node
Better, have the JVM do it for you at the moment of truth, before any human is awake:
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/data/dumps/
Those two flags belong in every production Cassandra JVM config. An OOM without a dump is an incident you get to have twice.
Related discipline, keep the heap at or under 8GB and let G1 target around 200ms pauses. Bigger heaps mostly buy you longer pauses and bigger dumps, and Cassandra's real caching happens off heap and in the page cache anyway.
Step 2: Eclipse MAT, dominator tree, ten minutes
Open the .hprof in Eclipse MAT and go straight to the dominator tree, objects ranked by retained memory, which answers what is actually holding the heap hostage. Ignore the histogram of a billion byte[]. The dominator tree tells you whose bytes.
What the tree shows, mapped to what it means, every one of these from real incidents:
| Dominating the heap | Actual problem |
|---|---|
A few enormous byte[] or cell containers under a read path |
Large partition being materialized. Get the key from the referencing objects, confirm with nodetool tablehistograms or sstablepartitions. Fix is to bucket the partition, which is a later post in this series. |
| Memtable objects, many tables | Over wide schema, or memtable flush thresholds set too generous for the heap |
| Tombstone or range tombstone structures under a query | A tombstone farm read that survived just long enough to dump |
| Netty buffers or inflight requests | Clients hammering with no paging. Check fetch size and unpaged full partition reads |
The punchline repeats across years and companies. The heap contains the workload. I have almost never opened a dump and found a Cassandra bug. I have regularly opened one and found a 4GB partition someone swore was just a busy customer.
Step 3: close the loop in the model
The dump names the object. The fix lives in the schema.
Large partition, time bucket the partition key. Unpaged reads, driver fetch size. Tombstone reads, the modeling fixes from two posts ago. The JVM flags and MAT are diagnosis. Cassandra data modeling is treatment.
Blaming the JVM is comfortable because nobody owns the JVM. The heap dump takes that comfort away. It shows you, in retained bytes, precisely which design decision you are looking at.
Top comments (0)