DEV Community: Saqib Ameen Subhan

Large partitions don't fail loudly. They fail everywhere at once.

Saqib Ameen Subhan — Fri, 04 Sep 2026 17:29:00 +0000

Most Cassandra problems announce themselves. A timeout, an exception, a graph going vertical.

Large partitions do not. They degrade reads, compaction, repair and GC simultaneously and gradually, which is why the incident channel fills with four unrelated looking symptoms and no suspect.

If you have been following this series, you have already met large partitions twice without the name. They are the humongous allocations in the heap dump post, and they are the overstreaming amplifier in the repair post. This post is about the disease itself.

What large means and why it hurts

A partition is the unit Cassandra reads, compacts, streams and materializes. The rule of thumb that has served me well is to aim under 100MB per partition and treat anything past that as debt. Old timers remember the hard 2GB era limits. Modern Cassandra handles big partitions better, which mostly means they hurt you more gradually.

Where the pain lands:

Reads. A partition read materializes structures proportional to what is scanned. In G1 terms, a multi hundred MB partition read is a parade of humongous allocations. GC pause spikes on whichever node holds it, the cluster marks it slow, and speculative retries fan the load elsewhere.
Compaction. Partitions are compacted as units. Giant partitions make giant, slow compaction tasks that starve neighbours, and pending tasks climb.
Repair. One mismatched cell in a giant partition can stream the whole thing. The overstreaming problem, concentrated.
Hotspotting. Big partitions are usually hot partitions, the same key taking disproportionate traffic. Size and heat compound.

How partitions get big, always the same story

The partition key models an unbounded thing.

-- looks innocent in the design review
CREATE TABLE events_by_device (
  device_id uuid,
  ts timestamp,
  payload blob,
  PRIMARY KEY (device_id, ts)
);

A partition key of device_id means every event that device ever produces, forever, in one partition. A chatty device, a popular customer, a busy trading day. Growth has no ceiling because the model gave it none.

I have seen the one big customer partition at multiple companies. It is practically a genre.

Finding them

nodetool tablehistograms ks events_by_device
# Percentile      Partition Size
# 50%             61,214 bytes
# 99%             386,857,368 bytes     <- there it is
# Max             2,395,318,855 bytes

The p50 to p99 gap is the signature. A healthy median hiding a monster tail.

To name the offenders on disk:

# per SSTable, list partitions over a threshold
sstablepartitions -t 100 /var/lib/cassandra/data/ks/events_by_device-*/

And Cassandra logs them at write time. Grep system.log for Writing large partition. Those log lines are the cheapest early warning system you will ever ignore.

The fix: give the partition a ceiling

You cannot cap a device's lifetime events. You can cap a partition, by putting time into the key.

CREATE TABLE events_by_device_v2 (
  device_id uuid,
  day date,                       -- the bucket
  ts timestamp,
  payload blob,
  PRIMARY KEY ((device_id, day), ts)
);

Now a partition is one device, one day. Bounded by physics instead of hope. Two design decisions follow.

Bucket size is a calculation, not a vibe. Estimate rows per day times row size, then pick the bucket, hour, day or week, that lands typical partitions in single digit MB and your worst realistic case under about 100MB. Do the arithmetic for your loudest tenant, not your average one. Averages are how the enormous customer surprises you.

Multi bucket reads are now yours to orchestrate. Last 6 hours spanning midnight means two partitions. Drivers make querying a known list of buckets cheap. Just design the access pattern alongside the schema, not after it. This pairs well with TWCS from the compaction post, because bucketed, TTL'd time series is the workload TWCS was born for.

Migration reality. Existing giants must be rewritten into v2. Dual write new data, backfill old in throttled batches, the same discipline as the 2TB MongoDB delete earlier in this series, chunk, sleep, watch the cluster. Then cut reads over and drop v1. Weeks, calmly, not a weekend, heroically.

The review question that prevents all of it

Every Cassandra schema review I run ends with one question. What bounds this partition?

If the answer is a business quantity, customers will not have many orders, then it is unbounded, because business quantities grow. That is their job. The only acceptable answers have units of time, or a hard modulo in the key.

Large partitions are never an ops problem. They are a design review that ended one question too early.

Repair at 1,000 nodes: why we stopped running nodetool repair by hand

Saqib Ameen Subhan — Fri, 04 Sep 2026 17:28:30 +0000

At small scale, repair is a cron job. At 1,000 nodes it is a scheduling problem, and if you treat a scheduling problem like a cron job, the cluster will teach you the difference during business hours.

I spent years operating Cassandra estates past the thousand node mark. Here is what repair actually does, why it is not optional, and how the operational model has to change with scale.

Why repair is a deadline, not a chore

Cassandra replicas drift. Hinted handoff covers short outages, but hints have a window, three hours by default. A replica down longer than that has permanently missed writes, and only anti entropy repair reconciles it.

The part that turns repair from hygiene into a deadline is tombstones. Deleted data is protected from resurrection by grave markers that compaction may purge after gc_grace_seconds, ten days by default. The safety of that purge rests on one assumption. Every replica heard about the delete before the marker vanished. Repair is how they hear.

So the rule is absolute. Every node completes repair at least once per gc_grace_seconds. Miss it, and a replica that slept through a delete will happily hand the missing row back to the cluster. Zombie data, silent, and by the time anyone notices, days of writes have been made against resurrected garbage.

Repair is not cleanup. It is the second half of every delete you have ever issued.

What a repair costs

Mechanically, nodes build Merkle trees over their data ranges, exchange them, and stream any ranges whose hashes disagree. Three costs follow.

Building trees reads data. Sequential I/O competing with your workload.
Streaming mismatches saturates network and creates new SSTables, which means the third cost.
Compaction debt. A big repair is followed by a compaction hangover, and nodetool compactionstats pending tasks tell the story.

Merkle tree resolution is finite, so one tree over a huge range means one tiny mismatch streams a disproportionately large chunk. This is overstreaming, and it is why repairing a giant range in one shot is both slow and wasteful.

Full, incremental, subrange, and what breaks at scale

Full repair, whole range, everything rehashed every time. Correct, brutal, does not scale. At 1,000 nodes, naive nodetool repair across the fleet means the cluster is effectively always repairing, and repair sessions colliding on shared ranges fail in tedious ways.

Incremental repair marks SSTables repaired so they are skipped next time. On paper, the fix. In practice it drags anticompaction behind it, rewriting SSTables to segregate repaired from unrepaired data, and its operational history, particularly before Cassandra 4.0's fixes, was rocky enough that many large operators simply did not trust it. We did not.

Subrange repair breaks the token range into small segments repaired one at a time. Small Merkle trees, precise comparisons, minimal overstreaming, and each segment is a small retryable unit of work. This is the primitive that actually scales.

But subrange repair across 1,000 nodes is thousands upon thousands of segments needing ordering, throttling, retries and collision avoidance. Congratulations, it is a scheduling problem.

Reaper: repair as a system, not a command

We ran Cassandra Reaper, open source, originally Spotify's and now community maintained, as the scheduler.

Splits every table's range into segments and runs them steadily with concurrency caps
An intensity knob throttles repair pressure so p99s do not feel it. We tuned it to stay invisible.
Failed segments retry individually, so a node blip costs one segment, not the whole run
A web UI that answers the only question leadership asks, which is whether we are inside the gc_grace window, yes or no

The operating policy that came out of years of this:

- every table repaired well inside gc_grace, target a complete cycle in ~7 days against a 10 day grace
- repair pressure tuned to be invisible in p99 read latency
- alert not on "repair failed" but on "time since last completed repair per table"
- pause repairs during topology changes, resume, never skip

That last alert framing matters. Failures are noise. Staleness is the actual risk. Measure the deadline, not the activity.

The small scale corollary

Under about 50 nodes you do not need Reaper's ceremony, but you do need the same two invariants. Every node repaired inside gc_grace_seconds, and repairs that do not collide. A boring, monitored schedule beats heroics.

Repair is the tax on eventual consistency. You can pay it on a schedule you chose, or all at once with interest, with zombie data as the collection notice.

The heap dump usually exonerates Cassandra

Saqib Ameen Subhan — Fri, 04 Sep 2026 17:26:34 +0000

When a Cassandra node OOMs, or GC pauses start stretching into seconds, the reflex is to blame Cassandra, or its favourite scapegoat, the JVM.

I have read a lot of Cassandra heap dumps, and here is the uncomfortable pattern. The heap usually contains exactly what the data model asked for. The dump does not convict the database. It convicts a partition.

This post is the workflow I actually use, in order, because the order saves hours.

Step 0: GC logs before heap dumps

A heap dump is a biopsy. GC logs are the patient history, and they are already on disk. Two things I look for first:

# gc.log (G1)
Pause Young (Normal) ... 180ms
Pause Young (Normal) ... 210ms
Pause Full (Allocation Failure) ... 8.4s      <- the incident
...
Humongous Allocation ... 18MB
Humongous Allocation ... 22MB

Full GC pauses. G1 doing an emergency stop the world collection because normal cycles could not keep up. On a Cassandra node, an 8 second pause does not just slow queries. The node misses gossip, gets marked down by peers, and hints start piling up elsewhere. One node's GC problem becomes the cluster's problem.

Humongous allocations. G1 divides the heap into regions, typically 4 to 32MB. Any single allocation larger than half a region is humongous, allocated awkwardly across contiguous regions and collected inefficiently.

And what does Cassandra allocate in one giant contiguous chunk? A large partition being materialized for a read. Humongous allocation warnings in a Cassandra gc.log are large partitions announcing themselves. You can often skip the heap dump entirely at this point and go straight to nodetool tablehistograms to find the table with a monster p99 partition size.

Step 1: take the dump without causing an incident

jmap on a live heap stops the world for the duration of the dump. On a 20GB heap, that is long enough for the cluster to declare the node dead. Never dump a node that is still serving.

# isolate first, node stays alive but stops serving
nodetool disablebinary && nodetool disablegossip

jmap -dump:live,format=b,file=/data/dumps/cass_$(hostname)_$(date +%s).hprof <pid>

# afterwards: rejoin or just restart the node

Better, have the JVM do it for you at the moment of truth, before any human is awake:

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/data/dumps/

Those two flags belong in every production Cassandra JVM config. An OOM without a dump is an incident you get to have twice.

Related discipline, keep the heap at or under 8GB and let G1 target around 200ms pauses. Bigger heaps mostly buy you longer pauses and bigger dumps, and Cassandra's real caching happens off heap and in the page cache anyway.

Step 2: Eclipse MAT, dominator tree, ten minutes

Open the .hprof in Eclipse MAT and go straight to the dominator tree, objects ranked by retained memory, which answers what is actually holding the heap hostage. Ignore the histogram of a billion byte[]. The dominator tree tells you whose bytes.

What the tree shows, mapped to what it means, every one of these from real incidents:

Dominating the heap	Actual problem
A few enormous `byte[]` or cell containers under a read path	Large partition being materialized. Get the key from the referencing objects, confirm with `nodetool tablehistograms` or `sstablepartitions`. Fix is to bucket the partition, which is a later post in this series.
Memtable objects, many tables	Over wide schema, or memtable flush thresholds set too generous for the heap
Tombstone or range tombstone structures under a query	A tombstone farm read that survived just long enough to dump
Netty buffers or inflight requests	Clients hammering with no paging. Check fetch size and unpaged full partition reads

The punchline repeats across years and companies. The heap contains the workload. I have almost never opened a dump and found a Cassandra bug. I have regularly opened one and found a 4GB partition someone swore was just a busy customer.

Step 3: close the loop in the model

The dump names the object. The fix lives in the schema.

Large partition, time bucket the partition key. Unpaged reads, driver fetch size. Tombstone reads, the modeling fixes from two posts ago. The JVM flags and MAT are diagnosis. Cassandra data modeling is treatment.

Blaming the JVM is comfortable because nobody owns the JVM. The heap dump takes that comfort away. It shows you, in retained bytes, precisely which design decision you are looking at.

A compaction strategy is a bet about your workload

Saqib Ameen Subhan — Fri, 04 Sep 2026 17:25:24 +0000

Choosing a Cassandra compaction strategy is placing a bet. Each strategy optimizes one thing by deliberately sacrificing another, and the honest way to choose is to decide which failure mode you would rather operate.

I have run all three majors across 1,000+ node estates. Here is the bet each one makes.

Why compaction exists at all

Cassandra never updates in place. Writes land in a memtable, flush to immutable SSTables, and a row's fragments accumulate across many files as it is updated over time. Reads must merge every relevant fragment.

Compaction is the background process that merges SSTables, consolidating fragments and purging expired tombstones after gc_grace_seconds, so reads touch fewer files.

The metric that keeps compaction honest:

nodetool tablehistograms ks table
# Percentile  SSTables     ...
# 50%             1.00
# 99%             4.00

SSTables per read at p99 is the number compaction exists to keep small.

STCS, SizeTiered, the write optimized default

Merges SSTables of similar size into bigger ones, repeatedly. Minimal write amplification, maximal ingest throughput.

The bet you are making: reads and disk headroom are negotiable.

A hot row's fragments can sit in many SSTables of different generations, so SSTables per read climbs.
Space amplification is the operational trap. Compacting large tiers requires holding input and output simultaneously. Plan for 50% free disk. I have watched teams treat 70% disk usage as plenty left, then be unable to run the very compaction that would reclaim space. That is a corner with no good exits.
Tombstones buried in giant, rarely recompacted SSTables can linger far past gc_grace_seconds.

Right when: write heavy, append mostly, reads tolerate variance. It is the default because it is the least dangerous average bet, not because it is right for you.

LCS, Leveled, paying writes to buy reads

Organizes SSTables into levels, L1, L2 and so on, each 10 times larger, guaranteeing non overlapping key ranges within a level. A read touches at most one SSTable per level, and in practice the vast majority of reads are satisfied by a single SSTable.

The bet: you will pay roughly 10x write amplification, because every row is rewritten as it migrates down levels, to make read latency tight and predictable.

On read heavy tables with strict p99s, it is the correct bet.
On write heavy tables, compaction falls behind, pending tasks pile up in nodetool compactionstats, and ironically reads degrade anyway because L0 accumulates overlapping files.
Easier on disk headroom than STCS, works in around 10% free, but harder on I/O and CPU, continuously.

Right when: read dominated, update in place workloads. Wrong when your disks are already busy keeping up with ingest.

TWCS, TimeWindow, the time series specialist

Groups SSTables by time window, say one per day. Within the current window, STCS as usual. Once a window closes, its SSTables compact once into one file and are never touched again.

When TTL expires a whole window, Cassandra drops the entire file, so tombstone processing effectively vanishes for aged out data. This is the single biggest tombstone fix available for time series, and the punchline of the previous post.

The bet: your data is truly time ordered and immutable once written.

Write out of order data or update old windows and you create cross window overlaps that can never compact away. The strategy's guarantees quietly die while everything still appears to work.
Rule of thumb, aim for a window size that yields 20 to 50 windows over the table's TTL.

Right when: metrics, events, telemetry, anything with TTL and append only semantics. IoT and monitoring tables at scale should almost always be TWCS.

The bet, stated plainly

Strategy	You optimize	You pay with	Operational trap
STCS	write throughput	read variance	needs around 50% free disk to compact
LCS	read p99	10x write amplification	falls behind under heavy ingest
TWCS	TTL'd time series	inflexibility	out of order writes break it silently

Per table, not per cluster. A keyspace legitimately mixes all three. And changing the bet is an online operation:

ALTER TABLE ks.events WITH compaction =
  {'class': 'TimeWindowCompactionStrategy',
   'compaction_window_unit': 'DAYS', 'compaction_window_size': 1};

Cassandra will re sort existing SSTables over time. On a big table, schedule the transition like the migration it is.

There is no best compaction strategy. There is only the failure mode you have chosen on purpose, versus the one that chose you.

mtools does not parse MongoDB 4.4+ JSON logs. So I built mdbkit.

Saqib Ameen Subhan — Mon, 24 Aug 2026 19:28:11 +0000

If you have operated MongoDB for any length of time, you know mtools. For years it was how we all read production logs. mloginfo, mplotqueries, and a slow query was suddenly visible.

Then MongoDB 4.4 changed the log format to structured JSON, and the log tools stopped understanding what they were reading.

That was six years ago. The support issue was opened in June 2020 and it is still open. We are on 8.0 now, and a lot of us have been getting by with grep, jq pipelines held together by hope, and a certain amount of squinting.

I got tired of it and built the replacement. It is called mdbkit, it is MIT licensed, and it is public as of this week.

Try it in sixty seconds without touching a cluster

This is the part I would want first if this were someone else's tool.

pip install --upgrade mdbkit
# macOS ships no pip: brew install pipx && pipx install mdbkit

mdbkit demo -o demo.log
mdbkit queries demo.log

mdbkit demo generates a realistic MongoDB log containing an actual incident. No cluster, no connection string, nothing of yours involved. You get to judge the tool on a problem you can inspect before you point it at anything that matters.

Here is what queries gives you on that demo log:

saqib@MacbookPro mongodb_logs % mdbkit queries demo.log
== mdbkit queries (slow query shapes) ==
lines: 1,004  parsed: 1,004  unparsed: 0  span: 2026-07-01T08:00:00+04:00 -> 2026-07-01T09:29:44.615000+04:00

namespace      op         count  cumMs  mean   max    docsEx      scan     plan            shape
-------------  ---------  -----  -----  -----  -----  ----------  -------  --------------  ---------------------------------------------
shop.events    aggregate  53     5.6m   6.3s   8.9s   47,170,000  98889:1  COLLSCAN+SORT   {tenantId:eq, ts:gte} sort:{ts:-1}
shop.orders    find       89     2.4m   1.6s   2.4s   11,125,000  2976:1   COLLSCAN+SORT   {createdAt:gt, status:eq} sort:{createdAt:-1}
shop.sessions  find       58     34.7s  597ms  888ms  2,784,000   155:1    IXSCAN{active}  {active:eq, lastSeen:gt} sort:{lastSeen:-1}
shop.products  update     42     25.7s  612ms  799ms  2,268,000   54000:1  COLLSCAN        {sku:eq}
shop.users     find       56     6.7s   119ms  139ms  56          1:1      IXSCAN{email}   {email:eq}

cumMs  = total wall time accumulated across ALL occurrences (not one query)
docsEx = total documents examined across all occurrences
scan   = docsExamined per returned doc (high = index missing or weak)
plan   = most common query plan; COLLSCAN/+SORT = index needed
         empty plan (?) = plan not present in log (below slowms threshold)
next   : mdbkit advise <log> [--ns <namespace>] for index candidates

That is not a screenshot I curated. demo is seeded, so those two commands produce that exact table on your machine too, down to the last digit. It behaves the same on my laptop, on yours, and on a conference projector.

The column that matters is the scan ratio. It is documents examined against documents returned, per query shape. A shape sitting at 98889:1 is reading roughly ninety nine thousand documents to hand back one. A shape at 1:1 is doing exactly the work it needs to.

That single ratio is most of slow query triage. Everything else is detail.

The workflow I actually use

Three commands, in order.

First, find the shape that is hurting.

mdbkit queries /var/log/mongodb/mongod.log

Slow operations grouped by query shape rather than listed individually. This matters more than it sounds. A log with 40,000 slow query lines is usually eight or nine distinct shapes repeating, and the one costing you the most is rarely the one that appears most often.

Then, ask what would fix it.

mdbkit advise /var/log/mongodb/mongod.log

This proposes candidate indexes based on the shapes it observed. More on how that works below, because it is the part people should be suspicious of.

And when something has already gone wrong, build the timeline.

mdbkit triage /var/log/mongodb/mongod.log

Elections, rollbacks, stalls, connection storms, in order, with timestamps. When you are twenty minutes into an incident and someone senior is asking what happened, a timeline you did not have to assemble by hand is worth a great deal.

The log cannot tell you everything

There is one failure mode a mongod log structurally cannot explain: the log simply stops, then starts again a minute later with no error in between.

The process was killed before it could write anything. That answer lives in the system log, one line up from where you were looking.

mdbkit triage mongod.log --oslog /var/log/syslog

[CRIT] Process start(s) in window: mongod startup marker at 09:14:35
[CRIT] System: oom-kill: The kernel OOM killer terminated a task.
         1 occurrence (last at 09:14:03). Processes: mongod.
         next: This explains an unexplained restart in the log above.

oslog reads /var/log/syslog or /var/log/messages and looks for the things that end database processes: OOM kills, file descriptor limits, segmentation faults, I/O errors, filesystems remounted read only, systemd service exits. Used with triage, the unexplained restart gets matched to the kernel line that caused it.

On journald systems there is no text log to read, and mdbkit will not run commands on your behalf, so it prints the journalctl invocation for you to capture and feed back in.

There is also ftdc, which decodes diagnostic.data offline. Every one of your nodes has been recording CPU, cache, queue depth and disk every second since the day you installed it, whether or not you run any monitoring. Plus serverstatus for making sense of a serverStatus dump, and compare for answering whether the index you added last week actually helped.

It never connects to your database

No driver. No URI. No network code anywhere in it.

mdbkit reads log files you hand it. It is strictly read only. Where an action would help, it prints the command for you to review and run yourself, rather than running anything on your behalf.

I built it this way because I would not run a stranger's tool on a production host, and I do not expect you to either. So rather than asking you to trust me, the README shows you how to verify it independently. grep the source for socket and driver imports. Run it under strace and watch it make no network calls. It takes about a minute and you should do it.

Zero runtime dependencies too, nothing in the supply chain beyond the Python standard library. That was a deliberate constraint and it cost me some convenience, but a tool that ships onto database servers should not drag a dependency tree behind it.

The index advice is rules, not AI

I gave a talk at the Dubai MongoDB User Group recently about how confidently language models describe a MongoDB that no longer exists. They will happily reference plannerVersion, a field removed in 5.0. They do not know about EXPRESS stages in 8.0. And more importantly, they cannot know your cardinality, your write mix, or what is currently sitting in your plan cache, because none of that is in your query text.

So mdbkit's index advice is deterministic. Rules over observed query shapes. The same log always produces the same recommendation.

Every suggestion comes with the evidence it reasoned from, a confidence level, and how to validate it before you act. It says candidate, never command. And it will never tell you to drop an index, because deciding an index is unused requires knowing about traffic that a log file cannot show you.

That is a deliberately narrow promise. I would rather it be right about a small thing than confident about a large one.

What would help most

Two things, and they are both easy for you and hard for me.

Real world log lines that parse incorrectly. Every MongoDB deployment logs slightly differently, and I have only seen the estates I have worked on. If mdbkit chokes on a line, that line is the most useful thing you can send me.

Index advice that is wrong or unhelpful. If it suggests something you know is a bad idea for your workload, I want to know why, because that is a rule that needs fixing.

Every release so far has come from someone doing exactly that. An FTDC decode that took twenty five minutes and pinned a CPU. A crash on a batched write. A restart with no explanation, which is why oslog exists at all.

Supports MongoDB 4.4 through 8.0. Python 3.8+.

Source: https://github.com/saqibameen86/mdbkit
Package: https://pypi.org/project/mdbkit

Free, MIT, and built because the gap annoyed me for long enough.

If you liked this, I also wrote about deleting 2TB from a live MongoDB cluster without anyone noticing and why I take down a healthy primary on purpose.

In Cassandra, a delete is a write (and reads pay for it)

Saqib Ameen Subhan — Sun, 23 Aug 2026 16:36:09 +0000

In Cassandra, a delete is a write.

That sentence sounds like trivia until it takes down a read path. DELETE doesn't remove anything. It writes a tombstone, a marker saying "this data is dead as of timestamp T." The actual removal happens later, during compaction, and only after a grace period. Between those two moments, your deleted data has negative value. It's gone from your application's point of view but still physically present, and every read that touches its range must process it.

Why the marker has to exist

Cassandra is a distributed, eventually consistent system. Suppose a delete simply removed data, and one replica was down when the delete happened. When it comes back, it still has the row, and to the rest of the cluster, that looks like data the others are missing. Repair would helpfully copy the deleted row back to everyone.

Deleted data returning from the dead is called zombie data, and tombstones exist to prevent it. The tombstone outranks the older value everywhere it's seen.

Which is why tombstones must survive for gc_grace_seconds, default 864000, ten days, before compaction may purge them. Ten days is the window you have to repair a down replica. Shrink gc_grace_seconds without shrinking your repair interval and you've quietly signed up for zombies. Repair at scale is its own post, and it's coming.

How reads pay

A read merges data across memtable and SSTables. Every tombstone in the requested range must be read, held, and reconciled against live data before Cassandra can answer. A partition that has accumulated a million tombstones makes you process a million dead cells to return whatever's alive.

Cassandra tells you when this is happening, in two escalating tones:

WARN  ReadCommand - Read 812 live rows and 104,832 tombstone cells for query ...
ERROR ... Scanned over 100001 tombstones ... query aborted

The warning fires past tombstone_warn_threshold, 1,000 by default. Past tombstone_failure_threshold, 100,000, the read is killed mid flight with TombstoneOverwhelmingException. That's the database choosing to fail your query rather than let it OOM the node.

The thresholds are guardrails, not tuning knobs. Raising them treats the symptom and keeps the disease.

Where tombstone farms come from

Every one of these I've met in production.

Queue like workloads. Insert, process, delete, repeat, in the same partition. The partition becomes a graveyard the consumer must scan past on every poll. Cassandra as a queue is the canonical anti pattern for exactly this reason.
Collection overwrites. UPDATE t SET mymap = {...} replaces a whole collection, which writes a range tombstone over the old one first. Prefer additive updates, mymap = mymap + {...}.
Inserting NULLs. Binding null in a prepared statement writes a tombstone for that cell. ORMs and "just bind every column" code generate these invisibly. Use unset values, not nulls.
TTL everywhere. Expired TTL cells become tombstones too. Fine if your compaction strategy is built for it.

Finding them, then fixing them

# per-SSTable estimate
sstablemetadata /var/lib/cassandra/data/ks/table-*/ *-Data.db | grep -i droppable
# Estimated droppable tombstones: 0.83

0.83 means an estimated 83% of that SSTable is purgeable tombstones.

For query level visibility, TRACING ON in cqlsh shows tombstone cells scanned per query. For a live table, nodetool tablestats and watch average tombstones per slice.

The durable fixes are modeling fixes.

Time series with TTL, use TimeWindowCompactionStrategy. Whole SSTables age out and get dropped as files, so tombstone processing largely disappears. Compaction strategy trade offs are the next post.
Stop deleting in place in hot partitions. Partition by time bucket and let whole partitions expire instead.
Audit for null binding and collection overwrites. These are the tombstones nobody meant to write.

Deletes in Cassandra are cheap to issue and expensive to have issued. Model as if every delete is a small loan against your read latency, because it is, and gc_grace_seconds is the repayment schedule.

How to delete 2TB from a live MongoDB cluster without anyone noticing

Saqib Ameen Subhan — Sun, 23 Aug 2026 16:04:47 +0000

The request was simple. "There's about 2TB of expired data in this collection. Can you delete it tonight?"

The honest answer was no, and being able to explain why, not just refuse, is most of the job. A missing TTL index had let short lived data pile up for months in one of our busiest collections. The team wanted one big deleteMany overnight. Here's what that would have actually done.

Why a mass delete hurts, and it isn't locking

People reach for "locks" as the explanation. Wrong database era. WiredTiger uses document level concurrency, so a huge delete doesn't freeze the collection. The damage arrives through three quieter channels.

1. The oplog becomes a firehose. Every deleted document is an individual entry in the oplog. Delete 500 million documents and you've written 500 million oplog entries, which every secondary must pull and apply. Replication lag climbs. If lag exceeds what the oplog window can hold, a secondary falls off the oplog entirely and needs a full resync. Now your delete has cost you redundancy.

2. Cache churn. To delete a document, WiredTiger loads its pages into cache. A 2TB sweep drags cold data through a cache that was carefully full of hot data. Your working set gets evicted, p99 latency on unrelated queries climbs, and someone opens an incident that will never mention the word "delete."

3. Acknowledgment pressure. If your writes use w:"majority" (they should, see the write concern post), the delete's progress is gated on those same struggling secondaries. Everything compounds.

I demonstrated exactly this in a lower environment. Kicked off the naive delete, watched rs.printSecondaryReplicationInfo() lag grow, and the "tonight" deadline converted itself into a plan.

The pattern: chunk, sleep, watch the lag

const CHUNK = 10000;
const SLEEP_MS = 500;
const cutoff = ISODate("2026-01-01");

let total = 0;
while (true) {
  const ids = db.events.find({ created: { $lt: cutoff } },
                              { _id: 1 }).limit(CHUNK).toArray().map(d => d._id);
  if (ids.length === 0) break;

  db.events.deleteMany({ _id: { $in: ids } });
  total += ids.length;

  sleep(SLEEP_MS);   // let the oplog breathe

  if (total % 1000000 === 0) print(`${total} deleted, lag check...`);
}

Two operational rules around it.

Run in off peak windows only. We ran nightly windows and paused at business hours. Duration was about a week and a half for the full 2TB. Nobody noticed, which was the definition of success.

Watch replication lag as your throttle. If lag climbs past your comfort line, raise SLEEP_MS. The loop's speed limit is the cluster's health, not your patience.

sleep() between chunks looks unbearably conservative to developers. That is the point. The delete has no deadline once the growth is stopped, and the growth is stopped by the real fix.

The real fix: TTL, so this never becomes a project again

db.events.createIndex({ created: 1 }, { expireAfterSeconds: 2592000 })  // 30 days

Details worth knowing about the TTL monitor, because they surprise people.

It wakes every 60 seconds, so expiry is approximate, not instant. Fine for cleanup, wrong tool for business logic that needs precise expiry.
It deletes in background batches, effectively running my chunked loop forever, gently. On huge backlogs it can lag, which is why we cleared the 2TB manually first and let TTL own steady state.
Index the field the data actually ages by. TTL on the wrong date field is a slow motion data loss incident.

We closed it with a runbook, the chunk script, the lag thresholds and the window schedule, so the next 2TB request is a lookup, not a debate.

The fastest way to delete 2TB is slowly.

I take down a healthy primary on purpose. You should too.

Saqib Ameen Subhan — Sun, 23 Aug 2026 15:58:41 +0000

At one company I built an autoscaling pipeline that replaced MongoDB cluster nodes automatically. CloudWatch caught the CPU threshold, and the automation rolled the cluster node by node. Secondary first, second secondary next, primary last. Twelve minutes, no human, no application impact.

The order is the interesting part. Rolling the primary last means the cluster experiences exactly one election per cycle, at a moment the automation chooses. Which only works if you trust elections. And you only get to trust elections by triggering them yourself, regularly, and watching what your application does.

What actually happens in an election

The mechanics, compressed.

Every node heartbeats every 2 seconds. A node that misses heartbeats for electionTimeoutMillis (default 10 seconds) is presumed gone.
An electable secondary calls an election. Voting is not seniority. A node only wins if its oplog is at least as recent as each voter's. Freshness outranks priority.
priority settings bias who stands for election, not who deserves to win. A priority 10 node with a stale oplog loses to a priority 1 node that's caught up.
Majority vote, new primary, secondaries sync from it.

Wall clock cost is typically a few seconds for the election itself, bounded by that 10 second detection window on the front. Call it 5 to 15 seconds of no primary, during which writes cannot be acknowledged.

The part that pages you isn't the election

It's what your application does during those seconds. Three behaviours, in ascending order of maturity.

Errors bubble to users. Driver throws NotWritablePrimary, nobody catches it, checkout fails.
Blind retry loops. Sometimes double applies the write. Worse than failing.
retryWrites=true plus a sane serverSelectionTimeoutMS. The driver holds the write, discovers the new primary from the topology update, retries exactly once. To the user, one slow request.

You do not find out which of these you are during a real incident. That's the wrong time.

The drill

Quarterly. In staging first, then production during a low traffic window. Yes, production. Staging tells you the drill works, production tells you the truth.

// 1. Note current topology
rs.status().members.map(m => ({ name: m.name, state: m.stateStr }))

// 2. Ask the primary to step down gracefully
rs.stepDown(60)   // refuses re-election for 60s

// 3. Watch the election from a secondary
rs.status()

While it runs, the numbers I capture:

detection to new primary elected : ___ s
application error count          : ___
p99 latency during window        : ___ ms
first write after failover       : double applied? (check by unique key)

Those blanks are the deliverable. Fill them from your own drill.

rs.stepDown() is the polite version. The primary flushes and closes cleanly. Once that's boring, graduate to the impolite version. kill -9 the primary's mongod, or drop its network. That's the one that resembles an actual cloud incident, and it exercises the full 10 second detection window instead of skipping it.

Two configuration notes from doing this at scale

Even numbered voting members are a bug. Four voters can split 2 to 2 and elect nobody. Keep voting members odd. Use an arbiter only if you truly can't afford a third data node, and know that arbiters don't help w:"majority" acknowledge anything.

Priorities encode your topology intent. In multi region clusters we set higher priorities in the primary region so failback happens automatically once nodes recover. But again, priority only nominates. The oplog decides.

An election you've rehearsed is an operational event. An election you haven't is an outage with paperwork.

"The write was acknowledged" means less than you think

Saqib Ameen Subhan — Sun, 23 Aug 2026 15:49:20 +0000

I spent years operating databases for payment platforms, where a lost write isn't a bug, it's a regulatory conversation. That environment teaches you to read the phrase "the write was acknowledged" the way a lawyer reads a contract. Acknowledged by whom? Durable against what?

In MongoDB, that one sentence has at least four different meanings, controlled by two settings most applications leave at defaults without ever deciding to.

The four contracts

// 1. w:1, the primary has it in memory
db.payments.insertOne(doc, { writeConcern: { w: 1 } })

// 2. w:1, j:true, the primary has it in its journal on disk
db.payments.insertOne(doc, { writeConcern: { w: 1, j: true } })

// 3. w:"majority", a majority of the replica set has it
db.payments.insertOne(doc, { writeConcern: { w: "majority" } })

// 4. w:"majority", j:true, a majority has it journaled

Each level survives a different failure.

w:1 survives nothing interesting. If the primary's mongod crashes before the next journal flush, the write is gone, while your application already told the user "success."
w:1, j:true survives a process crash on the primary. It does not survive the primary failing over.
w:"majority" survives failover. This is the one that matters, and here's why.

Where acknowledged writes go to die

A replica set election isn't polite. If the primary accepts writes at w:1, then loses connectivity before replicating them, a secondary gets elected and moves on. When the old primary rejoins, it discovers history diverged. It has writes the new primary never saw.

Those writes get rolled back. MongoDB doesn't delete them silently, it writes them to BSON files in the rollback directory, where, in my experience, they are examined by precisely no one until an auditor asks a question.

w:"majority" closes this hole. The write isn't acknowledged until enough nodes have it that any electable primary must have it too. It cannot be elected away.

The read side of the same contract

Durable writes with careless reads is half a system. Two settings complete it:

// Don't show me data that could still be rolled back
db.payments.find({...}).readConcern("majority")

// My own session should see its own writes, in order
const session = client.startSession({ causalConsistency: true })

And on the driver, retryWrites=true (default in modern drivers) makes the failover window survivable. A write interrupted by an election is retried exactly once against the new primary, safely, because every write carries a unique transaction number.

Which is also why "just wrap it in a retry loop yourself" is worse than the built in one. Your loop can double apply. The driver's cannot.

What I actually deploy

Tiering, not one global setting.

Data	Write concern	Why
Payments, ledger, anything auditable	`w:"majority"`	Rollback is not an acceptable word here
User profile updates	`w:"majority"`	Cheap insurance, users notice lost edits
High volume telemetry, click events	`w:1`	Losing two seconds of events in a rare failover is a fair trade for throughput

The latency cost of majority is one replication round trip, single digit milliseconds inside a region. I've watched teams run payments at w:1 to save five milliseconds, which is a way of saying they priced a lost financial record at five milliseconds.

The lab

Start a 3 node replica set locally (mongod --replSet three times, or docker), then:

// terminal 1: insert at w:1 in a loop
// terminal 2: rs.stepDown() on the primary mid-loop
// then check the old primary's rollback directory

ls /data/rs1/rollback/

Run it. Seeing your own acknowledged write sitting in a rollback file is worth more than this whole post.

Defaults are decisions someone else made. For write concern, make your own, per collection, on purpose, written down.

cannot index parallel arrays, the MongoDB error that improves your schema

Saqib Ameen Subhan — Sun, 16 Aug 2026 21:41:53 +0000

MongoServerError: cannot index parallel arrays [certifications] [skills]

If you've never hit this error, your schemas have been lucky. A team once brought me a document shaped like this during a design review:

{
  _id: 1,
  name: "engineer-42",
  skills:         ["mongodb", "cassandra", "python"],
  certifications: ["dba", "atlas-admin"]
}

They wanted fast queries on skill and certification combinations, so the natural move was:

db.engineers.createIndex({ skills: 1, certifications: 1 })
// MongoServerError: cannot index parallel arrays

MongoDB refuses. Not "performs badly", refuses. And the reason is one of those internals that, once you see it, changes how you model.

Why: multikey indexes are per element

When you index an array field, MongoDB creates a multikey index, one index entry per array element. Index skills on the document above and you get three entries, one for "mongodb", one for "cassandra", one for "python", each pointing at the same document.

Now imagine a compound index across two independent arrays. To answer any combination query, MongoDB would need an entry for every element of the cross product. Three skills times two certifications is six entries. Harmless here. But two arrays of 1,000 elements each is a million index entries for a single document. One insertOne fans out into a million B-tree writes. The server declines to let you build that footgun, and honestly, good.

There's a quieter problem hiding in the schema too. The combinations are meaningless. Nothing in two parallel arrays says which skill relates to which certification. The data model can't answer the question the index was supposed to serve.

The fix: one array of subdocuments

{
  _id: 1,
  name: "engineer-42",
  quals: [
    { skill: "mongodb",   cert: "dba" },
    { skill: "mongodb",   cert: "atlas-admin" },
    { skill: "cassandra", cert: null }
  ]
}

db.engineers.createIndex({ "quals.skill": 1, "quals.cert": 1 })
// works

This is allowed because both indexed paths traverse the same array. Index entries are generated per array element, so each entry is a real (skill, cert) pair that actually existed together in the data, the correlation the parallel arrays destroyed. The index gets smaller and the model gets more truthful. That combination is rare.

One caveat worth knowing before you rely on it. To match both fields within the same element, query with $elemMatch:

db.engineers.find({ quals: { $elemMatch: { skill: "mongodb", cert: "dba" } } })

Without $elemMatch, MongoDB matches documents where some element has the skill and some other element has the cert. The parallel arrays bug, reincarnated at query time.

The lab

db.t.insertOne({ a: [1,2,3], b: ["x","y"] })
db.t.createIndex({ a: 1, b: 1 })                          // watch it fail

db.t2.insertOne({ pairs: [{a:1,b:"x"},{a:2,b:"y"}] })
db.t2.createIndex({ "pairs.a": 1, "pairs.b": 1 })         // watch it work

db.t2.find({ pairs: { $elemMatch: { a: 1, b: "y" } } })   // zero results, and that's correct

That last line is the whole lesson in one query. a:1 and b:"y" never occurred in the same element, so the document doesn't match. Parallel arrays would have happily lied to you.

Some errors are the database being difficult. This one is the database refusing to let your schema make a promise it can't keep.

The explain plan I read before the CEO called back!

Saqib Ameen Subhan — Tue, 04 Aug 2026 05:09:05 +0000

The escalation reached our CEO before it reached my team.

An enterprise customer, one of the largest on the platform, was seeing latency spikes bad enough that they raised it commercially, not technically. I had about twenty minutes before a call with their leadership, so I did the only thing that ever works in that situation: I stopped guessing and pulled the profiler.

db.setProfilingLevel(1, { slowms: 1000 })
// ...wait one cycle...
db.system.profile.find().sort({ millis: -1 }).limit(3)

I set the threshold (1000 ms) high on purpose. The default log already captures slow queries over 100 ms, but the profiler gives me something the log doesn't: structured records I can sort and aggregate. I want it catching only the genuinely pathological queries, so the culprit stands out and the profiler itself stays light on an already stressed system.

One aggregation stood out. It ran every five minutes, scanned a collection of roughly 200 million documents, and had no index supporting it. Every five minutes, a full collection scan, on shared infrastructure. The "noisy neighbor" wasn't a mystery, it was on a schedule.

How I actually read an explain plan

Run the slow query with execution stats:

db.orders.explain("executionStats").aggregate([
  { $match: { accountId: "ACME", status: "active", created: { $gte: ISODate("2026-08-01") } } },
  { $sort: { created: -1 } }
])

I look at exactly three numbers before anything else:

"nReturned"          : 214,
"totalKeysExamined"  : 0,
"totalDocsExamined"  : 198236411

The ratio is the diagnosis. Healthy queries examine roughly as many keys as they return. This one examined 198 million documents to return 214. totalKeysExamined: 0 means no index was touched at all. The winning plan was COLLSCAN.

That's the whole skill, honestly. Everything else in the explain output is detail. If totalDocsExamined is orders of magnitude above nReturned, the query is doing the database's job by hand.

ESR: the field order that makes or breaks the index

The fix was not "add an index." It was "add the right index" and the difference is field order. MongoDB compound indexes follow what the docs call the ESR rule, Equality, Sort, Range:

Equality predicates first (accountId, status)
Sort fields next (created as sort key)
Range predicates last (created: { $gte: ... }, here sort and range share a field, which is the friendly case)

db.orders.createIndex({ accountId: 1, status: 1, created: -1 })

Get the order wrong, range before sort, and the index still gets used, but MongoDB has to fetch and sort in memory. You'll see it in the plan as a SORT stage instead of the sort being absorbed by index order. On 200 million documents, an in-memory sort is not a detail.

One refinement we used because the query only ever cared about active records:

db.orders.createIndex(
  { accountId: 1, created: -1 },
  { partialFilterExpression: { status: "active" } }
)

A partial index keeps only the entries that match the filter. Smaller index, less RAM, faster writes, and the query planner picks it up as long as the query includes the same predicate.

The lab, if you want to see it yourself

Five minutes on a local mongod:

for (let i = 0; i < 500000; i++) {
  db.orders.insertOne({ accountId: "A" + (i % 50), status: i % 3 ? "active" : "closed",
                        created: new Date(Date.now() - i * 60000), amount: i % 997 })
}

db.orders.explain("executionStats").find(
  { accountId: "A7", status: "active" }).sort({ created: -1 })

// note totalDocsExamined, then:
db.orders.createIndex({ accountId: 1, status: 1, created: -1 })

// run the explain again and compare

The before and after on totalDocsExamined makes the argument better than any slide deck.

What happened with the customer

Root cause explained in plain language on the call. I described the collection as a library with no catalogue, where every request meant walking every aisle. Partial index shipped the same day. Latency resolved within six hours of the escalation, and two weeks later I presented them a health report instead of an apology. They renewed.

Twenty minutes, the profiler, and three numbers. The hard part was knowing which three.