DEV Community: Saqib Ameen Subhan

In Cassandra, a delete is a write (and reads pay for it)

Saqib Ameen Subhan — Sun, 23 Aug 2026 16:36:09 +0000

In Cassandra, a delete is a write.

That sentence sounds like trivia until it takes down a read path. DELETE doesn't remove anything. It writes a tombstone, a marker saying "this data is dead as of timestamp T." The actual removal happens later, during compaction, and only after a grace period. Between those two moments, your deleted data has negative value. It's gone from your application's point of view but still physically present, and every read that touches its range must process it.

Why the marker has to exist

Cassandra is a distributed, eventually consistent system. Suppose a delete simply removed data, and one replica was down when the delete happened. When it comes back, it still has the row, and to the rest of the cluster, that looks like data the others are missing. Repair would helpfully copy the deleted row back to everyone.

Deleted data returning from the dead is called zombie data, and tombstones exist to prevent it. The tombstone outranks the older value everywhere it's seen.

Which is why tombstones must survive for gc_grace_seconds, default 864000, ten days, before compaction may purge them. Ten days is the window you have to repair a down replica. Shrink gc_grace_seconds without shrinking your repair interval and you've quietly signed up for zombies. Repair at scale is its own post, and it's coming.

How reads pay

A read merges data across memtable and SSTables. Every tombstone in the requested range must be read, held, and reconciled against live data before Cassandra can answer. A partition that has accumulated a million tombstones makes you process a million dead cells to return whatever's alive.

Cassandra tells you when this is happening, in two escalating tones:

WARN  ReadCommand - Read 812 live rows and 104,832 tombstone cells for query ...
ERROR ... Scanned over 100001 tombstones ... query aborted

The warning fires past tombstone_warn_threshold, 1,000 by default. Past tombstone_failure_threshold, 100,000, the read is killed mid flight with TombstoneOverwhelmingException. That's the database choosing to fail your query rather than let it OOM the node.

The thresholds are guardrails, not tuning knobs. Raising them treats the symptom and keeps the disease.

Where tombstone farms come from

Every one of these I've met in production.

Queue like workloads. Insert, process, delete, repeat, in the same partition. The partition becomes a graveyard the consumer must scan past on every poll. Cassandra as a queue is the canonical anti pattern for exactly this reason.
Collection overwrites. UPDATE t SET mymap = {...} replaces a whole collection, which writes a range tombstone over the old one first. Prefer additive updates, mymap = mymap + {...}.
Inserting NULLs. Binding null in a prepared statement writes a tombstone for that cell. ORMs and "just bind every column" code generate these invisibly. Use unset values, not nulls.
TTL everywhere. Expired TTL cells become tombstones too. Fine if your compaction strategy is built for it.

Finding them, then fixing them

# per-SSTable estimate
sstablemetadata /var/lib/cassandra/data/ks/table-*/ *-Data.db | grep -i droppable
# Estimated droppable tombstones: 0.83

0.83 means an estimated 83% of that SSTable is purgeable tombstones.

For query level visibility, TRACING ON in cqlsh shows tombstone cells scanned per query. For a live table, nodetool tablestats and watch average tombstones per slice.

The durable fixes are modeling fixes.

Time series with TTL, use TimeWindowCompactionStrategy. Whole SSTables age out and get dropped as files, so tombstone processing largely disappears. Compaction strategy trade offs are the next post.
Stop deleting in place in hot partitions. Partition by time bucket and let whole partitions expire instead.
Audit for null binding and collection overwrites. These are the tombstones nobody meant to write.

Deletes in Cassandra are cheap to issue and expensive to have issued. Model as if every delete is a small loan against your read latency, because it is, and gc_grace_seconds is the repayment schedule.

How to delete 2TB from a live MongoDB cluster without anyone noticing

Saqib Ameen Subhan — Sun, 23 Aug 2026 16:04:47 +0000

The request was simple. "There's about 2TB of expired data in this collection. Can you delete it tonight?"

The honest answer was no, and being able to explain why, not just refuse, is most of the job. A missing TTL index had let short lived data pile up for months in one of our busiest collections. The team wanted one big deleteMany overnight. Here's what that would have actually done.

Why a mass delete hurts, and it isn't locking

People reach for "locks" as the explanation. Wrong database era. WiredTiger uses document level concurrency, so a huge delete doesn't freeze the collection. The damage arrives through three quieter channels.

1. The oplog becomes a firehose. Every deleted document is an individual entry in the oplog. Delete 500 million documents and you've written 500 million oplog entries, which every secondary must pull and apply. Replication lag climbs. If lag exceeds what the oplog window can hold, a secondary falls off the oplog entirely and needs a full resync. Now your delete has cost you redundancy.

2. Cache churn. To delete a document, WiredTiger loads its pages into cache. A 2TB sweep drags cold data through a cache that was carefully full of hot data. Your working set gets evicted, p99 latency on unrelated queries climbs, and someone opens an incident that will never mention the word "delete."

3. Acknowledgment pressure. If your writes use w:"majority" (they should, see the write concern post), the delete's progress is gated on those same struggling secondaries. Everything compounds.

I demonstrated exactly this in a lower environment. Kicked off the naive delete, watched rs.printSecondaryReplicationInfo() lag grow, and the "tonight" deadline converted itself into a plan.

The pattern: chunk, sleep, watch the lag

const CHUNK = 10000;
const SLEEP_MS = 500;
const cutoff = ISODate("2026-01-01");

let total = 0;
while (true) {
  const ids = db.events.find({ created: { $lt: cutoff } },
                              { _id: 1 }).limit(CHUNK).toArray().map(d => d._id);
  if (ids.length === 0) break;

  db.events.deleteMany({ _id: { $in: ids } });
  total += ids.length;

  sleep(SLEEP_MS);   // let the oplog breathe

  if (total % 1000000 === 0) print(`${total} deleted, lag check...`);
}

Two operational rules around it.

Run in off peak windows only. We ran nightly windows and paused at business hours. Duration was about a week and a half for the full 2TB. Nobody noticed, which was the definition of success.

Watch replication lag as your throttle. If lag climbs past your comfort line, raise SLEEP_MS. The loop's speed limit is the cluster's health, not your patience.

sleep() between chunks looks unbearably conservative to developers. That is the point. The delete has no deadline once the growth is stopped, and the growth is stopped by the real fix.

The real fix: TTL, so this never becomes a project again

db.events.createIndex({ created: 1 }, { expireAfterSeconds: 2592000 })  // 30 days

Details worth knowing about the TTL monitor, because they surprise people.

It wakes every 60 seconds, so expiry is approximate, not instant. Fine for cleanup, wrong tool for business logic that needs precise expiry.
It deletes in background batches, effectively running my chunked loop forever, gently. On huge backlogs it can lag, which is why we cleared the 2TB manually first and let TTL own steady state.
Index the field the data actually ages by. TTL on the wrong date field is a slow motion data loss incident.

We closed it with a runbook, the chunk script, the lag thresholds and the window schedule, so the next 2TB request is a lookup, not a debate.

The fastest way to delete 2TB is slowly.

I take down a healthy primary on purpose. You should too.

Saqib Ameen Subhan — Sun, 23 Aug 2026 15:58:41 +0000

At one company I built an autoscaling pipeline that replaced MongoDB cluster nodes automatically. CloudWatch caught the CPU threshold, and the automation rolled the cluster node by node. Secondary first, second secondary next, primary last. Twelve minutes, no human, no application impact.

The order is the interesting part. Rolling the primary last means the cluster experiences exactly one election per cycle, at a moment the automation chooses. Which only works if you trust elections. And you only get to trust elections by triggering them yourself, regularly, and watching what your application does.

What actually happens in an election

The mechanics, compressed.

Every node heartbeats every 2 seconds. A node that misses heartbeats for electionTimeoutMillis (default 10 seconds) is presumed gone.
An electable secondary calls an election. Voting is not seniority. A node only wins if its oplog is at least as recent as each voter's. Freshness outranks priority.
priority settings bias who stands for election, not who deserves to win. A priority 10 node with a stale oplog loses to a priority 1 node that's caught up.
Majority vote, new primary, secondaries sync from it.

Wall clock cost is typically a few seconds for the election itself, bounded by that 10 second detection window on the front. Call it 5 to 15 seconds of no primary, during which writes cannot be acknowledged.

The part that pages you isn't the election

It's what your application does during those seconds. Three behaviours, in ascending order of maturity.

Errors bubble to users. Driver throws NotWritablePrimary, nobody catches it, checkout fails.
Blind retry loops. Sometimes double applies the write. Worse than failing.
retryWrites=true plus a sane serverSelectionTimeoutMS. The driver holds the write, discovers the new primary from the topology update, retries exactly once. To the user, one slow request.

You do not find out which of these you are during a real incident. That's the wrong time.

The drill

Quarterly. In staging first, then production during a low traffic window. Yes, production. Staging tells you the drill works, production tells you the truth.

// 1. Note current topology
rs.status().members.map(m => ({ name: m.name, state: m.stateStr }))

// 2. Ask the primary to step down gracefully
rs.stepDown(60)   // refuses re-election for 60s

// 3. Watch the election from a secondary
rs.status()

While it runs, the numbers I capture:

detection to new primary elected : ___ s
application error count          : ___
p99 latency during window        : ___ ms
first write after failover       : double applied? (check by unique key)

Those blanks are the deliverable. Fill them from your own drill.

rs.stepDown() is the polite version. The primary flushes and closes cleanly. Once that's boring, graduate to the impolite version. kill -9 the primary's mongod, or drop its network. That's the one that resembles an actual cloud incident, and it exercises the full 10 second detection window instead of skipping it.

Two configuration notes from doing this at scale

Even numbered voting members are a bug. Four voters can split 2 to 2 and elect nobody. Keep voting members odd. Use an arbiter only if you truly can't afford a third data node, and know that arbiters don't help w:"majority" acknowledge anything.

Priorities encode your topology intent. In multi region clusters we set higher priorities in the primary region so failback happens automatically once nodes recover. But again, priority only nominates. The oplog decides.

An election you've rehearsed is an operational event. An election you haven't is an outage with paperwork.

"The write was acknowledged" means less than you think

Saqib Ameen Subhan — Sun, 23 Aug 2026 15:49:20 +0000

I spent years operating databases for payment platforms, where a lost write isn't a bug, it's a regulatory conversation. That environment teaches you to read the phrase "the write was acknowledged" the way a lawyer reads a contract. Acknowledged by whom? Durable against what?

In MongoDB, that one sentence has at least four different meanings, controlled by two settings most applications leave at defaults without ever deciding to.

The four contracts

// 1. w:1, the primary has it in memory
db.payments.insertOne(doc, { writeConcern: { w: 1 } })

// 2. w:1, j:true, the primary has it in its journal on disk
db.payments.insertOne(doc, { writeConcern: { w: 1, j: true } })

// 3. w:"majority", a majority of the replica set has it
db.payments.insertOne(doc, { writeConcern: { w: "majority" } })

// 4. w:"majority", j:true, a majority has it journaled

Each level survives a different failure.

w:1 survives nothing interesting. If the primary's mongod crashes before the next journal flush, the write is gone, while your application already told the user "success."
w:1, j:true survives a process crash on the primary. It does not survive the primary failing over.
w:"majority" survives failover. This is the one that matters, and here's why.

Where acknowledged writes go to die

A replica set election isn't polite. If the primary accepts writes at w:1, then loses connectivity before replicating them, a secondary gets elected and moves on. When the old primary rejoins, it discovers history diverged. It has writes the new primary never saw.

Those writes get rolled back. MongoDB doesn't delete them silently, it writes them to BSON files in the rollback directory, where, in my experience, they are examined by precisely no one until an auditor asks a question.

w:"majority" closes this hole. The write isn't acknowledged until enough nodes have it that any electable primary must have it too. It cannot be elected away.

The read side of the same contract

Durable writes with careless reads is half a system. Two settings complete it:

// Don't show me data that could still be rolled back
db.payments.find({...}).readConcern("majority")

// My own session should see its own writes, in order
const session = client.startSession({ causalConsistency: true })

And on the driver, retryWrites=true (default in modern drivers) makes the failover window survivable. A write interrupted by an election is retried exactly once against the new primary, safely, because every write carries a unique transaction number.

Which is also why "just wrap it in a retry loop yourself" is worse than the built in one. Your loop can double apply. The driver's cannot.

What I actually deploy

Tiering, not one global setting.

Data	Write concern	Why
Payments, ledger, anything auditable	`w:"majority"`	Rollback is not an acceptable word here
User profile updates	`w:"majority"`	Cheap insurance, users notice lost edits
High volume telemetry, click events	`w:1`	Losing two seconds of events in a rare failover is a fair trade for throughput

The latency cost of majority is one replication round trip, single digit milliseconds inside a region. I've watched teams run payments at w:1 to save five milliseconds, which is a way of saying they priced a lost financial record at five milliseconds.

The lab

Start a 3 node replica set locally (mongod --replSet three times, or docker), then:

// terminal 1: insert at w:1 in a loop
// terminal 2: rs.stepDown() on the primary mid-loop
// then check the old primary's rollback directory

ls /data/rs1/rollback/

Run it. Seeing your own acknowledged write sitting in a rollback file is worth more than this whole post.

Defaults are decisions someone else made. For write concern, make your own, per collection, on purpose, written down.

cannot index parallel arrays, the MongoDB error that improves your schema

Saqib Ameen Subhan — Sun, 16 Aug 2026 21:41:53 +0000

MongoServerError: cannot index parallel arrays [certifications] [skills]

If you've never hit this error, your schemas have been lucky. A team once brought me a document shaped like this during a design review:

{
  _id: 1,
  name: "engineer-42",
  skills:         ["mongodb", "cassandra", "python"],
  certifications: ["dba", "atlas-admin"]
}

They wanted fast queries on skill and certification combinations, so the natural move was:

db.engineers.createIndex({ skills: 1, certifications: 1 })
// MongoServerError: cannot index parallel arrays

MongoDB refuses. Not "performs badly", refuses. And the reason is one of those internals that, once you see it, changes how you model.

Why: multikey indexes are per element

When you index an array field, MongoDB creates a multikey index, one index entry per array element. Index skills on the document above and you get three entries, one for "mongodb", one for "cassandra", one for "python", each pointing at the same document.

Now imagine a compound index across two independent arrays. To answer any combination query, MongoDB would need an entry for every element of the cross product. Three skills times two certifications is six entries. Harmless here. But two arrays of 1,000 elements each is a million index entries for a single document. One insertOne fans out into a million B-tree writes. The server declines to let you build that footgun, and honestly, good.

There's a quieter problem hiding in the schema too. The combinations are meaningless. Nothing in two parallel arrays says which skill relates to which certification. The data model can't answer the question the index was supposed to serve.

The fix: one array of subdocuments

{
  _id: 1,
  name: "engineer-42",
  quals: [
    { skill: "mongodb",   cert: "dba" },
    { skill: "mongodb",   cert: "atlas-admin" },
    { skill: "cassandra", cert: null }
  ]
}

db.engineers.createIndex({ "quals.skill": 1, "quals.cert": 1 })
// works

This is allowed because both indexed paths traverse the same array. Index entries are generated per array element, so each entry is a real (skill, cert) pair that actually existed together in the data, the correlation the parallel arrays destroyed. The index gets smaller and the model gets more truthful. That combination is rare.

One caveat worth knowing before you rely on it. To match both fields within the same element, query with $elemMatch:

db.engineers.find({ quals: { $elemMatch: { skill: "mongodb", cert: "dba" } } })

Without $elemMatch, MongoDB matches documents where some element has the skill and some other element has the cert. The parallel arrays bug, reincarnated at query time.

The lab

db.t.insertOne({ a: [1,2,3], b: ["x","y"] })
db.t.createIndex({ a: 1, b: 1 })                          // watch it fail

db.t2.insertOne({ pairs: [{a:1,b:"x"},{a:2,b:"y"}] })
db.t2.createIndex({ "pairs.a": 1, "pairs.b": 1 })         // watch it work

db.t2.find({ pairs: { $elemMatch: { a: 1, b: "y" } } })   // zero results, and that's correct

That last line is the whole lesson in one query. a:1 and b:"y" never occurred in the same element, so the document doesn't match. Parallel arrays would have happily lied to you.

Some errors are the database being difficult. This one is the database refusing to let your schema make a promise it can't keep.

The explain plan I read before the CEO called back!

Saqib Ameen Subhan — Tue, 04 Aug 2026 05:09:05 +0000

The escalation reached our CEO before it reached my team.

An enterprise customer, one of the largest on the platform, was seeing latency spikes bad enough that they raised it commercially, not technically. I had about twenty minutes before a call with their leadership, so I did the only thing that ever works in that situation: I stopped guessing and pulled the profiler.

db.setProfilingLevel(1, { slowms: 1000 })
// ...wait one cycle...
db.system.profile.find().sort({ millis: -1 }).limit(3)

I set the threshold (1000 ms) high on purpose. The default log already captures slow queries over 100 ms, but the profiler gives me something the log doesn't: structured records I can sort and aggregate. I want it catching only the genuinely pathological queries, so the culprit stands out and the profiler itself stays light on an already stressed system.

One aggregation stood out. It ran every five minutes, scanned a collection of roughly 200 million documents, and had no index supporting it. Every five minutes, a full collection scan, on shared infrastructure. The "noisy neighbor" wasn't a mystery, it was on a schedule.

How I actually read an explain plan

Run the slow query with execution stats:

db.orders.explain("executionStats").aggregate([
  { $match: { accountId: "ACME", status: "active", created: { $gte: ISODate("2026-08-01") } } },
  { $sort: { created: -1 } }
])

I look at exactly three numbers before anything else:

"nReturned"          : 214,
"totalKeysExamined"  : 0,
"totalDocsExamined"  : 198236411

The ratio is the diagnosis. Healthy queries examine roughly as many keys as they return. This one examined 198 million documents to return 214. totalKeysExamined: 0 means no index was touched at all. The winning plan was COLLSCAN.

That's the whole skill, honestly. Everything else in the explain output is detail. If totalDocsExamined is orders of magnitude above nReturned, the query is doing the database's job by hand.

ESR: the field order that makes or breaks the index

The fix was not "add an index." It was "add the right index" and the difference is field order. MongoDB compound indexes follow what the docs call the ESR rule, Equality, Sort, Range:

Equality predicates first (accountId, status)
Sort fields next (created as sort key)
Range predicates last (created: { $gte: ... }, here sort and range share a field, which is the friendly case)

db.orders.createIndex({ accountId: 1, status: 1, created: -1 })

Get the order wrong, range before sort, and the index still gets used, but MongoDB has to fetch and sort in memory. You'll see it in the plan as a SORT stage instead of the sort being absorbed by index order. On 200 million documents, an in-memory sort is not a detail.

One refinement we used because the query only ever cared about active records:

db.orders.createIndex(
  { accountId: 1, created: -1 },
  { partialFilterExpression: { status: "active" } }
)

A partial index keeps only the entries that match the filter. Smaller index, less RAM, faster writes, and the query planner picks it up as long as the query includes the same predicate.

The lab, if you want to see it yourself

Five minutes on a local mongod:

for (let i = 0; i < 500000; i++) {
  db.orders.insertOne({ accountId: "A" + (i % 50), status: i % 3 ? "active" : "closed",
                        created: new Date(Date.now() - i * 60000), amount: i % 997 })
}

db.orders.explain("executionStats").find(
  { accountId: "A7", status: "active" }).sort({ created: -1 })

// note totalDocsExamined, then:
db.orders.createIndex({ accountId: 1, status: 1, created: -1 })

// run the explain again and compare

The before and after on totalDocsExamined makes the argument better than any slide deck.

What happened with the customer

Root cause explained in plain language on the call. I described the collection as a library with no catalogue, where every request meant walking every aisle. Partial index shipped the same day. Latency resolved within six hours of the escalation, and two weeks later I presented them a health report instead of an apology. They renewed.

Twenty minutes, the profiler, and three numbers. The hard part was knowing which three.