DEV Community: Saqib Ameen Subhan

They did it, and I meant it

Saqib Ameen Subhan — Wed, 30 Sep 2026 10:24:14 +0000

In an interview, I was asked about the achievement I was most proud of. I talked about a migration my team did, moving a high traffic product from RDS MySQL to Aurora Serverless with zero downtime.

Then the interviewer asked what my role in it was. I gave an answer that can feel risky in an interview. I said I did not build it, my engineers did. My job was managing the stakeholders, getting the approvals, and making sure nothing got in their way.

I know the usual interview advice is to take more of the credit, to say things like "I designed the approach" or "I made the key decisions." Some of that would even be partly true. But the answer I gave is how I actually think about management, and it would not mean much if I only said it when it was easy.

Why the credit belongs to the team

The way I see it, a manager's work is the team's results. That is the job. If a manager takes personal credit for something the team delivered, they are counting the same work twice. Once as "my team delivered this" and again as "I delivered this." Only one of those is true.

The engineer who wrote the cutover script built the cutover. What I did was create the conditions for it. I got the support to do it properly instead of quickly, I made sure more than one person could run it, and I kept the stakeholders calm while the work was happening. That is real work and I am happy to take credit for exactly that. But the migration belongs to them.

What giving credit actually looks like

Saying "great job, team" in a meeting is nice, but it does not change anything for anyone. The credit that matters is the credit that reaches the people who decide promotions and careers.

Their names go in the email to leadership. Not "the team delivered this," but the actual names of the people who did it. Leadership should know who the strong engineers are without having to ask me.
They present their own work. When leadership wants a walkthrough of the migration, the engineer who built it presents it. I am in the room only to handle any difficult questions, not to explain their work for them.
My name comes up when something breaks. This is the other half of the same rule. Credit goes down to the team, and responsibility for problems comes up to me. You cannot do one without the other.

You can only give credit for work you let them own

There is one condition that people usually skip. You cannot give someone credit for work you did not let them own.

If the manager makes every important decision, reviews every line and solves every hard problem before the team gets to it, then "the credit goes to my team" is just something nice to say. Everyone in the room knows who really did it. To give real credit, you first have to hand over real ownership, with real responsibility and real production risk, before you know how it will turn out. That migration was my engineers' achievement because it was genuinely theirs to get wrong.

This is also why it builds over time. People who get real credit for real ownership take on more ownership next time. That is how two engineers I hired as interns went on to lead teams of their own.

The cost of doing it this way

In the short term, this makes you less visible to leadership as an individual. While you are putting your engineers' names in the email, another manager might be presenting their team's work as their own, and for a while it may work for them. That can be frustrating to watch.

I will not pretend it always balances out quickly. But when a manager's team keeps doing well, quarter after quarter, people start to notice who built that team. It takes longer, but it is the honest way to get there.

The migration is still one of my proudest achievements, and it still belongs to them.

I stopped being the leave referee

Saqib Ameen Subhan — Wed, 30 Sep 2026 10:20:07 +0000

Three engineers, three festivals, and one on call rota that has to be covered every single day.

Every year, Diwali, Christmas and New Year fall within a few weeks of each other. Every database team I have run has been small enough that not everyone can be on leave at the same time. For a long time I handled this the way most managers are taught to. The leave requests came to me, I looked at them, and I decided.

I tried hard to be fair. I kept notes on who got which holidays last year, and I balanced it as carefully as I could. But every time, my fair and well documented decision ended the same way. Someone did not get the days they wanted, and they were unhappy with me for it.

The problem was not how fair I was. The problem was that I was the one deciding. When a manager decides, someone always loses, and that person does not blame the situation. They blame the decision, because the decision has the manager's name on it. You can be as fair as you like and still end the quarter with someone on the team quietly unhappy with you.

What I do now

I set the rule, and I let the team decide the calendar.

The rule is about coverage, not about people, and it is the only part I keep. There must always be enough people available to cover production. Production does not stop for festivals. Who takes which days, as long as that rule holds, is not my decision anymore.

The first time I did this, I expected to be pulled back in within a day. Instead, the team just talked it out between themselves. One of them said, "You take Diwali, it matters more to you than Christmas does to me. I will take Christmas. I will cover New Year this time if I can have it next year." It took about ten minutes, and the calendar was done.

What is different here is that nobody lost. When I decided, one person lost. When they agreed on it themselves, both of them owned the decision. The engineer covering New Year is not unhappy with me, because he made a trade he thinks is fair, with a colleague who now owes him one. They both remember the deal better than any spreadsheet I could have kept.

Why it keeps working

When I was the one making sure things were fair, it felt like I was keeping score on everyone. When the team does it themselves, it becomes a habit they follow on their own. They remember who covered last winter without anyone telling them. They sort it out before I even see the calendar. They even protect each other's leave when other teams ask for help during holidays, which is something no rule of mine ever managed.

There was one more thing I did not expect. Working it out together brought the team closer. When someone gives up their days so that a colleague can be with family for a festival, it changes something between those two people. They have shown each other, not just said it, that they will cover for each other. That is exactly what you need later during an incident, when covering for each other is the whole job.

The real message underneath all this is simple. I trust you to sort this out like the professionals you are. Letting them decide is itself a way of showing respect, and people notice when they are treated like adults. They give that respect back.

Where this does not work

This does not work in every team, and I would not be honest if I said it did.

It only works if the team already feels safe speaking up. If you hand the calendar to a team where one loud person dominates, that person will take December every year, and the quieter people will tell themselves they did not really mind. Letting the team decide does not create fairness. It makes whatever culture you already have stronger. If the team respects each other, you get the kind of conversation I described. If they do not, you have just made the unfairness automatic and removed the one person who could have stopped it.

So the order matters. Build respect in the team first, and only then let them decide these things themselves. I still keep two things with me. The coverage rule, and stepping in if a deal ever breaks that rule. In all the years I have run teams this way, I have almost never had to step in.

The calendar gets sorted on its own now. Nobody has missed a festival in years, and nobody thanks me for it, which is how it should be.

You can buy attendance, You can buy hours, But the feeling that "this system is mine" is Not for Sale!

Saqib Ameen Subhan — Wed, 30 Sep 2026 09:08:29 +0000

The night an entire cloud region went down under us, I watched something that no on call policy can produce. Engineers joining the incident bridge who were not on call, had not been paged, and had every right to be asleep.

Nobody summoned them. The recovery ran for thirty hours, and the people who owned those systems treated the outage the way you treat a fire in your own house. Not because a rota said so, but because somewhere along the way, the production had become theirs.

The lazy explanation is great work ethic, which is a way of crediting the individuals so you do not have to examine the system. I have managed long enough to know better. Work ethic is real, but it does not explain why the same engineers who show up at 2 AM on one team quietly ghost their pagers on another. The variable is not the people.

The 2 AM behaviour was decided months earlier, at 2 PM, when nothing was burning.

What actually earns it

It is not perks and it is not speeches. In my experience it is four specific, boring, daily things.

Their leave is sacred. Nobody gets called on vacation. We cross train specifically so that is true, and the team settles its own leave calendar like adults. An engineer whose rest is protected does not ration their energy defensively. They give it freely, because they have learned it will not be stolen.

Credit flows down, in public. When the project lands, their names go in the email to leadership, and they present their own work upward. My name appears when something breaks. An engineer who gets the credit for a system starts to feel what is literally true. It is their system.

They are in the decisions. Tooling, process, rota design. The team decides, with full context, because they live inside the consequences. You cannot hand someone the outcomes and withhold the choices and then act surprised that they feel like a bystander.

They are never resources. Not in meetings, not in planning docs, not in my head. The word matters because the word leaks into behaviour. Resources get allocated. People get respected. Only one of those shows up at 2 AM.

None of this is charity. It is a chain with no skippable links. Respect makes the production theirs, ownership follows, and once ownership exists, accountability stops being a conversation you ever need to have. I have never once had to ask an owner whether they will handle something. The question does not survive contact with genuine ownership.

You can buy attendance. You can buy hours, and with enough money you can even buy compliance. The feeling that this system is mine is not for sale. It is reciprocated or it is absent.

The boundary that keeps this honest

Here is where I have to be careful, because this philosophy has an evil twin.

If a manager reads "my people show up at 2 AM" and hears free capacity, they have turned respect into exploitation with better branding. The willingness is a gift. The moment you treat it as a budget line, planning around it, expecting it, spending it, you are strip mining the very trust that created it, and it does not grow back at the rate you are extracting it.

So the same philosophy comes with obligations that run in my direction. I answer first at 2 AM. The leader eats the first hour before anyone else is asked to. Incident nights are followed by comp time off without anyone having to request it. And I watch for the pattern where the same engineer is the hero twice in a row, because the person who always shows up is one quarter away from being the person who used to work here.

Protect the gift and it compounds. Spend it and it is gone.

The morning after

Somewhere past hour thirty, the estate was back, the region was healing, and everyone finally got some well deserved rest.

The next morning I did not assign the postmortem. Drafts were already appearing in the channel, from the people who had been on the bridge, unprompted, because owners do not wait to be asked what happened to their own systems.

That is the whole thing, really. It was never about the night.

It was about all the days before it.

A junior engineer accidentally dropped a staging database!

Saqib Ameen Subhan — Tue, 15 Sep 2026 05:20:48 +0000

A junior engineer on my team once ran a cleanup command on staging, believing he was on dev. The data was gone before the prompt returned.

He came to me directly, shaking, honestly, and told me what happened. Which, I want to point out before anything else, is the single most important fact in this story.

On a lot of teams, that engineer spends an hour trying to quietly fix it himself first, and the incident gets worse in the dark. He came straight to me. That is not a personality trait. That is a property of the team, and you build it or you destroy it long before the incident happens.

The first hour

The first hour is triage, not analysis.

I took over communication with the affected dev team myself. Not because the engineer could not speak, but because the person with the least organizational armor should not be the face of an incident. We stabilized, restored from backup, verified, and closed the loop with the dev team the same day.

At no point did anyone ask who did it in a public channel. The dev team was told: we made a mistake, we have restored it, here is what we are changing.

Accountability flows up. A manager who forwards blame downward is just a router.

The interesting question

Once it was over, the question I sat with was not how do I make sure he is more careful.

Careful is not a system. Everyone is careful until they are tired, or it is 2 AM, or two terminal windows look identical, which is the actual root cause here. He was not careless. He was a human being using a tool that made two very different worlds look exactly the same.

So we changed the tools.

Terminal backgrounds now encode the environment. Dev is default. Staging is blue. Production is red, a red you cannot ignore, a red that makes you sit up slightly before you type. It costs nothing and it works on the tired version of you, which is the version that makes mistakes.

A pre execution checklist for destructive commands. Three questions, ten seconds. Where am I? What system is this? Am I on a read node or a write node? We wrote it into the runbook.

I told the team the analogy I actually think in. You cut vegetables outside the vessel, then put them in. You do not chop directly into the cooking pot, because there, every slip is dinner.

Neither fix is clever. That is the point. Clever fixes depend on people being sharp. Boring fixes work when they are not.

What happened to the engineer

Nothing. That is the answer, and it was deliberate and visible. No formal note, no development area in his next review, nothing.

The incident was the process's fault, the process got fixed, and the fix was named after the problem, not the person.

He stayed. He got better, genuinely better, the kind of better that comes from learning viscerally that mistakes are survivable here. Years later he was one of the people I trusted most in a production window.

If I had made that first incident expensive for him, I would have taught him, and everyone watching, because everyone is always watching, that the smart move next time is to hide it.

The general rule

When something breaks, there are always two available explanations. The person, or the system the person was inside.

Choosing the person is emotionally satisfying and fixes nothing. The next human inherits the same trap. Choosing the system is boring and permanent.

We never had that class of incident again. Not because the team became more careful, but because the trap was removed.

The terminal colors outlived everyone on that team, including me.

Large partitions don't fail loudly. They fail everywhere at once.

Saqib Ameen Subhan — Fri, 04 Sep 2026 17:29:00 +0000

Most Cassandra problems announce themselves. A timeout, an exception, a graph going vertical.

Large partitions do not. They degrade reads, compaction, repair and GC simultaneously and gradually, which is why the incident channel fills with four unrelated looking symptoms and no suspect.

If you have been following this series, you have already met large partitions twice without the name. They are the humongous allocations in the heap dump post, and they are the overstreaming amplifier in the repair post. This post is about the disease itself.

What large means and why it hurts

A partition is the unit Cassandra reads, compacts, streams and materializes. The rule of thumb that has served me well is to aim under 100MB per partition and treat anything past that as debt. Old timers remember the hard 2GB era limits. Modern Cassandra handles big partitions better, which mostly means they hurt you more gradually.

Where the pain lands:

Reads. A partition read materializes structures proportional to what is scanned. In G1 terms, a multi hundred MB partition read is a parade of humongous allocations. GC pause spikes on whichever node holds it, the cluster marks it slow, and speculative retries fan the load elsewhere.
Compaction. Partitions are compacted as units. Giant partitions make giant, slow compaction tasks that starve neighbours, and pending tasks climb.
Repair. One mismatched cell in a giant partition can stream the whole thing. The overstreaming problem, concentrated.
Hotspotting. Big partitions are usually hot partitions, the same key taking disproportionate traffic. Size and heat compound.

How partitions get big, always the same story

The partition key models an unbounded thing.

-- looks innocent in the design review
CREATE TABLE events_by_device (
  device_id uuid,
  ts timestamp,
  payload blob,
  PRIMARY KEY (device_id, ts)
);

A partition key of device_id means every event that device ever produces, forever, in one partition. A chatty device, a popular customer, a busy trading day. Growth has no ceiling because the model gave it none.

I have seen the one big customer partition at multiple companies. It is practically a genre.

Finding them

nodetool tablehistograms ks events_by_device
# Percentile      Partition Size
# 50%             61,214 bytes
# 99%             386,857,368 bytes     <- there it is
# Max             2,395,318,855 bytes

The p50 to p99 gap is the signature. A healthy median hiding a monster tail.

To name the offenders on disk:

# per SSTable, list partitions over a threshold
sstablepartitions -t 100 /var/lib/cassandra/data/ks/events_by_device-*/

And Cassandra logs them at write time. Grep system.log for Writing large partition. Those log lines are the cheapest early warning system you will ever ignore.

The fix: give the partition a ceiling

You cannot cap a device's lifetime events. You can cap a partition, by putting time into the key.

CREATE TABLE events_by_device_v2 (
  device_id uuid,
  day date,                       -- the bucket
  ts timestamp,
  payload blob,
  PRIMARY KEY ((device_id, day), ts)
);

Now a partition is one device, one day. Bounded by physics instead of hope. Two design decisions follow.

Bucket size is a calculation, not a vibe. Estimate rows per day times row size, then pick the bucket, hour, day or week, that lands typical partitions in single digit MB and your worst realistic case under about 100MB. Do the arithmetic for your loudest tenant, not your average one. Averages are how the enormous customer surprises you.

Multi bucket reads are now yours to orchestrate. Last 6 hours spanning midnight means two partitions. Drivers make querying a known list of buckets cheap. Just design the access pattern alongside the schema, not after it. This pairs well with TWCS from the compaction post, because bucketed, TTL'd time series is the workload TWCS was born for.

Migration reality. Existing giants must be rewritten into v2. Dual write new data, backfill old in throttled batches, the same discipline as the 2TB MongoDB delete earlier in this series, chunk, sleep, watch the cluster. Then cut reads over and drop v1. Weeks, calmly, not a weekend, heroically.

The review question that prevents all of it

Every Cassandra schema review I run ends with one question. What bounds this partition?

If the answer is a business quantity, customers will not have many orders, then it is unbounded, because business quantities grow. That is their job. The only acceptable answers have units of time, or a hard modulo in the key.

Large partitions are never an ops problem. They are a design review that ended one question too early.

Repair at 1,000 nodes: why we stopped running nodetool repair by hand

Saqib Ameen Subhan — Fri, 04 Sep 2026 17:28:30 +0000

At small scale, repair is a cron job. At 1,000 nodes it is a scheduling problem, and if you treat a scheduling problem like a cron job, the cluster will teach you the difference during business hours.

I spent years operating Cassandra estates past the thousand node mark. Here is what repair actually does, why it is not optional, and how the operational model has to change with scale.

Why repair is a deadline, not a chore

Cassandra replicas drift. Hinted handoff covers short outages, but hints have a window, three hours by default. A replica down longer than that has permanently missed writes, and only anti entropy repair reconciles it.

The part that turns repair from hygiene into a deadline is tombstones. Deleted data is protected from resurrection by grave markers that compaction may purge after gc_grace_seconds, ten days by default. The safety of that purge rests on one assumption. Every replica heard about the delete before the marker vanished. Repair is how they hear.

So the rule is absolute. Every node completes repair at least once per gc_grace_seconds. Miss it, and a replica that slept through a delete will happily hand the missing row back to the cluster. Zombie data, silent, and by the time anyone notices, days of writes have been made against resurrected garbage.

Repair is not cleanup. It is the second half of every delete you have ever issued.

What a repair costs

Mechanically, nodes build Merkle trees over their data ranges, exchange them, and stream any ranges whose hashes disagree. Three costs follow.

Building trees reads data. Sequential I/O competing with your workload.
Streaming mismatches saturates network and creates new SSTables, which means the third cost.
Compaction debt. A big repair is followed by a compaction hangover, and nodetool compactionstats pending tasks tell the story.

Merkle tree resolution is finite, so one tree over a huge range means one tiny mismatch streams a disproportionately large chunk. This is overstreaming, and it is why repairing a giant range in one shot is both slow and wasteful.

Full, incremental, subrange, and what breaks at scale

Full repair, whole range, everything rehashed every time. Correct, brutal, does not scale. At 1,000 nodes, naive nodetool repair across the fleet means the cluster is effectively always repairing, and repair sessions colliding on shared ranges fail in tedious ways.

Incremental repair marks SSTables repaired so they are skipped next time. On paper, the fix. In practice it drags anticompaction behind it, rewriting SSTables to segregate repaired from unrepaired data, and its operational history, particularly before Cassandra 4.0's fixes, was rocky enough that many large operators simply did not trust it. We did not.

Subrange repair breaks the token range into small segments repaired one at a time. Small Merkle trees, precise comparisons, minimal overstreaming, and each segment is a small retryable unit of work. This is the primitive that actually scales.

But subrange repair across 1,000 nodes is thousands upon thousands of segments needing ordering, throttling, retries and collision avoidance. Congratulations, it is a scheduling problem.

Reaper: repair as a system, not a command

We ran Cassandra Reaper, open source, originally Spotify's and now community maintained, as the scheduler.

Splits every table's range into segments and runs them steadily with concurrency caps
An intensity knob throttles repair pressure so p99s do not feel it. We tuned it to stay invisible.
Failed segments retry individually, so a node blip costs one segment, not the whole run
A web UI that answers the only question leadership asks, which is whether we are inside the gc_grace window, yes or no

The operating policy that came out of years of this:

- every table repaired well inside gc_grace, target a complete cycle in ~7 days against a 10 day grace
- repair pressure tuned to be invisible in p99 read latency
- alert not on "repair failed" but on "time since last completed repair per table"
- pause repairs during topology changes, resume, never skip

That last alert framing matters. Failures are noise. Staleness is the actual risk. Measure the deadline, not the activity.

The small scale corollary

Under about 50 nodes you do not need Reaper's ceremony, but you do need the same two invariants. Every node repaired inside gc_grace_seconds, and repairs that do not collide. A boring, monitored schedule beats heroics.

Repair is the tax on eventual consistency. You can pay it on a schedule you chose, or all at once with interest, with zombie data as the collection notice.

The heap dump usually exonerates Cassandra

Saqib Ameen Subhan — Fri, 04 Sep 2026 17:26:34 +0000

When a Cassandra node OOMs, or GC pauses start stretching into seconds, the reflex is to blame Cassandra, or its favourite scapegoat, the JVM.

I have read a lot of Cassandra heap dumps, and here is the uncomfortable pattern. The heap usually contains exactly what the data model asked for. The dump does not convict the database. It convicts a partition.

This post is the workflow I actually use, in order, because the order saves hours.

Step 0: GC logs before heap dumps

A heap dump is a biopsy. GC logs are the patient history, and they are already on disk. Two things I look for first:

# gc.log (G1)
Pause Young (Normal) ... 180ms
Pause Young (Normal) ... 210ms
Pause Full (Allocation Failure) ... 8.4s      <- the incident
...
Humongous Allocation ... 18MB
Humongous Allocation ... 22MB

Full GC pauses. G1 doing an emergency stop the world collection because normal cycles could not keep up. On a Cassandra node, an 8 second pause does not just slow queries. The node misses gossip, gets marked down by peers, and hints start piling up elsewhere. One node's GC problem becomes the cluster's problem.

Humongous allocations. G1 divides the heap into regions, typically 4 to 32MB. Any single allocation larger than half a region is humongous, allocated awkwardly across contiguous regions and collected inefficiently.

And what does Cassandra allocate in one giant contiguous chunk? A large partition being materialized for a read. Humongous allocation warnings in a Cassandra gc.log are large partitions announcing themselves. You can often skip the heap dump entirely at this point and go straight to nodetool tablehistograms to find the table with a monster p99 partition size.

Step 1: take the dump without causing an incident

jmap on a live heap stops the world for the duration of the dump. On a 20GB heap, that is long enough for the cluster to declare the node dead. Never dump a node that is still serving.

# isolate first, node stays alive but stops serving
nodetool disablebinary && nodetool disablegossip

jmap -dump:live,format=b,file=/data/dumps/cass_$(hostname)_$(date +%s).hprof <pid>

# afterwards: rejoin or just restart the node

Better, have the JVM do it for you at the moment of truth, before any human is awake:

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/data/dumps/

Those two flags belong in every production Cassandra JVM config. An OOM without a dump is an incident you get to have twice.

Related discipline, keep the heap at or under 8GB and let G1 target around 200ms pauses. Bigger heaps mostly buy you longer pauses and bigger dumps, and Cassandra's real caching happens off heap and in the page cache anyway.

Step 2: Eclipse MAT, dominator tree, ten minutes

Open the .hprof in Eclipse MAT and go straight to the dominator tree, objects ranked by retained memory, which answers what is actually holding the heap hostage. Ignore the histogram of a billion byte[]. The dominator tree tells you whose bytes.

What the tree shows, mapped to what it means, every one of these from real incidents:

Dominating the heap	Actual problem
A few enormous `byte[]` or cell containers under a read path	Large partition being materialized. Get the key from the referencing objects, confirm with `nodetool tablehistograms` or `sstablepartitions`. Fix is to bucket the partition, which is a later post in this series.
Memtable objects, many tables	Over wide schema, or memtable flush thresholds set too generous for the heap
Tombstone or range tombstone structures under a query	A tombstone farm read that survived just long enough to dump
Netty buffers or inflight requests	Clients hammering with no paging. Check fetch size and unpaged full partition reads

The punchline repeats across years and companies. The heap contains the workload. I have almost never opened a dump and found a Cassandra bug. I have regularly opened one and found a 4GB partition someone swore was just a busy customer.

Step 3: close the loop in the model

The dump names the object. The fix lives in the schema.

Large partition, time bucket the partition key. Unpaged reads, driver fetch size. Tombstone reads, the modeling fixes from two posts ago. The JVM flags and MAT are diagnosis. Cassandra data modeling is treatment.

Blaming the JVM is comfortable because nobody owns the JVM. The heap dump takes that comfort away. It shows you, in retained bytes, precisely which design decision you are looking at.

A compaction strategy is a bet about your workload

Saqib Ameen Subhan — Fri, 04 Sep 2026 17:25:24 +0000

Choosing a Cassandra compaction strategy is placing a bet. Each strategy optimizes one thing by deliberately sacrificing another, and the honest way to choose is to decide which failure mode you would rather operate.

I have run all three majors across 1,000+ node estates. Here is the bet each one makes.

Why compaction exists at all

Cassandra never updates in place. Writes land in a memtable, flush to immutable SSTables, and a row's fragments accumulate across many files as it is updated over time. Reads must merge every relevant fragment.

Compaction is the background process that merges SSTables, consolidating fragments and purging expired tombstones after gc_grace_seconds, so reads touch fewer files.

The metric that keeps compaction honest:

nodetool tablehistograms ks table
# Percentile  SSTables     ...
# 50%             1.00
# 99%             4.00

SSTables per read at p99 is the number compaction exists to keep small.

STCS, SizeTiered, the write optimized default

Merges SSTables of similar size into bigger ones, repeatedly. Minimal write amplification, maximal ingest throughput.

The bet you are making: reads and disk headroom are negotiable.

A hot row's fragments can sit in many SSTables of different generations, so SSTables per read climbs.
Space amplification is the operational trap. Compacting large tiers requires holding input and output simultaneously. Plan for 50% free disk. I have watched teams treat 70% disk usage as plenty left, then be unable to run the very compaction that would reclaim space. That is a corner with no good exits.
Tombstones buried in giant, rarely recompacted SSTables can linger far past gc_grace_seconds.

Right when: write heavy, append mostly, reads tolerate variance. It is the default because it is the least dangerous average bet, not because it is right for you.

LCS, Leveled, paying writes to buy reads

Organizes SSTables into levels, L1, L2 and so on, each 10 times larger, guaranteeing non overlapping key ranges within a level. A read touches at most one SSTable per level, and in practice the vast majority of reads are satisfied by a single SSTable.

The bet: you will pay roughly 10x write amplification, because every row is rewritten as it migrates down levels, to make read latency tight and predictable.

On read heavy tables with strict p99s, it is the correct bet.
On write heavy tables, compaction falls behind, pending tasks pile up in nodetool compactionstats, and ironically reads degrade anyway because L0 accumulates overlapping files.
Easier on disk headroom than STCS, works in around 10% free, but harder on I/O and CPU, continuously.

Right when: read dominated, update in place workloads. Wrong when your disks are already busy keeping up with ingest.

TWCS, TimeWindow, the time series specialist

Groups SSTables by time window, say one per day. Within the current window, STCS as usual. Once a window closes, its SSTables compact once into one file and are never touched again.

When TTL expires a whole window, Cassandra drops the entire file, so tombstone processing effectively vanishes for aged out data. This is the single biggest tombstone fix available for time series, and the punchline of the previous post.

The bet: your data is truly time ordered and immutable once written.

Write out of order data or update old windows and you create cross window overlaps that can never compact away. The strategy's guarantees quietly die while everything still appears to work.
Rule of thumb, aim for a window size that yields 20 to 50 windows over the table's TTL.

Right when: metrics, events, telemetry, anything with TTL and append only semantics. IoT and monitoring tables at scale should almost always be TWCS.

The bet, stated plainly

Strategy	You optimize	You pay with	Operational trap
STCS	write throughput	read variance	needs around 50% free disk to compact
LCS	read p99	10x write amplification	falls behind under heavy ingest
TWCS	TTL'd time series	inflexibility	out of order writes break it silently

Per table, not per cluster. A keyspace legitimately mixes all three. And changing the bet is an online operation:

ALTER TABLE ks.events WITH compaction =
  {'class': 'TimeWindowCompactionStrategy',
   'compaction_window_unit': 'DAYS', 'compaction_window_size': 1};

Cassandra will re sort existing SSTables over time. On a big table, schedule the transition like the migration it is.

There is no best compaction strategy. There is only the failure mode you have chosen on purpose, versus the one that chose you.

mtools does not parse MongoDB 4.4+ JSON logs. So I built mdbkit.

Saqib Ameen Subhan — Mon, 24 Aug 2026 19:28:11 +0000

If you have operated MongoDB for any length of time, you know mtools. For years it was how we all read production logs. mloginfo, mplotqueries, and a slow query was suddenly visible.

Then MongoDB 4.4 changed the log format to structured JSON, and the log tools stopped understanding what they were reading.

That was six years ago. The support issue was opened in June 2020 and it is still open. We are on 8.0 now, and a lot of us have been getting by with grep, jq pipelines held together by hope, and a certain amount of squinting.

I got tired of it and built the replacement. It is called mdbkit, it is MIT licensed, and it is public as of this week.

Try it in sixty seconds without touching a cluster

This is the part I would want first if this were someone else's tool.

pip install --upgrade mdbkit
# macOS ships no pip: brew install pipx && pipx install mdbkit

mdbkit demo -o demo.log
mdbkit queries demo.log

mdbkit demo generates a realistic MongoDB log containing an actual incident. No cluster, no connection string, nothing of yours involved. You get to judge the tool on a problem you can inspect before you point it at anything that matters.

Here is what queries gives you on that demo log:

saqib@MacbookPro mongodb_logs % mdbkit queries demo.log
== mdbkit queries (slow query shapes) ==
lines: 1,004  parsed: 1,004  unparsed: 0  span: 2026-07-01T08:00:00+04:00 -> 2026-07-01T09:29:44.615000+04:00

namespace      op         count  cumMs  mean   max    docsEx      scan     plan            shape
-------------  ---------  -----  -----  -----  -----  ----------  -------  --------------  ---------------------------------------------
shop.events    aggregate  53     5.6m   6.3s   8.9s   47,170,000  98889:1  COLLSCAN+SORT   {tenantId:eq, ts:gte} sort:{ts:-1}
shop.orders    find       89     2.4m   1.6s   2.4s   11,125,000  2976:1   COLLSCAN+SORT   {createdAt:gt, status:eq} sort:{createdAt:-1}
shop.sessions  find       58     34.7s  597ms  888ms  2,784,000   155:1    IXSCAN{active}  {active:eq, lastSeen:gt} sort:{lastSeen:-1}
shop.products  update     42     25.7s  612ms  799ms  2,268,000   54000:1  COLLSCAN        {sku:eq}
shop.users     find       56     6.7s   119ms  139ms  56          1:1      IXSCAN{email}   {email:eq}

cumMs  = total wall time accumulated across ALL occurrences (not one query)
docsEx = total documents examined across all occurrences
scan   = docsExamined per returned doc (high = index missing or weak)
plan   = most common query plan; COLLSCAN/+SORT = index needed
         empty plan (?) = plan not present in log (below slowms threshold)
next   : mdbkit advise <log> [--ns <namespace>] for index candidates

That is not a screenshot I curated. demo is seeded, so those two commands produce that exact table on your machine too, down to the last digit. It behaves the same on my laptop, on yours, and on a conference projector.

The column that matters is the scan ratio. It is documents examined against documents returned, per query shape. A shape sitting at 98889:1 is reading roughly ninety nine thousand documents to hand back one. A shape at 1:1 is doing exactly the work it needs to.

That single ratio is most of slow query triage. Everything else is detail.

The workflow I actually use

Three commands, in order.

First, find the shape that is hurting.

mdbkit queries /var/log/mongodb/mongod.log

Slow operations grouped by query shape rather than listed individually. This matters more than it sounds. A log with 40,000 slow query lines is usually eight or nine distinct shapes repeating, and the one costing you the most is rarely the one that appears most often.

Then, ask what would fix it.

mdbkit advise /var/log/mongodb/mongod.log

This proposes candidate indexes based on the shapes it observed. More on how that works below, because it is the part people should be suspicious of.

And when something has already gone wrong, build the timeline.

mdbkit triage /var/log/mongodb/mongod.log

Elections, rollbacks, stalls, connection storms, in order, with timestamps. When you are twenty minutes into an incident and someone senior is asking what happened, a timeline you did not have to assemble by hand is worth a great deal.

The log cannot tell you everything

There is one failure mode a mongod log structurally cannot explain: the log simply stops, then starts again a minute later with no error in between.

The process was killed before it could write anything. That answer lives in the system log, one line up from where you were looking.

mdbkit triage mongod.log --oslog /var/log/syslog

[CRIT] Process start(s) in window: mongod startup marker at 09:14:35
[CRIT] System: oom-kill: The kernel OOM killer terminated a task.
         1 occurrence (last at 09:14:03). Processes: mongod.
         next: This explains an unexplained restart in the log above.

oslog reads /var/log/syslog or /var/log/messages and looks for the things that end database processes: OOM kills, file descriptor limits, segmentation faults, I/O errors, filesystems remounted read only, systemd service exits. Used with triage, the unexplained restart gets matched to the kernel line that caused it.

On journald systems there is no text log to read, and mdbkit will not run commands on your behalf, so it prints the journalctl invocation for you to capture and feed back in.

There is also ftdc, which decodes diagnostic.data offline. Every one of your nodes has been recording CPU, cache, queue depth and disk every second since the day you installed it, whether or not you run any monitoring. Plus serverstatus for making sense of a serverStatus dump, and compare for answering whether the index you added last week actually helped.

It never connects to your database

No driver. No URI. No network code anywhere in it.

mdbkit reads log files you hand it. It is strictly read only. Where an action would help, it prints the command for you to review and run yourself, rather than running anything on your behalf.

I built it this way because I would not run a stranger's tool on a production host, and I do not expect you to either. So rather than asking you to trust me, the README shows you how to verify it independently. grep the source for socket and driver imports. Run it under strace and watch it make no network calls. It takes about a minute and you should do it.

Zero runtime dependencies too, nothing in the supply chain beyond the Python standard library. That was a deliberate constraint and it cost me some convenience, but a tool that ships onto database servers should not drag a dependency tree behind it.

The index advice is rules, not AI

I gave a talk at the Dubai MongoDB User Group recently about how confidently language models describe a MongoDB that no longer exists. They will happily reference plannerVersion, a field removed in 5.0. They do not know about EXPRESS stages in 8.0. And more importantly, they cannot know your cardinality, your write mix, or what is currently sitting in your plan cache, because none of that is in your query text.

So mdbkit's index advice is deterministic. Rules over observed query shapes. The same log always produces the same recommendation.

Every suggestion comes with the evidence it reasoned from, a confidence level, and how to validate it before you act. It says candidate, never command. And it will never tell you to drop an index, because deciding an index is unused requires knowing about traffic that a log file cannot show you.

That is a deliberately narrow promise. I would rather it be right about a small thing than confident about a large one.

What would help most

Two things, and they are both easy for you and hard for me.

Real world log lines that parse incorrectly. Every MongoDB deployment logs slightly differently, and I have only seen the estates I have worked on. If mdbkit chokes on a line, that line is the most useful thing you can send me.

Index advice that is wrong or unhelpful. If it suggests something you know is a bad idea for your workload, I want to know why, because that is a rule that needs fixing.

Every release so far has come from someone doing exactly that. An FTDC decode that took twenty five minutes and pinned a CPU. A crash on a batched write. A restart with no explanation, which is why oslog exists at all.

Supports MongoDB 4.4 through 8.0. Python 3.8+.

Source: https://github.com/saqibameen86/mdbkit
Package: https://pypi.org/project/mdbkit

Free, MIT, and built because the gap annoyed me for long enough.

If you liked this, I also wrote about deleting 2TB from a live MongoDB cluster without anyone noticing and why I take down a healthy primary on purpose.

In Cassandra, a delete is a write (and reads pay for it)

Saqib Ameen Subhan — Sun, 23 Aug 2026 16:36:09 +0000

In Cassandra, a delete is a write.

That sentence sounds like trivia until it takes down a read path. DELETE doesn't remove anything. It writes a tombstone, a marker saying "this data is dead as of timestamp T." The actual removal happens later, during compaction, and only after a grace period. Between those two moments, your deleted data has negative value. It's gone from your application's point of view but still physically present, and every read that touches its range must process it.

Why the marker has to exist

Cassandra is a distributed, eventually consistent system. Suppose a delete simply removed data, and one replica was down when the delete happened. When it comes back, it still has the row, and to the rest of the cluster, that looks like data the others are missing. Repair would helpfully copy the deleted row back to everyone.

Deleted data returning from the dead is called zombie data, and tombstones exist to prevent it. The tombstone outranks the older value everywhere it's seen.

Which is why tombstones must survive for gc_grace_seconds, default 864000, ten days, before compaction may purge them. Ten days is the window you have to repair a down replica. Shrink gc_grace_seconds without shrinking your repair interval and you've quietly signed up for zombies. Repair at scale is its own post, and it's coming.

How reads pay

A read merges data across memtable and SSTables. Every tombstone in the requested range must be read, held, and reconciled against live data before Cassandra can answer. A partition that has accumulated a million tombstones makes you process a million dead cells to return whatever's alive.

Cassandra tells you when this is happening, in two escalating tones:

WARN  ReadCommand - Read 812 live rows and 104,832 tombstone cells for query ...
ERROR ... Scanned over 100001 tombstones ... query aborted

The warning fires past tombstone_warn_threshold, 1,000 by default. Past tombstone_failure_threshold, 100,000, the read is killed mid flight with TombstoneOverwhelmingException. That's the database choosing to fail your query rather than let it OOM the node.

The thresholds are guardrails, not tuning knobs. Raising them treats the symptom and keeps the disease.

Where tombstone farms come from

Every one of these I've met in production.

Queue like workloads. Insert, process, delete, repeat, in the same partition. The partition becomes a graveyard the consumer must scan past on every poll. Cassandra as a queue is the canonical anti pattern for exactly this reason.
Collection overwrites. UPDATE t SET mymap = {...} replaces a whole collection, which writes a range tombstone over the old one first. Prefer additive updates, mymap = mymap + {...}.
Inserting NULLs. Binding null in a prepared statement writes a tombstone for that cell. ORMs and "just bind every column" code generate these invisibly. Use unset values, not nulls.
TTL everywhere. Expired TTL cells become tombstones too. Fine if your compaction strategy is built for it.

Finding them, then fixing them

# per-SSTable estimate
sstablemetadata /var/lib/cassandra/data/ks/table-*/ *-Data.db | grep -i droppable
# Estimated droppable tombstones: 0.83

0.83 means an estimated 83% of that SSTable is purgeable tombstones.

For query level visibility, TRACING ON in cqlsh shows tombstone cells scanned per query. For a live table, nodetool tablestats and watch average tombstones per slice.

The durable fixes are modeling fixes.

Time series with TTL, use TimeWindowCompactionStrategy. Whole SSTables age out and get dropped as files, so tombstone processing largely disappears. Compaction strategy trade offs are the next post.
Stop deleting in place in hot partitions. Partition by time bucket and let whole partitions expire instead.
Audit for null binding and collection overwrites. These are the tombstones nobody meant to write.

Deletes in Cassandra are cheap to issue and expensive to have issued. Model as if every delete is a small loan against your read latency, because it is, and gc_grace_seconds is the repayment schedule.

How to delete 2TB from a live MongoDB cluster without anyone noticing

Saqib Ameen Subhan — Sun, 23 Aug 2026 16:04:47 +0000

The request was simple. "There is about 2TB of expired data in this collection. Can you delete it tonight?"

My answer was no, and being able to explain why is most of the job. A missing TTL index had let short lived data pile up for months in one of our busiest collections, and the team wanted to run one big deleteMany overnight. Here is what that would have actually done.

Why a big delete hurts, and it is not locking

Most people think a big delete is risky because it locks the collection. In modern MongoDB it does not, because WiredTiger works at the document level, so a huge delete does not freeze the collection. The damage comes from three other places.

1. The oplog fills up. Every deleted document becomes its own entry in the oplog (the log that secondaries copy from). Delete 500 million documents and you have written 500 million oplog entries, and every secondary has to copy and apply each one of them. Replication lag starts climbing. If a secondary falls further behind than the oplog can hold, it can no longer catch up and needs a full resync. At that point your delete has cost you a copy of your data.

2. The cache gets flushed out. To delete a document, WiredTiger first has to load it into memory. A 2TB delete pulls a lot of old data into the cache and pushes out the data your application actually uses. Queries that have nothing to do with the delete start getting slow, and whoever looks into it will probably never connect it back to the delete.

3. Majority writes slow down. If your writes use w:"majority" (and they should, I covered why in an earlier post in this series), every write now waits on the same secondaries that are already struggling to keep up with the delete. The problems add up.

I showed this to the team in a lower environment. I started the big delete, ran rs.printSecondaryReplicationInfo(), and let them watch the lag grow. After that, "tonight" turned into a plan.

The approach: small chunks, a pause, and watching the lag

const CHUNK = 10000;
const SLEEP_MS = 500;
const cutoff = ISODate("2026-01-01");

let total = 0;
while (true) {
  const ids = db.events.find({ created: { $lt: cutoff } },
                              { _id: 1 }).limit(CHUNK).toArray().map(d => d._id);
  if (ids.length === 0) break;

  db.events.deleteMany({ _id: { $in: ids } });
  total += ids.length;

  sleep(SLEEP_MS);   // give the secondaries time to catch up

  if (total % 1000000 === 0) print(`${total} deleted, check the lag`);
}

Two rules we followed while running it.

Only run it in off peak hours. We ran it every night and stopped it before business hours. The full 2TB took around a week and a half, and nobody noticed anything, which was exactly the goal.

Let replication lag decide the speed. If the lag went above the level we were comfortable with, we increased SLEEP_MS. The speed of the loop is decided by how healthy the cluster is, not by how quickly we want it done.

The pause between chunks can look far too slow to developers. That is fine. Once the data has stopped growing, the delete has no deadline, and what stops the data from growing is the real fix below.

The real fix: a TTL index, so this never happens again

db.events.createIndex({ created: 1 }, { expireAfterSeconds: 2592000 })  // 30 days

A few things about TTL indexes that surprise people.

The TTL process runs every 60 seconds, so documents are not deleted at the exact second they expire. That is fine for cleaning up old data, but do not rely on it for business logic that needs exact timing.
It deletes in batches in the background, which is basically the same chunked loop as above, running forever. On a very large backlog it can fall behind, which is why we cleared the 2TB by hand first and then let TTL handle it from there.
Put the TTL index on the field that actually decides when the data is old. A TTL index on the wrong date field will quietly delete data you wanted to keep.

We finished with a runbook covering the script, the lag limits, and the schedule, so the next time someone asks to delete 2TB, the answer is already written down.

Try it yourself

If you want to see this on your laptop, mdbkit lab gives you a 3 node replica set.

pip install mdbkit
mdbkit lab start          # 3 node replica set on 127.0.0.1:28110-28112

Connect to the primary and load a million documents, about half of them older than 60 days.

// mongosh --port 28110
use shop
for (let b = 0; b < 100; b++) {
  const docs = []
  for (let i = 0; i < 10000; i++) {
    docs.push({ created: new Date(Date.now() - Math.random() * 120 * 86400000), payload: "x".repeat(200) })
  }
  db.events.insertMany(docs)
}

Then run the chunked delete. This version prints the time, how many documents have been deleted so far, and how far behind the secondaries are.

function secondaryLag() {
  const s = rs.status()
  const p = s.members.find(m => m.stateStr === "PRIMARY")
  const lags = s.members.filter(m => m.stateStr === "SECONDARY")
                        .map(m => (p.optimeDate - m.optimeDate) / 1000)
  return Math.max(...lags)
}

const cutoff = new Date(Date.now() - 60 * 86400000)
let total = 0
while (true) {
  const ids = db.events.find({ created: { $lt: cutoff } }, { _id: 1 })
                       .limit(10000).toArray().map(d => d._id)
  if (ids.length === 0) break
  db.events.deleteMany({ _id: { $in: ids } })
  total += ids.length
  const t = new Date().toISOString().substr(11, 8)
  print(t + "  deleted " + total + " old documents   secondary lag " + secondaryLag() + "s")
  sleep(500)
}
print("done, deleted " + total + " old documents")

You will see the lag stay close to zero the whole way through. If you want to compare, reload the data and run the same delete as one single deleteMany, then check rs.printSecondaryReplicationInfo() while it runs.

When you are done:

mdbkit lab destroy --yes

When you have to delete a lot of data from a live system, going slowly is the safe way to do it.

I take down a healthy primary on purpose. You should too.

Saqib Ameen Subhan — Sun, 23 Aug 2026 15:58:41 +0000

In one of my previous companies, I built an automation that used to scale our MongoDB clusters automatically. When CPU crossed a threshold in CloudWatch, the automation upgraded the nodes one by one. First a secondary, then the second secondary, and the primary last. It took around 12 minutes, nobody had to log in, and the application was not impacted.

The order matters here. Restarting the primary last means the cluster goes through only one election for every upscale, and the automation takes care of it. But this only works if you trust elections, and you will trust elections once you have triggered them yourself and seen how your application behaves when it happens.

What happens during an election?

In simple terms, this is how it works.

Every node in a replica set checks on the other nodes every 2 seconds (heartbeats).
If the primary does not respond for 10 seconds (electionTimeoutMillis), the other nodes assume it is gone.
A secondary asks the other nodes to vote for it. It can only win if its data is at least as up to date as the nodes voting for it (i.e its oplog is as recent as theirs).
The node that gets the majority of votes becomes the new primary, and the other secondaries start copying data from it.

One thing that surprises a lot of people is the priority setting. Priority decides which nodes get preference to become primary, but it cannot make a node with old data win. A node with priority 10 that is behind will lose to a node with priority 1 that is fully up to date. Up to date data matters more than priority. Once the higher priority node catches up with the others, it calls another election and takes over as primary, as the priority dictates.

In total, there can be up to 15 seconds with no primary, and during that time MongoDB cannot accept any writes.

The real problem is your application

The election itself only takes a few seconds. What really matters is what your application does during those seconds. I have seen three kinds of behaviour.

Users see errors. If the driver throws a NotWritablePrimary error and nobody has handled it, users start getting errors, and things like checkout start failing, which directly impacts the business.
The application retries blindly. Someone wrote their own retry loop, and sometimes the same write gets saved twice. This is actually worse than failing.
The driver handles it. With retryWrites=true and a sensible serverSelectionTimeoutMS, the driver holds the write, waits for the new primary, and retries it exactly once. The user just sees one slow request.

You don't want to find out which one you have during an incident.

The drill

I run this every quarter. First in staging, then in production during a low traffic window. Yes, production too. Staging tells you that the drill works. Production tells you how your real application behaves.

// 1. Check which node is primary right now
rs.status().members.map(m => ({ name: m.name, state: m.stateStr }))

// 2. Ask the primary to step down
rs.stepDown(60)   // it will not try to become primary again for 60 seconds

// 3. Check from a secondary which node became the new primary
rs.status()

While the drill runs, I note down these numbers.

time taken to elect a new primary : ___ seconds
errors seen by the application    : ___
p99 latency during the election   : ___ ms
any write saved twice?            : ___ (check using a unique key)

These numbers are the real output of the drill, so fill them in from your own run.

rs.stepDown() is the gentle way to do this. The primary finishes what it is doing and hands over cleanly. Once you are comfortable with that, try the harder version. Kill the primary's mongod process with kill -9, or cut its network. This is much closer to what happens in a real cloud outage, because the other nodes have to wait the full 10 seconds before they start an election.

Two things I learned doing this at scale

Always keep an odd number of voting members. With four voting members, the votes can split 2 and 2, and then nobody becomes primary. If you really cannot afford a third data node, you can use an arbiter, but remember that an arbiter does not help w:"majority" writes get acknowledged, because it does not hold any data.

Use priority to decide which region should be primary. In our multi region clusters, we gave higher priority to the nodes in the main region, so that once those nodes recovered, the primary moved back there automatically. But as I said above, priority only gives preference. The node with the most up to date data still wins.

Try it yourself

If you want to try this without touching a real cluster, mdbkit lab gives you a 3 node replica set on your laptop.

pip install mdbkit
mdbkit lab start          # 3 node replica set on 127.0.0.1:28110-28112

Connect to the replica set and run a loop that saves an order every second, with the time printed next to it.

// mongosh "mongodb://127.0.0.1:28110,127.0.0.1:28111,127.0.0.1:28112/"
let i = 0
while (true) {
  const t = new Date().toISOString().substr(11, 8)
  try {
    db.drill.insertOne({ order: ++i })
    print(t + "  order " + i + " saved")
  } catch (e) {
    print(t + "  order failed, no primary available")
  }
  sleep(1000)
}

In a second terminal, connect to the primary and run rs.stepDown(30). Then watch the first terminal and look at the timestamps. You will see exactly how long your orders were affected, and whether they failed or just waited.

When you are done:

mdbkit lab destroy --yes

An election you have practised is just a normal operational event. An election you have never practised is an outage.