DEV Community: Saqib Ameen Subhan

Nobody should call you on leave

Saqib Ameen Subhan — Thu, 15 Oct 2026 15:56:21 +0000

We had just started supporting a new database technology, and we hired an engineer who already had experience with it. He became the person managing it. The product using it was not in production yet. It was still in pre-production, where testing was going on, and it was running into some issues.

Then he went on personal leave for a wedding.

The other engineers on the team had a basic idea of the technology, and they tried to answer the questions that came up during testing. They could not solve them, so they had to call him. He stepped out of the wedding and helped them over the phone for a while.

Nobody did anything wrong here. The team tried their best, and he helped because he cared. But it showed me a problem we had built ourselves. Only one person on the team knew this system well enough, so being the expert meant he could never really be on leave. It was not even production yet, and we still needed him.

Managers sometimes talk about people like him as "critical resources," and they say it with pride. I do not think it is something to be proud of. A person who cannot take leave is not a strength for the team. He is a single point of failure, and he usually knows it, which makes it worse.

The rule

Every system gets a primary owner and a backup owner. That is not a new idea, most teams already have a "backup owner" column somewhere in a wiki. The difference is that the backup actually does the work.

Having your name in a wiki does not make you able to run a system. So in my teams, the backup does the real work while the primary watches. The backup runs the version upgrade, does the failover drill and runs the maintenance window. The primary sits with them and only steps in if something is about to go wrong. Next time, they swap.

We still write documentation, and we keep it up to date. But a runbook only describes the system the way someone once wrote it down. Working on it with the primary watching teaches you how the system actually behaves, like the node that always lags after a restart, the alert that is not reliable, or the step everyone skips because they "just know." That gap between what is written and how the system really behaves is exactly what leads to a call during someone's leave.

What you actually get from it

It looks like a training plan, but it does three other things at the same time.

It protects leave. If someone can be called at any time, they are not really on leave, they are just working from somewhere else. Once the backup can genuinely run the system, the primary's phone stays quiet and the leave becomes real rest.

It keeps people in the team. People rarely leave because of one bad quarter. They leave after years of never being fully off. Protecting leave is one of the cheapest ways I have found to keep good people.

It helps during incidents. On the night something big goes down, "only one person knows this system" is a real risk to the business. A team where everyone has worked on everything can handle it even when someone is away.

There is one more benefit that is easy to miss. Everyone depends on everyone, and they can see it. A senior engineer watches a junior run the upgrade on their system, and the junior gets to work on systems that would normally take years to get access to. In one of the teams I ran a few years ago, this is how two engineers I had hired as interns went on to lead teams of their own.

The one number I watch for all of this is how many times someone on leave had to take a work call. The target is zero.

What it costs

Cross training slows the team down in the short term. The backup takes twice as long to run a change as the expert would, and the expert is sitting right next to them, wanting to just do it. If your team measures everything by how fast work gets done, this looks like waste, and you will have to defend it more than once.

The harder cost is less visible. Your most experienced engineer might see this as taking away what makes them special. Being the only person who knows a system can feel like job security, and you are deliberately removing that. The real management work is the conversation where you help them see that nobody gets promoted while they are the only one who can run a system. Setting up the rota is easy compared to that conversation.

When it works, nobody notices

When this works, nobody talks about it. People plan weddings and festivals and just go, without checking their phone every hour, the way people in normal jobs do.

They did it, and I meant it

Saqib Ameen Subhan — Wed, 30 Sep 2026 10:24:14 +0000

In an interview, I was asked about the achievement I was most proud of. I talked about a migration my team did, moving a high traffic product from RDS MySQL to Aurora Serverless with zero downtime.

Then the interviewer asked what my role in it was. I gave an answer that can feel risky in an interview. I said I did not build it, my engineers did. My job was managing the stakeholders, getting the approvals, and making sure nothing got in their way.

I know the usual interview advice is to take more of the credit, to say things like "I designed the approach" or "I made the key decisions." Some of that would even be partly true. But the answer I gave is how I actually think about management, and it would not mean much if I only said it when it was easy.

Why the credit belongs to the team

The way I see it, a manager's work is the team's results. That is the job. If a manager takes personal credit for something the team delivered, they are counting the same work twice. Once as "my team delivered this" and again as "I delivered this." Only one of those is true.

The engineer who wrote the cutover script built the cutover. What I did was create the conditions for it. I got the support to do it properly instead of quickly, I made sure more than one person could run it, and I kept the stakeholders calm while the work was happening. That is real work and I am happy to take credit for exactly that. But the migration belongs to them.

What giving credit actually looks like

Saying "great job, team" in a meeting is nice, but it does not change anything for anyone. The credit that matters is the credit that reaches the people who decide promotions and careers.

Their names go in the email to leadership. Not "the team delivered this," but the actual names of the people who did it. Leadership should know who the strong engineers are without having to ask me.
They present their own work. When leadership wants a walkthrough of the migration, the engineer who built it presents it. I am in the room only to handle any difficult questions, not to explain their work for them.
My name comes up when something breaks. This is the other half of the same rule. Credit goes down to the team, and responsibility for problems comes up to me. You cannot do one without the other.

You can only give credit for work you let them own

There is one condition that people usually skip. You cannot give someone credit for work you did not let them own.

If the manager makes every important decision, reviews every line and solves every hard problem before the team gets to it, then "the credit goes to my team" is just something nice to say. Everyone in the room knows who really did it. To give real credit, you first have to hand over real ownership, with real responsibility and real production risk, before you know how it will turn out. That migration was my engineers' achievement because it was genuinely theirs to get wrong.

This is also why it builds over time. People who get real credit for real ownership take on more ownership next time. That is how two engineers I hired as interns went on to lead teams of their own.

The cost of doing it this way

In the short term, this makes you less visible to leadership as an individual. While you are putting your engineers' names in the email, another manager might be presenting their team's work as their own, and for a while it may work for them. That can be frustrating to watch.

I will not pretend it always balances out quickly. But when a manager's team keeps doing well, quarter after quarter, people start to notice who built that team. It takes longer, but it is the honest way to get there.

The migration is still one of my proudest achievements, and it still belongs to them.

I stopped being the leave referee

Saqib Ameen Subhan — Wed, 30 Sep 2026 10:20:07 +0000

Three engineers, three festivals, and one on call rota that has to be covered every single day.

Every year, Diwali, Christmas and New Year fall within a few weeks of each other. Every database team I have run has been small enough that not everyone can be on leave at the same time. For a long time I handled this the way most managers are taught to. The leave requests came to me, I looked at them, and I decided.

I tried hard to be fair. I kept notes on who got which holidays last year, and I balanced it as carefully as I could. But every time, my fair and well documented decision ended the same way. Someone did not get the days they wanted, and they were unhappy with me for it.

The problem was not how fair I was. The problem was that I was the one deciding. When a manager decides, someone always loses, and that person does not blame the situation. They blame the decision, because the decision has the manager's name on it. You can be as fair as you like and still end the quarter with someone on the team quietly unhappy with you.

What I do now

I set the rule, and I let the team decide the calendar.

The rule is about coverage, not about people, and it is the only part I keep. There must always be enough people available to cover production. Production does not stop for festivals. Who takes which days, as long as that rule holds, is not my decision anymore.

The first time I did this, I expected to be pulled back in within a day. Instead, the team just talked it out between themselves. One of them said, "You take Diwali, it matters more to you than Christmas does to me. I will take Christmas. I will cover New Year this time if I can have it next year." It took about ten minutes, and the calendar was done.

What is different here is that nobody lost. When I decided, one person lost. When they agreed on it themselves, both of them owned the decision. The engineer covering New Year is not unhappy with me, because he made a trade he thinks is fair, with a colleague who now owes him one. They both remember the deal better than any spreadsheet I could have kept.

Why it keeps working

When I was the one making sure things were fair, it felt like I was keeping score on everyone. When the team does it themselves, it becomes a habit they follow on their own. They remember who covered last winter without anyone telling them. They sort it out before I even see the calendar. They even protect each other's leave when other teams ask for help during holidays, which is something no rule of mine ever managed.

There was one more thing I did not expect. Working it out together brought the team closer. When someone gives up their days so that a colleague can be with family for a festival, it changes something between those two people. They have shown each other, not just said it, that they will cover for each other. That is exactly what you need later during an incident, when covering for each other is the whole job.

The real message underneath all this is simple. I trust you to sort this out like the professionals you are. Letting them decide is itself a way of showing respect, and people notice when they are treated like adults. They give that respect back.

Where this does not work

This does not work in every team, and I would not be honest if I said it did.

It only works if the team already feels safe speaking up. If you hand the calendar to a team where one loud person dominates, that person will take December every year, and the quieter people will tell themselves they did not really mind. Letting the team decide does not create fairness. It makes whatever culture you already have stronger. If the team respects each other, you get the kind of conversation I described. If they do not, you have just made the unfairness automatic and removed the one person who could have stopped it.

So the order matters. Build respect in the team first, and only then let them decide these things themselves. I still keep two things with me. The coverage rule, and stepping in if a deal ever breaks that rule. In all the years I have run teams this way, I have almost never had to step in.

The calendar gets sorted on its own now. Nobody has missed a festival in years, and nobody thanks me for it, which is how it should be.

You can buy attendance, You can buy hours, But the feeling that "this system is mine" is Not for Sale!

Saqib Ameen Subhan — Wed, 30 Sep 2026 09:08:29 +0000

The night an entire cloud region went down under us, I watched something that no on call policy can produce. Engineers joining the incident bridge who were not on call, had not been paged, and had every right to be asleep.

Nobody summoned them. The recovery ran for thirty hours, and the people who owned those systems treated the outage the way you treat a fire in your own house. Not because a rota said so, but because somewhere along the way, the production had become theirs.

The lazy explanation is great work ethic, which is a way of crediting the individuals so you do not have to examine the system. I have managed long enough to know better. Work ethic is real, but it does not explain why the same engineers who show up at 2 AM on one team quietly ghost their pagers on another. The variable is not the people.

The 2 AM behaviour was decided months earlier, at 2 PM, when nothing was burning.

What actually earns it

It is not perks and it is not speeches. In my experience it is four specific, boring, daily things.

Their leave is sacred. Nobody gets called on vacation. We cross train specifically so that is true, and the team settles its own leave calendar like adults. An engineer whose rest is protected does not ration their energy defensively. They give it freely, because they have learned it will not be stolen.

Credit flows down, in public. When the project lands, their names go in the email to leadership, and they present their own work upward. My name appears when something breaks. An engineer who gets the credit for a system starts to feel what is literally true. It is their system.

They are in the decisions. Tooling, process, rota design. The team decides, with full context, because they live inside the consequences. You cannot hand someone the outcomes and withhold the choices and then act surprised that they feel like a bystander.

They are never resources. Not in meetings, not in planning docs, not in my head. The word matters because the word leaks into behaviour. Resources get allocated. People get respected. Only one of those shows up at 2 AM.

None of this is charity. It is a chain with no skippable links. Respect makes the production theirs, ownership follows, and once ownership exists, accountability stops being a conversation you ever need to have. I have never once had to ask an owner whether they will handle something. The question does not survive contact with genuine ownership.

You can buy attendance. You can buy hours, and with enough money you can even buy compliance. The feeling that this system is mine is not for sale. It is reciprocated or it is absent.

The boundary that keeps this honest

Here is where I have to be careful, because this philosophy has an evil twin.

If a manager reads "my people show up at 2 AM" and hears free capacity, they have turned respect into exploitation with better branding. The willingness is a gift. The moment you treat it as a budget line, planning around it, expecting it, spending it, you are strip mining the very trust that created it, and it does not grow back at the rate you are extracting it.

So the same philosophy comes with obligations that run in my direction. I answer first at 2 AM. The leader eats the first hour before anyone else is asked to. Incident nights are followed by comp time off without anyone having to request it. And I watch for the pattern where the same engineer is the hero twice in a row, because the person who always shows up is one quarter away from being the person who used to work here.

Protect the gift and it compounds. Spend it and it is gone.

The morning after

Somewhere past hour thirty, the estate was back, the region was healing, and everyone finally got some well deserved rest.

The next morning I did not assign the postmortem. Drafts were already appearing in the channel, from the people who had been on the bridge, unprompted, because owners do not wait to be asked what happened to their own systems.

That is the whole thing, really. It was never about the night.

It was about all the days before it.

A junior engineer accidentally dropped a staging database!

Saqib Ameen Subhan — Tue, 15 Sep 2026 05:20:48 +0000

A junior engineer on my team once ran a cleanup command on staging, believing he was on dev. The data was gone before the prompt returned.

He came to me directly, shaking, honestly, and told me what happened. Which, I want to point out before anything else, is the single most important fact in this story.

On a lot of teams, that engineer spends an hour trying to quietly fix it himself first, and the incident gets worse in the dark. He came straight to me. That is not a personality trait. That is a property of the team, and you build it or you destroy it long before the incident happens.

The first hour

The first hour is triage, not analysis.

I took over communication with the affected dev team myself. Not because the engineer could not speak, but because the person with the least organizational armor should not be the face of an incident. We stabilized, restored from backup, verified, and closed the loop with the dev team the same day.

At no point did anyone ask who did it in a public channel. The dev team was told: we made a mistake, we have restored it, here is what we are changing.

Accountability flows up. A manager who forwards blame downward is just a router.

The interesting question

Once it was over, the question I sat with was not how do I make sure he is more careful.

Careful is not a system. Everyone is careful until they are tired, or it is 2 AM, or two terminal windows look identical, which is the actual root cause here. He was not careless. He was a human being using a tool that made two very different worlds look exactly the same.

So we changed the tools.

Terminal backgrounds now encode the environment. Dev is default. Staging is blue. Production is red, a red you cannot ignore, a red that makes you sit up slightly before you type. It costs nothing and it works on the tired version of you, which is the version that makes mistakes.

A pre execution checklist for destructive commands. Three questions, ten seconds. Where am I? What system is this? Am I on a read node or a write node? We wrote it into the runbook.

I told the team the analogy I actually think in. You cut vegetables outside the vessel, then put them in. You do not chop directly into the cooking pot, because there, every slip is dinner.

Neither fix is clever. That is the point. Clever fixes depend on people being sharp. Boring fixes work when they are not.

What happened to the engineer

Nothing. That is the answer, and it was deliberate and visible. No formal note, no development area in his next review, nothing.

The incident was the process's fault, the process got fixed, and the fix was named after the problem, not the person.

He stayed. He got better, genuinely better, the kind of better that comes from learning viscerally that mistakes are survivable here. Years later he was one of the people I trusted most in a production window.

If I had made that first incident expensive for him, I would have taught him, and everyone watching, because everyone is always watching, that the smart move next time is to hide it.

The general rule

When something breaks, there are always two available explanations. The person, or the system the person was inside.

Choosing the person is emotionally satisfying and fixes nothing. The next human inherits the same trap. Choosing the system is boring and permanent.

We never had that class of incident again. Not because the team became more careful, but because the trap was removed.

The terminal colors outlived everyone on that team, including me.

Large partitions don't fail loudly. They fail everywhere at once.

Saqib Ameen Subhan — Fri, 04 Sep 2026 17:29:00 +0000

Most Cassandra problems announce themselves. A timeout, an exception, a graph going vertical.

Large partitions do not. They degrade reads, compaction, repair and GC simultaneously and gradually, which is why the incident channel fills with four unrelated looking symptoms and no suspect.

If you have been following this series, you have already met large partitions twice without the name. They are the humongous allocations in the heap dump post, and they are the overstreaming amplifier in the repair post. This post is about the disease itself.

What large means and why it hurts

A partition is the unit Cassandra reads, compacts, streams and materializes. The rule of thumb that has served me well is to aim under 100MB per partition and treat anything past that as debt. Old timers remember the hard 2GB era limits. Modern Cassandra handles big partitions better, which mostly means they hurt you more gradually.

Where the pain lands:

Reads. A partition read materializes structures proportional to what is scanned. In G1 terms, a multi hundred MB partition read is a parade of humongous allocations. GC pause spikes on whichever node holds it, the cluster marks it slow, and speculative retries fan the load elsewhere.
Compaction. Partitions are compacted as units. Giant partitions make giant, slow compaction tasks that starve neighbours, and pending tasks climb.
Repair. One mismatched cell in a giant partition can stream the whole thing. The overstreaming problem, concentrated.
Hotspotting. Big partitions are usually hot partitions, the same key taking disproportionate traffic. Size and heat compound.

How partitions get big, always the same story

The partition key models an unbounded thing.

-- looks innocent in the design review
CREATE TABLE events_by_device (
  device_id uuid,
  ts timestamp,
  payload blob,
  PRIMARY KEY (device_id, ts)
);

A partition key of device_id means every event that device ever produces, forever, in one partition. A chatty device, a popular customer, a busy trading day. Growth has no ceiling because the model gave it none.

I have seen the one big customer partition at multiple companies. It is practically a genre.

Finding them

nodetool tablehistograms ks events_by_device
# Percentile      Partition Size
# 50%             61,214 bytes
# 99%             386,857,368 bytes     <- there it is
# Max             2,395,318,855 bytes

The p50 to p99 gap is the signature. A healthy median hiding a monster tail.

To name the offenders on disk:

# per SSTable, list partitions over a threshold
sstablepartitions -t 100 /var/lib/cassandra/data/ks/events_by_device-*/

And Cassandra logs them at write time. Grep system.log for Writing large partition. Those log lines are the cheapest early warning system you will ever ignore.

The fix: give the partition a ceiling

You cannot cap a device's lifetime events. You can cap a partition, by putting time into the key.

CREATE TABLE events_by_device_v2 (
  device_id uuid,
  day date,                       -- the bucket
  ts timestamp,
  payload blob,
  PRIMARY KEY ((device_id, day), ts)
);

Now a partition is one device, one day. Bounded by physics instead of hope. Two design decisions follow.

Bucket size is a calculation, not a vibe. Estimate rows per day times row size, then pick the bucket, hour, day or week, that lands typical partitions in single digit MB and your worst realistic case under about 100MB. Do the arithmetic for your loudest tenant, not your average one. Averages are how the enormous customer surprises you.

Multi bucket reads are now yours to orchestrate. Last 6 hours spanning midnight means two partitions. Drivers make querying a known list of buckets cheap. Just design the access pattern alongside the schema, not after it. This pairs well with TWCS from the compaction post, because bucketed, TTL'd time series is the workload TWCS was born for.

Migration reality. Existing giants must be rewritten into v2. Dual write new data, backfill old in throttled batches, the same discipline as the 2TB MongoDB delete earlier in this series, chunk, sleep, watch the cluster. Then cut reads over and drop v1. Weeks, calmly, not a weekend, heroically.

The review question that prevents all of it

Every Cassandra schema review I run ends with one question. What bounds this partition?

If the answer is a business quantity, customers will not have many orders, then it is unbounded, because business quantities grow. That is their job. The only acceptable answers have units of time, or a hard modulo in the key.

Large partitions are never an ops problem. They are a design review that ended one question too early.

Repair at 1,000 nodes: why we stopped running nodetool repair by hand

Saqib Ameen Subhan — Fri, 04 Sep 2026 17:28:30 +0000

At small scale, repair is a cron job. At 1,000 nodes it is a scheduling problem, and if you treat a scheduling problem like a cron job, the cluster will teach you the difference during business hours.

I spent years operating Cassandra estates past the thousand node mark. Here is what repair actually does, why it is not optional, and how the operational model has to change with scale.

Why repair is a deadline, not a chore

Cassandra replicas drift. Hinted handoff covers short outages, but hints have a window, three hours by default. A replica down longer than that has permanently missed writes, and only anti entropy repair reconciles it.

The part that turns repair from hygiene into a deadline is tombstones. Deleted data is protected from resurrection by grave markers that compaction may purge after gc_grace_seconds, ten days by default. The safety of that purge rests on one assumption. Every replica heard about the delete before the marker vanished. Repair is how they hear.

So the rule is absolute. Every node completes repair at least once per gc_grace_seconds. Miss it, and a replica that slept through a delete will happily hand the missing row back to the cluster. Zombie data, silent, and by the time anyone notices, days of writes have been made against resurrected garbage.

Repair is not cleanup. It is the second half of every delete you have ever issued.

What a repair costs

Mechanically, nodes build Merkle trees over their data ranges, exchange them, and stream any ranges whose hashes disagree. Three costs follow.

Building trees reads data. Sequential I/O competing with your workload.
Streaming mismatches saturates network and creates new SSTables, which means the third cost.
Compaction debt. A big repair is followed by a compaction hangover, and nodetool compactionstats pending tasks tell the story.

Merkle tree resolution is finite, so one tree over a huge range means one tiny mismatch streams a disproportionately large chunk. This is overstreaming, and it is why repairing a giant range in one shot is both slow and wasteful.

Full, incremental, subrange, and what breaks at scale

Full repair, whole range, everything rehashed every time. Correct, brutal, does not scale. At 1,000 nodes, naive nodetool repair across the fleet means the cluster is effectively always repairing, and repair sessions colliding on shared ranges fail in tedious ways.

Incremental repair marks SSTables repaired so they are skipped next time. On paper, the fix. In practice it drags anticompaction behind it, rewriting SSTables to segregate repaired from unrepaired data, and its operational history, particularly before Cassandra 4.0's fixes, was rocky enough that many large operators simply did not trust it. We did not.

Subrange repair breaks the token range into small segments repaired one at a time. Small Merkle trees, precise comparisons, minimal overstreaming, and each segment is a small retryable unit of work. This is the primitive that actually scales.

But subrange repair across 1,000 nodes is thousands upon thousands of segments needing ordering, throttling, retries and collision avoidance. Congratulations, it is a scheduling problem.

Reaper: repair as a system, not a command

We ran Cassandra Reaper, open source, originally Spotify's and now community maintained, as the scheduler.

Splits every table's range into segments and runs them steadily with concurrency caps
An intensity knob throttles repair pressure so p99s do not feel it. We tuned it to stay invisible.
Failed segments retry individually, so a node blip costs one segment, not the whole run
A web UI that answers the only question leadership asks, which is whether we are inside the gc_grace window, yes or no

The operating policy that came out of years of this:

- every table repaired well inside gc_grace, target a complete cycle in ~7 days against a 10 day grace
- repair pressure tuned to be invisible in p99 read latency
- alert not on "repair failed" but on "time since last completed repair per table"
- pause repairs during topology changes, resume, never skip

That last alert framing matters. Failures are noise. Staleness is the actual risk. Measure the deadline, not the activity.

The small scale corollary

Under about 50 nodes you do not need Reaper's ceremony, but you do need the same two invariants. Every node repaired inside gc_grace_seconds, and repairs that do not collide. A boring, monitored schedule beats heroics.

Repair is the tax on eventual consistency. You can pay it on a schedule you chose, or all at once with interest, with zombie data as the collection notice.

The heap dump usually exonerates Cassandra

Saqib Ameen Subhan — Fri, 04 Sep 2026 17:26:34 +0000

When a Cassandra node OOMs, or GC pauses start stretching into seconds, the reflex is to blame Cassandra, or its favourite scapegoat, the JVM.

I have read a lot of Cassandra heap dumps, and here is the uncomfortable pattern. The heap usually contains exactly what the data model asked for. The dump does not convict the database. It convicts a partition.

This post is the workflow I actually use, in order, because the order saves hours.

Step 0: GC logs before heap dumps

A heap dump is a biopsy. GC logs are the patient history, and they are already on disk. Two things I look for first:

# gc.log (G1)
Pause Young (Normal) ... 180ms
Pause Young (Normal) ... 210ms
Pause Full (Allocation Failure) ... 8.4s      <- the incident
...
Humongous Allocation ... 18MB
Humongous Allocation ... 22MB

Full GC pauses. G1 doing an emergency stop the world collection because normal cycles could not keep up. On a Cassandra node, an 8 second pause does not just slow queries. The node misses gossip, gets marked down by peers, and hints start piling up elsewhere. One node's GC problem becomes the cluster's problem.

Humongous allocations. G1 divides the heap into regions, typically 4 to 32MB. Any single allocation larger than half a region is humongous, allocated awkwardly across contiguous regions and collected inefficiently.

And what does Cassandra allocate in one giant contiguous chunk? A large partition being materialized for a read. Humongous allocation warnings in a Cassandra gc.log are large partitions announcing themselves. You can often skip the heap dump entirely at this point and go straight to nodetool tablehistograms to find the table with a monster p99 partition size.

Step 1: take the dump without causing an incident

jmap on a live heap stops the world for the duration of the dump. On a 20GB heap, that is long enough for the cluster to declare the node dead. Never dump a node that is still serving.

# isolate first, node stays alive but stops serving
nodetool disablebinary && nodetool disablegossip

jmap -dump:live,format=b,file=/data/dumps/cass_$(hostname)_$(date +%s).hprof <pid>

# afterwards: rejoin or just restart the node

Better, have the JVM do it for you at the moment of truth, before any human is awake:

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/data/dumps/

Those two flags belong in every production Cassandra JVM config. An OOM without a dump is an incident you get to have twice.

Related discipline, keep the heap at or under 8GB and let G1 target around 200ms pauses. Bigger heaps mostly buy you longer pauses and bigger dumps, and Cassandra's real caching happens off heap and in the page cache anyway.

Step 2: Eclipse MAT, dominator tree, ten minutes

Open the .hprof in Eclipse MAT and go straight to the dominator tree, objects ranked by retained memory, which answers what is actually holding the heap hostage. Ignore the histogram of a billion byte[]. The dominator tree tells you whose bytes.

What the tree shows, mapped to what it means, every one of these from real incidents:

Dominating the heap	Actual problem
A few enormous `byte[]` or cell containers under a read path	Large partition being materialized. Get the key from the referencing objects, confirm with `nodetool tablehistograms` or `sstablepartitions`. Fix is to bucket the partition, which is a later post in this series.
Memtable objects, many tables	Over wide schema, or memtable flush thresholds set too generous for the heap
Tombstone or range tombstone structures under a query	A tombstone farm read that survived just long enough to dump
Netty buffers or inflight requests	Clients hammering with no paging. Check fetch size and unpaged full partition reads

The punchline repeats across years and companies. The heap contains the workload. I have almost never opened a dump and found a Cassandra bug. I have regularly opened one and found a 4GB partition someone swore was just a busy customer.

Step 3: close the loop in the model

The dump names the object. The fix lives in the schema.

Large partition, time bucket the partition key. Unpaged reads, driver fetch size. Tombstone reads, the modeling fixes from two posts ago. The JVM flags and MAT are diagnosis. Cassandra data modeling is treatment.

Blaming the JVM is comfortable because nobody owns the JVM. The heap dump takes that comfort away. It shows you, in retained bytes, precisely which design decision you are looking at.

A compaction strategy is a bet about your workload

Saqib Ameen Subhan — Fri, 04 Sep 2026 17:25:24 +0000

Choosing a Cassandra compaction strategy is placing a bet. Each strategy optimizes one thing by deliberately sacrificing another, and the honest way to choose is to decide which failure mode you would rather operate.

I have run all three majors across 1,000+ node estates. Here is the bet each one makes.

Why compaction exists at all

Cassandra never updates in place. Writes land in a memtable, flush to immutable SSTables, and a row's fragments accumulate across many files as it is updated over time. Reads must merge every relevant fragment.

Compaction is the background process that merges SSTables, consolidating fragments and purging expired tombstones after gc_grace_seconds, so reads touch fewer files.

The metric that keeps compaction honest:

nodetool tablehistograms ks table
# Percentile  SSTables     ...
# 50%             1.00
# 99%             4.00

SSTables per read at p99 is the number compaction exists to keep small.

STCS, SizeTiered, the write optimized default

Merges SSTables of similar size into bigger ones, repeatedly. Minimal write amplification, maximal ingest throughput.

The bet you are making: reads and disk headroom are negotiable.

A hot row's fragments can sit in many SSTables of different generations, so SSTables per read climbs.
Space amplification is the operational trap. Compacting large tiers requires holding input and output simultaneously. Plan for 50% free disk. I have watched teams treat 70% disk usage as plenty left, then be unable to run the very compaction that would reclaim space. That is a corner with no good exits.
Tombstones buried in giant, rarely recompacted SSTables can linger far past gc_grace_seconds.

Right when: write heavy, append mostly, reads tolerate variance. It is the default because it is the least dangerous average bet, not because it is right for you.

LCS, Leveled, paying writes to buy reads

Organizes SSTables into levels, L1, L2 and so on, each 10 times larger, guaranteeing non overlapping key ranges within a level. A read touches at most one SSTable per level, and in practice the vast majority of reads are satisfied by a single SSTable.

The bet: you will pay roughly 10x write amplification, because every row is rewritten as it migrates down levels, to make read latency tight and predictable.

On read heavy tables with strict p99s, it is the correct bet.
On write heavy tables, compaction falls behind, pending tasks pile up in nodetool compactionstats, and ironically reads degrade anyway because L0 accumulates overlapping files.
Easier on disk headroom than STCS, works in around 10% free, but harder on I/O and CPU, continuously.

Right when: read dominated, update in place workloads. Wrong when your disks are already busy keeping up with ingest.

TWCS, TimeWindow, the time series specialist

Groups SSTables by time window, say one per day. Within the current window, STCS as usual. Once a window closes, its SSTables compact once into one file and are never touched again.

When TTL expires a whole window, Cassandra drops the entire file, so tombstone processing effectively vanishes for aged out data. This is the single biggest tombstone fix available for time series, and the punchline of the previous post.

The bet: your data is truly time ordered and immutable once written.

Write out of order data or update old windows and you create cross window overlaps that can never compact away. The strategy's guarantees quietly die while everything still appears to work.
Rule of thumb, aim for a window size that yields 20 to 50 windows over the table's TTL.

Right when: metrics, events, telemetry, anything with TTL and append only semantics. IoT and monitoring tables at scale should almost always be TWCS.

The bet, stated plainly

Strategy	You optimize	You pay with	Operational trap
STCS	write throughput	read variance	needs around 50% free disk to compact
LCS	read p99	10x write amplification	falls behind under heavy ingest
TWCS	TTL'd time series	inflexibility	out of order writes break it silently

Per table, not per cluster. A keyspace legitimately mixes all three. And changing the bet is an online operation:

ALTER TABLE ks.events WITH compaction =
  {'class': 'TimeWindowCompactionStrategy',
   'compaction_window_unit': 'DAYS', 'compaction_window_size': 1};

Cassandra will re sort existing SSTables over time. On a big table, schedule the transition like the migration it is.

There is no best compaction strategy. There is only the failure mode you have chosen on purpose, versus the one that chose you.

mtools does not parse MongoDB 4.4+ JSON logs. So I built mdbkit.

Saqib Ameen Subhan — Mon, 24 Aug 2026 19:28:11 +0000

If you have operated MongoDB for any length of time, you know mtools. For years it was how we all read production logs. mloginfo, mplotqueries, and a slow query was suddenly visible.

Then MongoDB 4.4 changed the log format to structured JSON, and the log tools stopped understanding what they were reading.

That was six years ago. The support issue was opened in June 2020 and it is still open. We are on 8.0 now, and a lot of us have been getting by with grep, jq pipelines held together by hope, and a certain amount of squinting.

I got tired of it and built the replacement. It is called mdbkit, it is MIT licensed, and it is public as of this week.

Try it in sixty seconds without touching a cluster

This is the part I would want first if this were someone else's tool.

pip install --upgrade mdbkit
# macOS ships no pip: brew install pipx && pipx install mdbkit

mdbkit demo -o demo.log
mdbkit queries demo.log

mdbkit demo generates a realistic MongoDB log containing an actual incident. No cluster, no connection string, nothing of yours involved. You get to judge the tool on a problem you can inspect before you point it at anything that matters.

Here is what queries gives you on that demo log:

saqib@MacbookPro mongodb_logs % mdbkit queries demo.log
== mdbkit queries (slow query shapes) ==
lines: 1,004  parsed: 1,004  unparsed: 0  span: 2026-07-01T08:00:00+04:00 -> 2026-07-01T09:29:44.615000+04:00

namespace      op         count  cumMs  mean   max    docsEx      scan     plan            shape
-------------  ---------  -----  -----  -----  -----  ----------  -------  --------------  ---------------------------------------------
shop.events    aggregate  53     5.6m   6.3s   8.9s   47,170,000  98889:1  COLLSCAN+SORT   {tenantId:eq, ts:gte} sort:{ts:-1}
shop.orders    find       89     2.4m   1.6s   2.4s   11,125,000  2976:1   COLLSCAN+SORT   {createdAt:gt, status:eq} sort:{createdAt:-1}
shop.sessions  find       58     34.7s  597ms  888ms  2,784,000   155:1    IXSCAN{active}  {active:eq, lastSeen:gt} sort:{lastSeen:-1}
shop.products  update     42     25.7s  612ms  799ms  2,268,000   54000:1  COLLSCAN        {sku:eq}
shop.users     find       56     6.7s   119ms  139ms  56          1:1      IXSCAN{email}   {email:eq}

cumMs  = total wall time accumulated across ALL occurrences (not one query)
docsEx = total documents examined across all occurrences
scan   = docsExamined per returned doc (high = index missing or weak)
plan   = most common query plan; COLLSCAN/+SORT = index needed
         empty plan (?) = plan not present in log (below slowms threshold)
next   : mdbkit advise <log> [--ns <namespace>] for index candidates

That is not a screenshot I curated. demo is seeded, so those two commands produce that exact table on your machine too, down to the last digit. It behaves the same on my laptop, on yours, and on a conference projector.

The column that matters is the scan ratio. It is documents examined against documents returned, per query shape. A shape sitting at 98889:1 is reading roughly ninety nine thousand documents to hand back one. A shape at 1:1 is doing exactly the work it needs to.

That single ratio is most of slow query triage. Everything else is detail.

The workflow I actually use

Three commands, in order.

First, find the shape that is hurting.

mdbkit queries /var/log/mongodb/mongod.log

Slow operations grouped by query shape rather than listed individually. This matters more than it sounds. A log with 40,000 slow query lines is usually eight or nine distinct shapes repeating, and the one costing you the most is rarely the one that appears most often.

Then, ask what would fix it.

mdbkit advise /var/log/mongodb/mongod.log

This proposes candidate indexes based on the shapes it observed. More on how that works below, because it is the part people should be suspicious of.

And when something has already gone wrong, build the timeline.

mdbkit triage /var/log/mongodb/mongod.log

Elections, rollbacks, stalls, connection storms, in order, with timestamps. When you are twenty minutes into an incident and someone senior is asking what happened, a timeline you did not have to assemble by hand is worth a great deal.

The log cannot tell you everything

There is one failure mode a mongod log structurally cannot explain: the log simply stops, then starts again a minute later with no error in between.

The process was killed before it could write anything. That answer lives in the system log, one line up from where you were looking.

mdbkit triage mongod.log --oslog /var/log/syslog

[CRIT] Process start(s) in window: mongod startup marker at 09:14:35
[CRIT] System: oom-kill: The kernel OOM killer terminated a task.
         1 occurrence (last at 09:14:03). Processes: mongod.
         next: This explains an unexplained restart in the log above.

oslog reads /var/log/syslog or /var/log/messages and looks for the things that end database processes: OOM kills, file descriptor limits, segmentation faults, I/O errors, filesystems remounted read only, systemd service exits. Used with triage, the unexplained restart gets matched to the kernel line that caused it.

On journald systems there is no text log to read, and mdbkit will not run commands on your behalf, so it prints the journalctl invocation for you to capture and feed back in.

There is also ftdc, which decodes diagnostic.data offline. Every one of your nodes has been recording CPU, cache, queue depth and disk every second since the day you installed it, whether or not you run any monitoring. Plus serverstatus for making sense of a serverStatus dump, and compare for answering whether the index you added last week actually helped.

It never connects to your database

No driver. No URI. No network code anywhere in it.

mdbkit reads log files you hand it. It is strictly read only. Where an action would help, it prints the command for you to review and run yourself, rather than running anything on your behalf.

I built it this way because I would not run a stranger's tool on a production host, and I do not expect you to either. So rather than asking you to trust me, the README shows you how to verify it independently. grep the source for socket and driver imports. Run it under strace and watch it make no network calls. It takes about a minute and you should do it.

Zero runtime dependencies too, nothing in the supply chain beyond the Python standard library. That was a deliberate constraint and it cost me some convenience, but a tool that ships onto database servers should not drag a dependency tree behind it.

The index advice is rules, not AI

I gave a talk at the Dubai MongoDB User Group recently about how confidently language models describe a MongoDB that no longer exists. They will happily reference plannerVersion, a field removed in 5.0. They do not know about EXPRESS stages in 8.0. And more importantly, they cannot know your cardinality, your write mix, or what is currently sitting in your plan cache, because none of that is in your query text.

So mdbkit's index advice is deterministic. Rules over observed query shapes. The same log always produces the same recommendation.

Every suggestion comes with the evidence it reasoned from, a confidence level, and how to validate it before you act. It says candidate, never command. And it will never tell you to drop an index, because deciding an index is unused requires knowing about traffic that a log file cannot show you.

That is a deliberately narrow promise. I would rather it be right about a small thing than confident about a large one.

What would help most

Two things, and they are both easy for you and hard for me.

Real world log lines that parse incorrectly. Every MongoDB deployment logs slightly differently, and I have only seen the estates I have worked on. If mdbkit chokes on a line, that line is the most useful thing you can send me.

Index advice that is wrong or unhelpful. If it suggests something you know is a bad idea for your workload, I want to know why, because that is a rule that needs fixing.

Every release so far has come from someone doing exactly that. An FTDC decode that took twenty five minutes and pinned a CPU. A crash on a batched write. A restart with no explanation, which is why oslog exists at all.

Supports MongoDB 4.4 through 8.0. Python 3.8+.

Source: https://github.com/saqibameen86/mdbkit
Package: https://pypi.org/project/mdbkit

Free, MIT, and built because the gap annoyed me for long enough.

If you liked this, I also wrote about deleting 2TB from a live MongoDB cluster without anyone noticing and why I take down a healthy primary on purpose.

In Cassandra, a delete is a write (and reads pay for it)

Saqib Ameen Subhan — Sun, 23 Aug 2026 16:36:09 +0000

In Cassandra, a delete is actually a write.

That sounds like trivia until it slows down your reads in production. When you run DELETE, Cassandra does not remove anything. It writes a marker called a tombstone, which says "this data was deleted at this time." The data is only removed later, during compaction, and only after a waiting period. Until then, the data is gone as far as your application is concerned, but it is still physically there, and every read that touches it has to go through it.

Why Cassandra needs the tombstone

Cassandra keeps copies of your data on multiple nodes (replicas). Imagine a delete simply removed the data, and one replica was down when the delete happened. When that replica comes back, it still has the row. To the rest of the cluster, it looks like the other nodes are missing data, so repair copies the deleted row back to every node.

Deleted data coming back like this is called zombie data, and tombstones exist to prevent it. The tombstone is newer than the old value, so wherever both are seen, the tombstone wins.

This is also why tombstones have to stay for gc_grace_seconds before compaction is allowed to remove them. The default is 864000 seconds, which is 10 days. Those 10 days are the time you have to run repair on a node that was down. If you reduce gc_grace_seconds without also running repair more often, you are setting yourself up for zombie data. I have written about repair at scale in a later post in this series.

How reads pay for it

When you read data, Cassandra has to merge what is in memory (the memtable) with what is on disk (SSTables). Every tombstone in the range you are reading has to be read and compared with the live data before Cassandra can give you an answer. If a partition has collected a million tombstones, Cassandra has to go through a million deleted entries just to return whatever is still alive.

Cassandra warns you when this happens, first with a warning and then with an error.

WARN  ReadCommand - Read 812 live rows and 104,832 tombstone cells for query ...
ERROR ... Scanned over 100001 tombstones ... query aborted

The warning shows up when a read goes past tombstone_warn_threshold, which is 1,000 by default. If it goes past tombstone_failure_threshold, which is 100,000 by default, Cassandra stops the read and throws a TombstoneOverwhelmingException. Cassandra fails your query on purpose, so that it does not run the node out of memory.

It can be tempting to just raise these limits, but that only hides the problem. The tombstones are still there.

Where all these tombstones come from

I have seen every one of these in production.

Using Cassandra like a queue. Insert, process, delete, and repeat, all in the same partition. The partition slowly fills with tombstones, and every time the consumer reads the next item, it has to go past all of them. This is exactly why using Cassandra as a queue is a well known anti pattern.
Overwriting a whole collection. Running UPDATE t SET mymap = {...} replaces the entire collection, and Cassandra writes a tombstone for the old one first. If you only want to add items, use mymap = mymap + {...} instead.
Inserting nulls. If your application binds a null value in a prepared statement, Cassandra writes a tombstone for that column. ORMs and code that "just binds every column" can create thousands of these without anyone noticing. Leave the value unset instead of setting it to null.
TTL on everything. When data with a TTL expires, it also turns into a tombstone. This is fine, as long as your compaction strategy is designed for it.

How to find them

To check how many tombstones an SSTable has:

sstablemetadata /var/lib/cassandra/data/ks/table-*/*-Data.db | grep -i droppable
# Estimated droppable tombstones: 0.83

0.83 means that about 83% of that SSTable is tombstones that can be removed. To see tombstones for a single query, run TRACING ON in cqlsh before the query. For a table that is live, nodetool tablestats shows the average tombstones per slice.

How to fix them

The real fixes are in how you model the data.

For time series data with TTL, use TimeWindowCompactionStrategy (TWCS). Whole SSTables expire together and get dropped as files, so Cassandra hardly has to deal with individual tombstones. I have covered compaction strategies in the next post.
Stop deleting rows one by one in busy partitions. Put time into the partition key, so that old partitions expire as a whole instead.
Check your code for null binding and collection overwrites. These are the tombstones nobody meant to create.

Try it yourself

You can see this on your laptop with Docker. Start Cassandra:

docker run --name cass -d cassandra:4.1

Wait about a minute for it to start, then create a table that we will use like a queue:

docker exec -it cass cqlsh -e "
CREATE KEYSPACE lab WITH replication = {'class':'SimpleStrategy','replication_factor':1};
CREATE TABLE lab.queue (pk int, ck int, PRIMARY KEY (pk, ck));"

Add 10,000 jobs to the same partition, then delete all but the last 10, the way a queue consumer would:

docker exec cass bash -c 'for i in $(seq 1 10000); do echo "INSERT INTO lab.queue (pk, ck) VALUES (1, $i);"; done > /tmp/ins.cql && cqlsh -f /tmp/ins.cql'
docker exec cass bash -c 'for i in $(seq 1 9990); do echo "DELETE FROM lab.queue WHERE pk = 1 AND ck = $i;"; done > /tmp/del.cql && cqlsh -f /tmp/del.cql'

Now read the partition with tracing on:

docker exec -it cass cqlsh

TRACING ON;
SELECT * FROM lab.queue WHERE pk = 1;

In the trace, look for the line that says how many live rows and how many tombstones were read. To return 10 rows, Cassandra had to go through all 9,990 deleted ones.

When you are done:

docker rm -f cass

In Cassandra, a delete is cheap to run but expensive to live with, because every delete stays in the way of your reads until gc_grace_seconds has passed and compaction removes it.

How to delete 2TB from a live MongoDB cluster without anyone noticing

Saqib Ameen Subhan — Sun, 23 Aug 2026 16:04:47 +0000

The request was simple. "There is about 2TB of expired data in this collection. Can you delete it tonight?"

My answer was no, and being able to explain why is most of the job. A missing TTL index had let short lived data pile up for months in one of our busiest collections, and the team wanted to run one big deleteMany overnight. Here is what that would have actually done.

Why a big delete hurts, and it is not locking

Most people think a big delete is risky because it locks the collection. In modern MongoDB it does not, because WiredTiger works at the document level, so a huge delete does not freeze the collection. The damage comes from three other places.

1. The oplog fills up. Every deleted document becomes its own entry in the oplog (the log that secondaries copy from). Delete 500 million documents and you have written 500 million oplog entries, and every secondary has to copy and apply each one of them. Replication lag starts climbing. If a secondary falls further behind than the oplog can hold, it can no longer catch up and needs a full resync. At that point your delete has cost you a copy of your data.

2. The cache gets flushed out. To delete a document, WiredTiger first has to load it into memory. A 2TB delete pulls a lot of old data into the cache and pushes out the data your application actually uses. Queries that have nothing to do with the delete start getting slow, and whoever looks into it will probably never connect it back to the delete.

3. Majority writes slow down. If your writes use w:"majority" (and they should, I covered why in an earlier post in this series), every write now waits on the same secondaries that are already struggling to keep up with the delete. The problems add up.

I showed this to the team in a lower environment. I started the big delete, ran rs.printSecondaryReplicationInfo(), and let them watch the lag grow. After that, "tonight" turned into a plan.

The approach: small chunks, a pause, and watching the lag

const CHUNK = 10000;
const SLEEP_MS = 500;
const cutoff = ISODate("2026-01-01");

let total = 0;
while (true) {
  const ids = db.events.find({ created: { $lt: cutoff } },
                              { _id: 1 }).limit(CHUNK).toArray().map(d => d._id);
  if (ids.length === 0) break;

  db.events.deleteMany({ _id: { $in: ids } });
  total += ids.length;

  sleep(SLEEP_MS);   // give the secondaries time to catch up

  if (total % 1000000 === 0) print(`${total} deleted, check the lag`);
}

Two rules we followed while running it.

Only run it in off peak hours. We ran it every night and stopped it before business hours. The full 2TB took around a week and a half, and nobody noticed anything, which was exactly the goal.

Let replication lag decide the speed. If the lag went above the level we were comfortable with, we increased SLEEP_MS. The speed of the loop is decided by how healthy the cluster is, not by how quickly we want it done.

The pause between chunks can look far too slow to developers. That is fine. Once the data has stopped growing, the delete has no deadline, and what stops the data from growing is the real fix below.

The real fix: a TTL index, so this never happens again

db.events.createIndex({ created: 1 }, { expireAfterSeconds: 2592000 })  // 30 days

A few things about TTL indexes that surprise people.

The TTL process runs every 60 seconds, so documents are not deleted at the exact second they expire. That is fine for cleaning up old data, but do not rely on it for business logic that needs exact timing.
It deletes in batches in the background, which is basically the same chunked loop as above, running forever. On a very large backlog it can fall behind, which is why we cleared the 2TB by hand first and then let TTL handle it from there.
Put the TTL index on the field that actually decides when the data is old. A TTL index on the wrong date field will quietly delete data you wanted to keep.

We finished with a runbook covering the script, the lag limits, and the schedule, so the next time someone asks to delete 2TB, the answer is already written down.

Try it yourself

If you want to see this on your laptop, mdbkit lab gives you a 3 node replica set.

pip install mdbkit
mdbkit lab start          # 3 node replica set on 127.0.0.1:28110-28112

Connect to the primary and load a million documents, about half of them older than 60 days.

// mongosh --port 28110
use shop
for (let b = 0; b < 100; b++) {
  const docs = []
  for (let i = 0; i < 10000; i++) {
    docs.push({ created: new Date(Date.now() - Math.random() * 120 * 86400000), payload: "x".repeat(200) })
  }
  db.events.insertMany(docs)
}

Then run the chunked delete. This version prints the time, how many documents have been deleted so far, and how far behind the secondaries are.

function secondaryLag() {
  const s = rs.status()
  const p = s.members.find(m => m.stateStr === "PRIMARY")
  const lags = s.members.filter(m => m.stateStr === "SECONDARY")
                        .map(m => (p.optimeDate - m.optimeDate) / 1000)
  return Math.max(...lags)
}

const cutoff = new Date(Date.now() - 60 * 86400000)
let total = 0
while (true) {
  const ids = db.events.find({ created: { $lt: cutoff } }, { _id: 1 })
                       .limit(10000).toArray().map(d => d._id)
  if (ids.length === 0) break
  db.events.deleteMany({ _id: { $in: ids } })
  total += ids.length
  const t = new Date().toISOString().substr(11, 8)
  print(t + "  deleted " + total + " old documents   secondary lag " + secondaryLag() + "s")
  sleep(500)
}
print("done, deleted " + total + " old documents")

You will see the lag stay close to zero the whole way through. If you want to compare, reload the data and run the same delete as one single deleteMany, then check rs.printSecondaryReplicationInfo() while it runs.

When you are done:

mdbkit lab destroy --yes

When you have to delete a lot of data from a live system, going slowly is the safe way to do it.