DEV Community: Kyryl

Your Familiar Java Classes Just Lost Identity 🧬

Kyryl — Wed, 12 Aug 2026 15:42:56 +0000

Project Valhalla's JEP 401 is integrated into JDK 28 as a preview. It needs --enable-preview at both compile and runtime, so nothing you ship today changes. But once this graduates, a class you already use constantly is going to stop having identity, and the compiler is not going to tell you.

What a value class actually changes

A value class in JEP 401 gets four behavioral changes:

Fields are implicitly final.
== compares field values instead of object identity.
Construction must fully initialize every field before the constructor returns.
Instance methods on a value class cannot be synchronized.

Several existing JDK classes become value classes the moment the preview flag is on. Primitive wrappers, Integer, Long, and friends, are named examples. So is LocalDate. Backward compatibility is preserved with the flag off, so this is opt-in territory today. But Integer and LocalDate are about as core to a Java codebase as it gets, which is exactly why this matters more than a typical preview feature.

Why value classes exist at all

Identity has a cost. Every identity-bearing object carries a header, cannot be unboxed cleanly, and cannot be packed into arrays or CPU registers the way a primitive can. The JVM has to always go through a pointer.

For types where nobody actually relies on identity, Integer and LocalDate being the textbook cases, that cost buys nothing. Nobody writes code that depends on two LocalDate instances representing January 1st being the same object, only that they represent the same date. Removing identity where nobody needs it lets the JVM flatten these objects and inline them instead of always chasing a pointer. That is the performance case for Valhalla in one sentence.

The first thing that breaks: `==`

// Before JDK 28 preview
LocalDate d1 = LocalDate.of(2026, 1, 1);
LocalDate d2 = LocalDate.of(2026, 1, 1);

System.out.println(d1 == d2);
// false, identity compare, two different objects

// JDK 28, --enable-preview, LocalDate is now a value class
LocalDate d1 = LocalDate.of(2026, 1, 1);
LocalDate d2 = LocalDate.of(2026, 1, 1);

System.out.println(d1 == d2);
// true, value compare, no identity left to compare

Boxed-primitive == has always been a footgun, thanks to the autoboxing integer cache (-128 to 127) making small values compare equal by accident while larger ones do not. This is a different, sharper version of the same footgun. It is not that identity comparison is unreliable anymore. There is no identity left to be unreliable. Any code that used == as an intentional identity check, a cache lookup, a pooling pattern, an object-identity-based Set, silently gets different results the moment the underlying type becomes a value class. No compile error. No exception. Just a different answer at a line nobody thought to re-read.

The second thing that breaks: `synchronized`

// Before: legal, LocalDate is an ordinary identity-bearing object
LocalDate d1 = LocalDate.of(2026, 1, 1);
synchronized (d1) {
    // ...
}

// After: LocalDate is a value class, this is no longer valid
LocalDate d1 = LocalDate.of(2026, 1, 1);
synchronized (d1) {
    // compile-time error, value classes cannot be used as monitors
}

This one at least fails loudly, at compile time, which is the better failure mode of the two. But it means any code locking on an instance of a type that becomes a value class needs to be found and fixed before upgrading, and today there is no tooling that flags "this class might become a value class someday" ahead of time.

The honest trade-off

Preview-only, opt-in, nothing breaks in production code today. That is the reassuring half. The other half: the classes affected are not obscure corners of the JDK, they are Integer and LocalDate, used in essentially every nontrivial Java codebase. And the failure mode on == specifically is silent. A compile-time break, like the synchronized case, gets caught in CI. A silent behavior change in an == comparison gets caught in production, if it gets caught at all.

The safe posture right now, while this is still behind a flag, is to audit for two patterns before JEP 401 graduates out of preview: == used intentionally for identity on boxed primitive types or LocalDate, and synchronized blocks or methods locking on instances of those same types. Neither pattern is common in well-written code, .equals() should be doing the comparison work already in most cases, but "uncommon" is not the same as "absent," and this is the kind of bug that survives code review because it used to be correct.

Found either pattern in your own codebase while reading this?

The Shared Cache Tier Is Disappearing 🗄️

Kyryl — Wed, 12 Aug 2026 15:42:35 +0000

Canva just published how they rebuilt session revocation without a shared cache tier, and it is the third architecture writeup I have run into this month that lands on the exact same shape. Different companies, different problems, same answer: durable object storage as the source of truth, plus a locally rebuilt index, no shared cache in the hot path.

The problem: revocation has to be fast and shared

Session revocation is a specific kind of hard problem. When a token needs to die before its natural expiry, every gateway in the fleet needs to find out, fast, without every request round-tripping to a central store. The obvious answer for years has been a shared cache, Redis cluster or Memcached fleet, that every gateway queries or subscribes to.

A shared cache tier solves the propagation problem, but it becomes its own liability. It is a single point of contention under load. It has its own failure modes: eviction storms, hot keys, painful cluster rebalances. And it needs its own on-call rotation, separate from the service that actually owns the data.

What Canva built instead

Canva stores revocation records as 16-byte binary entries in S3, sliced into 30-minute objects across a 12-hour revocation window. Instead of gateways querying a shared store on every request, each gateway pulls the recent object slices and rebuilds a local, in-memory sorted-array index from them.

S3 layout (conceptual):
  revocations/2026-08-11T00:00-00:30.bin
  revocations/2026-08-11T00:30-01:00.bin
  revocations/2026-08-11T01:00-01:30.bin
  ...

Concurrency is handled with conditional GETs and PUTs, giving optimistic concurrency without a lock service. ZooKeeper leader election is used to reduce write conflicts further, but Canva is explicit that it is an optimization, not a correctness dependency. The system is correct without it, just slightly noisier under contention.

The results, from their own numbers: cache memory down 87.5%. Workers sustaining 2,000+ revocations per second. The database footprint down to two read replicas. A million revocations fit in roughly 16MB.

The pattern, not just the case study

Strip away the specifics and the shape is: a workload with small records, a bounded time window, and tolerance for a few seconds of staleness, served by cheap durable object storage instead of a shared, stateful cache. Each worker rebuilds its own disposable index locally instead of depending on a shared service to hold the current state for everyone.

This shape fits more than session revocation. Feature flags, entity caches, compacted-topic-style event state, anything that is read far more than it is written, has a small per-record footprint, and does not need every reader to agree on the exact same value at the exact same instant.

Seeing three unrelated teams land here independently in the same month is a stronger signal than any one writeup. It suggests the shared cache tier's default status, "many workers need the same fast-changing data, reach for Redis," is no longer the obvious answer for a meaningful slice of these workloads.

The honest trade-off

This pattern is not a free upgrade. Two real costs:

It only works when the dataset has a natural staleness tolerance and a bounded window. Canva's 12-hour revocation window caps how much history a worker ever needs to rebuild from scratch. Without a bound like that, "rebuild the index from object storage" gets slower and the object count grows without limit.

It trades a shared cache's single-source consistency for eventual consistency across workers. A shared Redis instance gives every reader the same answer the instant a write lands. Here, a revocation is only as fresh as the last object slice a given gateway has pulled. If your correctness requirements demand that every worker sees a write at the exact same millisecond, no propagation lag tolerated, this pattern is the wrong tool and a shared cache (or a shared database) is still the right one.

Canva's own use of ZooKeeper leader election as an optional optimization, not a dependency, is a tell about how the trade-off was made: they accepted some write-conflict noise in exchange for not needing a coordination service to be up for the system to be correct.

Where this fits

If you are running a workload that looks like "many stateless workers need to check a small, frequently-updated dataset, and a few seconds of staleness is fine," this is worth evaluating before reaching for another Redis cluster. The infrastructure is simpler to operate (no separate cache fleet to keep alive), the failure mode is more boring (a worker with a stale local index, not a cache stampede), and the cost profile at scale, per Canva's own numbers, is dramatically smaller.

Have you replaced a shared cache tier with this shape? What broke first when you tried it, and did the staleness window end up being the real constraint?

97% of Your AI Approval Clicks Were Reflexes 🤖

Kyryl — Wed, 12 Aug 2026 15:42:15 +0000

Anthropic ran a study with 1,053 testers before making a call most tools never make honestly about their own permission model: the approval prompt was not working, and here are the numbers to prove it.

Auto mode, the classifier that decides which tool calls need a human, caught 89% of harmful actions in that study. Humans clicking through the old per-call approval prompt caught 13.6%. And 97% of those prompts got rubber-stamped without a second look, whether the action was harmless or not.

Starting August 14, new Claude Code sessions on Pro, Max, and Team plans switch from "ask before every tool call" to "route through a classifier, only interrupt for irreversible, destructive, or outward-facing actions." Enterprise, API, and cloud-partner deployments (Bedrock, Vertex, Foundry) stay opt-in for now, with a default flip planned there within the month. Classifier overhead is not billed separately.

The 97% number is the actual story

A permission prompt only works as a safety mechanism if the human reading it is actually evaluating each one. At 97% rubber-stamped, that was not happening. People were clicking yes on reflex, the same way you click through a terms-of-service dialog. The prompt was theater. It gave the appearance of oversight without providing any, and the one time it mattered, the reflex fired the same as every other time.

This matters beyond Claude Code specifically. Any tool that gates action behind a per-call human approval and expects that approval to be a meaningful check is making the same bet Claude Code just admitted it lost. If your CI pipeline, your deploy tooling, or your own internal agent framework has a "click to confirm" step that nobody actually reads anymore, this study is a preview of what an audit of your own approval logs would probably show.

What "irreversible, destructive, outward-facing" likely means in practice

Anthropic has not published the exact classifier boundary, but the categories map to what teams running agents in CI already learned to gate manually:

Irreversible: force-pushes, dropped database tables, deleted branches, anything git reset --hard adjacent.
Destructive: bulk file deletion, overwriting uncommitted work, rm -rf on anything wider than a scratch directory.
Outward-facing: pushing code, opening or closing PRs, sending messages to Slack or email, posting to external services, anything visible to someone besides the person running the session.

If you already run Claude Code with a pinned permission mode (auto-accept, plan mode, a custom allowlist), none of this changes for you. This only affects sessions that were relying on the default, which was most sessions belonging to people who never touched the setting.

The honest trade-off

A classifier is not you. It was trained on what 1,053 testers, in aggregate, considered harmful. It was not trained on your specific codebase, your specific generated/ folder that looks disposable but is not, or the one script in your repo where a "routine" bulk delete is actually catastrophic.

89% caught is an average across a study population, not a guarantee for any individual team's blast radius. A destructive action that looks routine in general (deleting files in a folder named tmp or build) may not trip the classifier even when, in your repo, that folder is load-bearing. Conversely, something narrow and specific to your setup that the classifier has never seen might get flagged when it is actually fine.

If you run agents against infrastructure with unusual failure modes, the safe move is still to pin an explicit permission mode rather than trust the default classifier to have learned your repo's edge cases. The classifier reduces the average number of things you have to think about. It does not remove the need to think about the things that are specific to you.

What actually shifted

This was framed as a UX improvement, fewer interruptions, faster flow. The more accurate framing is that Anthropic looked at its own approval funnel, found it was not catching anything a human was not already rubber-stamping, and moved the actual judgment call from a human reflex to a trained classifier. That is a better bet on the numbers given here. It is not the same as saying the judgment call is now being made well for every possible repo, it is being made by something with a measured track record instead of something with none.

Teams already running Claude Code semi-autonomous in CI or across a fleet of agents were building workarounds for exactly this gap. This is Anthropic catching up to how those teams were already using the tool, not adding a new capability.

Were you already running Claude Code with a pinned permission mode before this? What did you have to build around the old approve-every-call default?

🔥 The Real Latency Is the Human Handshake

Kyryl — Tue, 11 Aug 2026 20:13:12 +0000

Cross-team API changes are the real latency in microservices, and it is measured in weeks. Nobody puts that number on a dashboard because it does not show up in any trace, any p99, any APM tool. The network call is fast. The thing that is slow never touches the network at all.

The mechanism

Say you need one new field on an order object returned by another team's service. Maybe it is a discountCode, maybe it is a fulfillmentCenterId, does not matter. What matters is who owns the schema.

In a monolith, that field lives in a shared model. You open a PR, a teammate reviews it, it merges by lunch. One commit, one deploy, done.

Across a service boundary, the field lives in a service you do not control. The sequence looks like this:

You message the owning team, or file a ticket.
The ticket sits in their backlog until their next planning session.
Someone on that team estimates it, prioritizes it against their own roadmap, and schedules it.
They write the PR, review it, and ship it, on their timeline, not yours.
You write your own PR to consume the new field, once it exists.

Two weeks if the owning team is fast and the field is trivial. A quarter if it competes with their own sprint commitments, which it usually does, because your one field is nobody's priority but yours.

It is not a network problem

The instinct is to treat this as a technical latency problem, the same category as a slow query or a chatty API. It gets a technical response: better documentation, faster CI, an internal API gateway, more automation around service discovery.

None of that touches the actual bottleneck. The request-response cycle between the two services takes milliseconds whether the field exists or not. The delay lives entirely in the gap between "we filed the ask" and "the owning team scheduled the work." That gap is a queue, and it is a queue that belongs to a team whose backlog you have no authority over.

This is the part people miss: microservices did not introduce a network problem into your architecture. They introduced an organizational dependency graph, and every edge in that graph now has to be negotiated instead of just written. A monolith's "dependency" is a function call. A microservices "dependency" is a relationship between two teams' roadmaps.

A concrete version of the problem

Picture a checkout service that needs a loyaltyTier field from the customer service to compute a discount correctly. The checkout team does not own customer service. They file a request.

Ticket: Add `loyaltyTier` (enum: BRONZE, SILVER, GOLD) to GET /customers/{id}
Requested by: checkout-team
Priority: P2 (their P2, not yours)
Status: backlog

The checkout team cannot ship their discount logic until this lands. They are not blocked by a database, a deploy pipeline, or a load balancer. They are blocked by another team's sprint board. If the customer team is mid-migration, or short-staffed, or simply has three P1s ahead of it, the checkout team waits. There is no retry, no timeout, no circuit breaker for an organizational queue.

The honest trade-off

There are real fixes for this, and every one of them costs something up front. None of them are free lunches, and anyone pitching one as a strict win is skipping the cost column.

Embed a rep from the consuming team in the owning team's planning. This buys real speed: the field gets prioritized because a human advocating for it is sitting in the room. It costs headcount. Someone's calendar now has a recurring meeting with a team they do not report to, and someone's manager has to sign off on that time.

Go contract-first with a schema registry. Define the contract before either side writes code, validate changes against it in CI, and both teams catch breaking changes before they ship instead of after. This buys predictability. It costs discipline: every team touching the registry has to actually write contracts before code, review schema diffs like they review code diffs, and resist the shortcut of just shipping a field because it is faster this one time.

Build self-serve schema evolution for optional, additive fields. Let consuming teams add new optional fields to a schema without a round trip through the owning team's backlog, gated by validation rather than a human approval queue. This cuts the wait to near zero for the common case. It costs governance: someone has to build the tooling, define what counts as safely additive versus breaking, and maintain the guardrails so "self-serve" does not turn into "anyone can silently change the contract."

None of these fixes are cheap. All of them beat waiting a quarter for one field.

What actually changes

Adding a field is not slow because computers are slow. It is slow because prioritization is a human process, and cross-team prioritization has no SLA. The teams that ship fast across service boundaries are not the ones with the fastest network. They are the ones who paid the coordination cost before they needed the field, not after.

How long does a one-field ask actually take to land across a team boundary where you work, and which of these fixes, if any, does your org already run?

🔥 Two Services, One Table, Zero Isolation

Kyryl — Mon, 10 Aug 2026 21:39:57 +0000

Two services writing to the same table is a distributed monolith wearing a microservice badge. You split the codebase, you kept the database, and the split you actually shipped is cosmetic. Every cost of the split is real. The isolation you were supposed to get for that cost is not.

The decision that does not feel like an architecture decision

Nobody sits in a design review and votes to build a distributed monolith. It happens one convenience call at a time. Team A owns orders, team B needs order data for its own service, and the fastest way to get it is to point B's code at the same Postgres instance and the same orders table. No new endpoint to build, no contract to agree on, no deploy to coordinate. It ships this sprint instead of next quarter.

That decision gets filed under data access. It is not a data-access decision. It is the architectural boundary, decided by default, by whoever wrote the first query. The table has one schema, and from that moment both services are load-bearing on that schema staying exactly the way it is. Nobody wrote that dependency down anywhere. It lives in two codebases that do not import each other and cannot see each other's queries.

What the coupling looks like when it breaks

Here is the incident, not hypothetically, the way it actually shows up.

Service A owns the table by every reasonable definition, it was there first, its team migrated the schema originally. A needs to ship a feature. The feature needs a new required field, so someone adds:

ALTER TABLE orders ADD COLUMN fulfillment_channel VARCHAR(32) NOT NULL DEFAULT 'standard';

A's migration tooling runs it, A's tests pass, because A's tests only ever insert through A's own code path, which now always sets fulfillment_channel. A's deploy goes out clean. Nobody on A's team thinks about service B, because as far as A's team is concerned, B is not part of this change. A does not know B writes to this table at all, or if it does, nobody flagged this migration for review.

Service B writes to orders directly, from its own repository, using its own hand-rolled insert statement that predates the new column entirely. Two things can happen next, and both are bad.

If the column had no default, B's inserts start failing immediately with a constraint violation. That is the good outcome, loud and fast, a page goes off, someone finds the cause within the hour.

The worse outcome is what actually happened above: a default value. B's inserts keep succeeding. Every row B writes silently gets fulfillment_channel = 'standard', whether or not that is true. No error. No failed health check. No alert. The data is just wrong, quietly, for every order B touches, from the moment A's migration landed until someone downstream notices the fulfillment numbers do not add up. That could be hours. It is more often days, and by the time someone in the incident review asks "wait, since when has B been writing that channel," the wrong data is already in reports, already fed into whatever dashboard finance trusts.

Nobody connects the two events, the migration and the corrupted writes, because nothing in either codebase points at the other. The only shared artifact is the table itself, and tables do not send review requests.

Why splitting the codebase did not split the coupling

The instinct once you are burned by this is to blame the migration, add a review step, require anyone touching a shared table to ping the other team first. That is a process patch on an architecture problem, and it will hold until someone forgets, or a new hire does not know the informal rule exists, or the "other team" has been renamed twice since the rule was written down in a wiki nobody reads.

The actual problem is that two services with independent deploys, independent on-call, independent codebases are still, underneath all of that, one system with one shared piece of mutable state. A network call between them would at least force a versioned contract, something with a schema of its own that changes deliberately, gets reviewed, gets a deprecation window. A shared table has no such contract. Its schema is the contract, enforced by nothing except discipline, and discipline is the thing that fails first under deadline pressure.

You paid for the split. Two deploy pipelines, two on-call rotations, a network hop between the two halves whenever one calls the other for anything else. What you did not get, in exchange for that cost, is the one thing the split was supposed to buy: a boundary strong enough that one team's change cannot silently break the other. That is what "distributed monolith" means in practice. Every operational cost of microservices, none of the isolation.

The honest trade-off

There are two real fixes, and neither one is free.

The first: pick one service as the actual owner of the table, and every other consumer goes through that owner's API. Whatever queries B was running directly against orders become HTTP or gRPC calls to A. A's schema changes are now A's problem to manage behind a versioned interface, and B is protected from anything A does internally as long as the contract holds. The cost is real: you have to build that API surface, migrate every direct query B has, and accept the latency and availability coupling that comes with a synchronous call replacing a local join.

The second: actually split the data. B gets its own table, its own copy of whatever fields it needs, kept in sync through an event stream, A publishes an OrderUpdated event whenever the row changes, B consumes it and updates its own copy. Now B genuinely does not depend on A's schema, A can rename columns internally all day without touching B. What you buy instead is eventual consistency: B's copy is sometimes a few seconds or minutes behind, and now you own a second, denormalized dataset that can drift from the source if the event pipeline ever drops a message. Reconciliation becomes its own ongoing job.

Neither of these is a migration you do over a sprint. Both are projects: mapping every access pattern the "other" service currently has against the shared table, deciding what actually needs to move, coordinating a cutover that does not lose writes in the transition. It is slower and more expensive than the shared table ever was, which is exactly why the shared table happened in the first place. It was the fast option. It is just not the option that gives you the isolation you were charging yourself for.

How many "microservices" in your own systems still share write access to a table, and has anyone actually mapped what breaks if either side changes the schema?

🧹 Kafka Is Deleting the Wrong Data: Retention vs Compaction

Kyryl — Mon, 27 Jul 2026 19:22:06 +0000

Kafka is a log, and logs grow. At some point the broker has to reclaim disk, and it has two completely different ways to do it. They are not variations on a theme. They delete on different rules, and confusing them is a quiet data-loss bug that looks like corruption weeks later.

Retention: delete by age

Retention is a stopwatch. You set a window, and once data is older than that window it is deleted, segment by segment, regardless of what is inside.

log.retention.hours=168      # delete anything older than 7 days
log.retention.bytes=-1       # or cap by size per partition

Kafka stores a partition as a sequence of segment files. Retention works at segment granularity: when the newest record in a segment is older than the window (or the partition is over its byte cap), the whole segment is dropped. It never looks at record keys. It does not care whether a key still matters. Old is old.

This is exactly right for a stream of events. A click from last month, a sensor reading from an hour ago, a request log from yesterday. The value is in the recent flow; the old entries can leave.

Compaction: keep the latest per key

Compaction is a dedupe. Instead of asking "how old is this?", it asks "is there a newer record with the same key?" If yes, the older one can go.

cleanup.policy=compact

With compaction, Kafka guarantees that for every key, the latest record is retained. Older records for that same key are eventually removed by the log cleaner. A key that is written once and never updated survives forever, no matter how old it is. That is the crucial difference from retention.

This is exactly right for state. A changelog topic, a Kafka Streams KTable, a database outbox, a topic that materializes "current value per entity". The whole point is that a consumer can replay the topic from the beginning and rebuild the current state of every key.

You can also combine them:

cleanup.policy=compact,delete

Now the latest value per key is kept, but truly ancient records can still age out. Useful for a changelog where very old, untouched keys are acceptable to drop.

The incident this prevents

Here is the failure I have watched happen more than once.

Someone models current entity state as a Kafka topic: key is the entity id, value is its latest state. A downstream service rebuilds a local view by consuming the topic from offset zero. Works perfectly in testing.

The topic was left on the default cleanup.policy=delete with a 7-day retention. Most entities update often, so nobody notices. Then a slow-moving entity goes untouched for eight days. Its only record ages out. Now when the downstream service rebuilds, that entity is simply missing. No delete event, no error, no log line. It looks like data corruption, and you will spend a day chasing a bug that is really one config line.

The fix is one property: that topic should have been compact.

The honest trade-off

Compaction is not free, and it is not a drop-in for retention.

The log cleaner spends CPU and I/O rewriting segments. On a high-churn topic that is real, ongoing work.
Compaction only bounds size by number of distinct keys, not by time. A topic with an unbounded key space (say, keying by request id) under compact will grow without limit, because every key is unique and nothing is a duplicate. That is a disk-filling trap in the opposite direction.
Deletes in a compacted topic need tombstones: a record with the key and a null value. Miss that and a "deleted" key lives forever.

So the rule is not "compaction is better". It is: retention for events, compaction for state, compact,delete when state has a long tail you are willing to drop, and never compact on an unbounded key space.

Takeaway

Retention deletes by time. Compaction keeps the latest per key. They solve different problems, and the default (delete) is the wrong choice for any topic that represents state.

Go look at your most important topic. Is it delete or compact, and is that on purpose or just the default nobody changed?

🗑️ Kafka Keeps Data Longer Than retention.ms

Kyryl — Mon, 20 Jul 2026 18:52:46 +0000

You set retention.ms to one hour. Six hours later the data is still on disk. That is not a bug, and it is not your cleanup thread being slow.

Kafka deletes data very differently from how most people picture it.

Kafka deletes segments, not records

A partition is not one big file. It is an ordered sequence of segments, each a file on disk. Retention operates at the segment level, never at the record level.

Kafka will never open a segment and rewrite it to drop a handful of expired records. That would be expensive and would break the append-only design the whole system is built on. Instead it waits until an entire segment is eligible, then deletes the whole file in one move.

When a segment is actually eligible

A closed segment becomes eligible for deletion only when both are true:

The segment is closed (rolled). A segment rolls when it fills up (segment.bytes, default 1 GB) or ages out (segment.ms, default 7 days).
Its newest record is older than retention.ms (or the partition is over retention.bytes).

Note it is the newest record in the segment that has to age out, not the oldest. One young record keeps the whole segment, and every older record in it, alive.

The active segment never dies

Here is the part that surprises people, and the reason your data outlives its retention.

The active segment, the one currently being written, is never eligible for deletion, no matter what. Retention only ever considers closed segments.

So on a low-traffic topic, the active segment fills slowly and rolls rarely. The oldest record sitting in it can be hours or days past retention.ms, just waiting for the segment to finally roll so it can even be considered for deletion.

[ segment 0 ][ segment 1 ][ segment 2 ][ segment 3 (active) ]
   closed       closed       closed        being written
   deletable    deletable    deletable     NEVER deletable
   once past retention.ms                  regardless of age

Your retention is effectively retention.ms plus however long it takes the active segment to roll.

The honest trade-off

The fix is not to lower retention.ms further. That does nothing while the segment has not rolled. Lower segment.ms so segments roll more often and become deletable sooner.

But smaller segments are not free: more files, more open file handles, more frequent rolls, and more index overhead. On a high-throughput topic the default 1 GB / 7 day segments are fine and you will never notice this. It only bites on low-traffic topics with a tight retention expectation: compliance windows, PII deletion SLAs, "we only keep 24 hours" promises. Those are exactly the cases where the gap matters, so size the segment to the retention you actually need.

This is separate from log compaction (cleanup.policy=compact), which keeps the latest value per key instead of deleting by age. Same segment mechanics underneath, different eligibility rule.

Have you ever had data outlive its configured retention on a quiet topic, and traced it back to a segment that just never rolled?

🔀 Your Kafka Group Freezes on Every Deploy

Kyryl — Thu, 16 Jul 2026 19:09:08 +0000

You have a Kafka consumer group with four consumers, humming along. Traffic climbs, so you add a fifth. For a few seconds, every consumer stops. Lag spikes. Then it recovers and nobody thinks about it again.

That pause is not a bug. It is eager rebalancing, and most groups still run it by default.

Rebalancing is not the enemy

A consumer group rebalances whenever membership changes: a consumer joins, leaves, crashes, or the topic gains partitions. The group has to agree on who owns which partitions. That part is unavoidable.

The cost is not the reassignment. It is how the group gets there. That is controlled by partition.assignment.strategy, and the choice you make there decides whether a deploy costs you a few milliseconds or a few seconds of frozen consumption.

The four assignors

RangeAssignor (the historical default) assigns each consumer a contiguous range of partitions, per topic.

6 partitions, 3 consumers:
A -> P0 P1   B -> P2 P3   C -> P4 P5

Clean when the numbers divide. The moment they do not, the lower-id consumers get the extra partitions, and if you subscribe to several topics they pile up on the same consumers.

RoundRobinAssignor deals every partition out one by one across all consumers.

6 partitions, 4 consumers:
A -> P0 P4   B -> P1 P5   C -> P2   D -> P3

Even spread. But it has no memory. Every rebalance recomputes the whole assignment from scratch, so partitions jump around even when they did not need to.

StickyAssignor keeps the spread even and tries to preserve the previous assignment, so it moves as few partitions as possible on each rebalance.

CooperativeStickyAssignor does the same assignment as sticky, but over a different rebalance protocol. This is the one that matters.

Eager vs cooperative: the real difference

Here is the part that trips people up. Sticky reduces how many partitions move. It does not change what happens to the ones that stay.

Range, round-robin, and plain sticky all use the eager protocol. On every rebalance the group does a full stop-the-world:

Every consumer revokes all of its partitions.
Nobody consumes anything.
The new assignment is computed.
Consumers pick their partitions back up.

So even a partition that ends up on the exact same consumer still gets revoked and paused. Add one consumer to a group of ten and all of them stop, for every partition, until the dust settles.

Cooperative-sticky (KIP-429, Kafka 2.4+) rebalances incrementally:

Only the partitions that actually need to move are revoked.
Every other partition keeps being consumed, right through the rebalance.
The moved partitions get reassigned in a second, short round.

One consumer joins, one or two partitions pause for a moment, and the rest of the group never notices. Stop-the-world becomes stop-one-partition.

props.put(
    ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
    List.of(CooperativeStickyAssignor.class.getName()));

You cannot just flip the config

This is the trap. You cannot take a running group on the eager protocol and switch it to cooperative in a single deploy. A group with some members speaking eager and some speaking cooperative will not rebalance correctly.

The supported path is a two-phase rolling upgrade:

// Phase 1: deploy every instance with BOTH strategies listed.
// The group stays on the old protocol until all members support the new one.
props.put(
    ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
    List.of(CooperativeStickyAssignor.class.getName(),
            RangeAssignor.class.getName()));

// Phase 2: after every instance is running phase 1, deploy again
// with only the cooperative strategy. Now the group flips protocols.
props.put(
    ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
    List.of(CooperativeStickyAssignor.class.getName()));

Skip phase 1 and you get a broken rebalance in production. Two deploys, in order, with the whole fleet on phase 1 before phase 2 starts.

The honest trade-off

Cooperative-sticky is not free.

The migration is a two-step rolling upgrade you have to get right, not a one-line change.
Incremental rebalancing can take more rounds than a single eager rebalance, so the total rebalance can be longer even though nobody is fully stopped.
Your onPartitionsRevoked and onPartitionsLost callbacks need to be correct, because partitions now come and go without a global reset to lean on.

For a group that rebalances rarely and briefly, the eager pause may be cheap enough to ignore. For a group that scales, deploys, or loses instances often, the stop-the-world pause is a tax you pay on every event, and cooperative-sticky removes it.

Which assignor does your consumer group run today, and have you ever actually measured how long a rolling deploy freezes it?

🔄 Your Database Is Already an Event Stream

Kyryl — Thu, 09 Jul 2026 20:35:01 +0000

You have a service that owns the orders table. Three other teams need to know when an order changes. So you do the obvious thing: after every write, you publish an event to Kafka.

Then someone adds a new write path and forgets the publish. Now the order exists in the database but no event ever fired. The downstream read models drift. You find out three weeks later when finance asks why the numbers do not match.

This is the dual-write problem, and you cannot discipline your way out of it. Every write path is a place someone forgets to emit the event.

There is a cleaner option. Stop publishing events by hand. Let the database do it.

The database already knows what changed

Change Data Capture (CDC) turns your database into an event source. Debezium reads the write-ahead log, the same log Postgres already uses for replication and crash recovery, and turns every committed row change into a Kafka event.

Point a connector at a table. Every insert, update, and delete becomes a message on a topic. Downstream services build their own read models from that stream and never query your tables directly.

The write already happened. The WAL already recorded it. There is nothing to remember and nothing to enforce in application code, because the event is a byproduct of the commit, not a second action you have to bolt onto every write path.

Wiring it up

A Debezium Postgres connector is a bit of connector config, not application code:

{
  "name": "orders-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres",
    "database.dbname": "shop",
    "plugin.name": "pgoutput",
    "table.include.list": "public.orders",
    "topic.prefix": "shop"
  }
}

Changes to public.orders now land on the shop.public.orders topic. Each message carries a before and after image of the row plus an op field: c for create, u for update, d for delete. A consumer decides what to do with each:

@KafkaListener(topics = "shop.public.orders")
public void onChange(ChangeEvent event) {
    switch (event.op()) {
        case "c", "u" -> readModel.upsert(event.after());
        case "d"      -> readModel.remove(event.before().id());
    }
}

No coordination with the owning service. No shared library. The consumer subscribes and builds exactly the projection it needs.

The outbox pattern is CDC with intent

Tailing the whole WAL means downstream consumers see your table exactly as it is. Sometimes that is fine. Sometimes you want to publish a deliberate event, not a raw row.

The outbox pattern is the same mechanism applied on purpose. Inside the same transaction as your business write, you insert a row into an outbox table shaped like the event you want the world to see:

BEGIN;
UPDATE orders SET status = 'SHIPPED' WHERE id = 42;
INSERT INTO outbox (aggregate, event_type, payload)
VALUES ('order', 'OrderShipped', '{"orderId": 42}');
COMMIT;

Debezium tails the outbox table instead of orders. The event is atomic with the business change because they share one transaction, and its shape is something you designed rather than whatever columns the table happens to have today.

CDC is the mechanism either way. The outbox just decides what becomes an event and what stays internal.

The honest trade-off

Here is the part nobody puts on the slide.

Raw-table CDC couples your event stream to the table's physical shape. The moment a downstream consumer maps shop.public.orders, your column names are a public contract, whether you meant them to be or not.

That means column renames and other non-backwards-compatible schema changes are off the table for you. Rename status to order_status during an internal refactor and every downstream mapper breaks. Silently. There is no compile error, no failed deploy on your side. The consumer just starts reading null because nobody told it the table changed.

Tailing the raw WAL is convenient right up until the day your schema is no longer just yours.

Shape the output on purpose

The fix is not avoiding CDC. It is refusing to treat the current table shape as an accident that leaks downstream.

Single Message Transforms (SMTs): rename, drop, or restructure fields in the connector pipeline, so an internal column rename does not change the published event.
A CDC-facing view or outbox table: publish from a surface you designed, and let the physical table underneath change freely.
A schema registry: enforce compatibility rules on the topic, so a breaking change fails loudly at publish time instead of silently downstream.

All three do the same thing: they put a deliberate boundary between your storage and your stream. The database can still emit every change for free. You just decide what that change looks like on the wire.

CDC is not "replication with extra steps." It is a coupling decision. Either downstream services couple to your database's physical shape, or they couple to a contract you shaped on purpose. Pick the second one.

Do you use Debezium to emit events? Do you tail the raw WAL or shape it through an outbox first, and has an internal refactor ever quietly broken a downstream consumer on you?

🔁 One Health Check Turned a 10-Minute Outage Into 40

Kyryl — Mon, 06 Jul 2026 18:05:44 +0000

The database went down for ten minutes. The restart storm it triggered lasted forty.

Every pod's /actuator/health turned red the moment the database became unreachable. Kubernetes read that as "the process is broken" and started killing pods, one after another, across the entire fleet. New pods booted, connected to nothing, failed the same check, and got killed again. The dependency was down for ten minutes. The self-inflicted damage ran four times longer.

The trigger was one line of config that looked like a simplification.

The setup that looks clean

Someone wired both the liveness and the readiness probe to the same endpoint.

livenessProbe:
  httpGet:
    path: /actuator/health
    port: 8080
readinessProbe:
  httpGet:
    path: /actuator/health
    port: 8080

One endpoint, one thing to configure, one thing to reason about. It reads as clean. It is actually a category error, because those two probes are asking completely different questions and Spring's default /actuator/health answers neither of them cleanly.

Two probes, two questions

The distinction is the whole point, so it is worth stating flatly.

Liveness answers: should I kill this process and start a new one? The only honest reason to say yes is that the process is wedged in a way a restart will fix. A deadlock, an unrecoverable internal state, a JVM that is thrashing. Restarting helps only when the problem lives inside the process.

Readiness answers: should I send this pod traffic right now? A pod can be perfectly alive and still be unable to serve, because something it depends on is unavailable. The fix there is not a restart. It is to stop routing requests until the dependency comes back.

/actuator/health by default aggregates everything, including the db health indicator, into one status. Point liveness at it and you have told Kubernetes to kill the process whenever the database is down. But a down database is not a process problem, and no number of restarts will bring it back.

The restart storm, step by step

Here is the loop that ate thirty extra minutes:

The database becomes unreachable.
The db health indicator goes DOWN, so aggregated /actuator/health returns DOWN.
The liveness probe fails. Kubernetes kills the pod.
A fresh pod boots, starts up, and immediately checks its health.
The database is still down, so the new pod's health is DOWN too.
Liveness fails again. Kubernetes kills it again.
Repeat, on every pod, for as long as the dependency stays down.

Now the failure is worse than the original outage. The pods are gone, so even reads that could have been served from cache are gone. Startup load hammers the recovering database the moment it comes back. And your dashboards are a wall of CrashLoopBackOff that makes it look like the application itself is broken, which sends everyone debugging the wrong thing.

What should have happened is boring by comparison. Readiness fails, Kubernetes pulls the pods out of the Service endpoints, traffic stops, the processes keep running. When the database returns, readiness goes green and traffic resumes. No restarts, no cold caches, no thundering herd.

The fix Actuator already ships

You do not need a custom endpoint or a sidecar. Spring Boot Actuator has health groups built for exactly this split.

management:
  endpoint:
    health:
      probes:
        enabled: true
      group:
        liveness:
          include: livenessState
        readiness:
          include: readinessState,db

That gives you two dedicated endpoints:

/actuator/health/liveness includes only livenessState, the process-internal signal. No database, no downstream calls. It answers "is this process wedged" and nothing else.
/actuator/health/readiness includes readinessState plus the real dependency checks, db here. It answers "can this pod serve traffic right now."

Point the probes at the right endpoints:

livenessProbe:
  httpGet:
    path: /actuator/health/liveness
    port: 8080
readinessProbe:
  httpGet:
    path: /actuator/health/readiness
    port: 8080

Now a database outage trips readiness only. Traffic drains, the pods live, and recovery is automatic. Liveness stays green because the process is, in fact, fine.

The honest trade-off

This is not a set-and-forget win, and pretending it is misses the way the bug comes back.

The config takes five minutes and then nobody looks at it again. That is the problem. Over the next year the service grows: a Redis cache, a Kafka producer, a call to some downstream API. Each of those registers a health indicator, and each is a decision about which group it belongs in. If a new dependency lands in the readiness group, great. If it silently ends up aggregated into liveness, or if someone adds a custom indicator without thinking about probes at all, you have quietly rebuilt the original bug with a new trigger.

The drift is invisible until the next outage. Nobody audits "which health indicators feed which probe" as part of adding a dependency, so the wrong wiring sits there dormant for months. The next time that specific dependency fails, the restart storm returns, and it looks brand new even though it is the same mistake.

The mitigation is process, not code: when you add a dependency with a health indicator, decide its group in the same PR. Treat "which probe does this affect" as part of the definition of done for any new external call.

Has your liveness probe ever quietly grown a dependency check nobody noticed, or has your fleet actually lived through a restart storm like this? Curious how other teams keep the liveness group honest as services accumulate dependencies.

🔒 CREATE INDEX Is a Write Outage in Disguise

Kyryl — Mon, 06 Jul 2026 17:55:29 +0000

CREATE INDEX without CONCURRENTLY locks out writes for the entire build.

On a large, live table that plain statement is not a migration. It is a multi-minute write outage for every endpoint touching that table. And the worst part is it sails through every test you have, because your test data is tiny and the build finishes before you can blink.

The lock nobody reads about

A plain CREATE INDEX acquires a SHARE lock on the table and holds it until the build completes.

CREATE INDEX idx_orders_customer_id ON orders (customer_id);

SHARE allows concurrent reads. It blocks anything that writes: INSERT, UPDATE, DELETE. Every write against that table queues up behind the index build and waits. Not for a moment at the start. For the entire duration.

On a small table that duration is milliseconds, so nothing queues and nobody notices. That is exactly why this bug survives code review and staging. The table you tested against had ten thousand rows. The table in production has two hundred million, and building the index on it takes four minutes. For those four minutes, every write to orders is stalled.

The application does not error. It hangs. Requests pile up, connection pools drain, timeouts cascade to services that never touched the database directly. From the outside it looks like a full outage, and the migration that caused it already "succeeded" in your CI.

The fix

CREATE INDEX CONCURRENTLY was built for exactly this.

CREATE INDEX CONCURRENTLY idx_orders_customer_id ON orders (customer_id);

It does not take the blocking SHARE lock. It builds the index in the background, scanning the table while writes keep flowing the whole time. Reads and writes both continue. This is the default you want for any table that is already taking production traffic.

It is not free

CONCURRENTLY trades a fast, clean build for a slower, riskier one. Three costs, all real:

It is slower. To build without blocking writes, Postgres does two full passes over the table instead of one. One pass to build, a second to catch rows that changed during the first. On a huge table that roughly doubles the build time. You are trading wall-clock time for availability, which is almost always the right trade, but it is a trade.

It cannot run inside a transaction block. This is the one that breaks migration tooling. Most migration frameworks wrap each migration in a transaction by default, and CREATE INDEX CONCURRENTLY will error out if it finds itself inside one. You cannot bundle it with other DDL in the same atomic step. It has to run on its own, outside a transaction, which means you lose the "all or nothing" guarantee for that migration.

It can fail into an INVALID index. This is the sharp edge. If a CONCURRENTLY build fails partway, a deadlock, a cancelled session, a statement timeout, Postgres leaves a half-built index behind and marks it INVALID. It does not roll back. It does not clean up. That index is not used by the planner, but it is still updated on every write and still consumes disk. It just sits there, dead weight, until a human notices.

You find them by asking the catalog directly:

SELECT c.relname AS index_name,
       t.relname AS table_name
FROM pg_index i
JOIN pg_class c ON c.oid = i.indexrelid
JOIN pg_class t ON t.oid = i.indrelid
WHERE i.indisvalid = false;

Anything that comes back has to be dropped and rebuilt:

DROP INDEX CONCURRENTLY idx_orders_customer_id;
-- then re-run the CREATE INDEX CONCURRENTLY

If you run migrations through automation, a failed CONCURRENTLY will not retry cleanly either, because the invalid index now occupies the name you are trying to create. Your retry fails with "relation already exists" until someone drops the corpse.

The one real exception

"Always use CONCURRENTLY in production" is a good rule with exactly one honest exception: a brand new, empty table you just created in the same migration.

CREATE TABLE audit_log (...);
CREATE INDEX idx_audit_log_created_at ON audit_log (created_at);  -- fine

Nothing is writing to audit_log yet. There are no concurrent writes to block, so the SHARE lock costs nothing, and the plain build is faster and runs cleanly inside the same transaction as the CREATE TABLE. Reaching for CONCURRENTLY here would be cargo-culting the rule past the point where it applies.

The tell is simple: if the table already has live traffic, use CONCURRENTLY. If you are indexing something that does not exist yet, do not bother.

The honest trade-off

CONCURRENTLY is not a free upgrade you flip on and forget. You trade a guaranteed-fast, guaranteed-clean, transaction-safe build for a slower one that runs outside your transaction and can fail into a mess you have to notice and clean up yourself.

For any table with real traffic, that trade is worth it every time. A slower build you have to babysit beats a multi-minute outage you cannot. But "worth it" is not "free," and pretending the invalid-index failure mode does not exist is how teams get surprised at 2am.

Has anyone actually gotten burned by an invalid index left behind after a failed CONCURRENTLY build? How did you find it, and do you check pg_index as part of your migration process or only after something breaks?

📈 One Path Variable Can Bankrupt Your Prometheus

Kyryl — Mon, 06 Jul 2026 17:30:00 +0000

A path variable in your metrics tag will quietly bankrupt your Prometheus backend.

It passes code review. It compiles. It works, for a while. Then one day the dashboards start timing out, and it takes the team days to trace the slowdown back to a single line of instrumentation that looked completely normal.

The line that looks fine

Here is the kind of code that ships this problem. A Timer around an order lookup, tagged with the orderId so you can slice latency per request.

@GetMapping("/api/orders/{orderId}")
public Order getOrder(@PathVariable String orderId) {
    return Timer.builder("order.lookup")
            .tag("orderId", orderId)   // <-- the problem
            .register(meterRegistry)
            .record(() -> orderService.findById(orderId));
}

Nothing about this fails a review. It reads as "measure how long an order lookup takes, and let me break it down by order." Reasonable intent. The metric even works when you test it locally with three orders.

What actually happens

Prometheus is a time series database. The identity of a time series is its metric name plus the full set of label key-value pairs. Change one label value and you do not add a point to an existing series. You create a brand new series.

So order.lookup{orderId="1001"} and order.lookup{orderId="1002"} are two completely separate series, each with its own storage, its own index entry, its own memory footprint.

Now run that in production. Every distinct orderId that flows through the endpoint mints a new series. A million orders means a million series from this one metric. Add a userId tag somewhere else and the counts multiply. This is cardinality explosion, and /actuator/prometheus will happily expose all of it.

The failure is gradual, which is what makes it nasty:

Storage grows far faster than your request volume would suggest.
The Prometheus head block balloons, memory pressure climbs, scrape durations creep up.
Queries that touch the metric get slow, then time out.
Someone files "Prometheus is slow" or "the orders dashboard times out."

Notice what is missing from that chain: nobody says "the orderId tag is the problem." The symptom is three steps removed from the cause. I have watched a team spend the better part of a week bisecting scrape configs and bumping memory limits before someone finally ran a cardinality check and found one metric responsible for millions of series.

You can catch it directly once you suspect it:

# top metrics by series count
topk(10, count by (__name__)({__name__=~".+"}))

But you have to suspect it first, and the whole point is that nothing pointed you there.

The fix is not a new tool

The instinct is to reach for a sampling library or a relabeling rule to drop the bad label. You do not need either. Spring already hands you the right value.

For every request, Spring resolves the matched route pattern, the same mechanism its built-in HTTP server metrics (http.server.requests) use to keep their uri tag bounded. The best-matching pattern is available on the request as an attribute.

@GetMapping("/api/orders/{orderId}")
public Order getOrder(@PathVariable String orderId, HttpServletRequest request) {
    String pattern = (String) request.getAttribute(
            HandlerMapping.BEST_MATCHING_PATTERN_ATTRIBUTE); // "/api/orders/{orderId}"

    return Timer.builder("order.lookup")
            .tag("route", pattern)     // bounded to the number of routes
            .register(meterRegistry)
            .record(() -> orderService.findById(orderId));
}

Now the tag value is /api/orders/{orderId}, the template, not the resolved id. Every request through this endpoint lands on the same series. Cardinality for this metric is bounded to the number of routes you have, which is a small, fixed number that does not grow with traffic.

If all you wanted was per-endpoint latency, you may not need a custom timer at all. http.server.requests already gives you templated URI, method, and status out of the box. Reach for a custom metric only when you need a dimension the built-in one does not expose, and when you do, tag it with something bounded.

The honest trade-off

Templating the URI collapses per-entity granularity in the metric itself. That is a real loss, not a footnote.

Once the tag is /api/orders/{orderId}, the metric can no longer answer "how slow was the lookup for order 1002 specifically." It can only tell you about the route as a whole: p50, p99, error rate across all orders.

If you genuinely need to investigate one entity, that is now a logs or traces question, not a metrics question. Attach the orderId to a span or a structured log field, where high cardinality is expected and the backend is built for it. Metrics are for bounded, aggregatable dimensions. Traces and logs are for the long tail of individual cases.

That split is the right one. Metrics answer "is the system healthy and how is this route trending." Traces answer "what happened to this one request." Putting a unique id in a metric tag is asking metrics to do the job of tracing, and the bill for that mistake is paid by your storage backend, quietly, until it is not quiet anymore.

The rule

Any tag value that is not drawn from a small, known set does not belong on a metric. User ids, order ids, request ids, email addresses, raw paths: all of them are cardinality bombs. Route patterns, status codes, HTTP methods, enum-like states: all fine.

The tell is simple. Before you add a tag, ask how many distinct values it can take over the life of the service. If the answer scales with your traffic, you have found the bug before it finds you.

Has your Prometheus setup ever fallen over from a label nobody flagged, and how long did it take to trace it back to the metric? Curious how other teams caught it, and what guardrails you put in place afterward.

DEV Community: Kyryl

Your Familiar Java Classes Just Lost Identity 🧬

What a value class actually changes

Why value classes exist at all

The first thing that breaks: ==

The second thing that breaks: synchronized

The honest trade-off

The Shared Cache Tier Is Disappearing 🗄️

The problem: revocation has to be fast and shared

What Canva built instead

The pattern, not just the case study

The honest trade-off

Where this fits

97% of Your AI Approval Clicks Were Reflexes 🤖

The 97% number is the actual story

What "irreversible, destructive, outward-facing" likely means in practice

The honest trade-off

What actually shifted

🔥 The Real Latency Is the Human Handshake

The mechanism

It is not a network problem

A concrete version of the problem

The honest trade-off

What actually changes

🔥 Two Services, One Table, Zero Isolation

The decision that does not feel like an architecture decision

What the coupling looks like when it breaks

Why splitting the codebase did not split the coupling

The honest trade-off

🧹 Kafka Is Deleting the Wrong Data: Retention vs Compaction

Retention: delete by age

Compaction: keep the latest per key

The incident this prevents

The honest trade-off

Takeaway

🗑️ Kafka Keeps Data Longer Than retention.ms

Kafka deletes segments, not records

When a segment is actually eligible

The active segment never dies

The honest trade-off

🔀 Your Kafka Group Freezes on Every Deploy

Rebalancing is not the enemy

The four assignors

Eager vs cooperative: the real difference

You cannot just flip the config

The honest trade-off

🔄 Your Database Is Already an Event Stream

The database already knows what changed

Wiring it up

The outbox pattern is CDC with intent

The honest trade-off

Shape the output on purpose

🔁 One Health Check Turned a 10-Minute Outage Into 40

The setup that looks clean

Two probes, two questions

The restart storm, step by step

The fix Actuator already ships

The honest trade-off

🔒 CREATE INDEX Is a Write Outage in Disguise

The lock nobody reads about

The fix

It is not free

The one real exception

The honest trade-off

📈 One Path Variable Can Bankrupt Your Prometheus

The line that looks fine

What actually happens

The fix is not a new tool

The honest trade-off

The rule

The first thing that breaks: `==`

The second thing that breaks: `synchronized`