DEV Community: Kyryl

🗑️ Kafka Keeps Data Longer Than retention.ms

Kyryl — Mon, 20 Jul 2026 18:52:46 +0000

You set retention.ms to one hour. Six hours later the data is still on disk. That is not a bug, and it is not your cleanup thread being slow.

Kafka deletes data very differently from how most people picture it.

Kafka deletes segments, not records

A partition is not one big file. It is an ordered sequence of segments, each a file on disk. Retention operates at the segment level, never at the record level.

Kafka will never open a segment and rewrite it to drop a handful of expired records. That would be expensive and would break the append-only design the whole system is built on. Instead it waits until an entire segment is eligible, then deletes the whole file in one move.

When a segment is actually eligible

A closed segment becomes eligible for deletion only when both are true:

The segment is closed (rolled). A segment rolls when it fills up (segment.bytes, default 1 GB) or ages out (segment.ms, default 7 days).
Its newest record is older than retention.ms (or the partition is over retention.bytes).

Note it is the newest record in the segment that has to age out, not the oldest. One young record keeps the whole segment, and every older record in it, alive.

The active segment never dies

Here is the part that surprises people, and the reason your data outlives its retention.

The active segment, the one currently being written, is never eligible for deletion, no matter what. Retention only ever considers closed segments.

So on a low-traffic topic, the active segment fills slowly and rolls rarely. The oldest record sitting in it can be hours or days past retention.ms, just waiting for the segment to finally roll so it can even be considered for deletion.

[ segment 0 ][ segment 1 ][ segment 2 ][ segment 3 (active) ]
   closed       closed       closed        being written
   deletable    deletable    deletable     NEVER deletable
   once past retention.ms                  regardless of age

Your retention is effectively retention.ms plus however long it takes the active segment to roll.

The honest trade-off

The fix is not to lower retention.ms further. That does nothing while the segment has not rolled. Lower segment.ms so segments roll more often and become deletable sooner.

But smaller segments are not free: more files, more open file handles, more frequent rolls, and more index overhead. On a high-throughput topic the default 1 GB / 7 day segments are fine and you will never notice this. It only bites on low-traffic topics with a tight retention expectation: compliance windows, PII deletion SLAs, "we only keep 24 hours" promises. Those are exactly the cases where the gap matters, so size the segment to the retention you actually need.

This is separate from log compaction (cleanup.policy=compact), which keeps the latest value per key instead of deleting by age. Same segment mechanics underneath, different eligibility rule.

Have you ever had data outlive its configured retention on a quiet topic, and traced it back to a segment that just never rolled?

🔀 Your Kafka Group Freezes on Every Deploy

Kyryl — Thu, 16 Jul 2026 19:09:08 +0000

You have a Kafka consumer group with four consumers, humming along. Traffic climbs, so you add a fifth. For a few seconds, every consumer stops. Lag spikes. Then it recovers and nobody thinks about it again.

That pause is not a bug. It is eager rebalancing, and most groups still run it by default.

Rebalancing is not the enemy

A consumer group rebalances whenever membership changes: a consumer joins, leaves, crashes, or the topic gains partitions. The group has to agree on who owns which partitions. That part is unavoidable.

The cost is not the reassignment. It is how the group gets there. That is controlled by partition.assignment.strategy, and the choice you make there decides whether a deploy costs you a few milliseconds or a few seconds of frozen consumption.

The four assignors

RangeAssignor (the historical default) assigns each consumer a contiguous range of partitions, per topic.

6 partitions, 3 consumers:
A -> P0 P1   B -> P2 P3   C -> P4 P5

Clean when the numbers divide. The moment they do not, the lower-id consumers get the extra partitions, and if you subscribe to several topics they pile up on the same consumers.

RoundRobinAssignor deals every partition out one by one across all consumers.

6 partitions, 4 consumers:
A -> P0 P4   B -> P1 P5   C -> P2   D -> P3

Even spread. But it has no memory. Every rebalance recomputes the whole assignment from scratch, so partitions jump around even when they did not need to.

StickyAssignor keeps the spread even and tries to preserve the previous assignment, so it moves as few partitions as possible on each rebalance.

CooperativeStickyAssignor does the same assignment as sticky, but over a different rebalance protocol. This is the one that matters.

Eager vs cooperative: the real difference

Here is the part that trips people up. Sticky reduces how many partitions move. It does not change what happens to the ones that stay.

Range, round-robin, and plain sticky all use the eager protocol. On every rebalance the group does a full stop-the-world:

Every consumer revokes all of its partitions.
Nobody consumes anything.
The new assignment is computed.
Consumers pick their partitions back up.

So even a partition that ends up on the exact same consumer still gets revoked and paused. Add one consumer to a group of ten and all of them stop, for every partition, until the dust settles.

Cooperative-sticky (KIP-429, Kafka 2.4+) rebalances incrementally:

Only the partitions that actually need to move are revoked.
Every other partition keeps being consumed, right through the rebalance.
The moved partitions get reassigned in a second, short round.

One consumer joins, one or two partitions pause for a moment, and the rest of the group never notices. Stop-the-world becomes stop-one-partition.

props.put(
    ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
    List.of(CooperativeStickyAssignor.class.getName()));

You cannot just flip the config

This is the trap. You cannot take a running group on the eager protocol and switch it to cooperative in a single deploy. A group with some members speaking eager and some speaking cooperative will not rebalance correctly.

The supported path is a two-phase rolling upgrade:

// Phase 1: deploy every instance with BOTH strategies listed.
// The group stays on the old protocol until all members support the new one.
props.put(
    ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
    List.of(CooperativeStickyAssignor.class.getName(),
            RangeAssignor.class.getName()));

// Phase 2: after every instance is running phase 1, deploy again
// with only the cooperative strategy. Now the group flips protocols.
props.put(
    ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
    List.of(CooperativeStickyAssignor.class.getName()));

Skip phase 1 and you get a broken rebalance in production. Two deploys, in order, with the whole fleet on phase 1 before phase 2 starts.

The honest trade-off

Cooperative-sticky is not free.

The migration is a two-step rolling upgrade you have to get right, not a one-line change.
Incremental rebalancing can take more rounds than a single eager rebalance, so the total rebalance can be longer even though nobody is fully stopped.
Your onPartitionsRevoked and onPartitionsLost callbacks need to be correct, because partitions now come and go without a global reset to lean on.

For a group that rebalances rarely and briefly, the eager pause may be cheap enough to ignore. For a group that scales, deploys, or loses instances often, the stop-the-world pause is a tax you pay on every event, and cooperative-sticky removes it.

Which assignor does your consumer group run today, and have you ever actually measured how long a rolling deploy freezes it?

🔄 Your Database Is Already an Event Stream

Kyryl — Thu, 09 Jul 2026 20:35:01 +0000

You have a service that owns the orders table. Three other teams need to know when an order changes. So you do the obvious thing: after every write, you publish an event to Kafka.

Then someone adds a new write path and forgets the publish. Now the order exists in the database but no event ever fired. The downstream read models drift. You find out three weeks later when finance asks why the numbers do not match.

This is the dual-write problem, and you cannot discipline your way out of it. Every write path is a place someone forgets to emit the event.

There is a cleaner option. Stop publishing events by hand. Let the database do it.

The database already knows what changed

Change Data Capture (CDC) turns your database into an event source. Debezium reads the write-ahead log, the same log Postgres already uses for replication and crash recovery, and turns every committed row change into a Kafka event.

Point a connector at a table. Every insert, update, and delete becomes a message on a topic. Downstream services build their own read models from that stream and never query your tables directly.

The write already happened. The WAL already recorded it. There is nothing to remember and nothing to enforce in application code, because the event is a byproduct of the commit, not a second action you have to bolt onto every write path.

Wiring it up

A Debezium Postgres connector is a bit of connector config, not application code:

{
  "name": "orders-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres",
    "database.dbname": "shop",
    "plugin.name": "pgoutput",
    "table.include.list": "public.orders",
    "topic.prefix": "shop"
  }
}

Changes to public.orders now land on the shop.public.orders topic. Each message carries a before and after image of the row plus an op field: c for create, u for update, d for delete. A consumer decides what to do with each:

@KafkaListener(topics = "shop.public.orders")
public void onChange(ChangeEvent event) {
    switch (event.op()) {
        case "c", "u" -> readModel.upsert(event.after());
        case "d"      -> readModel.remove(event.before().id());
    }
}

No coordination with the owning service. No shared library. The consumer subscribes and builds exactly the projection it needs.

The outbox pattern is CDC with intent

Tailing the whole WAL means downstream consumers see your table exactly as it is. Sometimes that is fine. Sometimes you want to publish a deliberate event, not a raw row.

The outbox pattern is the same mechanism applied on purpose. Inside the same transaction as your business write, you insert a row into an outbox table shaped like the event you want the world to see:

BEGIN;
UPDATE orders SET status = 'SHIPPED' WHERE id = 42;
INSERT INTO outbox (aggregate, event_type, payload)
VALUES ('order', 'OrderShipped', '{"orderId": 42}');
COMMIT;

Debezium tails the outbox table instead of orders. The event is atomic with the business change because they share one transaction, and its shape is something you designed rather than whatever columns the table happens to have today.

CDC is the mechanism either way. The outbox just decides what becomes an event and what stays internal.

The honest trade-off

Here is the part nobody puts on the slide.

Raw-table CDC couples your event stream to the table's physical shape. The moment a downstream consumer maps shop.public.orders, your column names are a public contract, whether you meant them to be or not.

That means column renames and other non-backwards-compatible schema changes are off the table for you. Rename status to order_status during an internal refactor and every downstream mapper breaks. Silently. There is no compile error, no failed deploy on your side. The consumer just starts reading null because nobody told it the table changed.

Tailing the raw WAL is convenient right up until the day your schema is no longer just yours.

Shape the output on purpose

The fix is not avoiding CDC. It is refusing to treat the current table shape as an accident that leaks downstream.

Single Message Transforms (SMTs): rename, drop, or restructure fields in the connector pipeline, so an internal column rename does not change the published event.
A CDC-facing view or outbox table: publish from a surface you designed, and let the physical table underneath change freely.
A schema registry: enforce compatibility rules on the topic, so a breaking change fails loudly at publish time instead of silently downstream.

All three do the same thing: they put a deliberate boundary between your storage and your stream. The database can still emit every change for free. You just decide what that change looks like on the wire.

CDC is not "replication with extra steps." It is a coupling decision. Either downstream services couple to your database's physical shape, or they couple to a contract you shaped on purpose. Pick the second one.

Do you use Debezium to emit events? Do you tail the raw WAL or shape it through an outbox first, and has an internal refactor ever quietly broken a downstream consumer on you?

🔁 One Health Check Turned a 10-Minute Outage Into 40

Kyryl — Mon, 06 Jul 2026 18:05:44 +0000

The database went down for ten minutes. The restart storm it triggered lasted forty.

Every pod's /actuator/health turned red the moment the database became unreachable. Kubernetes read that as "the process is broken" and started killing pods, one after another, across the entire fleet. New pods booted, connected to nothing, failed the same check, and got killed again. The dependency was down for ten minutes. The self-inflicted damage ran four times longer.

The trigger was one line of config that looked like a simplification.

The setup that looks clean

Someone wired both the liveness and the readiness probe to the same endpoint.

livenessProbe:
  httpGet:
    path: /actuator/health
    port: 8080
readinessProbe:
  httpGet:
    path: /actuator/health
    port: 8080

One endpoint, one thing to configure, one thing to reason about. It reads as clean. It is actually a category error, because those two probes are asking completely different questions and Spring's default /actuator/health answers neither of them cleanly.

Two probes, two questions

The distinction is the whole point, so it is worth stating flatly.

Liveness answers: should I kill this process and start a new one? The only honest reason to say yes is that the process is wedged in a way a restart will fix. A deadlock, an unrecoverable internal state, a JVM that is thrashing. Restarting helps only when the problem lives inside the process.

Readiness answers: should I send this pod traffic right now? A pod can be perfectly alive and still be unable to serve, because something it depends on is unavailable. The fix there is not a restart. It is to stop routing requests until the dependency comes back.

/actuator/health by default aggregates everything, including the db health indicator, into one status. Point liveness at it and you have told Kubernetes to kill the process whenever the database is down. But a down database is not a process problem, and no number of restarts will bring it back.

The restart storm, step by step

Here is the loop that ate thirty extra minutes:

The database becomes unreachable.
The db health indicator goes DOWN, so aggregated /actuator/health returns DOWN.
The liveness probe fails. Kubernetes kills the pod.
A fresh pod boots, starts up, and immediately checks its health.
The database is still down, so the new pod's health is DOWN too.
Liveness fails again. Kubernetes kills it again.
Repeat, on every pod, for as long as the dependency stays down.

Now the failure is worse than the original outage. The pods are gone, so even reads that could have been served from cache are gone. Startup load hammers the recovering database the moment it comes back. And your dashboards are a wall of CrashLoopBackOff that makes it look like the application itself is broken, which sends everyone debugging the wrong thing.

What should have happened is boring by comparison. Readiness fails, Kubernetes pulls the pods out of the Service endpoints, traffic stops, the processes keep running. When the database returns, readiness goes green and traffic resumes. No restarts, no cold caches, no thundering herd.

The fix Actuator already ships

You do not need a custom endpoint or a sidecar. Spring Boot Actuator has health groups built for exactly this split.

management:
  endpoint:
    health:
      probes:
        enabled: true
      group:
        liveness:
          include: livenessState
        readiness:
          include: readinessState,db

That gives you two dedicated endpoints:

/actuator/health/liveness includes only livenessState, the process-internal signal. No database, no downstream calls. It answers "is this process wedged" and nothing else.
/actuator/health/readiness includes readinessState plus the real dependency checks, db here. It answers "can this pod serve traffic right now."

Point the probes at the right endpoints:

livenessProbe:
  httpGet:
    path: /actuator/health/liveness
    port: 8080
readinessProbe:
  httpGet:
    path: /actuator/health/readiness
    port: 8080

Now a database outage trips readiness only. Traffic drains, the pods live, and recovery is automatic. Liveness stays green because the process is, in fact, fine.

The honest trade-off

This is not a set-and-forget win, and pretending it is misses the way the bug comes back.

The config takes five minutes and then nobody looks at it again. That is the problem. Over the next year the service grows: a Redis cache, a Kafka producer, a call to some downstream API. Each of those registers a health indicator, and each is a decision about which group it belongs in. If a new dependency lands in the readiness group, great. If it silently ends up aggregated into liveness, or if someone adds a custom indicator without thinking about probes at all, you have quietly rebuilt the original bug with a new trigger.

The drift is invisible until the next outage. Nobody audits "which health indicators feed which probe" as part of adding a dependency, so the wrong wiring sits there dormant for months. The next time that specific dependency fails, the restart storm returns, and it looks brand new even though it is the same mistake.

The mitigation is process, not code: when you add a dependency with a health indicator, decide its group in the same PR. Treat "which probe does this affect" as part of the definition of done for any new external call.

Has your liveness probe ever quietly grown a dependency check nobody noticed, or has your fleet actually lived through a restart storm like this? Curious how other teams keep the liveness group honest as services accumulate dependencies.

🔒 CREATE INDEX Is a Write Outage in Disguise

Kyryl — Mon, 06 Jul 2026 17:55:29 +0000

CREATE INDEX without CONCURRENTLY locks out writes for the entire build.

On a large, live table that plain statement is not a migration. It is a multi-minute write outage for every endpoint touching that table. And the worst part is it sails through every test you have, because your test data is tiny and the build finishes before you can blink.

The lock nobody reads about

A plain CREATE INDEX acquires a SHARE lock on the table and holds it until the build completes.

CREATE INDEX idx_orders_customer_id ON orders (customer_id);

SHARE allows concurrent reads. It blocks anything that writes: INSERT, UPDATE, DELETE. Every write against that table queues up behind the index build and waits. Not for a moment at the start. For the entire duration.

On a small table that duration is milliseconds, so nothing queues and nobody notices. That is exactly why this bug survives code review and staging. The table you tested against had ten thousand rows. The table in production has two hundred million, and building the index on it takes four minutes. For those four minutes, every write to orders is stalled.

The application does not error. It hangs. Requests pile up, connection pools drain, timeouts cascade to services that never touched the database directly. From the outside it looks like a full outage, and the migration that caused it already "succeeded" in your CI.

The fix

CREATE INDEX CONCURRENTLY was built for exactly this.

CREATE INDEX CONCURRENTLY idx_orders_customer_id ON orders (customer_id);

It does not take the blocking SHARE lock. It builds the index in the background, scanning the table while writes keep flowing the whole time. Reads and writes both continue. This is the default you want for any table that is already taking production traffic.

It is not free

CONCURRENTLY trades a fast, clean build for a slower, riskier one. Three costs, all real:

It is slower. To build without blocking writes, Postgres does two full passes over the table instead of one. One pass to build, a second to catch rows that changed during the first. On a huge table that roughly doubles the build time. You are trading wall-clock time for availability, which is almost always the right trade, but it is a trade.

It cannot run inside a transaction block. This is the one that breaks migration tooling. Most migration frameworks wrap each migration in a transaction by default, and CREATE INDEX CONCURRENTLY will error out if it finds itself inside one. You cannot bundle it with other DDL in the same atomic step. It has to run on its own, outside a transaction, which means you lose the "all or nothing" guarantee for that migration.

It can fail into an INVALID index. This is the sharp edge. If a CONCURRENTLY build fails partway, a deadlock, a cancelled session, a statement timeout, Postgres leaves a half-built index behind and marks it INVALID. It does not roll back. It does not clean up. That index is not used by the planner, but it is still updated on every write and still consumes disk. It just sits there, dead weight, until a human notices.

You find them by asking the catalog directly:

SELECT c.relname AS index_name,
       t.relname AS table_name
FROM pg_index i
JOIN pg_class c ON c.oid = i.indexrelid
JOIN pg_class t ON t.oid = i.indrelid
WHERE i.indisvalid = false;

Anything that comes back has to be dropped and rebuilt:

DROP INDEX CONCURRENTLY idx_orders_customer_id;
-- then re-run the CREATE INDEX CONCURRENTLY

If you run migrations through automation, a failed CONCURRENTLY will not retry cleanly either, because the invalid index now occupies the name you are trying to create. Your retry fails with "relation already exists" until someone drops the corpse.

The one real exception

"Always use CONCURRENTLY in production" is a good rule with exactly one honest exception: a brand new, empty table you just created in the same migration.

CREATE TABLE audit_log (...);
CREATE INDEX idx_audit_log_created_at ON audit_log (created_at);  -- fine

Nothing is writing to audit_log yet. There are no concurrent writes to block, so the SHARE lock costs nothing, and the plain build is faster and runs cleanly inside the same transaction as the CREATE TABLE. Reaching for CONCURRENTLY here would be cargo-culting the rule past the point where it applies.

The tell is simple: if the table already has live traffic, use CONCURRENTLY. If you are indexing something that does not exist yet, do not bother.

The honest trade-off

CONCURRENTLY is not a free upgrade you flip on and forget. You trade a guaranteed-fast, guaranteed-clean, transaction-safe build for a slower one that runs outside your transaction and can fail into a mess you have to notice and clean up yourself.

For any table with real traffic, that trade is worth it every time. A slower build you have to babysit beats a multi-minute outage you cannot. But "worth it" is not "free," and pretending the invalid-index failure mode does not exist is how teams get surprised at 2am.

Has anyone actually gotten burned by an invalid index left behind after a failed CONCURRENTLY build? How did you find it, and do you check pg_index as part of your migration process or only after something breaks?

📈 One Path Variable Can Bankrupt Your Prometheus

Kyryl — Mon, 06 Jul 2026 17:30:00 +0000

A path variable in your metrics tag will quietly bankrupt your Prometheus backend.

It passes code review. It compiles. It works, for a while. Then one day the dashboards start timing out, and it takes the team days to trace the slowdown back to a single line of instrumentation that looked completely normal.

The line that looks fine

Here is the kind of code that ships this problem. A Timer around an order lookup, tagged with the orderId so you can slice latency per request.

@GetMapping("/api/orders/{orderId}")
public Order getOrder(@PathVariable String orderId) {
    return Timer.builder("order.lookup")
            .tag("orderId", orderId)   // <-- the problem
            .register(meterRegistry)
            .record(() -> orderService.findById(orderId));
}

Nothing about this fails a review. It reads as "measure how long an order lookup takes, and let me break it down by order." Reasonable intent. The metric even works when you test it locally with three orders.

What actually happens

Prometheus is a time series database. The identity of a time series is its metric name plus the full set of label key-value pairs. Change one label value and you do not add a point to an existing series. You create a brand new series.

So order.lookup{orderId="1001"} and order.lookup{orderId="1002"} are two completely separate series, each with its own storage, its own index entry, its own memory footprint.

Now run that in production. Every distinct orderId that flows through the endpoint mints a new series. A million orders means a million series from this one metric. Add a userId tag somewhere else and the counts multiply. This is cardinality explosion, and /actuator/prometheus will happily expose all of it.

The failure is gradual, which is what makes it nasty:

Storage grows far faster than your request volume would suggest.
The Prometheus head block balloons, memory pressure climbs, scrape durations creep up.
Queries that touch the metric get slow, then time out.
Someone files "Prometheus is slow" or "the orders dashboard times out."

Notice what is missing from that chain: nobody says "the orderId tag is the problem." The symptom is three steps removed from the cause. I have watched a team spend the better part of a week bisecting scrape configs and bumping memory limits before someone finally ran a cardinality check and found one metric responsible for millions of series.

You can catch it directly once you suspect it:

# top metrics by series count
topk(10, count by (__name__)({__name__=~".+"}))

But you have to suspect it first, and the whole point is that nothing pointed you there.

The fix is not a new tool

The instinct is to reach for a sampling library or a relabeling rule to drop the bad label. You do not need either. Spring already hands you the right value.

For every request, Spring resolves the matched route pattern, the same mechanism its built-in HTTP server metrics (http.server.requests) use to keep their uri tag bounded. The best-matching pattern is available on the request as an attribute.

@GetMapping("/api/orders/{orderId}")
public Order getOrder(@PathVariable String orderId, HttpServletRequest request) {
    String pattern = (String) request.getAttribute(
            HandlerMapping.BEST_MATCHING_PATTERN_ATTRIBUTE); // "/api/orders/{orderId}"

    return Timer.builder("order.lookup")
            .tag("route", pattern)     // bounded to the number of routes
            .register(meterRegistry)
            .record(() -> orderService.findById(orderId));
}

Now the tag value is /api/orders/{orderId}, the template, not the resolved id. Every request through this endpoint lands on the same series. Cardinality for this metric is bounded to the number of routes you have, which is a small, fixed number that does not grow with traffic.

If all you wanted was per-endpoint latency, you may not need a custom timer at all. http.server.requests already gives you templated URI, method, and status out of the box. Reach for a custom metric only when you need a dimension the built-in one does not expose, and when you do, tag it with something bounded.

The honest trade-off

Templating the URI collapses per-entity granularity in the metric itself. That is a real loss, not a footnote.

Once the tag is /api/orders/{orderId}, the metric can no longer answer "how slow was the lookup for order 1002 specifically." It can only tell you about the route as a whole: p50, p99, error rate across all orders.

If you genuinely need to investigate one entity, that is now a logs or traces question, not a metrics question. Attach the orderId to a span or a structured log field, where high cardinality is expected and the backend is built for it. Metrics are for bounded, aggregatable dimensions. Traces and logs are for the long tail of individual cases.

That split is the right one. Metrics answer "is the system healthy and how is this route trending." Traces answer "what happened to this one request." Putting a unique id in a metric tag is asking metrics to do the job of tracing, and the bill for that mistake is paid by your storage backend, quietly, until it is not quiet anymore.

The rule

Any tag value that is not drawn from a small, known set does not belong on a metric. User ids, order ids, request ids, email addresses, raw paths: all of them are cardinality bombs. Route patterns, status codes, HTTP methods, enum-like states: all fine.

The tell is simple. Before you add a tag, ask how many distinct values it can take over the life of the service. If the answer scales with your traffic, you have found the bug before it finds you.

Has your Prometheus setup ever fallen over from a label nobody flagged, and how long did it take to trace it back to the metric? Curious how other teams caught it, and what guardrails you put in place afterward.

"🚩 readOnly = true Is Not a Comment"

Kyryl — Fri, 03 Jul 2026 18:54:38 +0000

@Transactional(readOnly = true) gets slapped on every query method out of habit, the way people sprinkle final on local variables. A marker for the next reader, a bit of documentation that says "this method does not write anything." Except it is not documentation. It is a flag that Hibernate and, if you have set it up, your Postgres routing DataSource both actually read and act on.

Ignore that and you get one of two failures. Either a write silently vanishes with no exception, or every read in your app hits the primary database because nothing ever told the driver it was allowed to go to a replica.

What people think readOnly = true does

The common mental model: it is a hint for whoever reads the code later, maybe a small optimization Spring does under the hood, nothing that changes behavior you would notice. Under that model, marking a method readOnly = true when it is not strictly read-only feels harmless. Worst case, a wasted annotation.

That model is wrong on the part that matters most: the flush mode.

What it actually does: changes Hibernate's flush mode

Inside a transaction marked readOnly = true, Spring propagates that flag down to the underlying Hibernate Session and sets its flush mode to manual. Normally, Hibernate auto-flushes: before you run a query, it checks the persistence context for dirty managed entities and writes them out first, so the query sees consistent state. Under manual flush mode, none of that happens.

@Transactional(readOnly = true)
Report load(Long id) {
    // Hibernate sets FlushMode.MANUAL here
    // no dirty-checking, no auto-flush
    return repo.findById(id);
}

For a genuinely read-only method, this is a pure win. Skipping dirty-checking on every entity in the persistence context is real overhead, and readOnly = true removes it. That is the entire justification for slapping it on query methods, and it is a good one, right up until someone mutates state inside that method.

The trap: a write inside a read-only transaction

Here is the version that gets shipped:

@Transactional(readOnly = true)
// "just documentation", right?
Report loadAndPatch(Long id) {
    Report r = repo.findById(id);
    r.setLastViewedAt(now()); // looks persisted
    return r;
}
// ...it never flushes. The update vanishes.

r is a managed entity. setLastViewedAt mutates its field. Every instinct built from normal Spring Data usage says this change will be picked up on flush, the way it would in any other @Transactional method. It will not. The flush mode is manual, nothing triggers a flush, and the transaction commits with the mutation sitting in memory and nowhere else.

No exception. Nothing in the logs. The method returns a Report object with the field looking correctly set, because the in-memory object really was mutated. Only the database was never told. The bug surfaces days or weeks later as "last viewed timestamps are not updating," and the first three places anyone looks are the query, the column mapping, and the transaction boundary. The annotation that caused it reads like the most innocent line in the method.

This is worse than a typo in a query. A typo throws. This just quietly does nothing, forever, until someone notices the data is stale.

The other half: it can route your read to a replica

The flush-mode change is reason enough to take readOnly = true seriously, but on a Postgres setup with read replicas it does a second job. If you have wired an AbstractRoutingDataSource that inspects the current transaction's read-only flag, that flag is the actual signal deciding which physical database the query hits:

class ReplicaRoutingDataSource
        extends AbstractRoutingDataSource {

    protected Object determineCurrentLookupKey() {
        boolean ro = TransactionSynchronizationManager
            .isCurrentTransactionReadOnly();
        return ro ? "replica" : "primary";
    }
}

TransactionSynchronizationManager.isCurrentTransactionReadOnly() returns exactly the value Spring set from your @Transactional(readOnly = true) annotation. Get the annotation right and your reads spread across replicas, taking load off the primary. Get it wrong, forget it on a genuinely read-only method, and that query hits the primary for no reason. Put it on a method that writes, and depending on your routing setup you can end up sending a write-carrying transaction at a replica connection that rejects writes outright, or worse, one that does not reject them and you get an inconsistency between primary and replica state.

The annotation is not decoration in either direction. It is the single signal two separate systems, your ORM's flush behavior and your connection routing, both key off.

The honest trade-off

Mislabeling a write path as readOnly = true fails silently, not loudly. That is the real cost here, and it is worse than it sounds. A method that should compile-error or throw at runtime instead just does nothing to the database, and the visible parts of the system, the returned object, the lack of any exception, all look correct. You do not get a stack trace pointing at the problem. You get a support ticket three weeks later asking why a field never updates.

The fix is not to avoid readOnly = true. It genuinely helps, both for the flush-mode overhead and for replica routing. The fix is discipline: only mark a method readOnly = true when you have actually checked that it never mutates a managed entity, directly or through a called method. Treat the annotation as a contract, not a habit. If a method's purpose changes later and someone adds a write to it, the missing exception means nobody will be warned. Code review is the only real defense, along with grep-ing for readOnly = true methods that call setters on entities pulled from a repository.

Have you tracked a "why is this not saving" bug back to a stray readOnly = true? What caught it, a test, a code review, or production data going stale?

"🔇 Calling Your Own @Async Method Does Nothing"

Kyryl — Fri, 03 Jul 2026 18:54:03 +0000

Called this.notify(o) from inside placeOrder(), both methods on the same @Service. No exception. No warning. It just ran the mailer call inline, blocking the request thread like @Async was never there.

Here is the setup that broke:

@Service
class OrderService {

    void placeOrder(Order o) {
        save(o);
        this.notify(o); // bypasses the proxy
    }

    @Async
    void notify(Order o) {
        mailer.send(o); // never actually async
    }
}

Nothing here looks wrong on a first read. notify() is annotated @Async, it lives on a @Service, the class is wired into the container correctly. And yet placeOrder() blocks on mailer.send() every single time. No thread pool, no exception, no log line telling you the annotation got ignored.

Why this.method() Skips the Proxy

Spring AOP does not rewrite your class. When you annotate a method with @Async, @Transactional, or @Cacheable, Spring wraps the bean in a proxy: a JDK dynamic proxy if the bean implements an interface, a CGLIB subclass otherwise. Every advice you rely on, the async executor handoff, the transaction interceptor, the cache lookup, lives on that proxy, not on your class.

When another bean calls orderService.notify(o), it is calling the proxy. The proxy runs its advice, then delegates to the real method. That is the whole mechanism working as designed.

When placeOrder() calls this.notify(o), there is no proxy in the picture. this inside a Spring-managed object is the raw, unproxied instance. It was never wrapped, because Spring only ever gets a chance to wrap the object other beans see when they ask the container for it. The object refers to itself directly, and Java resolves this.notify(o) as an ordinary virtual method call. Spring cannot intercept a call it never sees, because the call never leaves the object.

This is not a bug in Spring. It is a direct consequence of how proxy-based AOP works, and it applies to every annotation-driven aspect Spring ships: @Async, @Transactional, @Cacheable, @Retryable, custom @Aspect advice, all of it. Self-invocation quietly skips every one.

Fix 1: Split the Callee Into a Second Bean

The textbook answer is to make sure the call always crosses a real bean boundary. Pull the annotated method out into its own class:

@Service
class OrderService {
    private final NotifyService notifyService;

    void placeOrder(Order o) {
        save(o);
        notifyService.notify(o); // real proxy
    }
}

@Service
class NotifyService {
    @Async
    void notify(Order o) {
        mailer.send(o); // now actually async
    }
}

This works. notifyService is a container-managed reference, Spring hands OrderService the proxy, and the call to notify() goes through it like any other cross-bean call. @Async fires, mailer.send() runs on the executor, placeOrder() returns immediately.

The cost is a class that has no reason to exist except to dodge this one behavior. NotifyService is not a domain concept. It does not group related responsibilities, it does not hide an implementation detail worth hiding, it exists purely because Spring's proxy model requires a bean boundary between caller and callee. Every time someone reads the codebase and asks "why is notification logic split out into its own service", the honest answer is "so a self-invocation bug does not resurface", which is not an answer that belongs in a design review.

For a genuinely separate concern, this split is the right call regardless of the AOP issue. For a single method that happens to need @Async, it is architecture shaped by a framework limitation, not by the domain.

Fix 2: Inject the Bean Into Itself

The pragmatic fix keeps the method where it belongs and instead gets a real proxy reference inside the class:

@Service
class OrderService {

    @Lazy @Autowired
    private OrderService self; // proxy to itself

    void placeOrder(Order o) {
        save(o);
        self.notify(o); // through the proxy, for real
    }

    @Async
    void notify(Order o) {
        mailer.send(o);
    }
}

self is a field of the bean's own type, autowired like any other dependency. Spring resolves it to the actual proxy, the same object every other bean gets when it asks the container for an OrderService. Calling self.notify(o) instead of this.notify(o) sends the call through that proxy, the async advice runs, and mailer.send() finally executes on the executor instead of the caller's thread.

The @Lazy is not optional. Without it, Spring tries to fully construct OrderService, including resolving its self field, which means it needs a fully constructed OrderService to finish constructing OrderService. That is a circular dependency, and Spring fails at startup rather than silently accepting it. @Lazy tells Spring to inject a lazy proxy for self instead, one that only resolves the real bean the first time self.notify() (or any method on it) actually gets called, well after the container has finished wiring everything else. By then OrderService exists, the cycle never has to resolve eagerly, and startup succeeds.

The Honest Trade-off

Self-injection needs @Lazy to avoid that circular-dependency failure, which means the fix does not work by just adding a field, it works by adding a field plus an annotation whose job is to explain to Spring why injecting a bean into itself is not actually circular. Anyone unfamiliar with the trick reads @Lazy @Autowired private OrderService self; and reasonably assumes something is wrong with the code, because nothing about "inject the class into itself" looks intentional on a first pass.

It is still the better trade. Splitting notify() into NotifyService solves the same problem by growing the class hierarchy, and that growth has to be maintained forever: another file, another bean to wire in tests, another indirection for anyone tracing the call path, all to work around a framework detail that has nothing to do with what the domain actually looks like. The self-injected field is uglier at the point you read it, but it stays contained to the one class that needs it, and it does not force a design decision that only exists because of AOP.

Neither fix is free. Pick the one whose cost you would rather explain in a code review: a class that exists for no domain reason, or a field that looks wrong until someone tells you why it is there.

Ever shipped a silently-skipped @Async or @Transactional to production because of self-invocation? What finally gave it away?

"🥊 Your Retry Aspect Is Retrying a Dead Transaction"

Kyryl — Fri, 03 Jul 2026 18:53:28 +0000

A retry aspect and a @Transactional aspect were both wrapping the same method. Neither had @Order. I assumed Spring would nest them the way I had pictured in my head. It did not, and the failure mode was a UnexpectedRollbackException on a method that should have quietly succeeded on the third attempt.

Here is the setup:

@Aspect
@Component
class RetryAspect {
    @Around("@annotation(Retryable)")
    Object retry(ProceedingJoinPoint pjp) {
        // retry loop around pjp.proceed()
    }
}

@Aspect
@Component
class TxAspect {
    @Around("@annotation(Transactional)")
    Object wrap(ProceedingJoinPoint pjp) {
        return tx.execute(status -> proceed(pjp));
    }
}
// no @Order on either one

Both aspects match the same method through two different annotations. Nothing in that code tells Spring which one should run first. Spring AOP will still pick something, deterministically for a given build, but that something is not documented, not guaranteed across versions, and not something you chose.

Why nesting order matters here

An @Around advice wraps the method call. When two advices target the same join point, they nest like Russian dolls: the outer one runs first, calls into the inner one, which calls into the actual method. Which aspect ends up outer decides what the inner one is running inside of.

For a retry aspect and a transaction aspect, that is not a cosmetic detail. It decides whether "retry" means "retry the whole transaction" or "retry inside a transaction that has already failed once."

Wrong order: transaction outer, retry inner

@Aspect
@Order(1)   // OUTER: opens the tx first
class TxAspect {
    @Around("@annotation(Transactional)")
    Object wrap(ProceedingJoinPoint pjp) {
        return tx.execute(status -> proceed(pjp));
    }
}

@Aspect
@Order(2)   // INNER: retries land in that tx
class RetryAspect {
    @Around("@annotation(Retryable)")
    Object retry(ProceedingJoinPoint pjp) {
        // retry loop around pjp.proceed()
    }
}

Lower @Order values run first on the way in, which puts TxAspect on the outside. One transaction opens before the retry loop even starts, and every retry attempt runs inside that same transaction.

That is the bug. The moment the first attempt throws whatever transient exception the retry logic is supposed to swallow, Spring's transaction machinery marks the current transaction rollback-only. That flag does not clear on the next attempt. It cannot, it is the same transaction. So attempt two runs inside a transaction that is already condemned. It might even succeed on its own terms, the business logic completes fine, but when the retry aspect's @Around unwinds and the transaction tries to commit, it hits the rollback-only flag and throws UnexpectedRollbackException instead of committing anything.

From the caller's side this looks insane. The logs show the operation succeeding on retry, then the whole thing still fails. The retry loop did its job. The transaction it was retrying inside of was already dead before the second attempt began.

Right order: retry outer, transaction inner

@Aspect
@Order(1)   // OUTER: retries first
class RetryAspect {
    @Around("@annotation(Retryable)")
    Object retry(ProceedingJoinPoint pjp) {
        // retry loop around pjp.proceed()
    }
}

@Aspect
@Order(2)   // INNER: fresh tx per attempt
class TxAspect {
    @Around("@annotation(Transactional)")
    Object wrap(ProceedingJoinPoint pjp) {
        return tx.execute(status -> proceed(pjp));
    }
}

Swap the @Order values and the nesting flips. RetryAspect is now outer, so its retry loop calls into TxAspect fresh on every attempt. TxAspect opens a brand-new transaction each time it is invoked, because from its point of view each retry is a completely separate call. Attempt one fails and rolls back cleanly. Attempt two starts an unmarked, unrelated transaction and gets a real shot at succeeding. No transaction ever carries scar tissue from a previous attempt.

This is the nesting you actually want for "retry a transactional operation": each attempt is its own unit of work, committed or rolled back on its own, with no memory of the attempt before it.

The honest trade-off

A fresh transaction per retry attempt is not free. Each one is a new database round trip: begin, do the work, commit or roll back. If your retry policy allows five attempts, a failing call can now open five transactions instead of one, and depending on your isolation level and connection pool size, that adds real latency and real contention under load.

That cost is worth paying for genuinely transient failures: a deadlock victim getting picked, a connection reset mid-query, a lock wait timeout. Those are cases where the exact same operation, run again a moment later, plausibly succeeds because the condition that broke it was temporary.

It is not worth paying for a bug. If the method fails because of bad input, a null somewhere it should not be, a constraint violation that is always going to violate, retrying it just runs the same doomed logic multiple times against a fresh transaction each time, burning connections and latency for a result that was never going to change. Retry policies need to be scoped to exceptions that are actually retryable, not Exception.class as a catch-all, or this whole ordering fix just gives you a more expensive way to fail five times instead of once.

@Order is not decoration on these two annotations. It is the only thing that decides whether "retry" means a clean second attempt or a slow-motion replay inside a transaction that already gave up.

Have you hit an aspect ordering bug like this, and how did you end up debugging it back to @Order?

"🔁 Your Prototype Bean Is a Singleton in Disguise"

Kyryl — Fri, 03 Jul 2026 18:52:46 +0000

A prototype-scoped bean injected into a singleton is created exactly once. That is not a corner case, it is how Spring's dependency injection works by design, and it catches people who assume @Scope("prototype") means "new instance whenever someone asks."

Here is the setup that trips people up:

@Component
@Scope("prototype")
class TokenGenerator {
    private final String seed = UUID.randomUUID().toString();

    String next() {
        return seed + "-" + System.nanoTime();
    }
}

@Service
class ReportService {

    private final TokenGenerator gen;

    ReportService(TokenGenerator gen) {
        // resolved ONCE, right here, when the container wires ReportService
        this.gen = gen;
    }

    String generate() {
        return gen.next();
    }
}

ReportService is a singleton. Spring builds it once at startup, and to build it, it has to resolve the TokenGenerator constructor argument. That resolution happens exactly one time. The prototype scope on TokenGenerator never gets a second chance to do its job, because nothing ever asks the container for another one. ReportService just holds onto the first instance it got, for the rest of the application's life.

If TokenGenerator were stateless, nobody would ever notice. The bug shows up the moment the prototype bean carries state that is supposed to reset per use, a running total, a per-request seed, a cache that should not outlive one call. Then every "fresh" instance is actually the same object, and callers start stepping on each other's state.

Fix 1: ApplicationContext.getBean() — it works, but it is ugly

The instinctive patch is to stop injecting the bean directly and instead ask the container for it on demand:

@Service
class ReportService {

    @Autowired
    private ApplicationContext ctx;

    String generate() {
        TokenGenerator gen = ctx.getBean(TokenGenerator.class);
        return gen.next();
    }
}

This does fix the bug. Every call to generate() pulls a brand-new TokenGenerator from the container, because getBean() triggers scope resolution every time it runs, not just at wiring time.

The problem is what it costs you. ReportService now holds a live reference to the whole ApplicationContext, which means it can reach any bean in the application, not just the one it needs. Unit testing gets worse too: instead of mocking one collaborator, you have to mock the entire context or stand up a real one. And the lookup is stringly typed by class, which means a typo or a refactor that renames the bean fails at runtime, not at compile time. It works. It also does not belong in application code that has any other option.

Fix 2: an abstract @Lookup method — cleaner, still Spring-flavored

Spring has a purpose-built mechanism for exactly this case:

@Service
abstract class ReportService {

    @Lookup
    abstract TokenGenerator gen();

    String generate() {
        return gen().next();
    }
}

@Lookup tells Spring to generate a subclass of ReportService at runtime (via CGLIB) that overrides gen() to fetch a fresh TokenGenerator from the container every time it is called. Your code never touches ApplicationContext directly, never does a stringly-typed getBean() call, and the method signature documents exactly what type it returns.

The catch: ReportService and the gen() method cannot be final, because CGLIB needs to subclass them. Constructor injection for other dependencies still works fine alongside @Lookup, but the class itself has to stay proxyable. It is still coupled to Spring, just through an annotation instead of an API call, and it depends on a runtime-generated subclass existing, which occasionally surprises people debugging stack traces for the first time.

Fix 3: inject ObjectProvider — the one to actually reach for

The cleanest option skips both the container reference and the CGLIB proxy:

@Service
class ReportService {

    private final ObjectProvider<TokenGenerator> genProvider;

    ReportService(ObjectProvider<TokenGenerator> genProvider) {
        this.genProvider = genProvider;
    }

    String generate() {
        TokenGenerator gen = genProvider.getObject();
        return gen.next();
    }
}

ObjectProvider<T> is a plain constructor-injected dependency, no ApplicationContext, no abstract class, no CGLIB subclass. Calling .getObject() resolves a fresh prototype bean at that exact moment, and only at that moment. The rest of ReportService stays a normal, final, easily-mocked class. If you are on jakarta.inject, Provider<T> gives you the same behavior with a one-method interface, if you prefer not to depend on a Spring-specific type at all.

This is also the version that is easiest to test. Mock ObjectProvider<TokenGenerator> to return a stub TokenGenerator from getObject(), and ReportService never needs a real Spring context in the test.

The honest trade-off

None of these three fixes matter if TokenGenerator is stateless. A stateless prototype bean behaves identically to a singleton, so field-injecting it directly is not a bug, it is just an unnecessary scope declaration. Reach for ObjectProvider only when the prototype bean genuinely carries per-call state that would otherwise leak between callers. Adding indirection to fetch a bean that never needed to be re-created is its own kind of mistake, just a quieter one.

There is a real cost either way: ObjectProvider (or @Lookup, or getBean()) adds a layer of indirection that a reader has to understand before they see why the bean is not just constructor-injected like everything else. That indirection earns its keep exactly when the state leak is real, and nowhere else.

Have you shipped the ApplicationContext.getBean() version to production before catching it in review? What made you notice?

🧱 Your Test Fixtures Are Lying About What Matters

Kyryl — Fri, 03 Jul 2026 11:51:05 +0000

Open a test file with a thirty-argument constructor call and try to spot the one value the test actually cares about. You cannot do it at a glance. Every field looks equally important, because the constructor treats them that way.

Here is the domain object in question, a fairly ordinary User:

public class User {
    private final UUID id;
    private final String email;
    private final String firstName;
    private final String lastName;
    private final String phone;
    private final Address address;
    private final Role role;
    private final AccountStatus status;
    private final boolean emailVerified;
    private final boolean twoFactorEnabled;
    private final String sessionToken;
    private final Instant tokenIssuedAt;
    private final Instant createdAt;
    private final Instant lastLoginAt;
    // ... 16 more fields exactly like this

    public User(UUID id, String email, String firstName, String lastName,
                String phone, Address address, Role role, AccountStatus status,
                boolean emailVerified, boolean twoFactorEnabled, String sessionToken,
                Instant tokenIssuedAt, Instant createdAt, Instant lastLoginAt
                /* ...16 more parameters */) {
        // assign everything
    }
}

Now here is a test that exercises exactly one behavior: a session with an expired token gets rejected.

@Test
void expiredTokenIsRejected() {
    User user = new User(
        UUID.randomUUID(),
        "jane.doe@example.com",
        "Jane",
        "Doe",
        "+1-555-0100",
        Address.of("221B Baker St", "London", "NW1", "UK"),
        Role.CUSTOMER,
        AccountStatus.ACTIVE,
        true,
        false,
        "expired-token-abc123",           // <- the one value this test cares about
        Instant.now().minus(Duration.ofDays(30)),
        Instant.now(),
        null
        // ... 16 more arguments exactly like this
    );

    assertThat(sessionService.isValid(user)).isFalse();
}

The fact under test, a token issued thirty days ago, sits at position eleven out of thirty. A reviewer has to read the whole argument list to find it, and count commas to be sure they found the right one. Six months from now, when User grows a preferredLanguage field, every one of these constructor calls either breaks or silently gets a null nobody reviewed.

Why this happens

Nobody designs a test this way on purpose. It happens because the constructor is the only tool available, and the constructor's job is to build a fully valid object, not to communicate what a specific test is checking. The constructor is honest about the shape of User. It says nothing about which of those thirty values is the point.

That is the actual problem, and it is not verbosity. It is signal-to-noise. A fixture constructor forces every test to restate the entire object graph, every time, whether that test cares about eviction dates, marketing consent, or nothing but a single boolean.

The pattern

A test data builder inverts the default. It ships with sensible, valid values for everything, and exposes named methods for the one or two things a given test needs to override.

public class UserTestDataBuilder {
    private UUID id = UUID.randomUUID();
    private String email = "jane.doe@example.com";
    private String firstName = "Jane";
    private String lastName = "Doe";
    private Role role = Role.CUSTOMER;
    private AccountStatus status = AccountStatus.ACTIVE;
    private String sessionToken = "valid-token";
    private Instant tokenIssuedAt = Instant.now();
    // ... every other field, defaulted to something valid

    public static UserTestDataBuilder aUser() {
        return new UserTestDataBuilder();
    }

    public UserTestDataBuilder withExpiredToken() {
        this.sessionToken = "expired-token-abc123";
        this.tokenIssuedAt = Instant.now().minus(Duration.ofDays(30));
        return this;
    }

    public User build() {
        return new User(id, email, firstName, lastName, /* ... */ sessionToken, tokenIssuedAt, /* ... */);
    }
}

And the test collapses to this:

@Test
void expiredTokenIsRejected() {
    User user = aUser().withExpiredToken().build();

    assertThat(sessionService.isValid(user)).isFalse();
}

Everything except the token is defaulted. The reader does not scan fourteen positional arguments looking for the relevant one, they read withExpiredToken() and move on. When User gains a preferredLanguage field next sprint, it gets a default inside the builder once, and every existing test keeps compiling, untouched.

The honest trade-off

This is not free. Someone has to write the builder class, and someone has to keep its defaults sensible as the domain object evolves. That is a second place User lives, and it can drift out of sync with real constraints if nobody updates it when validation rules change.

There is a subtler cost too. A "just valid enough" default can silently satisfy a constraint the test author never thought about. If AccountStatus.ACTIVE is the builder's default and a bug only reproduces for AccountStatus.PENDING_VERIFICATION, the builder hides that gap exactly as effectively as a thirty-argument constructor call hides the one field that matters. Defaults reduce noise, they do not replace judgment about what a test should actually cover.

For a domain object with three or four fields, none of this is worth it, a plain constructor is fine. It starts paying for itself once an object crosses somewhere around ten to fifteen fields, or once it shows up in more than a handful of tests.

What you actually gain

Reviewers read intent, not positions. New fields do not break unrelated tests. And the one thing under test is loud, instead of buried in a wall of arguments a reviewer has learned to skim past, which is its own kind of risk: skimmed code is where the real bug in position eleven goes unnoticed.

What do you reach for once your domain objects outgrow a plain constructor call: an Object Mother, a builder like the one above, or a random test-data library like Instancio or EasyRandom? Curious what actually holds up at scale versus what looks good in a blog post.

"🧪 Good Enough Tests Died When AI Started Writing Them"

Kyryl — Tue, 30 Jun 2026 19:16:56 +0000

For years, integration tests got a quiet pass. "Good enough, not perfect" was an acceptable answer, because writing them by hand was expensive and somebody had to ship the feature. That excuse just expired. AI writes these tests now, the authoring cost collapsed, and the bar moves from good enough to exact.

Before the how, the where.

This is about the middle of the pyramid

Three layers, and this post is only about one of them.

Unit tests sit at the bottom. Fast, pure, no infrastructure. Not the topic here.
Integration tests sit in the middle. Your code against real infrastructure and mocked externals. This is the layer everyone gets wrong.
End-to-end tests sit at the top. A thin layer that hits the real vendor sandbox and proves the whole thing actually works.

Almost nobody does the middle layer properly. Teams fall into one of two traps. They mock everything, so the suite stays green while production breaks, because the mocks drifted and the tests only proved their own assumptions. Or they skip the middle entirely and lean on a handful of e2e tests to somehow cover the gap. A disciplined integration layer is rare.

The rule: split by ownership

The fix is not "mock or real". It is one question asked per dependency: do I own this?

Ownership decides fidelity. If you own it, run it for real. If you do not, mock it at the wire. That single cut resolves almost every "should I mock this" argument.

Infra you own: run it for real, at the prod version

Your database, your message broker, your cache, your sibling services. Run them in containers that match production. Not H2 standing in for Postgres. Not an embedded broker. The real engine.

Two ways to get there. Testcontainers spins real containers from inside the test:

@Container
static PostgreSQLContainer<?> db =
    new PostgreSQLContainer<>("postgres:16.3");

@Container
static KafkaContainer kafka =
    new KafkaContainer("apache/kafka:3.8.0");

Or a docker-compose that brings the whole stack up once and lets the suite run against it:

services:
  postgres:
    image: postgres:16.3          # match prod exactly
  kafka:
    image: apache/kafka:3.8.0     # KRaft, no ZooKeeper
    environment:
      KAFKA_PROCESS_ROLES: broker,controller

Both are valid. Testcontainers gives you per-test lifecycle and zero shared state. Compose gives you one warm stack and faster local loops. Pick per project.

The part people skip is the version. Matching the engine is not enough, you have to match the version. Production runs Kafka in KRaft mode, and the test suite still runs Kafka plus ZooKeeper from a template somebody copied in 2021. That drift used to be tolerable. It is not anymore. The behavior, the configs, the failure modes differ. Pin the test image to what production runs, and bump it when production bumps.

The third-party API you do not own: WireMock the wire

For the external HTTP API you do not control, mock it at the wire. WireMock is the tool (MockServer works too). It stands up a fake HTTP server, so you stub the response and verify the call:

stubFor(post("/charges")
    .willReturn(okJson("{ \"id\": \"ch_1\" }")));

// your real client runs against the fake server

verify(postRequestedFor(urlEqualTo("/charges"))
    .withRequestBody(matchingJsonPath("$.amount")));

The critical word is wire. Stub the wire, not your Java client. The moment you mock your own client class, you mock away serialization, the retry policy, timeouts, and error mapping. That is exactly the code most likely to carry the bug. WireMock leaves all of it running and only fakes the server on the other end. And in the AI-era bar, every outbound call gets both a stub and a verify, so a silently-dropped or malformed request fails the test instead of slipping through.

The safety net: a wire mock can lie

A stub is a snapshot. The day the vendor changes their contract, your mock keeps returning the old shape and your green suite is now fiction. This is the real cost of mocking, and pretending otherwise is dishonest.

That is precisely the job of the thin e2e-against-sandbox layer at the top of the pyramid. It runs against the vendor's real sandbox, out of band from the main suite, and catches the drift the mock cannot. Back it with recorded real responses or contract tests so the stubs stay honest.

The honest trade-off

AI collapsed the cost of writing these tests. It did not collapse the cost of running them.

You still pay for Docker in CI and the startup seconds each container costs, which you mitigate with container reuse, singletons, or a compose stack that comes up once. You now owe version upkeep every time production upgrades, because a pinned image that drifts behind prod is its own quiet lie. And the wire mocks still rot, so the e2e-on-sandbox layer is not optional.

The bar is higher because the labor is finally cheap. The runtime and maintenance bill is still real, and you should budget for it instead of pretending the AI made testing free.

Now that AI writes the tests, is "good enough" coverage still an acceptable answer? And has a lying mock or a version mismatch ever shipped a bug past your integration suite?