DEV Community: Kyryl

Can Kafka Replace Redis for Cache Synchronization? A Production Answer

Kyryl — Wed, 14 Oct 2026 08:40:04 +0000

I recently came across a question on r/apachekafka that captures a common cache-synchronization
problem:
Can Kafka Replace Redis for Cache Synchronization Across Multiple Spring Boot Pods?

The original question, paraphrased
Spring Boot on GCP, ~25 pods, Redis currently handles cache synchronization. Considering Kafka instead: publish an event when a cache entry changes, every pod consumes it and updates its own local cache. Is Kafka a good fit here? With 25 pods, does every pod get the update, or does Kafka hand it to only one consumer? Does each pod need its own consumer group? Has anyone actually run this — what are the pros and cons?

I have experience solving this class of problem. The approach described here — a compacted Kafka
topic used to materialize a local cache in every pod — is a common pattern and has been running in
production systems for years. It is not a universal replacement for Redis, but it is a practical
option when its consistency and operational trade-offs fit the workload. This article explains the
pattern, the details that matter in practice, and a small runnable example.

Each pod uses a separate consumer group, receives the complete topic, and serves reads from its own local cache.

The Problem: Shared Configuration on a Fast Read Path

Assume the application has configuration or reference data that every instance needs often:
feature rules, routing settings, tenant settings, or similar data. It needs to be quick to
retrieve, but it must also change without redeploying every pod.

A shared Redis cache is a natural solution. Every pod reads the same centrally managed value, so
after an update there is one current view to consult. The trade-off is that each read leaves the
process and depends on the cache service being available and responsive. Depending on the chosen
cache strategy, a Redis miss can also cause an additional lookup: with cache-aside, for example,
the application reads the source of truth, then populates Redis before the next request can use
the cache.

Kafka takes a different approach. An update is appended to a compacted topic, and every instance
replays that topic into its own in-memory cache. Reads are then local map lookups. The replicas are
eventually consistent: when a pod begins processing work, it may still have an older value while
it catches up — or no value at all if consumption has failed. Later, I will show how to prevent a
pod from doing real work until it has consumed the configuration. In return, Kafka provides a
durable log for replay and recovery, keeps the read path inside the pod, and can support explicit
schema-based serialization such as Avro, Protobuf, or JSON Schema.

Neither Kafka nor Redis is immune to failure. But if Kafka is already a dependency, using it to
distribute this configuration can avoid adding Redis as another runtime dependency and another
system to operate. That reduces the number of moving parts and failure modes to consider; it does
not make Kafka failures disappear. This is a worthwhile trade only when eventual consistency is
acceptable for the data.

Kafka can also decouple the service that publishes configuration from the services that consume
it. They only need to agree on the record contract and its schema; consumers do not need a direct
runtime dependency on the producer. With Redis, both sides typically need to agree on the key
format and the serialized value — often the same DTO or a shared representation — and consumers
must know where to read it.

Aspect	Redis (shared cache)	Kafka (compacted topic → local cache)
Data view	One centrally managed cache	One materialized cache per instance
Consistency	One shared cache view; source changes still require correct invalidation	Eventual consistency between local replicas
Read path	Network round-trip; misses may require a source-of-truth lookup	In-process map lookup after the initial replay
Recovery / new instances	Populate or warm the cache	Replay the compacted topic
Record history	Not inherent to the cache	Replayable current state; compaction removes superseded values, so this is not an audit log
Service coupling	Shared key/value format and Redis access	Shared record schema; no direct producer dependency
Operational cost	Shared cache tier, plus Kafka if it is also needed	Kafka storage, consumers, and one cache per instance

This article focuses on how to build the Kafka side safely: broadcasting every update to every
instance, rebuilding the cache on startup, and refusing to do real work until that rebuild is
complete.

A note on the code: the repository is a runnable demonstration, not a drop-in production
library. I verified its Docker flow end to end: startup replay, readiness, live updates, and
tombstone eviction. Before adopting the pattern, test it against your own volume, rebalance,
serialization, retry, and outage scenarios.

TL;DR

Yes, with a caveat: Kafka replaces Redis for broadcasting state to every pod, not for reading from one shared cache. Each pod gets its own eventually-consistent local copy.

Every pod needs its own Kafka consumer group. Pods sharing a group split the partitions (load-balanced); pods each in their own group each get the full topic (broadcast).

The topic must be compacted. Use more than one partition when startup replay needs concurrency, and make the local cache thread-safe when listener concurrency is enabled.

Keep the dataset small and relatively low-volume. This pattern fits feature flags, routing rules, and service configuration—not a large or high-churn business dataset.

Deletes are tombstones: a record with a null value, not a special "delete" message type.

Wait until startup replay is complete before interacting with the outside world. Do not serve endpoints, run scheduled jobs, process queue messages, or perform other externally visible work from a partial local cache. Open readiness only after the replay is complete.

Unique groups leave inactive offset metadata behind until Kafka's configured retention removes it. Expect temporary clutter in consumer-group tooling and monitor the relevant broker settings.

How the Whole Flow Works

There are two phases: rebuild the local replica, then keep it updated.

Startup has a fixed finish line. After every partition reaches it, the same listener keeps the ready pod's local cache updated.

A producer writes an upsert to the compacted topic. The Kafka record key is the configuration key; its value is the latest configuration. A null value is a tombstone that means delete.
A new pod starts with a new consumer group and auto.offset.reset=earliest, so Kafka assigns every topic partition to that pod's group and replay begins from the oldest retained offsets.
During assignment, the readiness gate reads two numbers for every partition: the consumer's current position and the partition's end offset. That end offset becomes the fixed startup target.
The listener applies each record to a thread-safe local map. Only after the listener returns successfully does the tracking aspect advance that partition's processed position. A listener failure therefore cannot make the gate claim that the failed record was applied.
When every partition reaches its captured target, readiness changes to UP. The deployment platform may now route requests to the pod, and readiness-gated scheduled jobs may run. Reads use the local map and do not call Kafka.
The same consumer remains active after startup. New records update or delete local entries as they arrive, so pods converge independently and are eventually consistent with one another.

The fixed target in step 3 is important. If a producer writes while replay is running, the new
record does not move the startup finish line forever. The consumer still processes it through the
normal listener path, but readiness is defined against the snapshot captured at assignment. What
can make startup slow is a large existing log, not a stream of later low-volume updates. This is
why the pattern is intended for small, low-churn upsert datasets such as feature flags, routing
rules, and service configuration.

Publish the Current State in a Compacted Topic

Start by creating the configuration topic with cleanup.policy=compact. Compaction retains the
latest record for each key, so a pod that starts later can replay the topic from the beginning and
reconstruct the current configuration without asking another service to warm its cache.

This works best for a small, relatively low-volume upsert topic: feature flags, routing rules,
tenant settings, or similar configuration. It is not a good default for a large or high-churn
dataset that every pod would need to replay and retain in memory.

Create enough partitions to let one pod replay the topic concurrently with several consumer
threads when startup time matters. The useful concurrency is capped by the partition count, and the
local cache must be thread-safe when listener concurrency is configured above one. In the example,
a one-shot kafka-topic-init service creates and seeds the topic before the application starts.

The next requirement is ensuring every pod receives that state.

Give Every Pod Its Own Consumer Group

A Kafka consumer group has one independent view of a topic: within that group, each partition is
assigned to one consumer. Therefore, pods that share a group split the work; pods in separate
groups each consume the complete topic.

A shared group divides the state. A unique group per pod gives every local cache the complete state.

Give each pod a unique group.id. In this example, a shared prefix identifies the deployment and
Spring's ${random.uuid} supplies the per-instance suffix:

app:
  config-topic: ${CONFIG_TOPIC:local-cache-config}
  kafka:
    bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS:localhost:9092}
    consumer-group: ${CONSUMER_GROUP_PREFIX:local-cache-demo}-${random.uuid}

Every startup creates a new group and must rebuild its local cache. For that reason, set
auto.offset.reset to earliest in the consumer properties. latest would leave a new pod empty
until the next update arrives.

Map<String, Object> consumerProperties = Map.of(
        // other consumer properties...
        ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");

With the topic and broadcast behavior in place, each pod can now materialize the same local state.

Materialize Updates in the Local Cache

Once the topic contains the current state and every pod receives it, the listener is deliberately
small: put normal records into the local cache and remove a key for a tombstone.

@Slf4j
@Component
@RequiredArgsConstructor
public class ConfigTopicListener {

    private final LocalConfigCache cache;

    @ConfigConsumer("config-topic")
    @KafkaListener(topics = "${app.config-topic}")
    public void onMessage(ConsumerRecord<String, String> record) {
        if (record.value() == null) {
            log.debug("Tombstone for key '{}' — evicting from local cache", record.key());
            cache.remove(record.key());
            return;
        }
        cache.put(record.key(), record.value());
    }
}

Normal records cover creates and updates. Deletion needs one additional convention.

Delete Configuration with Tombstones

A record with the same key and a null value is a tombstone. Compaction retains it long
enough for consumers to observe the delete, then eventually removes it. delete.retention.ms is
also the bound within which a consumer starting at offset zero must complete its scan to be
guaranteed a valid final-state snapshot, as the
Kafka broker configuration reference
explains. Configure that window from measured worst-case replay time; do not assume the default is
automatically safe for a large topic.

The tombstone is broadcast like any other record; each pod interprets its null value as an instruction to remove the key.

Hard deletion is appropriate only when the application should truly forget the configuration. If an
inactive configuration must remain available for an audit, older data, or later reactivation,
publish a normal record with active: false or deactivatedAt and keep it in the cache instead.
If the entry count is small, retaining every configuration locally can be simpler. A compacted topic
keeps the latest value per key, not every revision forever; use versioned keys or a separate history
topic when complete history is required.

The example uses a ConcurrentHashMap for LocalConfigCache. Spring Kafka starts with one
consumer thread by default, but configured listener concurrency can update the same cache from
multiple threads. The @ConfigConsumer annotation on the listener connects it to the readiness
gate described next.

At this point every pod can build the correct local state. It still must not use that state until
the replay has finished.

Don't Serve Traffic Until You've Caught Up

Do not receive traffic, run scheduled jobs, or perform other external work until the
configuration has been consumed.

This is the part that's easy to skip and expensive to skip. A pod that reports "ready" the moment
it's assigned partitions is ready in name only — it might have replayed three records out of three
thousand. Requests hit an almost-empty cache; a scheduled job runs off stale defaults; nobody
notices until someone asks why pod 14 behaved differently for two minutes after a deploy.

The example packages this concern as a small readiness-gate module. When Kafka assigns
partitions, the gate captures their current end offsets as the startup target. Each successfully
processed record advances the tracked position for its partition. Readiness opens only after every
partition reaches its captured target and no processing error remains. Empty topics work too:
their starting positions already equal their end offsets.

The target is a fixed snapshot. Records published after partition assignment do not extend it, so
a continuously updated topic cannot keep the pod in startup forever. Once the captured target is
reached, later records continue through the normal live-update path and remain subject to the
eventual-consistency trade-off described below.

The gate starts closed, tracks successful replay against fixed targets, and opens only when every assigned partition has caught up.

The flow has four responsibilities. First, a marker annotation identifies the @KafkaListener
method whose ConsumerRecord arguments the readiness gate tracks:

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface ConfigConsumer {
    String value() default "";
}

Second, the tracker records the target end offsets captured on assignment and the next offset after
each successfully processed record:

@Component
public class ConsumptionTracker {

    private final ConcurrentMap<String, ConsumerReplayProgress> progressByConsumerId =
            new ConcurrentHashMap<>();

    public void register(String consumerId) {
        progressByConsumerId.computeIfAbsent(consumerId, ignored -> new ConsumerReplayProgress());
    }

    public void onPartitionsAssigned(
            String consumerId,
            Map<TopicPartition, Long> currentPositions,
            Map<TopicPartition, Long> targetEndOffsets) {
        progressFor(consumerId).assign(currentPositions, targetEndOffsets);
    }

    public void onRecordProcessed(String consumerId, ConsumerRecord<?, ?> record) {
        progressFor(consumerId).recordSuccess(
                new TopicPartition(record.topic(), record.partition()),
                positionAfter(record));
    }

    public boolean isFullyConsumed() {
        return progressByConsumerId.isEmpty()
                || progressByConsumerId.values().stream()
                        .allMatch(ConsumerReplayProgress::isCaughtUp);
    }

    // Kafka positions identify the next record to read, not the last one already processed.
    private long positionAfter(ConsumerRecord<?, ?> record) {
        return Math.incrementExact(record.offset());
    }
}

ConsumerReplayProgress owns the assigned partitions, while a small
PartitionReplayProgress owns one partition's current position, fixed target, and error. Keeping
those responsibilities separate makes the readiness rule explicit: every assigned partition must
reach its own target, and a failure is cleared only by a successful record from that same
partition.

An @Around aspect reports every successfully processed ConsumerRecord to the tracker
automatically — the
listener method itself never calls it directly, which is why ConfigTopicListener.onMessage above
has no readiness-related code in it at all, just the @ConfigConsumer("config-topic") annotation.
A separate registrar pre-registers every @ConfigConsumer method at startup, before the first
message arrives. A ConsumerAwareRebalanceListener supplies assignments and their captured end
offsets. That keeps readiness DOWN from startup until the listener has processed through every
target, and recalculates the target after a rebalance.

Third, expose the gate through a normal HealthIndicator in Spring Boot's readiness health
group. This composes the cache state with Boot's own readiness state and gives the deployment
platform one stable endpoint to poll. It also avoids competing with Boot's application-availability
events during startup:

@Component("configConsumption")
@RequiredArgsConstructor
public class ConfigConsumptionHealthIndicator implements HealthIndicator {

    private final ConsumptionTracker tracker;

    @Override
    public Health health() {
        var details = tracker.describe();
        return tracker.isFullyConsumed()
                ? Health.up().withDetails(details).build()
                : Health.down().withDetails(details).build();
    }
}

management:
  endpoint:
    health:
      "group[readiness]":
        include: readinessState,configConsumption

/actuator/health/readiness now reports UP only after every assigned partition reaches its
captured startup target. Configure the deployment platform to hold traffic until that endpoint
returns success.

The example's Docker Compose service uses the same endpoint as its healthcheck, so Docker marks
the application healthy only after the replay completes. This mirrors a deployment platform's
availability check; Compose's health status itself does not control traffic routing.

Fourth, gating scheduled jobs (or any other bean method) the same way, without repeating the
if (!tracker.isFullyConsumed()) return; check at every call site: a second annotation,
@ReadinessGated, plus its own small aspect that skips the call entirely until the gate opens —
this is what CacheSnapshotJob.logSnapshot above is actually annotated with:

@Aspect
@Component
@RequiredArgsConstructor
public class ReadinessGateAspect {

    private final ConsumptionTracker tracker;

    @Around("@annotation(com.example.company.readinessgate.ReadinessGated)")
    public Object skipUntilReady(ProceedingJoinPoint joinPoint) throws Throwable {
        if (!tracker.isFullyConsumed()) {
            log.debug("Skipping {} — readiness gate not open yet", joinPoint.getSignature().toShortString());
            return null;
        }
        return joinPoint.proceed();
    }
}

Same shape as the @ConfigConsumer aspect earlier, just pointed the other direction: that one
reports activity to the tracker, this one reads it and decides whether to let the call through
at all. It only makes sense on void methods — a skipped call returns null — which is exactly
what a @Scheduled job is.

Applying the Readiness Gate

The application must block every interaction with the outside world that depends on this
configuration; otherwise, it can act on an empty or stale cache. The example implements two such
gates, but the same principle applies more broadly.

For HTTP traffic, the ConfigConsumptionHealthIndicator keeps the readiness endpoint DOWN until
the replay is complete. Controllers do not need to repeat tracker.isFullyConsumed() in every
endpoint when the deployment platform uses that endpoint to withhold traffic. For example, in
Kubernetes, use a readinessProbe for routing and give a startupProbe enough time for a large
initial replay:

@GetMapping("/api/cache")
public ResponseEntity<Map<String, CacheEntry>> all() {
    return ResponseEntity.ok(cache.snapshot());
}

For a scheduled job, @ReadinessGated prevents invocation until the same tracker is ready:

@ReadinessGated
@Scheduled(fixedDelayString = "${app.cache-snapshot-job.interval}")
public void logSnapshot() {
    log.info("Local cache holds {} entries", cache.size());
}

Other external interactions need an equivalent gate. For example, a queue listener that depends on
this configuration should not begin processing until the cache is ready; arrange that at the
listener/container level or with an acknowledgement strategy that does not acknowledge or lose
messages while the gate is closed. This example provides gates for HTTP traffic and scheduled jobs
only, but the same readiness signal can drive equivalent protection for queue consumers, outgoing
calls, or any other configuration-dependent work.

Where This Pattern Bites

The happy path is attractive: replay once, then serve local reads. The operational cost appears at
the boundaries—startup, lag, retention, and failure. These are the decisions I would make explicit
before choosing the pattern.

Every Restart Pays the Replay Cost

A new group has no offsets, so every pod rebuilds from the beginning. The cost therefore grows with
the records still present in the compacted log—not simply with the number of current keys, because
compaction is asynchronous and old segments may remain for a while. Measure replay time with
production-like data and give the startupProbe enough budget. Kafka already fetches records in
batches, so tune max.poll.records and keep per-record work cheap. For larger replays, switch the
listener to batch mode and apply each returned list in one invocation; the example's tracking
aspect supports both individual ConsumerRecord arguments and iterable batches. Add listener
concurrency only when the topic has enough partitions to use it. Three partitions allow at most
three consumer threads in one pod to make progress in parallel.
The worst-case replay must also fit inside the topic's tombstone retention window, or a scan from
offset zero is not guaranteed to reconstruct a valid snapshot.

The captured targets do not move when producers append new records during replay, so continuous
writes do not make startup unbounded. The existing log is what determines startup time. If that log
is large enough to make readiness slow or unpredictable, the topic is probably outside this
pattern's intended low-volume configuration use case.

Unique Groups Leave Temporary Metadata

Each pod generates a new consumer group. After the pod disappears, its committed offsets remain
visible until Kafka removes inactive group metadata according to broker retention settings. This is
usually operational clutter, not an ever-growing live workload, but it affects dashboards and any
automation that assumes every listed group is active. Use an identifiable prefix, monitor the
broker's
offset-retention policy,
and filter inactive groups in tooling.

Large Configuration Does Not Belong in Every Pod

This pattern is a good fit for a small, frequently read configuration set. If the configuration is
large, keeping the whole dataset in local memory is usually a bad idea. Every pod needs enough heap
for its own copy, and adding pods multiplies that memory cost. Keep large configuration in a shared
store, or materialize only the small subset each service actually needs.

Eventual Consistency Is Part of the Contract

Each pod consumes updates independently, so two pods can briefly return different values. A slow
consumer, rebalance, retry, or Kafka outage can extend that gap. This is acceptable for feature
flags, routing hints, and other configuration that tolerates short-lived staleness; it is a poor fit
for authorization decisions, inventory, balances, or any rule that must change everywhere at the
same instant. Define the tolerated lag, monitor it, and make the stale-data behavior explicit to
callers before choosing this design.

These costs do not make the pattern wrong. They define its useful range: compact, replayable,
eventually consistent reference data with a very hot read path and a team already comfortable
operating Kafka.

Interactive production lab

Can you keep this architecture alive?

Diagnose four failures involving consumer groups, startup replay, readiness, and tombstones.
It takes about five minutes.

Start the incident lab

Code

Code from this article is available at
github.com/javaAndScriptDeveloper/kafka-backed-local-read-replica-article.
The repository's README contains the setup and run instructions, including how to inspect consumer
groups, observe tombstone propagation, and scale the application locally.

Let's Talk

If you're running—or have run—something similar, I would like to hear where the trade-offs landed
for you. Contact links are in the footer.

FAQ

Can Kafka replace Redis for cache synchronization across pods?

Yes, for the specific job of broadcasting state to every instance — it is a different tool doing a related job, not a drop-in replacement. Redis gives every pod one shared cache view, while Kafka gives every pod its own eventually-consistent copy built by replaying a compacted topic. Use this pattern when local read latency matters and short-lived staleness after an update is acceptable.

With N pods, does every pod receive every Kafka message, or does Kafka split messages across them?

It depends entirely on consumer group, not on Kafka itself. Pods sharing one consumer group get partitions split between them (load-balanced) — each message goes to exactly one pod. Give every pod its own, distinct consumer group and each group gets the full topic independently, so every pod sees every message. That's the broadcast pattern this article covers.

How do I give every pod its own Kafka consumer group?

Combine a human-controlled prefix with Spring's per-instance ${random.uuid} value: group-id: ${CONSUMER_GROUP_PREFIX}-${random.uuid}. Each application instance gets a unique group, so every pod consumes the full topic independently.

How do you delete a key when the Kafka topic is compacted?

Publish a tombstone: a record with the same key and a null value. Consumers that encounter it evict the key instead of caching null. Kafka eventually removes both the superseded value and the tombstone; size delete.retention.ms so a full replay can finish within that window, which Kafka documents as the bound for reconstructing a valid snapshot from offset zero.

Should scheduled jobs and traffic wait until Kafka consumption catches up?

Yes. A pod that starts serving requests or running scheduled jobs before it has replayed the topic is working from an empty or partial cache. Keep readiness DOWN until startup replay is complete. The example captures each partition's end offset when it is assigned and opens readiness only after every processed position reaches its target.

Can continuous updates keep a new pod stuck in startup forever?

No. The readiness gate captures a fixed end offset for each partition when Kafka assigns it. Records appended after that snapshot do not move the startup target; the pod processes them as normal eventually-consistent updates. Startup can still take a long time when the existing compacted log is large, which is one reason this pattern is best for small, low-volume upsert topics such as feature flags and service configuration.

Originally published at codewithkyryl.dev.

The Transactional Outbox Pattern: From a Single Task to a Production-Grade Async Engine

Kyryl — Tue, 13 Oct 2026 08:40:09 +0000

There is a class of bugs that never shows up in your unit tests and rarely appears in staging.
It appears in production, at 2am, when your app saves a record to the database, then tries to
call an external service — and one of them fails. One side committed. The other didn't. Nobody
knows until a user complains.

This is the dual-write problem, and the Transactional Outbox Pattern is the standard answer
to it. The idea is simple: instead of calling the external system directly, write a task into the
same database transaction as your domain change. A background worker picks it up and executes it.
The write is atomic. The task cannot be lost.

What's less documented is everything that comes after the basic idea — what happens when tasks
fail, when two workers race for the same row, when your app crashes mid-execution, when you have
ten different task types living in one table, and when your stakeholders ask why the queue depth
spiked at 3am last Tuesday.

That's what this article covers.

I'll work through a single concrete scenario — a report generation service — and evolve it
step by step from the simplest possible outbox to something I'd be comfortable running in
production. Every code snippet is from a real working repo linked at the end.

Stack: Java, Spring Boot, PostgreSQL.

TL;DR

The Transactional Outbox Pattern fixes the dual-write problem by writing a task row in the same DB transaction as your domain change, then executing it from a background worker.

Poll the outbox table with SELECT ... FOR UPDATE SKIP LOCKED so multiple workers can run concurrently without double-processing.

Production-grade requires more than the basic idea: a state machine, retries with backoff, heartbeats/leases to recover crashed workers, ordering guarantees, and monitoring of queue depth.

You don't need Kafka — PostgreSQL alone is enough for reliable async execution.

The Scenario

sequenceDiagram
    actor User as User (Backoffice)
    participant API as ReportGenerationService
    participant DB as PostgreSQL

    User->>API: POST /reports (from, to)
    API->>DB: INSERT report_generation_tasks (PENDING)
    DB-->>API: task saved
    API-->>User: 202 Accepted

    loop Scheduler every 1s
        API->>DB: SELECT ... WHERE state = PENDING FOR UPDATE SKIP LOCKED
        DB-->>API: task row
        API->>API: generateReport()
        API->>DB: UPDATE state = COMPLETED
    end

We're building a CRM. Users can request monthly analytics reports from a backoffice UI.
Generating a report means aggregating tens of millions of rows — it can take minutes.
Obviously we can't do this synchronously in an HTTP request.

The user clicks a button. The backend accepts the request, saves it, and responds immediately.
A background worker picks up the task, runs the generation, and updates the status when done.

This is the simplest case where an outbox makes sense: work that is genuinely async by nature,
where the user already expects to wait.

Part 1 — The Basic Outbox

What Every Outbox Needs

At its core, any outbox table needs three things:

State — where is this task in its lifecycle?
Payload — what exactly needs to be executed?
Timestamps — when was it created, when did it last change?

CREATE TYPE task_state AS ENUM ('PENDING', 'COMPLETED');

CREATE TABLE report_generation_tasks (
                                       id         UUID       PRIMARY KEY DEFAULT gen_random_uuid(),
                                       state      task_state NOT NULL    DEFAULT 'PENDING',
                                       payload    JSONB      NOT NULL,
                                       created_at TIMESTAMP  NOT NULL    DEFAULT now(),
                                       updated_at TIMESTAMP  NOT NULL    DEFAULT now()
);

The payload for our use case is a date range:

@Value
@Builder
@Jacksonized
public class ReportGenerationPayload {
  Instant from;
  Instant to;
}

The JPA entity:

@Entity
@Table(name = "report_generation_tasks")
@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
public class ReportGenerationEntity {

    @Id
    @GeneratedValue(strategy = GenerationType.UUID)
    @Column(updatable = false, nullable = false)
    private UUID id;

    @Enumerated(EnumType.STRING)
    @Column(nullable = false, columnDefinition = "task_state")
    private TaskState state;

    @JdbcTypeCode(SqlTypes.JSON)
    @Column(nullable = false, updatable = false, columnDefinition = "jsonb")
    private ReportGenerationPayload payload;

    @CreationTimestamp
    @Column(nullable = false, updatable = false)
    private Instant createdAt;

    @UpdateTimestamp
    @Column(nullable = false)
    private Instant updatedAt;
}

Submitting a task:

@Service
@RequiredArgsConstructor
public class ReportGenerationService {

    private final ReportGenerationRepository repository;

    public void submit(ReportGenerationCommand command) {
        repository.save(ReportGenerationEntity.builder()
                .state(TaskState.PENDING)
                .payload(ReportGenerationPayload.builder()
                        .from(command.getFrom())
                        .to(command.getTo())
                        .build())
                .build());
    }
}

The Worker

A scheduled job polls for pending tasks and executes them:

@Component
@RequiredArgsConstructor
public class ReportGenerationScheduledJob {

    private final ReportGenerationTaskService taskService;

    @Scheduled(fixedDelay = 1000)
    public void run() {
        taskService.fetchAndProcess();
    }
}

@Service
@RequiredArgsConstructor
public class ReportGenerationTaskService {

    private final ReportGenerationRepository repository;

    public void fetchAndProcess() {
        repository.findFirstByStateOrderByCreatedAt(TaskState.PENDING)
                .ifPresent(task -> {
                    generateReport(task);
                    task.setState(TaskState.COMPLETED);
                    repository.save(task);
                });
    }

    private void generateReport(ReportGenerationEntity task) {
        // actual report generation logic
    }
}

public interface ReportGenerationRepository extends JpaRepository<ReportGenerationEntity, UUID> {
    Optional<ReportGenerationEntity> findFirstByStateOrderByCreatedAt(TaskState state);
}

This works in a single-instance app with no failures. In the real world, it has two problems:
two workers can claim the same task simultaneously, and tasks can fail without any recovery path.
Let's fix both.

Part 2 — Concurrency: Two Workers, One Task

When two scheduler threads — or two app instances — run the fetch query at the same moment,
both can read the same PENDING row and both will try to execute it. The result is duplicate work
at best, corrupted state at worst.

The fix is a database lock taken at fetch time. PostgreSQL's SELECT ... FOR UPDATE SKIP LOCKED
is purpose-built for exactly this use case: it locks the row for the current transaction and skips
any rows that are already locked by another transaction, rather than waiting.

public interface ReportGenerationRepository extends JpaRepository<ReportGenerationEntity, UUID> {

    @Lock(LockModeType.PESSIMISTIC_WRITE)
    @QueryHint(name = AvailableSettings.JAKARTA_LOCK_TIMEOUT, value = "-2") // -2 = SKIP_LOCKED
    Optional<ReportGenerationEntity> findFirstByStateOrderByCreatedAt(TaskState state);
}

The SKIP_LOCKED hint value -2 is what Hibernate translates to the SKIP LOCKED clause.
Since there's no first-class annotation for this, it's worth extracting into a reusable annotation
so you don't scatter the magic number across your repositories:

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.ANNOTATION_TYPE})
@QueryHints(@QueryHint(
    name = AvailableSettings.JAKARTA_LOCK_TIMEOUT,
    value = Timeouts.SKIP_LOCKED_MILLI + ""
))
public @interface SkipLocked {}

Much cleaner. Now the repository reads like intent, not plumbing:

@SkipLocked
@Lock(LockModeType.PESSIMISTIC_WRITE)
Optional<ReportGenerationEntity> findFirstByStateOrderByCreatedAt(TaskState state);

You can keep the lock while processing by wrapping the whole fetch-execute-update cycle in a
single transaction, but you generally should not:

// Technically works, but do not use this pattern for outbox processing.
@Transactional
public void fetchAndProcess() {
    repository.findFirstByStateOrderByCreatedAt(TaskState.PENDING)
            .ifPresent(task -> {
                generateReport(task);
                task.setState(TaskState.COMPLETED);
                repository.save(task);
            });
}

sequenceDiagram
    participant DB as PostgreSQL
    participant A as Worker A
    participant B as Worker B

    Note over DB: task row: PENDING (unlocked)

    par Worker A and Worker B fetch simultaneously
        A->>DB: SELECT ... FOR UPDATE SKIP LOCKED
        B->>DB: SELECT ... FOR UPDATE SKIP LOCKED
    end

    DB-->>A: row locked ✓ — returns task
    DB-->>B: row locked by A — SKIP LOCKED returns empty

    Note over B: no task found, exits cleanly

    A->>A: generateReport()
    A->>DB: UPDATE state = COMPLETED
    A->>DB: COMMIT — lock released

    Note over DB: task row: COMPLETED (unlocked)

Even if generateReport() is small enough that this technically works, running an outbox action
under an open transaction is an antipattern. Transaction duration should not depend on outbox
execution duration: the action can become slower over time, block on I/O, or call a downstream
service, leaving a database connection and row lock held for an unpredictable period.

Instead, claim the row in a short transaction by moving it to PROCESSING, commit and release the
lock, and only then execute the outbox action. Record COMPLETED or FAILED in another short
transaction afterward. The next section implements that state transition and its recovery path.

Part 2.5 — Reducing Latency with Application Events

The polling scheduler has an inherent latency floor. If your job fires every second, a task
submitted at t=0ms sits PENDING until the next tick — up to 1000ms later. For most async
workloads that's acceptable. For cases where you want near-immediate pickup — sending a
transactional email, triggering a webhook — it's not.

The pattern borrowed from Celery: save the outbox entity to the database, then immediately
publish a Spring ApplicationEvent carrying the new task's ID. A listener picks it up and
dispatches processing right away, without waiting for the next scheduler tick.

sequenceDiagram
    participant S as Service
    participant DB as PostgreSQL
    participant EL as Event Listener
    participant W as Worker

    rect rgb(240, 240, 240)
        Note over S,W: Polling approach — up to 1s latency
        S->>DB: INSERT task (PENDING)
        Note over W: ... waiting for next scheduler tick (0–1000ms) ...
        W->>DB: SELECT PENDING tasks
        DB-->>W: task found
        W->>W: process task
    end

    rect rgb(230, 247, 240)
        Note over S,W: Application event approach — near-zero latency
        S->>DB: INSERT task (PENDING)
        S->>EL: publishEvent(taskId) — fires AFTER_COMMIT
        EL->>W: trigger fetchAndProcess() immediately
        W->>DB: SELECT PENDING tasks
        DB-->>W: task found
        W->>W: process task
    end

The scheduler still runs as a safety net — it catches anything the event listener missed (app
restart, listener exception, missed event). But the happy path bypasses the polling delay
entirely.

Implementation

Publish the event inside the service method, after saving:

@Service
@RequiredArgsConstructor
public class ReportGenerationService {

    private final ReportGenerationRepository repository;
    private final ApplicationEventPublisher eventPublisher;

    @Transactional
    public void submit(ReportGenerationCommand command) {
        var entity = repository.save(ReportGenerationEntity.builder()
                .state(TaskState.PENDING)
                .payload(ReportGenerationPayload.builder()
                        .from(command.getFrom())
                        .to(command.getTo())
                        .build())
                .build());

        eventPublisher.publishEvent(new ReportGenerationTaskSubmittedEvent(entity.getId()));
    }
}

public record ReportGenerationTaskSubmittedEvent(UUID taskId) {}

The listener reacts and triggers processing immediately:

@Component
@RequiredArgsConstructor
public class ReportGenerationTaskEventListener {

    private final ReportGenerationTaskService taskService;

    @TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT)
    public void onTaskSubmitted(ReportGenerationTaskSubmittedEvent event) {
        taskService.fetchAndProcess();
    }
}

The Classic Mistake

The most common error here is using @EventListener instead of
@TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT).

@EventListener fires the moment publishEvent() is called — which is inside the open
transaction, before the INSERT has committed. The listener calls fetchAndProcess(),
which queries the database for the task by ID. The row does not exist yet. The task is silently
skipped, the event is lost, and you're left wondering why processing never started — even though
the scheduler eventually picks it up seconds later and everything looks fine in the logs.

// ❌ fires inside the transaction — entity not visible to other connections yet
@EventListener
public void onTaskSubmitted(ReportGenerationTaskSubmittedEvent event) {
    taskService.fetchAndProcess();
}

// ✅ fires after COMMIT — entity is visible, fetch succeeds
@TransactionalEventListener(phase = TransactionPhase.AFTER_COMMIT)
public void onTaskSubmitted(ReportGenerationTaskSubmittedEvent event) {
    taskService.fetchAndProcess();
}

AFTER_COMMIT guarantees the listener runs only after the transaction that published the event
has fully committed. By that point the row is visible to all connections and the fetch will
find it.

The scheduler remains unchanged — it acts as the recovery mechanism for any tasks the event
listener missed. The two approaches complement each other: events for latency, polling for
reliability.

Part 3 — Failures, Retries, and the PROCESSING State

stateDiagram-v2
    [*] --> PENDING: submit()

    PENDING --> PROCESSING: worker claims task\n(TX1 commits)

    PROCESSING --> COMPLETED: generateReport() succeeds\n(TX2 commits)
    PROCESSING --> FAILED: generateReport() throws\n(TX2 commits)
    PROCESSING --> PENDING: app crash detected\nstuck-task recovery job resets

    FAILED --> PROCESSING: retry eligible\n(retryCount ≤ MAX and executeAt ≤ now)
    FAILED --> [*]: retryCount > MAX\n(dead — manual ops)

    COMPLETED --> [*]: cleanup job removes\nafter retention period

Outbox actions should not run under an open database transaction. The locked-transaction approach
holds a database lock for the entire
duration of generateReport(). For a report that takes two minutes, that's two minutes of a
locked row — a connection tied up, other DB operations potentially queuing behind it. With
SKIP LOCKED workers don't block each other on the lock, but long-running transactions still
put pressure on the connection pool and make the database unhappy under load. A very small action
may technically complete without causing visible trouble, but coupling transaction duration to
outbox execution duration is still an antipattern.

The fix is to separate claiming a task from executing it. Flip the task to PROCESSING in a
short transaction, release the lock, then do the actual work outside any transaction. This way
the DB is only touched briefly at the start and briefly at the end — the expensive part happens
in between without holding anything.

First, extend the enum:

ALTER TYPE task_state ADD VALUE 'PROCESSING';
ALTER TYPE task_state ADD VALUE 'FAILED';

ALTER TABLE report_generation_tasks
    ADD COLUMN retry_count  INTEGER NOT NULL DEFAULT 0,
    ADD COLUMN error_message VARCHAR;

retry_count tracks how many times execution was attempted. We cap retries at 10 — beyond that,
something is genuinely broken and needs manual intervention. We'll wire alerting around this
later. error_message stores the last exception so you know why a task is stuck without
digging through logs.

The Java enum grows with it:

public enum TaskState {
    PENDING,
    PROCESSING,
    COMPLETED,
    FAILED
}

The service now runs two separate short transactions with the actual work in between:

@RequiredArgsConstructor
public class ReportGenerationTaskService {

    private final ReportGenerationRepository repository;
    private final TransactionTemplate transactionTemplate;

    private static final int MAX_RETRY_COUNT = 10;

    public void fetchAndProcess() {
        transactionTemplate
                .execute(status ->
                        repository.findFirstEligibleTask(
                                List.of(TaskState.PENDING, TaskState.FAILED),
                                MAX_RETRY_COUNT
                        ).map(this::markProcessing))
                .ifPresentOrElse(
                        task -> {
                            try {
                                generateReport(task);
                                markCompleted(task);
                            } catch (Exception e) {
                                markFailed(task, e);
                                log.error("Task {} failed on attempt {}", task.getId(), task.getRetryCount(), e);
                            }
                        },
                        () -> log.debug("No eligible tasks to process")
                );
    }

    private ReportGenerationEntity markProcessing(ReportGenerationEntity task) {
        task.setState(TaskState.PROCESSING);
        return repository.save(task);
    }

    private void markCompleted(ReportGenerationEntity task) {
        task.setState(TaskState.COMPLETED);
        repository.save(task);
    }

    private void markFailed(ReportGenerationEntity task, Exception e) {
        task.setState(TaskState.FAILED);
        task.setRetryCount(task.getRetryCount() + 1);
        task.setErrorMessage(e.getMessage()); // don't skip this — it's the only trace of what went wrong without digging through logs
        repository.save(task);
    }
}

The query fetches both PENDING and FAILED tasks — a failed task with retryCount < 10 is
just as eligible as a fresh one. Once retry count hits the cap, the task stops being fetched and
sits as a permanent FAILED record until someone looks at it. That's intentional — beyond 10
attempts something is genuinely broken and needs eyes on it, not another retry.

@SkipLocked
@Lock(LockModeType.PESSIMISTIC_WRITE)
@Query("""
    SELECT t FROM ReportGenerationEntity t
    WHERE t.state IN :states
      AND t.retryCount <= :maxRetryCount
    ORDER BY t.createdAt ASC
    LIMIT 1
    """)
Optional<ReportGenerationEntity> findFirstEligibleTask(
    @Param("states") List<TaskState> states,
    @Param("maxRetryCount") Integer maxRetryCount
);

But wait — should a just-failed task really be picked up again on the very next scheduler tick,
seconds later? If the downstream system is down, hammering it every second makes things worse.
That's what the next section covers.

Part 4 — Exponential Backoff

Without a delay between retries, a failed task re-enters the queue on the very next scheduler
tick — usually seconds later. If the downstream system is down, you're hammering it at full
polling rate. That's bad for the downstream system and bad for your DB.

The fix is an execute_at column: the task is not eligible until this timestamp is in the past.
Fresh tasks get execute_at = now() so they're picked up immediately. Failed tasks get
execute_at = now() + delay, where the delay grows with each retry.

ALTER TABLE report_generation_tasks
    ADD COLUMN execute_at TIMESTAMP NOT NULL DEFAULT NOW();

The delay schedule drives everything — MAX_RETRY_COUNT is derived from it, not declared
separately. This way the two can never drift out of sync:

private static final Map<Integer, Duration> RETRY_DELAYS = Map.of(
    0, Duration.ofMinutes(1),
    1, Duration.ofMinutes(5),
    2, Duration.ofMinutes(15),
    3, Duration.ofMinutes(30),
    4, Duration.ofHours(1),
    5, Duration.ofHours(2),
    6, Duration.ofHours(4),
    7, Duration.ofHours(8),
    8, Duration.ofHours(12),
    9, Duration.ofDays(1)
);

private static final int MAX_RETRY_COUNT = RETRY_DELAYS.size();

Add a new entry to RETRY_DELAYS and the retry cap automatically updates. No magic number to
keep in sync elsewhere.

In markFailed(), capture the current retry count before incrementing — that selects the
delay for the next attempt:

private void markFailed(ReportGenerationEntity task, Exception e) {
    int currentRetry = task.getRetryCount();
    task.setState(TaskState.FAILED);
    task.setRetryCount(currentRetry + 1);
    task.setErrorMessage(e.getMessage());
    task.setExecuteAt(Instant.now().plus(
            RETRY_DELAYS.getOrDefault(currentRetry, RETRY_DELAYS.get(RETRY_DELAYS.size() - 1))));
    repository.save(task);
}

The scheduler keeps running at the same rate. It just skips tasks whose window hasn't opened.
Failed tasks sit quietly until their time comes, then re-enter naturally.

Part 5 — Stuck Tasks and Heartbeats

sequenceDiagram
    participant W as Worker
    participant DB as PostgreSQL
    participant R as Recovery Job

    W->>DB: UPDATE state = PROCESSING (TX1 commits)
    W->>W: generateReport() — running...

    Note over W: 💥 app crashes mid-execution

    Note over DB: task stuck in PROCESSING<br/>heartbeat_at stops updating

    loop Recovery job runs every 5 min
        R->>DB: SELECT WHERE state = PROCESSING<br/>AND heartbeat_at < now() - 1h
        DB-->>R: stuck task found
        R->>DB: UPDATE state = PENDING
        Note over DB: task back in queue — will be retried
    end

With the two-transaction approach, consider what happens if the app crashes after marking a task
PROCESSING but before marking it COMPLETED or FAILED. The DB shows PROCESSING. Nobody is
processing it. It sits there forever.

The naive fix is a recovery job that finds tasks stuck in PROCESSING for too long and resets
them to PENDING:

@Transactional
public void recoverStuckTasks() {
    repository.findFirstByStateAndUpdatedAtLessThan(
                    TaskState.PROCESSING,
                    Instant.now().minus(1, ChronoUnit.HOURS))
            .ifPresent(task -> {
                log.warn("Resetting stuck task {} to PENDING", task.getId());
                task.setState(TaskState.PENDING);
                repository.save(task);
            });
}

This works for most cases. The problem: updated_at changes whenever anything changes on the
row — including unrelated state transitions. A task legitimately processing for 55 minutes looks
identical to a crashed task from the updated_at perspective.

The cleaner solution is a heartbeat: a dedicated column that gets bumped periodically only
while the task is being actively processed. The recovery job now checks this column, not
updated_at. A task that hasn't heartbeated in the last hour is genuinely stuck.

ALTER TABLE report_generation_tasks ADD COLUMN heartbeat_at TIMESTAMP;

The heartbeat runs on its own thread, inside its own transactions, independent of the processing
logic. The cleanest implementation is a try-with-resources — it starts when processing starts
and stops on any exit path:

public abstract class Heartbeat implements AutoCloseable {

    private final ScheduledExecutorService executor;

    protected Heartbeat(Runnable heartbeatFn) {
        executor = Executors.newSingleThreadScheduledExecutor();
        executor.scheduleAtFixedRate(heartbeatFn, 0, 10, TimeUnit.SECONDS);
    }

    @Override
    public void close() {
        executor.shutdownNow();
    }
}

public class ReportGenerationTaskHeartbeat extends Heartbeat {

    ReportGenerationTaskHeartbeat(UUID taskId, ReportGenerationRepository repository) {
        super(() -> repository.updateHeartbeatAt(taskId, Instant.now()));
    }
}

The repository method carries its own @Transactional — no need to wrap the call manually:

@Modifying
@Transactional
@Query("UPDATE ReportGenerationTaskEntity t SET t.heartbeatAt = :now WHERE t.id = :id")
void updateHeartbeatAt(@Param("id") UUID id, @Param("now") Instant now);

Wire it via a factory so the service stays clean:

@Component
@RequiredArgsConstructor
public class ReportGenerationTaskHeartbeatFactory {

    private final ReportGenerationRepository repository;

    public Heartbeat start(UUID taskId) {
        return new ReportGenerationTaskHeartbeat(taskId, repository);
    }
}

The processing block now looks like this:

public void fetchAndProcess() {
    transactionTemplate
            .execute(status -> repository.findFirstEligibleTask(
                    List.of(TaskState.PENDING, TaskState.FAILED),
                    MAX_RETRY_COUNT
            ).map(this::markProcessing))
            .ifPresentOrElse(
                    task -> {
                        try (var heartbeat = heartbeatFactory.start(task.getId())) {
                            try {
                                generateReport(task);
                                markCompleted(task);
                            } catch (Exception e) {
                                markFailed(task, e);
                                log.error("Task {} failed", task.getId(), e);
                            }
                        }
                    },
                    () -> log.debug("No eligible tasks to process")
            );
}

The recovery job uses heartbeat_at instead of updated_at:

@Modifying
@Transactional
@Query("UPDATE ReportGenerationEntity t SET t.heartbeatAt = :now WHERE t.id = :id")
void updateHeartbeatAt(@Param("id") UUID id, @Param("now") Instant now);

Optional<ReportGenerationEntity> findFirstByStateAndHeartbeatAtLessThan(
        TaskState state, Instant threshold);

updated_at reflects when the task state last changed. heartbeat_at reflects whether the
task is alive right now. Keeping them separate means you can answer both questions without
ambiguity.

Part 6 — Multi-Type Outbox

flowchart TB
    subgraph before["❌ Before — separate table per type"]
        direction TB
        W1[ReportWorker] --> T1[(report_generation_tasks)]
        W2[EmailWorker]  --> T2[(email_notification_tasks)]
        W3[WebhookWorker]--> T3[(webhook_delivery_tasks)]
    end

    before --> after

    subgraph after["✅ After — one table, handler per type"]
        direction TB
        OT[(outbox_tasks\ntask_type column)] --> D[Dispatcher]
        D --> H1[ReportHandler]
        D --> H2[EmailHandler]
        D --> H3[WebhookHandler]
    end

Six months in, your service handles more than report generation. You need email notifications.
Then data exports. Then webhook deliveries.

You have two options: a separate table per task type, or one shared table with a task_type
discriminator. The shared table wins when task types are numerous and you're comfortable
processing them from one place. The infrastructure — retries, heartbeats, backoff, monitoring —
stays in one place. Adding a new task type is just adding a new handler.

The tradeoffs worth knowing upfront:

A misbehaving new task type can slow down all other types — one bad handler burning retries creates backpressure across the whole table.
Ordering guarantees become harder — more on this below.

The Schema

CREATE TYPE outbox_task_type AS ENUM ('REPORT_GENERATION', 'EMAIL_NOTIFICATION');

CREATE TABLE outbox_tasks (
    id            UUID             PRIMARY KEY DEFAULT gen_random_uuid(),
    task_type     outbox_task_type NOT NULL,
    state         task_state       NOT NULL    DEFAULT 'PENDING',
    payload       JSONB            NOT NULL,
    retry_count   INTEGER          NOT NULL    DEFAULT 0,
    error_message VARCHAR,
    execute_at    TIMESTAMPTZ      NOT NULL    DEFAULT NOW(),
    heartbeat_at  TIMESTAMPTZ,
    created_at    TIMESTAMPTZ      NOT NULL    DEFAULT now(),
    updated_at    TIMESTAMPTZ      NOT NULL    DEFAULT now()
);

The Handler Interface

The key extensibility point — one implementation per task type:

public interface OutboxTaskHandler<P> {
    OutboxTaskType getType();
    Class<P> payloadClass();
    void handle(UUID taskId, P payload) throws Exception;
}

taskId is passed into every handler. You'll see why in the idempotency section.

A concrete handler:

@Component
public class EmailNotificationHandler implements OutboxTaskHandler<EmailNotificationPayload> {

    @Override
    public OutboxTaskType getType() {
        return OutboxTaskType.EMAIL_NOTIFICATION;
    }

    @Override
    public Class<EmailNotificationPayload> payloadClass() {
        return EmailNotificationPayload.class;
    }

    @Override
    public void handle(UUID taskId, EmailNotificationPayload payload) throws Exception {
        log.info("Sending email to {} — task {}", payload.getRecipient(), taskId);
        // call email provider
    }
}

The Dispatcher

The service collects all handler beans at startup, builds a lookup map, and routes each task by
taskType:

@Service
@RequiredArgsConstructor
public class OutboxTaskService {

    private final OutboxTaskRepository repository;
    private final ObjectMapper objectMapper;
    private final List<OutboxTaskHandler<?>> handlerList;

    private Map<OutboxTaskType, OutboxTaskHandler<?>> handlers;

    @PostConstruct
    void init() {
        handlers = handlerList.stream()
                .collect(Collectors.toMap(OutboxTaskHandler::getType, Function.identity()));
    }

    @SuppressWarnings("unchecked")
    private void dispatch(OutboxTaskEntity task) throws Exception {
        var handler = (OutboxTaskHandler<Object>) handlers.get(task.getTaskType());
        if (handler == null) {
            throw new IllegalStateException(
                    "No handler registered for task type: " + task.getTaskType());
        }
        var payload = objectMapper.readValue(task.getPayload(), handler.payloadClass());
        handler.handle(task.getId(), payload);
    }
}

Registering a new task type is: add enum value, add handler bean, done. The dispatcher, the
scheduler, the retry logic — none of it changes.

Part 7 — Ordering

If your use case doesn't care about processing order — fire-and-forget notifications, independent
background jobs — skip this section. But even then, fetching earlier-inserted tasks first is the
more honest behaviour: tasks submitted first have been waiting longer, and processing them in
submission order is what users would reasonably expect.

Without an explicit ORDER BY, PostgreSQL makes no guarantee about row return order. Always
sort by created_at ASC in your fetch query:

@Query("""
    SELECT t FROM OutboxTaskEntity t
    WHERE t.state IN :states
      AND t.retryCount <= :maxRetryCount
      AND t.executeAt <= :now
    ORDER BY t.createdAt ASC
    LIMIT 1
    """)
Optional<OutboxTaskEntity> findFirstEligibleTask(...);

Back this with a partial index:

CREATE INDEX idx_outbox_fetch
    ON outbox_tasks (created_at ASC)
    WHERE state IN ('PENDING', 'FAILED');

The Clock Skew Trap

First, be honest about whether this matters for your use case. If your tasks are independent —
report generation, email sending, webhook delivery — processing order usually doesn't matter at
all. Task A finishing before task B is fine regardless of which was submitted first. In that case
this whole section is not your problem.

If you do need FIFO — tasks for the same entity that must be processed in submission order —
then read on.

The issue is that created_at is generated on the application side. Each app instance has its
own system clock. If instance A is 200ms ahead of instance B, a task submitted by B at t=100ms
gets a created_at that is earlier than a task submitted by A at t=50ms — even though A's
task was inserted first. The fetch query sorts by created_at and picks up B's task first.
Submission order is inverted.

NTP keeps clocks close but not perfectly synchronized. Under normal conditions the drift is
milliseconds. Under load, during VM migrations, or after a container restart, it can be more.

The fix is to generate the ordering value on the database side — either let PostgreSQL assign
created_at via DEFAULT now() rather than letting the application set it, or use a
database-backed sequence which is guaranteed monotonically increasing regardless of which
instance inserted the row. Here's the sequence approach:

ALTER TABLE outbox_tasks ADD COLUMN seq BIGSERIAL;

@Column(insertable = false, updatable = false)
private Long seq;

Sort by seq ASC. The value is assigned by a single authoritative source — the database — so
it reflects true insertion order regardless of which instance submitted the task or what that
instance's clock said.

Keep created_at for observability. Use seq for ordering.

If strict FIFO matters in your system — don't forget this.

Part 8 — Indexes That Keep the Table Fast at Scale

Your outbox table is polled every second, often multiple times per second. Every row in it is
either active work (PENDING, FAILED, PROCESSING) or historical noise (COMPLETED). Without the
right indexes, each poll scans the entire table to find the handful of eligible rows.

The right tool is a partial index — one that only includes rows matching a predicate. When a
task transitions to COMPLETED, it falls out of the partial index automatically. The index stays
small forever, regardless of how many tasks the table has ever processed.

The Fetch Index — the hot path

Index for the fetch query:

CREATE INDEX idx_outbox_fetch
    ON outbox_tasks (created_at ASC)
    WHERE state IN ('PENDING', 'FAILED');

To test it: inserted 50,000 rows with only 7 in PENDING state, enabled Hibernate SQL debug
logging to get the exact generated query in the console, ran EXPLAIN ANALYZE over that SQL:

Limit  (cost=0.14..27.50 rows=1 width=102) (actual time=0.082..0.084 rows=1 loops=1)
  ->  LockRows  (actual time=0.081..0.082 rows=1 loops=1)
        ->  Index Scan using idx_outbox_fetch on outbox_tasks
              (actual time=0.030..0.030 rows=1 loops=1)
Planning Time: 1.230 ms
Execution Time: 0.139 ms

0.03ms to find the oldest eligible task. The 49,993 COMPLETED rows were never read. Looks good to me.

Two more indexes for the other two background operations:

For the recovery job:

CREATE INDEX idx_outbox_stuck
    ON outbox_tasks (heartbeat_at ASC, created_at ASC)
    WHERE state = 'PROCESSING';

For the cleanup job:

CREATE INDEX idx_outbox_cleanup
    ON outbox_tasks (updated_at ASC)
    WHERE state = 'COMPLETED';

Part 9 — Cleanup

Your outbox is an operational table. Every fetch query scans it. Old rows are dead weight.

The rule: only delete COMPLETED tasks. FAILED tasks are your audit trail. Leave them alone.
Handle them manually or via a separate ops process when needed.

@Modifying
@Transactional
@Query("""
    DELETE FROM OutboxTaskEntity t
    WHERE t.state IN :states
      AND t.updatedAt < :before
    """)
int deleteByStateInAndUpdatedAtBefore(
    @Param("states") List<TaskState> states,
    @Param("before") Instant before
);

private static final int CLEANUP_RETENTION_DAYS = 30;

@Transactional
public void cleanup() {
    var cutoff = Instant.now().minus(CLEANUP_RETENTION_DAYS, ChronoUnit.DAYS);
    int deleted = repository.deleteByStateInAndUpdatedAtBefore(
            List.of(TaskState.COMPLETED), cutoff);
    log.info("Cleanup: removed {} completed tasks older than {} days", deleted, CLEANUP_RETENTION_DAYS);
}

How long to retain: I've seen everything from 1 day to 3 months. If you're unsure — keep longer.
The cost of retaining old rows is a slightly larger table. The cost of deleting them too early
is losing your debugging history on the day you need it most.

Part 10 — Idempotency: Making Retries Safe

Nothing in this architecture guarantees exactly-once execution. The task runs and the state
update to COMPLETED are two separate operations. If the app crashes after execution but before
the state update commits, the task will be retried. It will run again with the same inputs.

This is at-least-once delivery — it's the only guarantee available here. The question is
whether a duplicate execution causes harm.

If your task is inherently idempotent (generating the same report twice produces the same report),
you're done. If it has side effects (sends an email, charges a card), a duplicate is a problem.

The fix: use the outbox task's own id as an idempotency key inside the handler. The task ID is
stable across retries — it never changes for the same row. The handler checks whether work for
this ID was already recorded. If yes, it's a no-op.

In our case, report generation ends by inserting a row into generated_reports — that's the
record of work done. We add a UNIQUE constraint on idempotency_key so that even if two
instances race, only one INSERT wins. The other gets a constraint violation and treats it as a
successful no-op:

CREATE TABLE generated_reports (
    id               UUID        PRIMARY KEY DEFAULT gen_random_uuid(),
    idempotency_key  UUID        NOT NULL,
    report_path      VARCHAR     NOT NULL,
    created_at       TIMESTAMPTZ NOT NULL    DEFAULT now(),

    CONSTRAINT uq_generated_reports_idempotency_key UNIQUE (idempotency_key)
);

@Component
@RequiredArgsConstructor
public class ReportGenerationHandler implements OutboxTaskHandler<ReportGenerationPayload> {

    private final GeneratedReportRepository reports;

    @Override
    public void handle(UUID taskId, ReportGenerationPayload payload) {
        if (reports.existsByIdempotencyKey(taskId)) {
            log.info("Report already generated for task {} — skipping", taskId);
            return;
        }

        var reportPath = runReportGeneration(payload);

        reports.save(GeneratedReport.builder()
                .idempotencyKey(taskId)
                .reportPath(reportPath)
                .build());
    }
}

If your task calls an external API instead of writing to the DB, check whether that API supports
idempotency keys — most payment processors and transactional email providers do. Pass the outbox
task ID as the idempotency key in whatever form the API requires (header, body field, query param).
The remote service handles deduplication on their end.

Part 11 — Monitoring

The outbox runs silently in the background. When it works, nobody notices. When it breaks,
nobody notices that either — at least not immediately. Stuck tasks pile up. Downstream services
stop receiving events. By the time someone looks at logs it's been two hours.

Three signals cover the failure space completely, per the Google SRE model:

Latency — how long does each task take from pick-up to completion? Spike here means the handler is slow or the DB is under pressure.
Throughput — how many tasks per unit time are finishing? Drops to zero means the scheduler isn't running or no tasks are being claimed.
Error rate — what fraction of attempts end in failure? Above ~5% means a downstream dependency is degraded and the retry backlog is growing.

Instrumentation

Add to build.gradle.kts:

implementation("org.springframework.boot:spring-boot-starter-actuator")
implementation("io.micrometer:micrometer-registry-prometheus")

@Bean
public TimedAspect timedAspect(MeterRegistry registry) {
    return new TimedAspect(registry);
}

Keep metric names as constants and increment from a dedicated component — not scattered across
service methods:

@Component
@RequiredArgsConstructor
public class OutboxMetrics {

  private static final String TASK_PROCESSED = "outbox.task.processed";
  private static final String OUTCOME_TAG    = "outcome";
  private static final String COMPLETED      = "completed";
  private static final String FAILED         = "failed";

  private final MeterRegistry meterRegistry;
  private final ReportGenerationRepository repository;

  public void recordCompleted() {
    meterRegistry.counter(TASK_PROCESSED, OUTCOME_TAG, COMPLETED).increment();
  }

  public void recordFailed() {
    meterRegistry.counter(TASK_PROCESSED, OUTCOME_TAG, FAILED).increment();
  }

  @PostConstruct
  void registerGauge() {
    Gauge.builder("outbox.tasks.pending", repository,
                    repo -> repo.countByStateIn(List.of(TaskState.PENDING, TaskState.FAILED)))
            .description("Tasks waiting to be processed")
            .register(meterRegistry);
  }
}

The service then just calls metrics.recordCompleted() and metrics.recordFailed() — no raw
strings in business logic.

After startup, curl localhost:8080/actuator/prometheus | grep outbox gives you:

outbox_task_duration_seconds{quantile="0.5"}  0.062
outbox_task_duration_seconds{quantile="0.95"} 0.145
outbox_task_duration_seconds{quantile="0.99"} 0.312
outbox_task_processed_total{outcome="completed"} 40.0
outbox_task_processed_total{outcome="failed"}    2.0
outbox_tasks_pending 0.0

Alerts

Dashboards are for humans watching screens. Alerts are for humans who are asleep.

Alert	Fires when	For	Meaning
`OutboxQueueBacklog`	pending > 50	5m	Processing can't keep up with ingest
`OutboxHighErrorRate`	error rate > 5%	2m	Handler or downstream degraded
`OutboxSlowProcessing`	p99 latency > 30s	5m	DB contention or slow external call
`OutboxStuckTasks`	pending > 0 and zero completions in 10m	10m	Scheduler crashed or all workers stuck

Tune thresholds to your own load. A queue depth of 50 may be normal for a high-throughput
service. For a low-volume internal tool, even 5 tasks sitting for 5 minutes warrants a look.
The signal logic is the same; only the numbers change.

Production Checklist

Before shipping any outbox to production:

☐ Indexes are efficient — partial indexes on PENDING/FAILED and PROCESSING states
☐ EXPLAIN ANALYZE confirms index usage on the fetch query
☐ Ordering is not a problem — ORDER BY seq or created_at, backed by index
☐ Processed data cleanup job running, COMPLETED rows removed after retention period
☐ Idempotency handled inside handlers with side effects
☐ Exponential backoff wired — or confirmed not needed for this use case
☐ Metrics in place — queue depth, throughput, error rate, latency
☐ Alerts configured — backlog, error rate, latency, stuck tasks

The Ecosystem — You Don't Always Have to Build This

Everything described in this article is something you can build yourself — and there is real
value in doing so at least once. You understand the failure modes, the index choices, the
ordering traps, because you designed around them. That knowledge doesn't go away when you switch
to a library.

But in production, you often don't need to maintain this yourself.

db-scheduler — Gustav Karlsson's
persistent cluster-friendly scheduler for Java. This is the library I keep coming back to and
the one whose internals most closely match what we built in this article. It uses a single
database table, pessimistic locking with SKIP LOCKED, heartbeats to detect stuck tasks, and
a clean task/handler interface. When I design self-rolled outboxes, I use db-scheduler as the
reference implementation — the interfaces, the patterns, the failure handling are all sound. If
you're starting a new project and want a battle-tested foundation rather than rolling your own,
start here. The abstractions are thin enough that you'll recognize everything from this article
in the source code.

Celery — the de-facto standard for async task execution in
the Python ecosystem. Celery supports PostgreSQL as a result backend and Redis or RabbitMQ as a
broker. The AFTER_COMMIT pattern from Part 2.5 applies here too — push to Celery only after
the transaction commits, not inside it.

Quartz Scheduler — the long-standing Java scheduling
library with JDBC-backed job persistence. More general than an outbox, with cron expressions,
triggers, and job stores. The operational overhead is higher than db-scheduler and the API is
verbose by modern standards, but it's proven at scale with first-class Spring integration via
spring-boot-starter-quartz.

Final Thoughts

Most of what's in this article isn't clever engineering — it's accumulated scar tissue. The
PROCESSING state exists because someone's app crashed. The heartbeat exists because
updated_at lied. The partial index exists because a full table scan at polling frequency
compounds quietly until it doesn't. The AFTER_COMMIT listener exists because someone published
an event inside a transaction and spent hours debugging why it never fired.

Each piece is boring on its own. Together they make the difference between an outbox that works
in development and one you can trust at 3am.

Code

Code from this article is available at github.com/javaAndScriptDeveloper/outboxes-article. Everything runs locally with a single command.

Let's Talk

If you hit something I missed, or you've solved one of these problems differently in your own
systems — I'd genuinely like to hear it. Contact links are in the footer.

FAQ

What is the Transactional Outbox Pattern?

The Transactional Outbox Pattern solves the dual-write problem by writing a task row into the same database transaction as your domain change instead of calling an external system directly. A background worker then reads the outbox table and executes the task. Because the domain write and the task write commit atomically, the task can never be lost even if the external call or the app crashes.

What problem does the outbox pattern solve?

It solves the dual-write problem: when an operation must both update the database and call an external system (publish an event, send an email, call an API), there is no shared transaction across the two. If one succeeds and the other fails you get inconsistent state. The outbox makes the second action part of the database transaction, so they succeed or fail together.

Do I need Kafka or a message broker to use the outbox pattern?

No. The outbox pattern works with just a relational database. A scheduled worker can poll the outbox table directly using SELECT ... FOR UPDATE SKIP LOCKED. A broker like Kafka is optional — you can publish from the outbox, but PostgreSQL alone is enough for reliable async execution.

How do multiple workers avoid processing the same outbox row?

Use SELECT ... FOR UPDATE SKIP LOCKED when claiming rows. Each worker locks the rows it picks and SKIP LOCKED makes other workers skip already-locked rows instead of blocking, so multiple instances can poll the same table concurrently without double-processing.

How do you handle failed tasks and retries in an outbox?

Give each task a state machine (PENDING → PROCESSING → COMPLETED/FAILED), a retry counter, and a next_attempt_at timestamp for exponential backoff. A heartbeat / lease timestamp lets you reclaim tasks orphaned by a crashed worker, and a max-retry threshold moves poison messages to a dead-letter state for manual inspection.

Originally published at codewithkyryl.dev.

🔀 Your Agent Is Paying an LLM to Return a Boolean

Kyryl — Tue, 22 Sep 2026 21:03:17 +0000

Count the model calls in your agent loop. Most of them do not write anything.

Is this ticket urgent. Which tool should run next. Is this shell command safe. Is the task finished. Should this go to the cheap model or the expensive one. Every one of those is a boolean or an enum. And the usual way to answer them is a generative model, a prompt that ends in "respond with only YES or NO", and a parser that hopes the model listened.

It works. It is also the wrong tool for the shape of the question.

The glue code nobody talks about

If you have built an agent harness, you have written this code. A prompt that begs for a single word. A regex or a strip().upper() == "YES" check. A retry when the model answers "Yes, because the deploy failed twice..." and your parser chokes. A second retry when it wraps the answer in markdown.

The backend version of this mistake is familiar. It is calling a full remote service for something a lookup table answers. Or running LIKE '%x%' where an equality check on an indexed column would do. The capability is there, so you use it, and you pay for the generality on every single call.

In an agent loop that cost compounds. Each step is another round trip to a model that is built to write paragraphs, asked to write one token.

What Jev does differently

Last week LangChain published a post, Building a harness with Jev, by Sydney Runkle and Hunter Lovell. Jev comes from TypeSafe AI and they call it a "System One model". It does not generate text. You send it a state (the context) and a set of typed questions, and it returns typed answers with a probability attached.

Three question types:

Choice picks one of several options, with a probability for each.
Score rates against ordered levels like low, medium, high.
Noul is a plain yes or no.

Adding more questions to one call barely moves the latency. You pay only for the tokens of the extra questions. So one call can answer "is it urgent", "which team owns it" and "how risky is it" at once.

The basic call from the post:

from langchain_typesafe import Noul, TypeSafeClassifier

classifier = TypeSafeClassifier()

response = classifier.invoke({
    "state": (
        "The deploy failed twice and customers are seeing 500s. "
        "Can someone look now?"
    ),
    "questions": {
        "urgent": Noul(
            instructions="Does this need attention right now?"
        ),
    },
})

urgency = response.nouls["urgent"].noul

No "answer with only YES or NO". No parsing. The contract is the type.

Two places it fits in a harness

The post shows two middlewares, both under langchain_typesafe.experimental.

The first is model routing. The classifier reads the request and picks the cheapest model that can handle it:

from langchain.agents import create_agent
from langchain_typesafe.experimental.middleware import (
    ModelChoice,
    ModelRouterMiddleware,
)

router = ModelRouterMiddleware(
    choices={
        "fast": ModelChoice(
            model="openai:luna",
            criteria="Direct lookups, extraction, and localized changes.",
        ),
        "powerful": ModelChoice(
            model="openai:sol",
            criteria="Architecture and high-stakes decisions.",
        ),
    },
    instructions="Choose the least costly model that can complete the task.",
)

agent = create_agent("openai:gpt-5.6-luna", middleware=[router])

The second is an auto-mode guardrail. Before a tool call runs, the classifier decides whether it looks risky:

from langchain.agents import create_agent
from langchain_typesafe.experimental.middleware import (
    AutoModeMiddleware,
)

guardrail = AutoModeMiddleware(tools=["bash"])

agent = create_agent("openai:gpt-5.6-luna", middleware=[guardrail])

If that sounds familiar, it is the same idea behind the auto-mode classifiers in Claude Code and Cursor. Something small and fast sits in front of the tool call and decides whether a human needs to see it.

The probability is the real feature

A generative yes/no gives you a string. You can ask the model how confident it is, but that is just more generated text, and it is not a number you can put a policy on.

A calibrated classifier gives you a probability. That turns a prompt into a routing policy you own. The post does not show how to read the probability off the response, so here is the shape of the policy with the field access left abstract:

CONFIDENT = 0.80

def route(answer: bool, p: float) -> str:
    # p = probability the classifier assigned to its answer
    if p >= CONFIDENT:
        return "act" if answer else "skip"
    # not sure: pay for the big model, or ask a human
    return "escalate"

That 0.80 is now a knob. Log every decision with its probability, look at the escalation rate and the error rate, move the threshold. You cannot do that with a prompt. You can only reword it and hope.

This is the actual shift. The cheaper model is the small story. The bigger one is a split between two workloads that agent harnesses have been treating as one. Classification wants a typed answer and a confidence. Generation wants text. Different jobs, different tools.

The honest trade-off

I have not run Jev in production. Everything above is about the pattern, and here is what I would want to check before trusting the product.

The numbers are the vendor's best case. TypeSafe AI claims "up to 200x faster inference and 400x lower cost than comparable LLMs on classification tasks". "Up to" on their benchmarks is not your workload. Replay a week of your own agent decisions through it and measure.

Calibration is the whole pitch, and it is a claim. The threshold logic only works if 0.9 actually means right about nine times in ten on your data. If it does not, the knob is decoration. Checking that needs a labeled sample of real decisions. Most teams do not have one, and building it is the real cost of adopting this.

It is experimental. Both middlewares live under experimental. New vendor, new model, sitting in the critical path of every agent step. Plan for the API to move.

The guardrail has the usual blind spot. A classifier that flags risky bash knows what "risky" looks like in general. It does not know your repo. A bulk delete in a generated folder is harmless for you and scary in general. A routine-looking command against the wrong database is the opposite. Same problem every auto-mode classifier has.

It does not replace the LLM. The post says so itself: Jev is not a drop-in replacement for an LLM. Anything open-ended still needs a generative model. You are adding a second model to operate, monitor and version, not removing the first.

Where I would start

Not with the guardrail. Start with the cheapest, most reversible decision in your loop: model routing, or "is this task done". Wire the classifier in shadow mode next to your current LLM check, log both answers with the classifier's probability, and compare them for a week. If the two agree on the easy cases and the probability drops on the cases where they disagree, calibration is doing its job on your data. Then move the threshold into the live path.

If they disagree randomly, you learned that cheaply, and your old regex is still there.

How many of your agent's LLM calls are really classification, and have you tried splitting them out yet?

🩸 acks=all Still Lost My Data: Kafka Unclean Leader Election

Kyryl — Sat, 22 Aug 2026 16:31:02 +0000

You did everything the durability docs told you to. acks=all. Replication factor 3. You even remembered min.insync.replicas. Then a broker failover ate a handful of records that your producer had already been told were committed.

acks=all did not lie to you. A different setting did.

What acks=all actually promises

When you produce with acks=all, the leader does not acknowledge the write until every replica in the in-sync replica set (the ISR) has it. Combined with min.insync.replicas=2, that means at least two replicas hold the record before your producer sees success. That is a real, strong guarantee.

The word doing all the work is in-sync. The guarantee is only as durable as the membership of that set, and that membership is not fixed.

Step one: the ISR shrinks

A follower has one job: keep pulling from the leader and stay caught up. When it cannot keep the pace, for a long enough window (replica.lag.time.max.ms), the leader removes it from the ISR.

Followers fall behind for boring reasons. A GC pause. A slow disk. A saturated network link. A node under memory pressure. None of these are exotic; they happen in normal operation.

Here is the subtle part. When the ISR shrinks from three replicas to two, writes keep committing. As long as the surviving set still meets min.insync.replicas, the producer never notices. The topic looks healthy. But the redundancy you were counting on is quietly gone.

Step two: the leader dies

Now the leader for that partition fails. Kafka must elect a new leader. It has two options.

Promote a replica that was in the ISR. That replica has every committed record, so nothing is lost. This is a clean election.
If no in-sync replica is available, either wait, or promote a replica that was not in sync.

That decision is controlled by a single broker/topic setting:

unclean.leader.election.enable=false

Step three: the flag pulls the trigger

With unclean.leader.election.enable=true, when there is no in-sync replica to promote, Kafka picks a lagging one anyway to keep the partition available.

That replica is missing the newest committed records, the ones that only lived on the ISR members that are now gone. The moment it becomes leader, it is the source of truth. Every other replica that rejoins truncates its log to match the new leader. The records that were ahead of the new leader's high-water mark are discarded across the whole partition.

Those were committed writes. Your producer got a success response for them. They are now gone, and nothing in the client will ever tell you.

ISR = {1, 2, 3}   acks=all commits, all good
broker 3 lags     ISR = {1, 2}, writes still commit
leader (1) dies   no in-sync replica free
unclean election  broker 3 (stale) becomes leader
                  brokers truncate to broker 3's offset
                  -> committed records 4,5 lost

The fix

unclean.leader.election.enable=false
min.insync.replicas=2
# producer
acks=all

With unclean election off, if there is no in-sync replica to promote, the partition goes offline and waits for one to come back. Producers get an error and retry or buffer. You lose availability for that partition, but you never serve or acknowledge data you cannot back up.

Keep min.insync.replicas at 2 (with RF 3) so a single lagging follower cannot drop you below the durability floor. Produce with acks=all so the two guarantees actually compose.

The honest trade-off

This is not free. You are explicitly choosing consistency over availability for that partition. If enough replicas are unhealthy at once, the partition stops accepting writes until Kafka has an in-sync replica to trust. For a payments ledger or an event source that must never lose a record, that is exactly what you want. For a firehose of metrics where a short gap is fine and downtime is not, you might genuinely prefer the opposite.

The mistake is not picking availability. The mistake is picking it by accident, because the setting defaulted that way on an old cluster and nobody revisited it. Durability you did not choose on purpose is not durability.

Takeaway

acks=all protects you while the ISR is healthy. Unclean leader election is what happens when it is not. Decide, deliberately, whether a partition should go dark or go stale when the replicas you trusted are gone.

Which way is your most important topic configured, and did someone actually choose it?

Your Familiar Java Classes Just Lost Identity 🧬

Kyryl — Wed, 12 Aug 2026 15:42:56 +0000

Project Valhalla's JEP 401 is integrated into JDK 28 as a preview. It needs --enable-preview at both compile and runtime, so nothing you ship today changes. But once this graduates, a class you already use constantly is going to stop having identity, and the compiler is not going to tell you.

What a value class actually changes

A value class in JEP 401 gets four behavioral changes:

Fields are implicitly final.
== compares field values instead of object identity.
Construction must fully initialize every field before the constructor returns.
Instance methods on a value class cannot be synchronized.

Several existing JDK classes become value classes the moment the preview flag is on. Primitive wrappers, Integer, Long, and friends, are named examples. So is LocalDate. Backward compatibility is preserved with the flag off, so this is opt-in territory today. But Integer and LocalDate are about as core to a Java codebase as it gets, which is exactly why this matters more than a typical preview feature.

Why value classes exist at all

Identity has a cost. Every identity-bearing object carries a header, cannot be unboxed cleanly, and cannot be packed into arrays or CPU registers the way a primitive can. The JVM has to always go through a pointer.

For types where nobody actually relies on identity, Integer and LocalDate being the textbook cases, that cost buys nothing. Nobody writes code that depends on two LocalDate instances representing January 1st being the same object, only that they represent the same date. Removing identity where nobody needs it lets the JVM flatten these objects and inline them instead of always chasing a pointer. That is the performance case for Valhalla in one sentence.

The first thing that breaks: `==`

// Before JDK 28 preview
LocalDate d1 = LocalDate.of(2026, 1, 1);
LocalDate d2 = LocalDate.of(2026, 1, 1);

System.out.println(d1 == d2);
// false, identity compare, two different objects

// JDK 28, --enable-preview, LocalDate is now a value class
LocalDate d1 = LocalDate.of(2026, 1, 1);
LocalDate d2 = LocalDate.of(2026, 1, 1);

System.out.println(d1 == d2);
// true, value compare, no identity left to compare

Boxed-primitive == has always been a footgun, thanks to the autoboxing integer cache (-128 to 127) making small values compare equal by accident while larger ones do not. This is a different, sharper version of the same footgun. It is not that identity comparison is unreliable anymore. There is no identity left to be unreliable. Any code that used == as an intentional identity check, a cache lookup, a pooling pattern, an object-identity-based Set, silently gets different results the moment the underlying type becomes a value class. No compile error. No exception. Just a different answer at a line nobody thought to re-read.

The second thing that breaks: `synchronized`

// Before: legal, LocalDate is an ordinary identity-bearing object
LocalDate d1 = LocalDate.of(2026, 1, 1);
synchronized (d1) {
    // ...
}

// After: LocalDate is a value class, this is no longer valid
LocalDate d1 = LocalDate.of(2026, 1, 1);
synchronized (d1) {
    // compile-time error, value classes cannot be used as monitors
}

This one at least fails loudly, at compile time, which is the better failure mode of the two. But it means any code locking on an instance of a type that becomes a value class needs to be found and fixed before upgrading, and today there is no tooling that flags "this class might become a value class someday" ahead of time.

The honest trade-off

Preview-only, opt-in, nothing breaks in production code today. That is the reassuring half. The other half: the classes affected are not obscure corners of the JDK, they are Integer and LocalDate, used in essentially every nontrivial Java codebase. And the failure mode on == specifically is silent. A compile-time break, like the synchronized case, gets caught in CI. A silent behavior change in an == comparison gets caught in production, if it gets caught at all.

The safe posture right now, while this is still behind a flag, is to audit for two patterns before JEP 401 graduates out of preview: == used intentionally for identity on boxed primitive types or LocalDate, and synchronized blocks or methods locking on instances of those same types. Neither pattern is common in well-written code, .equals() should be doing the comparison work already in most cases, but "uncommon" is not the same as "absent," and this is the kind of bug that survives code review because it used to be correct.

Found either pattern in your own codebase while reading this?

The Shared Cache Tier Is Disappearing 🗄️

Kyryl — Wed, 12 Aug 2026 15:42:35 +0000

Canva just published how they rebuilt session revocation without a shared cache tier, and it is the third architecture writeup I have run into this month that lands on the exact same shape. Different companies, different problems, same answer: durable object storage as the source of truth, plus a locally rebuilt index, no shared cache in the hot path.

The problem: revocation has to be fast and shared

Session revocation is a specific kind of hard problem. When a token needs to die before its natural expiry, every gateway in the fleet needs to find out, fast, without every request round-tripping to a central store. The obvious answer for years has been a shared cache, Redis cluster or Memcached fleet, that every gateway queries or subscribes to.

A shared cache tier solves the propagation problem, but it becomes its own liability. It is a single point of contention under load. It has its own failure modes: eviction storms, hot keys, painful cluster rebalances. And it needs its own on-call rotation, separate from the service that actually owns the data.

What Canva built instead

Canva stores revocation records as 16-byte binary entries in S3, sliced into 30-minute objects across a 12-hour revocation window. Instead of gateways querying a shared store on every request, each gateway pulls the recent object slices and rebuilds a local, in-memory sorted-array index from them.

S3 layout (conceptual):
  revocations/2026-08-11T00:00-00:30.bin
  revocations/2026-08-11T00:30-01:00.bin
  revocations/2026-08-11T01:00-01:30.bin
  ...

Concurrency is handled with conditional GETs and PUTs, giving optimistic concurrency without a lock service. ZooKeeper leader election is used to reduce write conflicts further, but Canva is explicit that it is an optimization, not a correctness dependency. The system is correct without it, just slightly noisier under contention.

The results, from their own numbers: cache memory down 87.5%. Workers sustaining 2,000+ revocations per second. The database footprint down to two read replicas. A million revocations fit in roughly 16MB.

The pattern, not just the case study

Strip away the specifics and the shape is: a workload with small records, a bounded time window, and tolerance for a few seconds of staleness, served by cheap durable object storage instead of a shared, stateful cache. Each worker rebuilds its own disposable index locally instead of depending on a shared service to hold the current state for everyone.

This shape fits more than session revocation. Feature flags, entity caches, compacted-topic-style event state, anything that is read far more than it is written, has a small per-record footprint, and does not need every reader to agree on the exact same value at the exact same instant.

Seeing three unrelated teams land here independently in the same month is a stronger signal than any one writeup. It suggests the shared cache tier's default status, "many workers need the same fast-changing data, reach for Redis," is no longer the obvious answer for a meaningful slice of these workloads.

The honest trade-off

This pattern is not a free upgrade. Two real costs:

It only works when the dataset has a natural staleness tolerance and a bounded window. Canva's 12-hour revocation window caps how much history a worker ever needs to rebuild from scratch. Without a bound like that, "rebuild the index from object storage" gets slower and the object count grows without limit.

It trades a shared cache's single-source consistency for eventual consistency across workers. A shared Redis instance gives every reader the same answer the instant a write lands. Here, a revocation is only as fresh as the last object slice a given gateway has pulled. If your correctness requirements demand that every worker sees a write at the exact same millisecond, no propagation lag tolerated, this pattern is the wrong tool and a shared cache (or a shared database) is still the right one.

Canva's own use of ZooKeeper leader election as an optional optimization, not a dependency, is a tell about how the trade-off was made: they accepted some write-conflict noise in exchange for not needing a coordination service to be up for the system to be correct.

Where this fits

If you are running a workload that looks like "many stateless workers need to check a small, frequently-updated dataset, and a few seconds of staleness is fine," this is worth evaluating before reaching for another Redis cluster. The infrastructure is simpler to operate (no separate cache fleet to keep alive), the failure mode is more boring (a worker with a stale local index, not a cache stampede), and the cost profile at scale, per Canva's own numbers, is dramatically smaller.

Have you replaced a shared cache tier with this shape? What broke first when you tried it, and did the staleness window end up being the real constraint?

97% of Your AI Approval Clicks Were Reflexes 🤖

Kyryl — Wed, 12 Aug 2026 15:42:15 +0000

Anthropic ran a study with 1,053 testers before making a call most tools never make honestly about their own permission model: the approval prompt was not working, and here are the numbers to prove it.

Auto mode, the classifier that decides which tool calls need a human, caught 89% of harmful actions in that study. Humans clicking through the old per-call approval prompt caught 13.6%. And 97% of those prompts got rubber-stamped without a second look, whether the action was harmless or not.

Starting August 14, new Claude Code sessions on Pro, Max, and Team plans switch from "ask before every tool call" to "route through a classifier, only interrupt for irreversible, destructive, or outward-facing actions." Enterprise, API, and cloud-partner deployments (Bedrock, Vertex, Foundry) stay opt-in for now, with a default flip planned there within the month. Classifier overhead is not billed separately.

The 97% number is the actual story

A permission prompt only works as a safety mechanism if the human reading it is actually evaluating each one. At 97% rubber-stamped, that was not happening. People were clicking yes on reflex, the same way you click through a terms-of-service dialog. The prompt was theater. It gave the appearance of oversight without providing any, and the one time it mattered, the reflex fired the same as every other time.

This matters beyond Claude Code specifically. Any tool that gates action behind a per-call human approval and expects that approval to be a meaningful check is making the same bet Claude Code just admitted it lost. If your CI pipeline, your deploy tooling, or your own internal agent framework has a "click to confirm" step that nobody actually reads anymore, this study is a preview of what an audit of your own approval logs would probably show.

What "irreversible, destructive, outward-facing" likely means in practice

Anthropic has not published the exact classifier boundary, but the categories map to what teams running agents in CI already learned to gate manually:

Irreversible: force-pushes, dropped database tables, deleted branches, anything git reset --hard adjacent.
Destructive: bulk file deletion, overwriting uncommitted work, rm -rf on anything wider than a scratch directory.
Outward-facing: pushing code, opening or closing PRs, sending messages to Slack or email, posting to external services, anything visible to someone besides the person running the session.

If you already run Claude Code with a pinned permission mode (auto-accept, plan mode, a custom allowlist), none of this changes for you. This only affects sessions that were relying on the default, which was most sessions belonging to people who never touched the setting.

The honest trade-off

A classifier is not you. It was trained on what 1,053 testers, in aggregate, considered harmful. It was not trained on your specific codebase, your specific generated/ folder that looks disposable but is not, or the one script in your repo where a "routine" bulk delete is actually catastrophic.

89% caught is an average across a study population, not a guarantee for any individual team's blast radius. A destructive action that looks routine in general (deleting files in a folder named tmp or build) may not trip the classifier even when, in your repo, that folder is load-bearing. Conversely, something narrow and specific to your setup that the classifier has never seen might get flagged when it is actually fine.

If you run agents against infrastructure with unusual failure modes, the safe move is still to pin an explicit permission mode rather than trust the default classifier to have learned your repo's edge cases. The classifier reduces the average number of things you have to think about. It does not remove the need to think about the things that are specific to you.

What actually shifted

This was framed as a UX improvement, fewer interruptions, faster flow. The more accurate framing is that Anthropic looked at its own approval funnel, found it was not catching anything a human was not already rubber-stamping, and moved the actual judgment call from a human reflex to a trained classifier. That is a better bet on the numbers given here. It is not the same as saying the judgment call is now being made well for every possible repo, it is being made by something with a measured track record instead of something with none.

Teams already running Claude Code semi-autonomous in CI or across a fleet of agents were building workarounds for exactly this gap. This is Anthropic catching up to how those teams were already using the tool, not adding a new capability.

Were you already running Claude Code with a pinned permission mode before this? What did you have to build around the old approve-every-call default?

🔥 The Real Latency Is the Human Handshake

Kyryl — Tue, 11 Aug 2026 20:13:12 +0000

Cross-team API changes are the real latency in microservices, and it is measured in weeks. Nobody puts that number on a dashboard because it does not show up in any trace, any p99, any APM tool. The network call is fast. The thing that is slow never touches the network at all.

The mechanism

Say you need one new field on an order object returned by another team's service. Maybe it is a discountCode, maybe it is a fulfillmentCenterId, does not matter. What matters is who owns the schema.

In a monolith, that field lives in a shared model. You open a PR, a teammate reviews it, it merges by lunch. One commit, one deploy, done.

Across a service boundary, the field lives in a service you do not control. The sequence looks like this:

You message the owning team, or file a ticket.
The ticket sits in their backlog until their next planning session.
Someone on that team estimates it, prioritizes it against their own roadmap, and schedules it.
They write the PR, review it, and ship it, on their timeline, not yours.
You write your own PR to consume the new field, once it exists.

Two weeks if the owning team is fast and the field is trivial. A quarter if it competes with their own sprint commitments, which it usually does, because your one field is nobody's priority but yours.

It is not a network problem

The instinct is to treat this as a technical latency problem, the same category as a slow query or a chatty API. It gets a technical response: better documentation, faster CI, an internal API gateway, more automation around service discovery.

None of that touches the actual bottleneck. The request-response cycle between the two services takes milliseconds whether the field exists or not. The delay lives entirely in the gap between "we filed the ask" and "the owning team scheduled the work." That gap is a queue, and it is a queue that belongs to a team whose backlog you have no authority over.

This is the part people miss: microservices did not introduce a network problem into your architecture. They introduced an organizational dependency graph, and every edge in that graph now has to be negotiated instead of just written. A monolith's "dependency" is a function call. A microservices "dependency" is a relationship between two teams' roadmaps.

A concrete version of the problem

Picture a checkout service that needs a loyaltyTier field from the customer service to compute a discount correctly. The checkout team does not own customer service. They file a request.

Ticket: Add `loyaltyTier` (enum: BRONZE, SILVER, GOLD) to GET /customers/{id}
Requested by: checkout-team
Priority: P2 (their P2, not yours)
Status: backlog

The checkout team cannot ship their discount logic until this lands. They are not blocked by a database, a deploy pipeline, or a load balancer. They are blocked by another team's sprint board. If the customer team is mid-migration, or short-staffed, or simply has three P1s ahead of it, the checkout team waits. There is no retry, no timeout, no circuit breaker for an organizational queue.

The honest trade-off

There are real fixes for this, and every one of them costs something up front. None of them are free lunches, and anyone pitching one as a strict win is skipping the cost column.

Embed a rep from the consuming team in the owning team's planning. This buys real speed: the field gets prioritized because a human advocating for it is sitting in the room. It costs headcount. Someone's calendar now has a recurring meeting with a team they do not report to, and someone's manager has to sign off on that time.

Go contract-first with a schema registry. Define the contract before either side writes code, validate changes against it in CI, and both teams catch breaking changes before they ship instead of after. This buys predictability. It costs discipline: every team touching the registry has to actually write contracts before code, review schema diffs like they review code diffs, and resist the shortcut of just shipping a field because it is faster this one time.

Build self-serve schema evolution for optional, additive fields. Let consuming teams add new optional fields to a schema without a round trip through the owning team's backlog, gated by validation rather than a human approval queue. This cuts the wait to near zero for the common case. It costs governance: someone has to build the tooling, define what counts as safely additive versus breaking, and maintain the guardrails so "self-serve" does not turn into "anyone can silently change the contract."

None of these fixes are cheap. All of them beat waiting a quarter for one field.

What actually changes

Adding a field is not slow because computers are slow. It is slow because prioritization is a human process, and cross-team prioritization has no SLA. The teams that ship fast across service boundaries are not the ones with the fastest network. They are the ones who paid the coordination cost before they needed the field, not after.

How long does a one-field ask actually take to land across a team boundary where you work, and which of these fixes, if any, does your org already run?

🔥 Two Services, One Table, Zero Isolation

Kyryl — Mon, 10 Aug 2026 21:39:57 +0000

Two services writing to the same table is a distributed monolith wearing a microservice badge. You split the codebase, you kept the database, and the split you actually shipped is cosmetic. Every cost of the split is real. The isolation you were supposed to get for that cost is not.

The decision that does not feel like an architecture decision

Nobody sits in a design review and votes to build a distributed monolith. It happens one convenience call at a time. Team A owns orders, team B needs order data for its own service, and the fastest way to get it is to point B's code at the same Postgres instance and the same orders table. No new endpoint to build, no contract to agree on, no deploy to coordinate. It ships this sprint instead of next quarter.

That decision gets filed under data access. It is not a data-access decision. It is the architectural boundary, decided by default, by whoever wrote the first query. The table has one schema, and from that moment both services are load-bearing on that schema staying exactly the way it is. Nobody wrote that dependency down anywhere. It lives in two codebases that do not import each other and cannot see each other's queries.

What the coupling looks like when it breaks

Here is the incident, not hypothetically, the way it actually shows up.

Service A owns the table by every reasonable definition, it was there first, its team migrated the schema originally. A needs to ship a feature. The feature needs a new required field, so someone adds:

ALTER TABLE orders ADD COLUMN fulfillment_channel VARCHAR(32) NOT NULL DEFAULT 'standard';

A's migration tooling runs it, A's tests pass, because A's tests only ever insert through A's own code path, which now always sets fulfillment_channel. A's deploy goes out clean. Nobody on A's team thinks about service B, because as far as A's team is concerned, B is not part of this change. A does not know B writes to this table at all, or if it does, nobody flagged this migration for review.

Service B writes to orders directly, from its own repository, using its own hand-rolled insert statement that predates the new column entirely. Two things can happen next, and both are bad.

If the column had no default, B's inserts start failing immediately with a constraint violation. That is the good outcome, loud and fast, a page goes off, someone finds the cause within the hour.

The worse outcome is what actually happened above: a default value. B's inserts keep succeeding. Every row B writes silently gets fulfillment_channel = 'standard', whether or not that is true. No error. No failed health check. No alert. The data is just wrong, quietly, for every order B touches, from the moment A's migration landed until someone downstream notices the fulfillment numbers do not add up. That could be hours. It is more often days, and by the time someone in the incident review asks "wait, since when has B been writing that channel," the wrong data is already in reports, already fed into whatever dashboard finance trusts.

Nobody connects the two events, the migration and the corrupted writes, because nothing in either codebase points at the other. The only shared artifact is the table itself, and tables do not send review requests.

Why splitting the codebase did not split the coupling

The instinct once you are burned by this is to blame the migration, add a review step, require anyone touching a shared table to ping the other team first. That is a process patch on an architecture problem, and it will hold until someone forgets, or a new hire does not know the informal rule exists, or the "other team" has been renamed twice since the rule was written down in a wiki nobody reads.

The actual problem is that two services with independent deploys, independent on-call, independent codebases are still, underneath all of that, one system with one shared piece of mutable state. A network call between them would at least force a versioned contract, something with a schema of its own that changes deliberately, gets reviewed, gets a deprecation window. A shared table has no such contract. Its schema is the contract, enforced by nothing except discipline, and discipline is the thing that fails first under deadline pressure.

You paid for the split. Two deploy pipelines, two on-call rotations, a network hop between the two halves whenever one calls the other for anything else. What you did not get, in exchange for that cost, is the one thing the split was supposed to buy: a boundary strong enough that one team's change cannot silently break the other. That is what "distributed monolith" means in practice. Every operational cost of microservices, none of the isolation.

The honest trade-off

There are two real fixes, and neither one is free.

The first: pick one service as the actual owner of the table, and every other consumer goes through that owner's API. Whatever queries B was running directly against orders become HTTP or gRPC calls to A. A's schema changes are now A's problem to manage behind a versioned interface, and B is protected from anything A does internally as long as the contract holds. The cost is real: you have to build that API surface, migrate every direct query B has, and accept the latency and availability coupling that comes with a synchronous call replacing a local join.

The second: actually split the data. B gets its own table, its own copy of whatever fields it needs, kept in sync through an event stream, A publishes an OrderUpdated event whenever the row changes, B consumes it and updates its own copy. Now B genuinely does not depend on A's schema, A can rename columns internally all day without touching B. What you buy instead is eventual consistency: B's copy is sometimes a few seconds or minutes behind, and now you own a second, denormalized dataset that can drift from the source if the event pipeline ever drops a message. Reconciliation becomes its own ongoing job.

Neither of these is a migration you do over a sprint. Both are projects: mapping every access pattern the "other" service currently has against the shared table, deciding what actually needs to move, coordinating a cutover that does not lose writes in the transition. It is slower and more expensive than the shared table ever was, which is exactly why the shared table happened in the first place. It was the fast option. It is just not the option that gives you the isolation you were charging yourself for.

How many "microservices" in your own systems still share write access to a table, and has anyone actually mapped what breaks if either side changes the schema?

🧹 Kafka Is Deleting the Wrong Data: Retention vs Compaction

Kyryl — Mon, 27 Jul 2026 19:22:06 +0000

Kafka is a log, and logs grow. At some point the broker has to reclaim disk, and it has two completely different ways to do it. They are not variations on a theme. They delete on different rules, and confusing them is a quiet data-loss bug that looks like corruption weeks later.

Retention: delete by age

Retention is a stopwatch. You set a window, and once data is older than that window it is deleted, segment by segment, regardless of what is inside.

log.retention.hours=168      # delete anything older than 7 days
log.retention.bytes=-1       # or cap by size per partition

Kafka stores a partition as a sequence of segment files. Retention works at segment granularity: when the newest record in a segment is older than the window (or the partition is over its byte cap), the whole segment is dropped. It never looks at record keys. It does not care whether a key still matters. Old is old.

This is exactly right for a stream of events. A click from last month, a sensor reading from an hour ago, a request log from yesterday. The value is in the recent flow; the old entries can leave.

Compaction: keep the latest per key

Compaction is a dedupe. Instead of asking "how old is this?", it asks "is there a newer record with the same key?" If yes, the older one can go.

cleanup.policy=compact

With compaction, Kafka guarantees that for every key, the latest record is retained. Older records for that same key are eventually removed by the log cleaner. A key that is written once and never updated survives forever, no matter how old it is. That is the crucial difference from retention.

This is exactly right for state. A changelog topic, a Kafka Streams KTable, a database outbox, a topic that materializes "current value per entity". The whole point is that a consumer can replay the topic from the beginning and rebuild the current state of every key.

You can also combine them:

cleanup.policy=compact,delete

Now the latest value per key is kept, but truly ancient records can still age out. Useful for a changelog where very old, untouched keys are acceptable to drop.

The incident this prevents

Here is the failure I have watched happen more than once.

Someone models current entity state as a Kafka topic: key is the entity id, value is its latest state. A downstream service rebuilds a local view by consuming the topic from offset zero. Works perfectly in testing.

The topic was left on the default cleanup.policy=delete with a 7-day retention. Most entities update often, so nobody notices. Then a slow-moving entity goes untouched for eight days. Its only record ages out. Now when the downstream service rebuilds, that entity is simply missing. No delete event, no error, no log line. It looks like data corruption, and you will spend a day chasing a bug that is really one config line.

The fix is one property: that topic should have been compact.

The honest trade-off

Compaction is not free, and it is not a drop-in for retention.

The log cleaner spends CPU and I/O rewriting segments. On a high-churn topic that is real, ongoing work.
Compaction only bounds size by number of distinct keys, not by time. A topic with an unbounded key space (say, keying by request id) under compact will grow without limit, because every key is unique and nothing is a duplicate. That is a disk-filling trap in the opposite direction.
Deletes in a compacted topic need tombstones: a record with the key and a null value. Miss that and a "deleted" key lives forever.

So the rule is not "compaction is better". It is: retention for events, compaction for state, compact,delete when state has a long tail you are willing to drop, and never compact on an unbounded key space.

Takeaway

Retention deletes by time. Compaction keeps the latest per key. They solve different problems, and the default (delete) is the wrong choice for any topic that represents state.

Go look at your most important topic. Is it delete or compact, and is that on purpose or just the default nobody changed?

🗑️ Kafka Keeps Data Longer Than retention.ms

Kyryl — Mon, 20 Jul 2026 18:52:46 +0000

You set retention.ms to one hour. Six hours later the data is still on disk. That is not a bug, and it is not your cleanup thread being slow.

Kafka deletes data very differently from how most people picture it.

Kafka deletes segments, not records

A partition is not one big file. It is an ordered sequence of segments, each a file on disk. Retention operates at the segment level, never at the record level.

Kafka will never open a segment and rewrite it to drop a handful of expired records. That would be expensive and would break the append-only design the whole system is built on. Instead it waits until an entire segment is eligible, then deletes the whole file in one move.

When a segment is actually eligible

A closed segment becomes eligible for deletion only when both are true:

The segment is closed (rolled). A segment rolls when it fills up (segment.bytes, default 1 GB) or ages out (segment.ms, default 7 days).
Its newest record is older than retention.ms (or the partition is over retention.bytes).

Note it is the newest record in the segment that has to age out, not the oldest. One young record keeps the whole segment, and every older record in it, alive.

The active segment never dies

Here is the part that surprises people, and the reason your data outlives its retention.

The active segment, the one currently being written, is never eligible for deletion, no matter what. Retention only ever considers closed segments.

So on a low-traffic topic, the active segment fills slowly and rolls rarely. The oldest record sitting in it can be hours or days past retention.ms, just waiting for the segment to finally roll so it can even be considered for deletion.

[ segment 0 ][ segment 1 ][ segment 2 ][ segment 3 (active) ]
   closed       closed       closed        being written
   deletable    deletable    deletable     NEVER deletable
   once past retention.ms                  regardless of age

Your retention is effectively retention.ms plus however long it takes the active segment to roll.

The honest trade-off

The fix is not to lower retention.ms further. That does nothing while the segment has not rolled. Lower segment.ms so segments roll more often and become deletable sooner.

But smaller segments are not free: more files, more open file handles, more frequent rolls, and more index overhead. On a high-throughput topic the default 1 GB / 7 day segments are fine and you will never notice this. It only bites on low-traffic topics with a tight retention expectation: compliance windows, PII deletion SLAs, "we only keep 24 hours" promises. Those are exactly the cases where the gap matters, so size the segment to the retention you actually need.

This is separate from log compaction (cleanup.policy=compact), which keeps the latest value per key instead of deleting by age. Same segment mechanics underneath, different eligibility rule.

Have you ever had data outlive its configured retention on a quiet topic, and traced it back to a segment that just never rolled?

🔀 Your Kafka Group Freezes on Every Deploy

Kyryl — Thu, 16 Jul 2026 19:09:08 +0000

You have a Kafka consumer group with four consumers, humming along. Traffic climbs, so you add a fifth. For a few seconds, every consumer stops. Lag spikes. Then it recovers and nobody thinks about it again.

That pause is not a bug. It is eager rebalancing, and most groups still run it by default.

Rebalancing is not the enemy

A consumer group rebalances whenever membership changes: a consumer joins, leaves, crashes, or the topic gains partitions. The group has to agree on who owns which partitions. That part is unavoidable.

The cost is not the reassignment. It is how the group gets there. That is controlled by partition.assignment.strategy, and the choice you make there decides whether a deploy costs you a few milliseconds or a few seconds of frozen consumption.

The four assignors

RangeAssignor (the historical default) assigns each consumer a contiguous range of partitions, per topic.

6 partitions, 3 consumers:
A -> P0 P1   B -> P2 P3   C -> P4 P5

Clean when the numbers divide. The moment they do not, the lower-id consumers get the extra partitions, and if you subscribe to several topics they pile up on the same consumers.

RoundRobinAssignor deals every partition out one by one across all consumers.

6 partitions, 4 consumers:
A -> P0 P4   B -> P1 P5   C -> P2   D -> P3

Even spread. But it has no memory. Every rebalance recomputes the whole assignment from scratch, so partitions jump around even when they did not need to.

StickyAssignor keeps the spread even and tries to preserve the previous assignment, so it moves as few partitions as possible on each rebalance.

CooperativeStickyAssignor does the same assignment as sticky, but over a different rebalance protocol. This is the one that matters.

Eager vs cooperative: the real difference

Here is the part that trips people up. Sticky reduces how many partitions move. It does not change what happens to the ones that stay.

Range, round-robin, and plain sticky all use the eager protocol. On every rebalance the group does a full stop-the-world:

Every consumer revokes all of its partitions.
Nobody consumes anything.
The new assignment is computed.
Consumers pick their partitions back up.

So even a partition that ends up on the exact same consumer still gets revoked and paused. Add one consumer to a group of ten and all of them stop, for every partition, until the dust settles.

Cooperative-sticky (KIP-429, Kafka 2.4+) rebalances incrementally:

Only the partitions that actually need to move are revoked.
Every other partition keeps being consumed, right through the rebalance.
The moved partitions get reassigned in a second, short round.

One consumer joins, one or two partitions pause for a moment, and the rest of the group never notices. Stop-the-world becomes stop-one-partition.

props.put(
    ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
    List.of(CooperativeStickyAssignor.class.getName()));

You cannot just flip the config

This is the trap. You cannot take a running group on the eager protocol and switch it to cooperative in a single deploy. A group with some members speaking eager and some speaking cooperative will not rebalance correctly.

The supported path is a two-phase rolling upgrade:

// Phase 1: deploy every instance with BOTH strategies listed.
// The group stays on the old protocol until all members support the new one.
props.put(
    ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
    List.of(CooperativeStickyAssignor.class.getName(),
            RangeAssignor.class.getName()));

// Phase 2: after every instance is running phase 1, deploy again
// with only the cooperative strategy. Now the group flips protocols.
props.put(
    ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
    List.of(CooperativeStickyAssignor.class.getName()));

Skip phase 1 and you get a broken rebalance in production. Two deploys, in order, with the whole fleet on phase 1 before phase 2 starts.

The honest trade-off

Cooperative-sticky is not free.

The migration is a two-step rolling upgrade you have to get right, not a one-line change.
Incremental rebalancing can take more rounds than a single eager rebalance, so the total rebalance can be longer even though nobody is fully stopped.
Your onPartitionsRevoked and onPartitionsLost callbacks need to be correct, because partitions now come and go without a global reset to lean on.

For a group that rebalances rarely and briefly, the eager pause may be cheap enough to ignore. For a group that scales, deploys, or loses instances often, the stop-the-world pause is a tax you pay on every event, and cooperative-sticky removes it.

Which assignor does your consumer group run today, and have you ever actually measured how long a rolling deploy freezes it?

DEV Community: Kyryl

Can Kafka Replace Redis for Cache Synchronization? A Production Answer

The Problem: Shared Configuration on a Fast Read Path

How the Whole Flow Works

Publish the Current State in a Compacted Topic

Give Every Pod Its Own Consumer Group

Materialize Updates in the Local Cache

Delete Configuration with Tombstones

Don't Serve Traffic Until You've Caught Up

Applying the Readiness Gate

Where This Pattern Bites

Every Restart Pays the Replay Cost

Unique Groups Leave Temporary Metadata

Large Configuration Does Not Belong in Every Pod

Eventual Consistency Is Part of the Contract

Can you keep this architecture alive?

Code

Let's Talk

FAQ

Can Kafka replace Redis for cache synchronization across pods?

With N pods, does every pod receive every Kafka message, or does Kafka split messages across them?

How do I give every pod its own Kafka consumer group?

How do you delete a key when the Kafka topic is compacted?

Should scheduled jobs and traffic wait until Kafka consumption catches up?

Can continuous updates keep a new pod stuck in startup forever?

The Transactional Outbox Pattern: From a Single Task to a Production-Grade Async Engine

The Scenario

Part 1 — The Basic Outbox

What Every Outbox Needs

The Worker

Part 2 — Concurrency: Two Workers, One Task

Part 2.5 — Reducing Latency with Application Events

Implementation

The Classic Mistake

Part 3 — Failures, Retries, and the PROCESSING State

Part 4 — Exponential Backoff

Part 5 — Stuck Tasks and Heartbeats

Part 6 — Multi-Type Outbox

The Schema

The Handler Interface

The Dispatcher

Part 7 — Ordering

The Clock Skew Trap

Part 8 — Indexes That Keep the Table Fast at Scale

The Fetch Index — the hot path

Part 9 — Cleanup

Part 10 — Idempotency: Making Retries Safe

Part 11 — Monitoring

Instrumentation

Alerts

Production Checklist

The Ecosystem — You Don't Always Have to Build This

Final Thoughts

Code

Let's Talk

FAQ

What is the Transactional Outbox Pattern?

What problem does the outbox pattern solve?

Do I need Kafka or a message broker to use the outbox pattern?

How do multiple workers avoid processing the same outbox row?

How do you handle failed tasks and retries in an outbox?

🔀 Your Agent Is Paying an LLM to Return a Boolean

The glue code nobody talks about

What Jev does differently

Two places it fits in a harness

The probability is the real feature

The honest trade-off

Where I would start

🩸 acks=all Still Lost My Data: Kafka Unclean Leader Election

What acks=all actually promises

Step one: the ISR shrinks

Step two: the leader dies

Step three: the flag pulls the trigger

The fix

The honest trade-off

Takeaway

Your Familiar Java Classes Just Lost Identity 🧬

What a value class actually changes

Why value classes exist at all

The first thing that breaks: ==

The first thing that breaks: `==`

The second thing that breaks: `synchronized`