DEV Community: Denis Toropov

When @Cacheable caches not the data, but the problem

Denis Toropov — Thu, 08 Oct 2026 17:53:09 +0000

A few days ago, I came across an unusual bug came from @Cacheable.

The method looked harmless: it returned a collection, and we cached the result. That method was used in several places. Some callers only read the collection. Others modified it - removing items, reordering it, or adding extra data for their own flow.

That worked fine until caching turned the returned collection into shared state.

After @Cacheable, callers were no longer getting a freshly built result. They were often getting the same object from the cache. So once one code path changed that collection, the cached value itself was changed. The next caller could receive an already mutated version.

In a multi-user system, this gets ugly fast. The bug looks random. One request changes something “locally,” another request sees unexpected data, and nothing in the logs clearly points to the real cause.

The issue was not the cache itself. The issue was caching a mutable object and then letting different parts of the code treat it as their own copy.

Since then, I’ve been much more careful with cached method results.

If a method is annotated with @Cacheable, I treat its return value as read-only.

In practice, that usually means:

return List.copyOf(...) or Set.copyOf(...)
avoid exposing collections that callers may modify
create a defensive copy if the ownership boundary is not obvious

A small optimization introduced shared mutable state across requests. That was the whole bug.

A good reminder that caching changes more than performance. It also changes the contract of a method.

When microservices make data harder than it needs to be

Denis Toropov — Sun, 04 Oct 2026 12:23:28 +0000

Datadog’s recent research captures a reality many teams already feel in practice: once companies move to microservices, their database landscape tends to fragment quickly. Based on data from more than 2.5 million services, Datadog found that more than half of organizations use three or more database technologies, about a quarter use five or more, and nearly half run SQL and NoSQL side by side.

That trend makes sense at scale. Different services have different workloads, and large organizations often have enough platform maturity to support multiple storage models. But the same pattern is much harder to justify in small companies. In many cases, teams adopt several databases not because the business truly needs them, but because microservices make local decisions easier than system-wide ones.

The result is a familiar architectural trade-off. Teams gain service autonomy, but they lose data simplicity.

In a monolith, the shared database is often a bottleneck for team independence. Schema changes require coordination, coupling grows over time, and one database has to satisfy many use cases. But there is also a major upside: joins, referential integrity, and transactional consistency stay close to the data. Complex business questions can often be answered inside one engine.

Microservices change that. As Datadog puts it, organizations replace one global schema with hundreds or even thousands of smaller schemas owned by individual services. That improves isolation, but it also makes cross-domain queries harder. Customer data lives in one place, orders in another, payments somewhere else, and support history in yet another store. The business still needs a unified view, but the data is no longer naturally unified.

This is where many teams discover that the join never disappeared. It just moved up the stack.

Datadog points to GraphQL as one way organizations handle this problem. Instead of joining data inside a relational database, they assemble it in an integration layer that queries multiple services and data stores behind a single schema. That works, but it comes with real cost. According to Datadog, 55% of GraphQL executions include more than 10 child resolve spans, and some services handle more than 100 resolves per execution. The median graphql.execute lasts about 200 ms, while individual field resolutions are closer to 40 ms, showing how query depth and resolver fan-out add latency even when work is parallelized.

This is an important architectural point. A database join and a distributed join are not equivalent. A relational database resolves joins close to the data, inside one execution engine. A GraphQL layer has to orchestrate calls across service boundaries, manage partial failures, control N+1 query behavior, and hide inconsistent latency from the client. It can be done well, but it is undeniably more complex.

For small companies, that complexity is often disproportionate. A team with limited platform capacity now has to operate multiple databases, maintain integration logic, and solve analytics separately. Datadog notes that analytics becomes harder in microservice environments because data from distributed systems has to be transformed and consolidated into OLAP platforms such as Snowflake, Redshift, Databricks, BigQuery, or ClickHouse; 44% of organizations in their data already use at least one such platform. Service integration gets harder too, which helps explain why nearly 70% of customers use message queues to decouple services asynchronously.

None of this means microservices are a mistake. It means they should not automatically lead to polyglot persistence. For a smaller company, a standard relational database plus a small number of carefully justified supporting technologies is often the better choice. PostgreSQL for transactional data, Redis for caching, Kafka only when event streaming is truly needed—this is usually easier to scale organizationally than letting every service pick its own storage engine.

The main lesson from Datadog’s research is not that using many databases is modern. It is that fragmentation has become common, and common is not the same as optimal. Large companies may benefit from specialized databases because they have the scale, workloads, and platform teams to support them. Smaller companies often inherit the downside first: more schemas, harder joins, more integration layers, and more operational surface area.

So the practical question is not whether a service could choose its own database. It is whether the organization can afford the data integration problem that follows.

If the answer depends on GraphQL layers, resolver batching, warehouses, queues, and custom aggregation logic, then the cost of database diversity is already showing up. At that point, the simplest architecture may not be the most fashionable one. It may be the one with fewer databases.

Source: Datadog, How microservice architectures have shaped the usage of database technologies.

Technical debt: why it appears and how to handle it systematically

Denis Toropov — Wed, 23 Sep 2026 19:23:32 +0000

Technical debt usually gets attention in two situations: when releases become painful, or when even a small change starts costing far more than it should. Until then, it is often treated as background noise: some dependencies are outdated, some areas have no tests, there are known bugs, documentation is weak, but overall the system still works.

That is exactly why it becomes dangerous. Technical debt rarely arrives as one major event. It accumulates through a series of small decisions that look reasonable at the time. A library upgrade gets postponed because a feature matters more right now. Performance work is delayed because the SLA is still acceptable. Tests are skipped to hit a release date. Documentation is left for later. After a while, the team is no longer gaining speed from those compromises. It is paying for them in every sprint.

The main mistake is treating technical debt as something abstract. If it is not captured in tasks, estimated, and brought into planning, it effectively does not exist. And if it does not exist in planning, it keeps growing.

What technical debt means in real development

Without the polished definitions, technical debt is everything that makes changes to a system more expensive and more risky than they need to be.

This is not only about code. Debt can sit in architecture, test coverage, dependencies, documentation, user workflows, and even in the operational processes around the system. The real sign of debt is not that something looks messy. It is that the team keeps paying for it through time, defects, manual effort, or elevated risk.

A few phrases usually show that debt has already become part of daily work:

“better not touch that module”
“release carefully, there are no tests there”
“yes, the library is old, but upgrading it is risky”
“that bug is known, just use the workaround”
“ask Ivan if you need it, he is the only one who knows how it works”

Once this becomes normal, the debt is no longer affecting only the codebase. It is affecting the team’s delivery speed.

1. Outdated library versions

One of the most common sources of technical debt is old dependencies and framework versions that stay untouched for years because the system is still running.

The problem is that this kind of stability is usually misleading. As long as the library is not blocking work today, the upgrade gets pushed back. Then one day the project is sitting on an unsupported version, security fixes are no longer available, newer parts of the ecosystem are incompatible, and moving to a supported baseline turns into a separate migration effort.

The longer the upgrade is delayed, the more expensive it becomes. There is a major difference between moving one minor version forward and jumping across several major versions. The first is a normal engineering task. The second often becomes a risky migration with regression concerns, API compatibility issues, configuration changes, and hard-to-predict side effects.

There is also a quieter risk: hidden constraints begin to pile up. The team can no longer adopt a new observability agent, a newer JDBC driver, a Kafka client update, or a security patch without triggering a chain of dependent upgrades. On paper the issue is just an old library. In practice it becomes a constraint on the system’s evolution.

2. Poor performance

Technical debt is not only about code maintainability. Performance degradation becomes debt very quickly, especially when the team gets used to it and starts treating it as normal.

In most systems, it does not begin with a disaster. It starts with small compromises. One heavy query. One extra synchronous service call. One place where caching was skipped. One batch job that runs longer than expected. While the load is moderate, the system survives. Then the data volume grows, more users arrive, more integrations are added, and it becomes clear that the safety margin disappeared a long time ago.

At that point, performance stops being a narrow technical topic. It starts affecting the business directly: users wait longer, batch jobs no longer fit into their windows, infrastructure costs go up, and the team hesitates to add new scenarios because the system is already close to its limit.

The worst case is when there are no baseline metrics and no history of degradation. If the team does not know its p95 or p99, does not understand the load profile, and cannot point to actual bottlenecks, every conversation about performance turns into guesswork. That is already technical debt: the problem exists, but it is not described in a way the team can manage.

3. Lack of tests

A lack of tests is one of the most expensive forms of debt because it affects every change, not just one specific part of the system.

For a while, a system without tests can look faster to build. But that gain does not last. Over time, every change demands more manual checking, more coordination, and more caution. Refactoring becomes risky. Releases become stressful. A bug fix in one area can create a new defect in another.

The issue is not limited to the absence of unit tests. A weak test setup also creates debt. For example:

Situation	What it turns into
Very few tests	regressions are found after release
Only integration tests	checks are too slow, so the team runs them inconsistently
Fragile tests	people stop trusting them
Tests tied to implementation rather than behavior	any refactoring breaks a large part of the suite
No coverage for critical business flows	the most important scenarios still rely on manual verification

In the end, a team without tests does not just have weaker quality control. It loses the ability to change the system safely. That directly affects delivery.

4. Bugs and known defects

Not every bug is technical debt. But a bug that everyone knows about, nobody fixes, and people work around on a daily basis almost always becomes part of it.

That kind of defect rarely lives only in the code. It usually grows its own support structure around it: manual checks, special instructions for support teams, extra business steps, repeated operations, and exceptions in process rules. From the outside, it may look manageable. In reality, the team keeps paying for it.

The most dangerous cases are the defects people get used to. They stop feeling urgent because they do not look like an outage. But those are often the issues that drain time for years: support handles the same cases again and again, analysts build new processes around the limitation, and developers remember that one fragile area must be treated carefully.

If a bug already has an oral workaround, that is usually enough reason to move it into the technical debt backlog instead of leaving it in a loose list of known issues.

5. Documentation

Poor documentation is easy to ignore in a small team where everyone already shares context. But once the system grows, new people join, integrations multiply, audits happen, incidents occur, or several teams need to work in parallel, missing documentation starts slowing work down as much as bad code does.

The issue is not simply that there is nothing to read. The issue is dependence on individual memory. Architectural constraints, past decisions, integration quirks, hidden contracts, and operational details live only in the heads of a few engineers. As long as those people are around, the system looks manageable. The moment someone is on vacation, leaves the project, or is unavailable during an incident, the cost of that approach becomes obvious.

Documentation debt does not mean everything must be written down. In practice, it is usually enough to keep a few critical layers up to date:

the architecture view and service boundaries
key integrations and contracts
a decision log for important technical choices
instructions for running, supporting, and troubleshooting the system
constraints that engineers must remember before making changes

This is not paperwork for its own sake. It is a way to transfer context and reduce bus factor.

6. User guide

This is often not treated as technical debt at all, especially in internal enterprise systems. In practice, though, the absence of a user guide creates measurable losses.

When users do not understand how to work with the system, the load shifts to support, analysts, and developers. Explanations move into chat threads, short calls, screen-sharing sessions, forwarded outdated instructions, and manual guidance for new scenarios. If the product is complex, such as a CRM, MDM platform, internal operational tool, or enterprise workflow system, those losses become visible very quickly.

There is also a second effect: changes become more expensive. Any change in the interface or process requires additional communication because users do not have a stable reference point in the form of current guidance.

A user guide is not cosmetic. In a mature product, it is part of the operating model. If it does not exist, the system starts depending on a live translator between functionality and the end user. That is a valid item for the technical debt backlog.

Why technical debt needs to be quantified

As long as technical debt lives in phrases like “everything is outdated here,” “this is a bottleneck,” or “that part is dangerous,” it will usually lose to product work during prioritization.

Not because product work is always more important. It loses because product work is usually described better: it has a goal, an expected effect, a deadline, and an outcome. Technical debt often has only team frustration attached to it.

To make debt manageable, it has to be converted into concrete units. A minimal template usually looks like this:

Field	What to capture
Area	dependencies, performance, tests, bugs, documentation, user guide
Problem	what exactly is wrong
Consequence	what it affects: releases, SLA, support, team speed, risk
Risk	low / medium / high
Estimate	hours, days, story points
Success metric	how the team will know the situation improved

For example, “we should update the stack” is not useful. “Spring Boot 2.x is no longer supported, some security patches are unavailable, and newer dependency versions are blocked; estimate: 8 working days; result: move to a supported baseline and remove upgrade constraints” is already good enough for planning.

Quantification is not about reporting for its own sake. It is what allows technical debt to be compared with other debt items and with product work in the same language.

A separate technical debt backlog

If technical debt is not kept in a separate list, it almost always dissolves inside the general task queue. In that form it is hard to discuss, hard to rank, and easy to ignore.

That is why a separate technical debt backlog works better in practice. It should not be isolated from normal planning. It should be part of it.

Each item in that backlog should have at least three things: a clear description of the problem, an estimate, and a reason for its priority. For prioritization, a few simple criteria are usually enough:

Criterion	Question
Business impact	does it affect customers, operations, or SLA
Delivery impact	does it slow down or destabilize delivery
Risk	can it lead to incidents, defects, vulnerabilities, or data loss
Cost of delay	how much more expensive will it be to fix next quarter
Effort	what is the real cost of doing the work

This also helps remove emotion from the discussion. Instead of saying “everyone hates this part of the system,” the conversation moves to risk, cost, and impact.

Example of a technical debt backlog

A working backlog might look like this:

Task	Reason	Priority	Estimate	Expected effect
Upgrade critical libraries and the framework	security and compatibility risks have accumulated	High	8 days	fewer vulnerabilities, easier maintenance
Optimize the 5 slowest SQL queries	p95 is above target SLA	High	5 days	faster core operations, lower database load
Add integration tests for a critical business flow	regressions keep appearing in releases	High	6 days	more predictable releases
Fix known defects with manual workarounds	support and business teams are compensating in process	Medium	4 days	fewer manual operations and support requests
Document key architectural decisions and integrations	too much knowledge depends on a few engineers	Medium	3 days	faster onboarding and easier incident analysis
Prepare a user guide for the top 5 scenarios	support load is too high	Medium	3 days	fewer repeated questions and user mistakes

At this point, it is no longer just a list of complaints. It becomes a list of tasks with meaning, cost, and expected results.

Why it makes sense to spend 20% of each sprint on debt

Trying to “fully eliminate technical debt first and return to product work later” almost never works. The business is usually not ready to stop product development for months, and without clear boundaries the team can easily drift into endless internal improvements with no visible result.

That is why reserving part of each sprint for technical debt is a practical approach. Starting with 20% is usually reasonable: not so little that nothing changes, and not so much that product delivery stalls.

There are several benefits to this.

First, it makes the work predictable. The team does not need to re-argue every sprint why technical debt deserves time.

Second, it reduces accumulation. Small, regular improvements are almost always cheaper than occasional large-scale cleanup efforts.

Third, after a few months the impact becomes visible in numbers: fewer incidents, fewer regressions, faster releases, less manual work, shorter onboarding, and more stable performance.

Of course, 20% is not a universal number. One team may need only 10 to 15%. Another team, especially in the middle of a heavy migration, may need 25 to 30%. But the principle of reserving capacity works better than relying on the idea that there will be a convenient moment later.

How to put this into practice

The mechanics are simple.

Start by collecting known debt items into one list. Do not try to fix everything immediately. Just make it visible. Then record the impact, risk, and estimate for each item. After that, rank the backlog and bring into the sprint work that fits roughly 20% of the team’s capacity.

The next step is to keep the work measurable. If the team takes on performance debt, it should know which metrics are expected to move. If the work is about tests, it should be clear where regression risk should go down. If the effort is about documentation, it should be clear which dependency on specific people is being reduced.

One more point matters: the backlog needs regular review. Some items lose importance over time. Others become more urgent. In some cases, an architectural change removes an old debt item entirely. In others, new debt appears that was not visible before.

When the Same Reference Data Lives in Three Services: Why We Moved It into a Dedicated Service

Denis Toropov — Sun, 20 Sep 2026 18:32:11 +0000

How to reduce data collisions when three services use different databases but rely on the same business semantics.

In distributed systems, problems rarely begin with something dramatic. On paper, everything usually looks rational: separate services, separate databases, clear responsibility boundaries. But at some point, the system starts arguing with itself.

I had a case where we had to split balance-related data across three different databases.

The first database was used by the account display service. The second one powered the service that showed customer-level balances. The third stored current account balances. Each database had its own service because each domain had different read patterns, different performance requirements, and a different data representation model.

At first, that separation looked reasonable. The real problem was elsewhere: all three services needed the same reference data.

That was where the real complexity began.

Where the Logic Started Breaking

When the same reference data exists in multiple services, the system almost inevitably starts producing different versions of the truth.

In one place, a status has already been updated. In another, an old category is still there. In a third, a local mapping was once introduced as a temporary workaround and somehow survived multiple releases. Formally, the data still looks similar. In practice, the business meaning starts to diverge.

For users, this looks like a simple inconsistency between screens:

the same account appears in one category in one view;
gets aggregated using another attribute in a different view;
and is displayed with yet another interpretation somewhere else.

For the engineering team, this is no longer a cosmetic issue. It becomes incidents, manual reconciliation, long investigations, and eventually a loss of trust in the data.

The Initial Setup

At the architecture level, the system looked roughly like this:

the account service worked with its own database;
the customer balance service worked with its own database;
the current account balance service worked with its own database;
each service used the same reference data: account types, statuses, product attributes, classifiers, and other reference entities.

On paper, this is a standard enterprise architecture pattern. But the moment you have multiple owners of the same business semantics, you start running into questions that are very hard to answer consistently.

Who is the source of truth for a reference entity? When should an update be considered applied? What happens if one system is already using the new version while another is still on the old one? Who owns the investigation when the same field is interpreted differently across systems?

If I answer honestly, at that stage we did not have one good answer to those questions.

Why Reference Data Is Not “Secondary Data”

This is a common design trap. Reference data is often treated as something secondary: not money, not transactions, not the main operational flow. So teams assume they can simply keep a local copy next to every service and sync it somehow later.

But in practice, reference data defines how the system interprets the primary data.

A balance by itself is just a number. Its meaning appears only in context: product type, account status, segment membership, aggregation attributes, and other business qualifiers. If that context differs across services, then even the same balance value stops meaning the same thing to the business.

At that point, this is no longer a technical inconsistency. It is a semantic inconsistency.

What Problems Started Showing Up

The issues did not appear as one major outage. They accumulated as a growing gray zone.

First, reference data versions drifted asynchronously. One service had already applied a change, another had not, and a third updated through a separate process. As a result, the same entity started looking different across user journeys.

Second, services developed different refresh rules. One updated on a schedule, another reacted to an event, and a third relied on a separate integration flow. Even if the source reference data is nominally the same, different delivery mechanisms quickly create drift.

Third, diagnostics became much harder. Any incident of the form “why is it one thing here and another there?” required investigating three databases, three services, three update histories, and often multiple teams. The worst part was that these incidents rarely looked like major failures. They lived as chronic operational pain.

The Options We Considered

Once it became clear that local duplication of reference data was getting too expensive, we considered several options.

Option 1. Keep Local Copies and Improve Synchronization

The most obvious path was to keep the architecture as is and simply make update delivery more disciplined.

The benefit is obvious: services keep local data, fast reads, and their familiar integration model.

The downside is that the ownership problem does not go away. Even with better synchronization, the system still has multiple places where the same reference data can be interpreted and changed differently.

Option 2. Create a Shared Reference Database

This is better than having copies in three different places. At least there is formally a single storage layer.

But this option often creates another problem: services start reading the same schema directly, rules get duplicated in consuming code, and the contract evolves without a clear owner. In the end, you may have one database, but you still do not have one place that actually owns reference data behavior.

Option 3. Build a Dedicated Reference Data Service

This was the option we ultimately chose.

The idea was simple: reference data stops being “something shared by everyone” and becomes its own domain with an explicit owner. One service is responsible for storage, versioning, validation, and change publication. The other services stop deciding for themselves which version of a reference entity is the correct one.

Why We Chose a Dedicated Reference Data Service

The key insight was this: our real problem was not data delivery by itself. It was multiple owners of the same business semantics.

As long as reference data lives in several services, the system is almost guaranteed to produce local interpretations:

one service introduces an extra status;
another adds a local mapping;
a third implements a special filtering rule;
a fourth leaves a temporary workaround in place until it quietly becomes permanent.

A dedicated reference data service does not magically make the system simple. But it removes the main source of collisions: multiple decision centers for the same reference entities.

Put differently, we stopped synchronizing three almost identical truths and started operating with one.

What Changed in the Architecture

After we introduced the service, the architecture became conceptually simpler:

the reference data service became the owner of the canonical model;
the account service, customer balance service, and current balance service stopped owning reference data independently;
validation, versioning, and update rules were centralized;
all collisions were investigated against one source of truth rather than three local copies.

The important nuance is that we centralized ownership, not necessarily every online read.

If every consumer depends on synchronous runtime calls to the reference data service, then instead of data inconsistency you may end up with cascading degradation when that service is unavailable. In systems like this, the better pattern is usually to centralize the write model and the rules, while shaping the read model according to each service’s load profile.

What a Real Reference Data Service Must Do

If you build this service seriously, not as a thin CRUD wrapper over a table, it has several non-negotiable responsibilities.

First, it must own the model. Not just store data, but define fields, statuses, relationships, constraints, and lifecycle rules.

Second, it must support versioning. At any moment, a consumer should be able to answer which version of a reference dataset it is currently using.

Third, it must publish changes predictably. That can be done through APIs, events, snapshots, or a hybrid model, but the delivery mechanism needs to be controlled and consistent.

Fourth, it must enforce data quality: validation, auditability, deduplication, mandatory field checks, and a transparent change process.

Fifth, it must be observable. If balances are interpreted through this reference data, then freshness, version, update errors, and consumer lag all become production metrics, not implementation details.

Trade-offs

Approach	Pros	Cons
Local copies of reference data in each service	Fast local reads, minimal network dependency, service autonomy	Data drift, difficult investigations, multiple sources of truth
Shared reference database	Formally centralized storage, relatively easy starting point	Strong schema coupling, high risk of rule leakage into consumers, no explicit contract ownership
Dedicated reference data service	Centralized ownership, transparent rules, fewer collisions, easier audit and version control	A new critical component, higher SLA requirements, separate design needed for change distribution

We chose the third option because in our case the main cost was not read latency. It was the cost of inconsistent data semantics across systems.

Where Teams Usually Get It Wrong

The most common mistake is to call something a reference data service when it is really just a database with a REST wrapper.

If that service does not own the model, has no versioning, no controlled change lifecycle, and no transparent data distribution model, then it does not solve the original problem. It merely moves it somewhere else.

The second mistake is forcing all consumers to read reference data synchronously and only online. That quickly turns an infrastructure component into a single point of cascading degradation.

In practice, a hybrid model is usually the right one:

a single service owns the reference data;
changes are published centrally;
consumers can keep local read-optimized projections where that makes sense for load and latency;
the right to change and interpret reference data remains centralized.

Decision Log

Context. Three services worked with three separate databases but used the same reference data to interpret balances.

Problem. We were seeing collisions, version drift, and difficult incidents where different systems showed different values or categories for the same entity.

Options. Keep local copies and improve synchronization; create a shared database; build a dedicated reference data service.

Decision. Build a dedicated reference data service as the single owner of reference entities.

Why. The core issue was not only data delivery. It was multiple ownership of the same business semantics.

Consequences. Fewer collisions, simpler investigations, and a clearer evolution path for reference data. The price was a new critical service that had to be designed as part of the core architecture rather than as a supporting utility.

What I Would Explicitly Build Into This System

If I were designing a similar setup for production from the start, I would explicitly include the following:

a single writer for reference data;
explicit versions and publication timestamps;
freshness controls on the consumer side;
auditability and clear ownership;
backward compatibility rules for contract changes;
observability not only for service availability, but for consumer data freshness as well.

Final Takeaway

When multiple services use the same reference data, the problem very quickly stops being about where to store tables. It becomes a question of who owns the meaning of the data.

In our case, we had three databases, three services, and one shared problem: the same reference data lived in several places and started drifting over time. Moving that logic into a dedicated reference data service did not remove all system complexity, but it removed the most dangerous kind of complexity: competing versions of the truth.

The short version is this: in a distributed system, you can live with different databases, different services, and different read models. But if reference data defines how the business understands its core data, it needs a single owner.

Transactional Outbox with Kafka: How to Stop Losing Events When Syncing Databases

Denis Toropov — Thu, 11 Jun 2026 19:07:00 +0000

Transactional Outbox with Kafka: How to Stop Losing Events When Syncing Databases

When you sync data between services (or databases) through Kafka, the classic failure looks like this: the database transaction commits, but the Kafka message never gets published (crash, network issue, timeout). Your systems diverge silently, and you only discover it when a user reports incorrect data.

Why the obvious fixes don’t work

Wrapping Kafka send in retries doesn’t guarantee delivery if the process dies after the DB commit. Sending to Kafka before committing the DB creates “phantom events” about changes that never made it into the database. Two-phase commit is usually too complex operationally and doesn’t fit Kafka in a clean, universal way.

The key idea

Make the critical operation one atomic DB transaction: write the business data and the event to an outbox table in the same commit. If the update exists, the event exists too.

Conceptually:

In one DB transaction: update domain tables + INSERT into outbox
Later: a separate mechanism reads outbox and publishes to Kafka
This makes delivery retryable because the event is durably stored

Two ways to deliver events from the outbox

1) Polling Relay

A background worker polls unsent rows (sent_at IS NULL), publishes them to Kafka, then marks them as sent. It’s simple, reliable, and often enough when sub-second latency isn’t a strict requirement. If you run multiple workers, you must ensure each event is claimed once (typically via row locking / “skip locked” patterns).

2) CDC via Debezium

Instead of polling, Debezium streams inserts from the database transaction log (WAL) to Kafka. This reduces latency and removes the need for a poller, but adds infrastructure/ops complexity (Debezium + Kafka Connect).

Don’t forget: duplicates will happen

Outbox pipelines are typically at-least-once, so consumers must be idempotent. The most robust approach is an inbox table with a unique constraint on event_id: first insert the event id; if it already exists, skip processing. This avoids race conditions that occur with “check then insert”.

Common pitfalls

If you run the relay inside every API instance, scaling your API can accidentally scale polling load and hammer the DB. Also, outbox/inbox tables will grow—plan retention/cleanup. Finally, monitor lag: the count of unsent events and the age of the oldest unsent event are simple, high-signal metrics.

Bottom line

Transactional Outbox prevents “DB committed but event lost” by making event creation part of the DB transaction, then reliably delivering events to Kafka via polling or CDC—while consumers protect themselves with idempotency.