When the Same Reference Data Lives in Three Services: Why We Moved It into a Dedicated Service

Denis Toropov — Sun, 20 Sep 2026 18:32:11 +0000

How to reduce data collisions when three services use different databases but rely on the same business semantics.

In distributed systems, problems rarely begin with something dramatic. On paper, everything usually looks rational: separate services, separate databases, clear responsibility boundaries. But at some point, the system starts arguing with itself.

I had a case where we had to split balance-related data across three different databases.

The first database was used by the account display service. The second one powered the service that showed customer-level balances. The third stored current account balances. Each database had its own service because each domain had different read patterns, different performance requirements, and a different data representation model.

At first, that separation looked reasonable. The real problem was elsewhere: all three services needed the same reference data.

That was where the real complexity began.

Where the Logic Started Breaking

When the same reference data exists in multiple services, the system almost inevitably starts producing different versions of the truth.

In one place, a status has already been updated. In another, an old category is still there. In a third, a local mapping was once introduced as a temporary workaround and somehow survived multiple releases. Formally, the data still looks similar. In practice, the business meaning starts to diverge.

For users, this looks like a simple inconsistency between screens:

the same account appears in one category in one view;
gets aggregated using another attribute in a different view;
and is displayed with yet another interpretation somewhere else.

For the engineering team, this is no longer a cosmetic issue. It becomes incidents, manual reconciliation, long investigations, and eventually a loss of trust in the data.

The Initial Setup

At the architecture level, the system looked roughly like this:

the account service worked with its own database;
the customer balance service worked with its own database;
the current account balance service worked with its own database;
each service used the same reference data: account types, statuses, product attributes, classifiers, and other reference entities.

On paper, this is a standard enterprise architecture pattern. But the moment you have multiple owners of the same business semantics, you start running into questions that are very hard to answer consistently.

Who is the source of truth for a reference entity? When should an update be considered applied? What happens if one system is already using the new version while another is still on the old one? Who owns the investigation when the same field is interpreted differently across systems?

If I answer honestly, at that stage we did not have one good answer to those questions.

Why Reference Data Is Not “Secondary Data”

This is a common design trap. Reference data is often treated as something secondary: not money, not transactions, not the main operational flow. So teams assume they can simply keep a local copy next to every service and sync it somehow later.

But in practice, reference data defines how the system interprets the primary data.

A balance by itself is just a number. Its meaning appears only in context: product type, account status, segment membership, aggregation attributes, and other business qualifiers. If that context differs across services, then even the same balance value stops meaning the same thing to the business.

At that point, this is no longer a technical inconsistency. It is a semantic inconsistency.

What Problems Started Showing Up

The issues did not appear as one major outage. They accumulated as a growing gray zone.

First, reference data versions drifted asynchronously. One service had already applied a change, another had not, and a third updated through a separate process. As a result, the same entity started looking different across user journeys.

Second, services developed different refresh rules. One updated on a schedule, another reacted to an event, and a third relied on a separate integration flow. Even if the source reference data is nominally the same, different delivery mechanisms quickly create drift.

Third, diagnostics became much harder. Any incident of the form “why is it one thing here and another there?” required investigating three databases, three services, three update histories, and often multiple teams. The worst part was that these incidents rarely looked like major failures. They lived as chronic operational pain.

The Options We Considered

Once it became clear that local duplication of reference data was getting too expensive, we considered several options.

Option 1. Keep Local Copies and Improve Synchronization

The most obvious path was to keep the architecture as is and simply make update delivery more disciplined.

The benefit is obvious: services keep local data, fast reads, and their familiar integration model.

The downside is that the ownership problem does not go away. Even with better synchronization, the system still has multiple places where the same reference data can be interpreted and changed differently.

Option 2. Create a Shared Reference Database

This is better than having copies in three different places. At least there is formally a single storage layer.

But this option often creates another problem: services start reading the same schema directly, rules get duplicated in consuming code, and the contract evolves without a clear owner. In the end, you may have one database, but you still do not have one place that actually owns reference data behavior.

Option 3. Build a Dedicated Reference Data Service

This was the option we ultimately chose.

The idea was simple: reference data stops being “something shared by everyone” and becomes its own domain with an explicit owner. One service is responsible for storage, versioning, validation, and change publication. The other services stop deciding for themselves which version of a reference entity is the correct one.

Why We Chose a Dedicated Reference Data Service

The key insight was this: our real problem was not data delivery by itself. It was multiple owners of the same business semantics.

As long as reference data lives in several services, the system is almost guaranteed to produce local interpretations:

one service introduces an extra status;
another adds a local mapping;
a third implements a special filtering rule;
a fourth leaves a temporary workaround in place until it quietly becomes permanent.

A dedicated reference data service does not magically make the system simple. But it removes the main source of collisions: multiple decision centers for the same reference entities.

Put differently, we stopped synchronizing three almost identical truths and started operating with one.

What Changed in the Architecture

After we introduced the service, the architecture became conceptually simpler:

the reference data service became the owner of the canonical model;
the account service, customer balance service, and current balance service stopped owning reference data independently;
validation, versioning, and update rules were centralized;
all collisions were investigated against one source of truth rather than three local copies.

The important nuance is that we centralized ownership, not necessarily every online read.

If every consumer depends on synchronous runtime calls to the reference data service, then instead of data inconsistency you may end up with cascading degradation when that service is unavailable. In systems like this, the better pattern is usually to centralize the write model and the rules, while shaping the read model according to each service’s load profile.

What a Real Reference Data Service Must Do

If you build this service seriously, not as a thin CRUD wrapper over a table, it has several non-negotiable responsibilities.

First, it must own the model. Not just store data, but define fields, statuses, relationships, constraints, and lifecycle rules.

Second, it must support versioning. At any moment, a consumer should be able to answer which version of a reference dataset it is currently using.

Third, it must publish changes predictably. That can be done through APIs, events, snapshots, or a hybrid model, but the delivery mechanism needs to be controlled and consistent.

Fourth, it must enforce data quality: validation, auditability, deduplication, mandatory field checks, and a transparent change process.

Fifth, it must be observable. If balances are interpreted through this reference data, then freshness, version, update errors, and consumer lag all become production metrics, not implementation details.

Trade-offs

Approach	Pros	Cons
Local copies of reference data in each service	Fast local reads, minimal network dependency, service autonomy	Data drift, difficult investigations, multiple sources of truth
Shared reference database	Formally centralized storage, relatively easy starting point	Strong schema coupling, high risk of rule leakage into consumers, no explicit contract ownership
Dedicated reference data service	Centralized ownership, transparent rules, fewer collisions, easier audit and version control	A new critical component, higher SLA requirements, separate design needed for change distribution

We chose the third option because in our case the main cost was not read latency. It was the cost of inconsistent data semantics across systems.

Where Teams Usually Get It Wrong

The most common mistake is to call something a reference data service when it is really just a database with a REST wrapper.

If that service does not own the model, has no versioning, no controlled change lifecycle, and no transparent data distribution model, then it does not solve the original problem. It merely moves it somewhere else.

The second mistake is forcing all consumers to read reference data synchronously and only online. That quickly turns an infrastructure component into a single point of cascading degradation.

In practice, a hybrid model is usually the right one:

a single service owns the reference data;
changes are published centrally;
consumers can keep local read-optimized projections where that makes sense for load and latency;
the right to change and interpret reference data remains centralized.

Decision Log

Context. Three services worked with three separate databases but used the same reference data to interpret balances.

Problem. We were seeing collisions, version drift, and difficult incidents where different systems showed different values or categories for the same entity.

Options. Keep local copies and improve synchronization; create a shared database; build a dedicated reference data service.

Decision. Build a dedicated reference data service as the single owner of reference entities.

Why. The core issue was not only data delivery. It was multiple ownership of the same business semantics.

Consequences. Fewer collisions, simpler investigations, and a clearer evolution path for reference data. The price was a new critical service that had to be designed as part of the core architecture rather than as a supporting utility.

What I Would Explicitly Build Into This System

If I were designing a similar setup for production from the start, I would explicitly include the following:

a single writer for reference data;
explicit versions and publication timestamps;
freshness controls on the consumer side;
auditability and clear ownership;
backward compatibility rules for contract changes;
observability not only for service availability, but for consumer data freshness as well.

Final Takeaway

When multiple services use the same reference data, the problem very quickly stops being about where to store tables. It becomes a question of who owns the meaning of the data.

In our case, we had three databases, three services, and one shared problem: the same reference data lived in several places and started drifting over time. Moving that logic into a dedicated reference data service did not remove all system complexity, but it removed the most dangerous kind of complexity: competing versions of the truth.

The short version is this: in a distributed system, you can live with different databases, different services, and different read models. But if reference data defines how the business understands its core data, it needs a single owner.

Transactional Outbox with Kafka: How to Stop Losing Events When Syncing Databases

Denis Toropov — Thu, 11 Jun 2026 19:07:00 +0000

Transactional Outbox with Kafka: How to Stop Losing Events When Syncing Databases

When you sync data between services (or databases) through Kafka, the classic failure looks like this: the database transaction commits, but the Kafka message never gets published (crash, network issue, timeout). Your systems diverge silently, and you only discover it when a user reports incorrect data.

Why the obvious fixes don’t work

Wrapping Kafka send in retries doesn’t guarantee delivery if the process dies after the DB commit. Sending to Kafka before committing the DB creates “phantom events” about changes that never made it into the database. Two-phase commit is usually too complex operationally and doesn’t fit Kafka in a clean, universal way.

The key idea

Make the critical operation one atomic DB transaction: write the business data and the event to an outbox table in the same commit. If the update exists, the event exists too.

Conceptually:

In one DB transaction: update domain tables + INSERT into outbox
Later: a separate mechanism reads outbox and publishes to Kafka
This makes delivery retryable because the event is durably stored

Two ways to deliver events from the outbox

1) Polling Relay

A background worker polls unsent rows (sent_at IS NULL), publishes them to Kafka, then marks them as sent. It’s simple, reliable, and often enough when sub-second latency isn’t a strict requirement. If you run multiple workers, you must ensure each event is claimed once (typically via row locking / “skip locked” patterns).

2) CDC via Debezium

Instead of polling, Debezium streams inserts from the database transaction log (WAL) to Kafka. This reduces latency and removes the need for a poller, but adds infrastructure/ops complexity (Debezium + Kafka Connect).

Don’t forget: duplicates will happen

Outbox pipelines are typically at-least-once, so consumers must be idempotent. The most robust approach is an inbox table with a unique constraint on event_id: first insert the event id; if it already exists, skip processing. This avoids race conditions that occur with “check then insert”.

Common pitfalls

If you run the relay inside every API instance, scaling your API can accidentally scale polling load and hammer the DB. Also, outbox/inbox tables will grow—plan retention/cleanup. Finally, monitor lag: the count of unsent events and the age of the oldest unsent event are simple, high-signal metrics.

Bottom line

Transactional Outbox prevents “DB committed but event lost” by making event creation part of the DB transaction, then reliably delivering events to Kafka via polling or CDC—while consumers protect themselves with idempotency.

DEV Community: Denis Toropov

When the Same Reference Data Lives in Three Services: Why We Moved It into a Dedicated Service

Where the Logic Started Breaking

The Initial Setup

Why Reference Data Is Not “Secondary Data”

What Problems Started Showing Up

The Options We Considered

Option 1. Keep Local Copies and Improve Synchronization

Option 2. Create a Shared Reference Database

Option 3. Build a Dedicated Reference Data Service

Why We Chose a Dedicated Reference Data Service

What Changed in the Architecture

What a Real Reference Data Service Must Do

Trade-offs

Where Teams Usually Get It Wrong

Decision Log

What I Would Explicitly Build Into This System

Final Takeaway

Transactional Outbox with Kafka: How to Stop Losing Events When Syncing Databases

Transactional Outbox with Kafka: How to Stop Losing Events When Syncing Databases

Why the obvious fixes don’t work

The key idea

Two ways to deliver events from the outbox

1) Polling Relay

2) CDC via Debezium

Don’t forget: duplicates will happen

Common pitfalls

Bottom line