DEV Community: Martin Simon

Shared library for Micro-Services: Why should you have one?

Martin Simon — Tue, 29 Apr 2025 17:34:49 +0000

When I talk to developers about microservices, one thing always stands out: everyone seems to define microservices slightly differently. From how small a service should be to how tightly they interact, the implementation varies widely. But one topic comes up frequently and often sparks debate — shared libraries. Should microservices share code, or does that undermine their independence?

Based on my experience, I believe that having a shared library can be a good idea — if you do it right. In this post, I’ll explore why shared libraries can be useful, where the risks are, and how to approach them responsibly.

What Do We Mean by a Shared Library?

A shared library is reusable code that multiple services can include as a dependency, typically managed using tools like Gradle or Maven in the Java ecosystem. These libraries usually provide utilities, technical abstractions, or common infrastructure integrations.

Yes, microservices promise language and framework freedom. But in small to medium-sized companies, that flexibility often becomes a burden. It's more efficient to standardize around a single tech stack — and in that environment, sharing well-abstracted code can offer real productivity benefits.

Think of frameworks like Spring Boot — we don’t write everything from scratch. In the same spirit, a shared library is your internal toolbox, tuned to your company’s needs.

Why Would You Want a Shared Library?

Here are some practical reasons shared libraries can be valuable:

Boost Productivity: Avoid re-implementing the same boilerplate in every service.
Standardize Best Practices: Enforce consistent logging, error handling, metrics, and more.
Encapsulate Technical Complexity: Implement things like multi-tenancy, authentication, or service-to-service communication once and reuse them.

Some examples from real-world projects:

Multi-tenant HTTP middleware
Shared Spring Security configuration
Multi-tenant MongoDB setup
Kafka message header propagators

These aren’t business rules — they’re technical concerns best solved once and reused.

Where’s the Catch?

Of course, shared libraries are not a silver bullet. They come with trade-offs that you should handle with care:

1. Avoid Business Logic

Never include business rules or data models in shared libraries. That creates tight coupling between services and violates microservice independence.

Bad examples:

Shared domain entities (JPA/Hibernate)
Shared data access layers
Shared decision logic

Services should own their own business logic and evolve independently.

2. Versioning Matters

Shared libraries should be versioned like any external dependency. Use semantic versioning, and assume services may not upgrade right away. This forces you to maintain backward compatibility and minimize breaking changes.

3. Ownership and Maintenance

Someone needs to own the library. Without clear responsibility, it can become outdated, bloated, or inconsistent. Ideally, a platform or infra team maintains it — or at least one team with a clear mandate and review process.

Combine Shared Library and the BOM

When using a shared library in your project alongside other external dependencies, managing dependency versions can become tricky. Inconsistent versions across microservices can lead to compatibility issues, especially as your codebase grows. Fortunately, frameworks like Spring Boot address this problem by providing a BOM (Bill of Materials), which predefines the version of each used dependency. This ensures that all dependencies are aligned and eliminates the risk of version mismatches.

It’s a good idea to apply the same concept to your shared library. By using a BOM for your internal dependencies, you can maintain consistent versions across all services that rely on it. When using Spring Boot, you can even inherit from its BOM and simply add your own dependencies as needed.

Simplified upgrades

One of the biggest advantages of using a BOM is the simplicity of upgrading project dependencies. In a system with multiple microservices, maintaining dependency versions independently in each service can be a significant burden. With a BOM, you can centralize version management, making upgrades much more straightforward. Instead of updating dependencies in each individual service, you only need to upgrade the version of your shared library in the BOM.

This means that when you need to update your shared library or any of its dependencies, you can apply that change across all services in a single step, rather than updating each service manually. This approach significantly reduces maintenance overhead and ensures that all services stay in sync with the latest versions of the shared libraries and dependencies.

Recommendations

✅ Use a BOM for version management

✅ Include technical utilities only (auth, logging, retries, etc.)

✅ Use semantic versioning and release notes

✅ Assign clear ownership and establish a contribution process

❌ Don’t share business logic or domain models

❌ Don’t force synchronous coupling through shared code

Treat Shared Libraries Like Third-Party Libraries

This is the best mindset. A shared library should feel like an external dependency. You use it because it solves a general technical problem well — not because someone dropped a bunch of code into a common repo.

Summary

Microservices are about independent, scalable, and loosely coupled components. But independence doesn’t mean isolation. Technical reuse — if done carefully — can boost productivity and consistency without violating the microservice philosophy.

Just remember: share the tools, not the rules.

[Boost]

Martin Simon — Thu, 13 Mar 2025 12:57:36 +0000

Ditching the Overhead: My Experience with Go, HTMX, Tailwind & Alpine.js

Martin Simon ・ Mar 12

Ditching the Overhead: My Experience with Go, HTMX, Tailwind & Alpine.js

Martin Simon — Wed, 12 Mar 2025 18:59:14 +0000

Prologue

As a software developer primarily focused on Java for backend systems, I occasionally enjoy developing simple web apps to stay up-to-date with frontend technologies. A few years ago, I developed a web app to display results from rope climbing competitions. To my surprise, it's still in use today. Back then, I had grand plans to enhance it with more features, but life had other plans, and the project gathered dust for several years.

The app was built using Angular 9, and to streamline development, I used PrimeNG as the component library. When I recently decided to update it, I was taken aback by the challenges of upgrading to Angular 19. Some dependencies, including the component library, were no longer compatible.

Honestly, this was frustrating. With a full-time job, family responsibilities, and other hobbies, I didn't want to spend my limited free time updating side projects every six months just to keep up with Angular's updates.

So, I faced a choice: invest time in updating the project or explore something new and exciting. I chose the latter and decided to use a tech stack that has gained popularity recently—Go for the backend, and HTMX, Alpine.js, and Tailwind CSS for the frontend.

Why I chose Go as Java developer

Firstly, I love learning new things. I don't expect my side projects to make me rich; I see them as opportunities to have fun and learn something new. Additionally, I don't want to spend much on hosting, and since Java with Spring Boot has significant memory demands, I sought a lightweight alternative that I could host on my VPS with 1 GB RAM—including a database and Traefik for handling certificates.

In this article, I want to motivate you to try out this stack. It brought me a lot of joy, and I'd like to pass on a few insights.

Setting goals

My primary goal was to rewrite an existing web application from Angular + Spring Boot into Go + HTMX. Beyond that, I set several other objectives.

Backend Goals

Learn Something New - I work with Java daily, so redoing this project in Java didn't make sense. I wanted a trending language with staying power. Go, introduced in 2007, has proven its market position.
KISS principle - Make the application easy to maintain in the coming years.
Low-memory footprint - Keep hosting costs low.

Frontend Goals

Avoid Dependency Hell - Managing dependencies can be challenging, especially in the frontend ecosystem, where libraries frequently release new versions with less emphasis on backward compatibility.
Find a Stable Stack - I want to spend less time maintaining the project while keeping it up-to-date.
Write Less JavaScript - Reduce complexity by writing less JavaScript for my side projects.
Write Less CSS - As a backend developer, I often feel overwhelmed by the amount of CSS in projects. I wanted to avoid the hassle of managing extensive CSS files.

Developing the Backend

What I liked about Go

Java has a stable and robust ecosystem, and it will take time for Go to reach a similar level. However, Go brings a refreshing perspective. I tried to use as few dependencies as possible to maximize performance and minimize overhead.

I particularly liked the following:

Slqc - A great library for generating code from plain SQL, focused on performance and utilization. I personally like this much more then ORM frameworks, where the control over generated SQL and its performance is only limited.
Quick Startup - Go apps start quickly compared to bloated Spring Boot applications, making hot deployments unnecessary.
Low Memory Footprint - As someone who values fun over profit with side projects, keeping hosting costs low is crucial.
Multiple Return Values - A feature I've often missed in Java. While some might argue it goes against OOP principles, it saves a lot of code.

What I miss so far

Exception Handling - Go's error handling is different and took some getting used to.
Stack Traces - Java's stack traces are more informative.
Excelize - A dependency for reading Excel files, similar to Apache POI. While easier to use than POI, it lacks some of POI's robustness and features.

Developing the Frontend

Choosing the Frontend-Stack

For my small projects, I value simplicity. Although I'm confident writing TypeScript in the Angular framework, I sought less overhead. I considered using plain JavaScript to avoid dependencies and breaking changes in Angular and related libraries. However, this would have introduced complexity elsewhere, requiring more code to handle basic tasks. I ended up with three minimal libraries that offer a relatively stable API with amazing additional value.

HTMX

When I first read about HTMX, it immediately piqued my interest. This tiny library challenges the common pattern of splitting frontend and backend development into two independent applications, which has become the standard in recent years. It reminded me of the good old days of full-stack development with JSP.

HTMX extends standard HTML with additional attributes to handle asynchronous calls to the backend, retrieving data to be displayed. Unlike JSON, HTMX works directly with the DOM, requiring the backend service to return the entire HTML, which is then automatically added to the DOM. Key features of HTMX include:

AJAX Requests: Easily fetch data from the server and update the page without a full reload.
Partial Page Updates: Dynamically update specific parts of a webpage.
Event Handling: Manage user interactions like clicks, hovers, and form submissions.

With HTMX, I easily implemented:

Loading Spinner - The hx-indicator attribute controls the display of a loading indicator with literally no additional code.
Pushing URL State - When a partial page update should also update the URL state, the attribute "hx-push-url" does the job.
Partial Update of Pages - Switching between menu items or triggering actions renders data in any defined area on the page.

Alpine.js

One of my goals was to write as little JavaScript as possible. Alpine.js complements HTMX perfectly, adding reactivity to the frontend without writing a single line of JavaScript.

With Alpine.js, I easily implemented:

Custom Dropdown Component - Without a component library, I had to implement my own custom dropdown. Surprisingly, this was easy using a few attributes to control the state and handle events.
Confirmation pop-ups - Showing or hiding elements on the page couldn't be easier with Alpine.js.
Menus with highlighted selected items - Highlighting the currently selected menu item is a local state that must be managed within the application. Alpine.js excels at handling such tasks.

Tailwind CSS

Tailwind CSS makes writing custom CSS obsolete. It offers predefined classes and an intuitive system that's easy to use. Plus, Tailwind is fully customizable, allowing you to extend and adapt styles and colors to your needs.

Moreover, Tailwind's stability regarding backward compatibility is commendable. The framework's developers prioritize maintaining a stable API, ensuring that updates don't break existing styles. This stability is crucial for long-term projects, as it minimizes the time spent on maintenance and allows developers to focus on adding new features rather than fixing broken styles. With Tailwind, you can confidently build and scale your projects, knowing that your styling foundation is robust and future-proof.

Summary

In conclusion, transitioning my web app from Angular and Spring Boot to Go, HTMX, Alpine.js, and Tailwind CSS was a rewarding experience that offered numerous benefits. This new stack not only enhanced my productivity but also provided a stable and efficient foundation for future development. Go's simplicity and performance, combined with the intuitive frontend solutions offered by HTMX and Alpine.js, made the development process enjoyable and less cumbersome. While there are challenges, such as the limited reusability of components in larger projects, the overall advantages of this stack make it a compelling choice for both personal and professional projects.

Multi-Tenancy in Java-Based Microservice-Platform

Martin Simon — Wed, 05 Feb 2025 20:07:10 +0000

Supporting multi-tenancy is often a crucial step for scaling your application and expanding its reach to a broader customer base. However, building this capability introduces a layer of complexity to your application, particularly when working within a microservice architecture. In this article, we’ll explore ways to manage this complexity effectively and discuss practical approaches to implementing multi-tenancy with ease.

What Is a Multi-Tenancy

Multi-tenancy is the capability of a system or application to support multiple tenants (such as customers or organizations) within a single running instance while maintaining logical isolation. This approach optimizes resource utilization, reducing infrastructure and maintenance costs by sharing hardware, software, and operational overhead.

A well-implemented multi-tenancy solution should be seamless for tenants—they shouldn’t notice that they are sharing the instance with others. Tenant isolation can be achieved at different levels, such as the database, authentication, or application code itself.

Since multi-tenancy introduces additional complexity, the architectural decisions made during implementation significantly impact scalability, security, and future costs. In this article, we will explore how to handle tenants across different layers of an application. We will specifically focus on isolation at the application layer and briefly discuss approaches for data separation.

Managing a Tenant Context

When implementing tenant isolation at the application level, you will quickly realize that you need to have a tenant identifier available in different parts of your application code. We will refer to this identifier as tenant-id.

Passing this ID across multiple layers of an application can be cumbersome, adding unnecessary complexity and making code maintenance and testability significantly more challenging. If every interaction with the application logic requires the tenant-id, we can introduce the concept of a tenant context, which centrally holds this information.

Since Java is a multi-threaded language, we can leverage thread-local storage to manage the tenant context efficiently. This approach allows us to associate the tenant-id with the current thread, making it accessible throughout the application's execution flow. However, it's important to note that this method does not work in reactive frameworks, as they handle concurrency differently from traditional thread-based processing.

Using ThreadLocal to Keep the Context

This simple class will be the core of our multi-tenancy implementation. It contains only three methods and very little logic, but it is powerful and provides incredible flexibility.

public class TenantContext {
    private static final ThreadLocal<String> tenantIdHolder = new ThreadLocal<>();

    public static void setTenantId(String tenantId) {
        if (tenantId == null) {
            throw new TenantContextException("Tenant ID cannot be null");
        }
        tenantIdHolder.set(tenantId);
        MDC.put("tenantId", tenantId);
    }

    public static String getTenantId() {
        String tenantId = tenantIdHolder.get();
        if (tenantId == null) {
            throw new TenantContextException("Tenant ID is not set in the current thread");
        }
        return tenantId;
    }

    public static void clear() {
        tenantIdHolder.remove();
        MDC.remove("tenantId"); 
    }
}

Whenever a request arrives at the application, we are expected to set the TenantContext using the setter, which stores the tenant-id in a ThreadLocal. This ensures that whenever we call getTenantId() within the same thread where the context was set, we reliably obtain the correct tenant-id.

The clear method is also very important. Since threads in Java can be reused (for example, in thread pools), we must clear the context once processing is complete to prevent unintended data leakage between requests. This cleanup is typically performed in an interceptor after a REST call or when Kafka message processing ends.

Additionally, you can see that we put the tenant ID into the MDC, which allows us to clearly associate logs with a particular tenant.

Passing a Tenant Context between Microservices

In a microservice architecture, requests often travel across multiple services, and it's crucial to pass tenant information between them. Ensuring proper tenant context propagation is essential for maintaining data isolation and security.

For simplicity, we'll focus on two of the most common communication methods:

Synchronous calls via REST API
Asynchronous calls via Kafka

Header vs. Body: Where to Pass the Tenant Id?

Regardless of the communication channel, one key decision is where to include the tenant-id — in the header or the body of the request? Both approaches have advantages and drawbacks. Let’s compare them:

Passing the Tenant ID in the Header
✅ Pros:

Standardized: HTTP headers are commonly used for metadata (e.g., authentication tokens).
Cleaner request bodies: The payload remains focused on business data.
Abstracted from business logic: Tenant ID handling can be managed at the infrastructure level (e.g., middleware, filters), reducing code duplication.

❌ Cons:

Risk of header loss: In multi-service calls, headers might be unintentionally stripped or modified, making tenant context propagation unreliable.
Requires explicit validation: You need additional checks to ensure the tenant ID is always present in each request.

Passing the Tenant ID in the Body
✅ Pros:

Self-contained: The request body holds all necessary context, making it easier to debug.

❌ Cons:

Requires modifying the data model for each service to include tenant information.
GET request limitation: Since request bodies are not supported in GET requests, the tenant ID must be passed as a query parameter instead.

Let's look at how to easily abstract the tenant context from the rest of the business logic when using header propagation of the tenant id.

REST

import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import org.springframework.stereotype.Component;
import org.springframework.web.servlet.HandlerInterceptor;

@Component
public class TenantInterceptor implements HandlerInterceptor {
    private static final String TENANT_HEADER = "X-Tenant-ID";

    @Override
    public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) {
        String tenantId = request.getHeader(TENANT_HEADER);
        if (tenantId != null) {
            TenantContext.setTenantId(tenantId);
        }
        return true;
    }

    @Override
    public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex) {
        TenantContext.clear(); // Clean up after request processing
    }
}

After registering the interceptor, we no longer need to explicitly retrieve the tenant id or pass it between calls. It will be automatically set in the TenantContext and work seamlessly.

To also automate the propagation of the tenant-id in outgoing REST calls, we can provide another interceptor and register it with RestTemplate. This ensures successful tenant context propagation across multiple services without requiring any handling in the business logic.

import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpRequest;
import org.springframework.http.client.ClientHttpRequestExecution;
import org.springframework.http.client.ClientHttpRequestInterceptor;
import org.springframework.http.client.ClientHttpResponse;
import org.springframework.stereotype.Component;
import java.io.IOException;

@Component
public class TenantPropagationInterceptor implements ClientHttpRequestInterceptor {
    private static final String TENANT_HEADER = "X-Tenant-ID";

    @Override
    public ClientHttpResponse intercept(HttpRequest request, byte[] body, 
                                        ClientHttpRequestExecution execution) throws IOException {
        String tenantId = TenantContext.getTenantId();
        if (tenantId != null) {
            request.getHeaders().add(TENANT_HEADER, tenantId);
        }
        return execution.execute(request, body);
    }
}

Registering the interceptor.

import org.springframework.boot.web.client.RestTemplateCustomizer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.client.RestTemplate;

import java.util.List;

@Configuration
public class RestTemplateConfig {

    @Bean
    public RestTemplate restTemplate(TenantPropagationInterceptor tenantInterceptor) {
        RestTemplate restTemplate = new RestTemplate();
        restTemplate.setInterceptors(List.of(tenantInterceptor));
        return restTemplate;
    }
}

Kafka

For Kafka, we can take a very similar approach by defining an interceptor that reads the message header and passes the value into the tenant context.

import org.apache.kafka.clients.consumer.ConsumerInterceptor;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import java.util.Map;

public class TenantConsumerInterceptor implements ConsumerInterceptor<String, String> {
    private static final String TENANT_HEADER = "X-Tenant-ID";

    @Override
    public ConsumerRecords<String, String> onConsume(ConsumerRecords<String, String> records) {
        for (ConsumerRecord<String, String> record : records) {
            if (record.headers().lastHeader(TENANT_HEADER) != null) {
                String tenantId = new String(record.headers().lastHeader(TENANT_HEADER).value());
                TenantContext.setTenantId(tenantId);
            }
        }
        return records;
    }

    @Override
    public void onCommit(Map offsets) {
        // No action needed on commit
    }

    @Override
    public void close() {
        // Cleanup if needed
    }

    @Override
    public void configure(Map<String, ?> configs) {
        // No configuration needed
    }
}

We will also provide an interceptor that writes the tenant-id into the header of outgoing messages.

import org.apache.kafka.clients.producer.ProducerInterceptor;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
import java.util.Map;

public class TenantProducerInterceptor implements ProducerInterceptor<String, String> {
    private static final String TENANT_HEADER = "X-Tenant-ID";

    @Override
    public ProducerRecord<String, String> onSend(ProducerRecord<String, String> record) {
        String tenantId = TenantContext.getTenantId();
        if (tenantId != null) {
            record.headers().add(TENANT_HEADER, tenantId.getBytes());
        }
        return record;
    }

    @Override
    public void onAcknowledgement(RecordMetadata metadata, Exception exception) {
        // No action needed on acknowledgment
    }

    @Override
    public void close() {
        // Cleanup if needed
    }

    @Override
    public void configure(Map<String, ?> configs) {
        // No configuration needed
    }
}

Passing Tenant-Id from Ingress

Tenants can have customized URLs specific to them. Our goal is to find an easy way to bind the URL of the tenant application to the tenant-id. When running in Kubernetes, we can use the Ingress definition and set the header in the Nginx configuration.

The following configuration sets the tenant ID in the header while also hiding any incoming tenant-id headers from the request. Why do this?

We want to ensure that clients cannot inject unexpected tenant-id, ensuring that only our system assigns and propagates the correct tenant-id.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-Tenant-Id "tenant123";
      proxy_hide_header X-Tenant-Id;

Splitting DB

Splitting the database layer is the next logical step in building a secure multi-tenant architecture. We will not dive into the implementation details now; instead, we will explore how separation can be designed using the previously implemented components.

By customizing the abstraction of our database layer—responsible for selecting the database connection and executing queries—we gain the ability to adapt this behavior and choose the appropriate database based on the current tenant context. This further separates multi-tenancy from the business logic, allowing the application to stay focused on business functionality rather than data separation.

Summary

Multi-tenancy shouldn’t be something to fear. Following basic principles will open the door to better scaling of your applications and allow you to focus on business logic rather than the technical overhead of passing context information.

The following diagram summarizes the article and illustrates the complete architecture that has been discussed.

Let me know in the comments, what would you improve or what would you do differently?

Building Real-Time Data Pipelines with Kafka Streams

Martin Simon — Fri, 17 Jan 2025 12:33:17 +0000

Microservice architecture may offer numerous benefits, but it also comes with its fair share of trade-offs. One of these is the challenge of managing distributed data across multiple databases. Providing even basic reporting can become a daunting task when data from various sources must be combined. For such scenarios, building a reporting service on top of a relational database can make a lot of sense. But how do you transfer the data efficiently without adding technical overhead with tools like Apache NiFi or Flink? In this article, I want to take a closer look at data streams using Kafka and discuss strategies to achieve this goal. I’d love to hear your thoughts and experiences as well.

Stream the Data, Not Just Events

Consider a simple microservice for processing orders. An order can transition through the following states:

CREATED
PAID
DELIVERING
DELIVERED

If we want to update the order data in the reporting system every time there’s a change, we need to listen to those changes. However, propagating data through streams should not rely solely on events but rather on the complete data itself. Avoid designing events that carry only partial state information about your domain.

Here’s an example of a poorly designed event:

{
  "specversion": "1.0",  
  "type": "OrderStatusUpdated",
  "source": "/orders-service",
  "id": "1234-5678-9012",
  "time": "2025-01-14T12:00:00Z",
  "datacontenttype": "application/json",        
  "data": {    
    "orderId": "order-001",    
    "status": "SHIPPED",  
    "timestamp": "2025-01-14T12:00:00Z"  
  }
}

The complexity of handling such events is significantly higher compared to handling data streams. Imagine a scenario where you want to restore the data warehouse by replaying all events from Kafka. With an event-per-status approach, the system would process each individual status change of an order, drastically increasing the load on the consumer. In contrast, handling complete data objects reduces the workload and ensures a more efficient pipeline.

A simple solution is to use event structures that encapsulate the full state of the entity. For example:

{
  "specversion": "1.0",
  "type": "OrderCreateUpdate",
  "source": "/orders-service",
  "id": "event-5678",
  "time": "2025-01-14T13:05:00Z",
  "datacontenttype": "application/json",
  "data": {
    "orderId": "order-001",
    "customer": {
      "id": "customer-123"
      "name": "John Doe",
      "email": "john.doe@example.com"
    },
    "items": [
      {
        "productId": "prod-123",
        "quantity": 2,
        "price": 20.0
      }
    ],
    "status": "SHIPPED",
    "statusHistory": [
      {
        "status": "PLACED",
        "timestamp": "2025-01-14T10:00:00Z"
      },
      {
        "status": "PROCESSING",
        "timestamp": "2025-01-14T11:00:00Z"
      },
      {
        "status": "SHIPPED",
        "timestamp": "2025-01-14T12:00:00Z"
      }
    ],
    "shippingDetails": {
      "address": "123 Main St, Springfield",
      "carrier": "DHL"
    },
    "totalAmount": 40.0,
    "lastUpdated": "2025-01-14T12:00:00Z"
  }
}

Idempotency: The Cornerstone of Data Consistency

Let’s revisit the example of a poorly designed event that contains only a partial status update. Beyond the increased load on the consumer when restoring data, the critical flaw is the lack of idempotency in the event design.

While the event includes an id, the status update itself lacks a unique identifier. This creates challenges during consumption. Although consumers can verify whether they’ve already processed an event based on its id and skip it if necessary, what happens if there’s an issue with the producer? Imagine the producer unintentionally generates multiple events for the same status change. Without a unique identifier for the status update, it becomes nearly impossible for consumers to determine whether the status has already been processed.

This flaw can lead to data duplication during event processing, where the same change is applied multiple times, resulting in inconsistencies in the downstream system. For instance, in the case of a reporting database, duplicated events could inflate metrics, misrepresent totals, or even corrupt the overall dataset.

To ensure idempotency, each event should carry a unique identifier not only for the event itself but also for the specific change it represents. This allows consumers to reliably deduplicate messages and maintain consistent data, even in the face of producer-side issues or message re-delivery.

Recreating Data in Your Data Store

When it comes to initially loading data into a reporting system or restoring data for any reason, it’s essential to consider various strategies. Let’s explore some techniques to handle these scenarios effectively.

Replaying the Topic

Resetting the offset of the Kafka consumer is the simplest way to retrieve all events from the beginning of the topic and restore data in your reporting system. However, this approach comes with several prerequisites and limitations:

Retention Time: The event retention time must be set to -1 (infinite retention) to ensure no messages are lost since the beginning. This is often impractical due to legal regulations or storage limitations.
Topic Existence: The topic for exposing data must have existed since the inception of the order service. However, as systems evolve and functionalities are added iteratively, this is rarely the case in reality.

These constraints necessitate alternative solutions to ensure reliable data restoration.

Republishing All Events

One effective solution is to provide a mechanism to republish all events from scratch. This involves reading all data from the source database, transforming it into events, and publishing those events to the topic. Here’s a simple way to implement this:

Providing an Admin Endpoint

Expose an admin endpoint that can trigger the republishing process. For example:

POST /admin/expose-order-events

This endpoint should have limited access, such as being protected behind authentication and authorization layers, to prevent misuse. When called, it starts reading data from the source database, transforms the data into events, and publishes them to Kafka.

Leveraging Idempotent Events

If the events are designed to be idempotent, you can safely call this endpoint multiple times without causing data duplication or side effects. Idempotency ensures that the same event can be processed multiple times with consistent results, making this approach both reliable and robust.

By combining these techniques, you can effectively manage data restoration scenarios while maintaining data consistency and reducing the complexity of your architecture.

Summary

This article focused on key recommendations for implementing data streams using Kafka, intentionally avoiding the inclusion of additional tools to keep the architecture as straightforward as possible.

By adhering to a few fundamental principles, you can significantly reduce future effort, minimize unnecessary load on consumers, and prevent potential data inconsistencies in your data warehouse.

To achieve these goals, I recommend focusing on:

Ensuring idempotency: Design events with unique identifiers to avoid data duplication and maintain consistency.
Streaming full data objects, not just events: Provide the complete state of domain entities to simplify downstream processing.
Enabling easy data restoration: Ensure data can be easily restored from the source database to handle failures or rebuild the warehouse efficiently.

By following these principles, you can create a robust and maintainable data pipeline that scales well and integrates seamlessly with your reporting systems.

Let me know about your thoughts and opinions!