Welcome! 👋
In this series, we're going to build a real-time backend system from scratch using a modern microservices architecture.
Our goal isn't just to build another project that works. Instead, we'll understand why we make certain architectural decisions, when to use specific technologies, and how individual services communicate to form a scalable distributed system.
We'll start by connecting all the pieces together. Along the way, we'll answer questions like:
- What does it actually take to build a real-time system?
- Which technologies are best suited for real-time communication, and why?
- How do we design services that are independent, loosely coupled, and easy to scale?
- How can we run multiple services locally without making development painful?
- How do we containerize our applications with Docker so that moving from local development to deployment becomes seamless?
- How do services communicate with each other in a distributed environment?
- Where does an API Gateway fit into the architecture?
- How do we secure our services?
- What happens when things go wrong? How do we handle failures, retries, and unexpected errors gracefully?
- What practices make a system reliable, maintainable, and production-ready?
Rather than jumping straight into coding, we'll first understand the architecture and the reasoning behind every component we introduce. Once we have a clear picture of the system, we'll implement each service step by step, gradually assembling a complete real-time backend.
By the end of this series, you won't just have a working project—you'll understand how modern backend systems are designed, how independent services collaborate, and how to build applications that are scalable, resilient, and ready for production.
So, let's get started and build something amazing together. 🚀
About Project
This project was built to understand how independent microservices run, communicate, and collaborate within a distributed system.
It is intended for intermediate to advanced developers who are already familiar with the fundamentals of backend development and want to explore concepts beyond the basics.
The project demonstrates a production-inspired architecture using Docker for containerization, Maven multi-module project structure for code organization, and service-to-service communication between independent services. It is designed to provide hands-on experience with building, running, and managing multiple services in a scalable backend environment.
Project Anatomy :
Project Folder Structure
What are covered :
⚡ Event-Driven Architecture
📡 Server-Sent Events (SSE)
📨 Kafka
🐳 Docker
🧩 Multi-Module Architecture
🌐 API Gateway
yes this episode will give you very brief about each one, but trust me going this is approach like ladder principle instead of learning each topic in depth just learn that much that just let you move in and later we make it more robust and secure.
Before we start building, let's first understand what an event-driven system is and why it has become a fundamental architectural pattern for modern distributed applications. We'll explore how services communicate through events, why this approach improves scalability and flexibility, and compare the different technologies available for implementing event-driven architectures.
What Is an Event-Driven System?
Before we start building, let's first understand what an event-driven system is and why it's widely used in modern distributed applications.
In an event-driven architecture, services communicate by producing and consuming events instead of calling each other directly. When something important happens—such as a user placing an order or a payment being completed—an event is published. Any interested service can then react to that event independently.
This approach makes systems more loosely coupled, scalable, resilient, and easier to extend as new services are added.
What Are Our Options?
There are several ways to implement event-driven communication, each suited to different use cases:
Apache Kafka – A distributed event streaming platform designed for high throughput, durability, and scalability.
RabbitMQ – A message broker that excels at reliable message delivery and task distribution.
Redis Pub/Sub – A lightweight publish-subscribe mechanism for simple, low-latency messaging.
Cloud Messaging Services – Managed solutions such as AWS SNS/SQS, Google Pub/Sub, and Azure Service Bus.
Throughout this series, we'll use Apache Kafka to build our event-driven system and understand why it's one of the most popular choices for scalable microservices.
OK OK.. Since now we have to deal with Kafka, let's understand about it from the surface level to gets started and later we will use it at it's full potential..maybe😅.
I have provide some diagrams and notes for each sub-topics you can use that as well in the diagrams and notes folder in Github Repo.
GitHub Link
Let's understand some terminologies associated with Kafka to get into it. You can also refer this diagram to get some high-level understanding first.
Terms associated with Kafka
Producer : A Producer publishes events to Kafka.A producer never knows who will consume the event.
Consumer : A Consumer reads events from Kafka and reacts to them.One event can trigger multiple independent services.
Event : An event represents something that has happened in your system.examples : UserRegistered,OrderPlaced etc.
Topic : A Topic is a logical category where related events are stored.Examples we place all order related events in order-service topic, shipment related events in the shipment-service topic and so on.
Message (Record) : A Message (also called a Record) is one event stored inside a topic.Each message contains key, value, timestamp and offset.
Partition : A topic is divided into partitions. The reason kafka does to achieve parallel processing, higher throughput and scalability.
Ordering is guaranteed only within a partition.
Offset : Every message inside a partition has an Offset.Offsets uniquely identify messages within a partition.Consumers use offsets to know where to resume reading.
Broker : A Broker is a Kafka server.A Kafka cluster consists of one or more brokers.
Cluster : A Cluster is a collection of brokers working together.Adding brokers improves scalability and fault tolerance.Every broker in a Kafka cluster knows about all the other brokers in that same cluster, they do this by maintaining up-to-date cluster metadata, which contains the unique IDs, hostnames, and ports of every single broker, as well as which broker is currently the leader for any given topic partition.
Consumer Group : Multiple consumers can belong to the same Consumer Group.Kafka distributes partitions among consumers so each message is processed only once per group.
Replication : Kafka copies partitions across brokers.
Key : The Key determines which partition stores a message.All events with the same key go to the same partition, preserving order.
Value : The Value is the actual event payload.
Serialization : Kafka transmits bytes.Objects must be converted into bytes before sending.
*Event Streaming *: Kafka continuously streams events.Unlike traditional messaging, Kafka retains events, allowing consumers to replay or process them later.
I think🤔 this must be good enough to sail our ship properly.Let's talk about the framework we are going to use and that willllllll.... beeee... our enterprise king Spring Boot👑.
Let's see how we are going to design since we are planned to create multiple modular maven project.So for this we have parent pom file that contains all the things and create docker docker compose file for running all the services. you can again check my project link and look for the docker compose services Docker file for reference. Link
Let's create events for that will be sending to Kafka topics since you Kafka accepts bytes so we have to serialize and consumer have to deseialize it.
@Data
@NoArgsConstructor
@AllArgsConstructor
@Builder
public class ShipmentStartedEvent {
private String orderId;
private String shipmentId;
private String customerId;
private String status;
private BigDecimal totalAmount;
private String shippingStreet;
private String shippingCity;
private String shippingState;
private String shippingZip;
private long timestamp;
}
@Data
@NoArgsConstructor
@AllArgsConstructor
@Builder
public class DeliveryStartedEvent {
private String deliveryId;
private String orderId;
private String shipmentId;
private String customerId;
private String status;
private BigDecimal totalAmount;
private String shippingStreet;
private String shippingCity;
private String shippingState;
private String shippingZip;
private long timestamp;
}
@Data
@NoArgsConstructor
@AllArgsConstructor
@Builder
public class OrderCreatedEvent {
private String orderId;
private String customerId;
private BigDecimal totalAmount;
private String shippingStreet;
private String shippingCity;
private String shippingState;
private String shippingZip;
private long timestamp;
}
We setup topics and other config related stuff in the application.properties or application.yaml.
For sending events to Kafka topics with data we can create some Beans in Spring to create KafkaTemplate which we use to send events first or act as event producer.
@Configuration
public class KafkaProducerConfig {
@Value("${spring.kafka.bootstrap-servers:localhost:9092}")
private String bootstrapServers;
@Bean
public ProducerFactory<String, DeliveryStartedEvent> deliveryStartedEventProducerFactory() {
Map<String, Object> config = new HashMap<>();
config.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
config.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
config.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
config.put(ProducerConfig.ACKS_CONFIG, "all");
config.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
return new DefaultKafkaProducerFactory<>(config);
}
@Bean
public KafkaTemplate<String, DeliveryStartedEvent> deliveryKafkaTemplate() {
return new KafkaTemplate<>(deliveryStartedEventProducerFactory());
}
}
Now we can rely on the Spring framework that it will add this as dependancy so that we can use in our services.
private final KafkaTemplate<String, DeliveryStartedEvent> kafkaTemplate;
public DeliveryService(DeliveryRepository deliveryRepository,
KafkaTemplate<String,
DeliveryStartedEvent> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
kafkaTemplate.send(deliveryEventsTopic, deliveryEvent.getDeliveryId(), deliveryEvent);
Now similarly we can create Beans for the Kafka Consumers to listen whenever Kafka topic receives event.
@Configuration
public class KafkaConsumerConfig {
@Value("${spring.kafka.bootstrap-servers:localhost:9092}")
private String bootstrapServers;
@Value("${spring.kafka.consumer.group-id:delivery-service-group}")
private String groupId;
@Bean
public ConsumerFactory<String, Object> shipmentStartedEventConsumerFactory() {
Map<String, Object> config = new HashMap<>();
config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
config.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
config.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);
config.put(JsonDeserializer.TRUSTED_PACKAGES, "com.tracker.common_events");
config.put(JsonDeserializer.TYPE_MAPPINGS,
"com.tracker.common_events.ShipmentStartedEvent:com.tracker.common_events.ShipmentStartedEvent");
config.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
config.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
return new DefaultKafkaConsumerFactory<>(config);
}
@Bean
public ConcurrentKafkaListenerContainerFactory<String, Object> shipmentStartedEventListenerFactory() {
ConcurrentKafkaListenerContainerFactory<String, Object> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(shipmentStartedEventConsumerFactory());
factory.setConcurrency(2);
factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL);
ExponentialBackOff backOff = new ExponentialBackOff(1000L, 2.0);
backOff.setMaxElapsedTime(30000L);
factory.setCommonErrorHandler(new DefaultErrorHandler(backOff));
return factory;
}
}
factory.setConsumerFactory(orderCreatedEventConsumerFactory());
Connecting to the Broker -> It injects a configuration blueprint (the ConsumerFactory) into your listener container factory.
factory.setConcurrency(3);
It tells Spring to spin up exactly 3 background consumer threads inside this single application instance.
factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL);
It completely disables Kafka's default "auto-commit" behavior, forcing the application to take manual control over when a message is marked as "read."
Why this is critical for Production: By default (enable.auto.commit = true), Kafka automatically commits messages periodically in the background, even if your Java code throws an error or crashes halfway through processing. This can result in permanently lost events.
With AckMode.MANUAL, Spring Boot stops auto-committing. It waits until your listener method finishes executing without throwing any exceptions. Once the method completes successfully, the framework automatically signals Kafka to commit that specific offset. If your code fails, no commit happens, protecting your data.
ExponentialBackOff backOff = new ExponentialBackOff(1000L, 2.0);
backOff.setMaxElapsedTime(30000L);
This instantiates an algorithmic delay calculator used when handling unexpected application failures.
How the math works: * 1000L: The initial interval. If your code fails, it will wait 1 second (1000ms) before trying again.
2.0: The multiplier. If the first retry fails, it multiplies the previous wait time by 2.0.
setMaxElapsedTime(30000L): The global time limit. It states that the loop can continue backing off and retrying, but the cumulative time spent retrying this single failing message cannot exceed 30 seconds.
factory.setCommonErrorHandler(new DefaultErrorHandler(backOff));
What it does: It registers a global error handler that wraps around your listener methods, utilizing the exponential back-off rules defined above.
Under the Hood: If your consumer throws an exception (like a database timeout or a network failure), the DefaultErrorHandler catches it before it can crash the background thread.
It pauses the consumer loop, rolls back the uncommitted manual offset, waits for the calculated back-off duration, and then feeds the exact same message back into your listener method. If the message still fails after the 30-second maximum elapsed time limit is reached, the DefaultErrorHandler logs a critical error and moves on to the next message, preventing a single "poison pill" from stopping your entire pipeline forever.
Now In order to listen we can make use of spring-boot-starter-kafka the KafkaListener annotation to listen to topics.
@KafkaListener(
topics = "${app.kafka.topics.order-events:order-events}",
containerFactory = "orderCreatedEventListenerFactory",
groupId = "${spring.kafka.consumer.group-id:shipment-service-group}"
)
public void onOrderCreated(
@Payload OrderCreatedEvent event,
@Header(KafkaHeaders.RECEIVED_KEY) String key,
@Header(KafkaHeaders.RECEIVED_PARTITION) int partition,
@Header(KafkaHeaders.OFFSET) long offset,
Acknowledgment ack) {
log.info("Received OrderCreatedEvent: key={}, partition={}, offset={}, orderId={}, customerId={}",
key, partition, offset, event.getOrderId(), event.getCustomerId());
try {
shipmentService.createShipment(event);
ack.acknowledge();
log.debug("Acknowledged offset={} for orderId={}", offset, event.getOrderId());
} catch (Exception e) {
log.error("Failed to process OrderCreatedEvent for orderId={}: {}", event.getOrderId(), e.getMessage(), e);
throw e;
}
}
You must have have Headers and Acknowledge in the function parameters
Acknowledge is used for Prevent Data Loss : You ensure a message is only committed after it is safely saved to your database.Error Handling: If your system crashes mid-process, the unacknowledged message will be re-delivered upon restart.
Header values are generated automatically by the Apache Kafka broker and the Spring Kafka framework during the message lifecycle.
Now at this moment we know how to produce events to Kafka topics and how Consumer can get those.
Right now, each of our microservices is exposed through its own URL. As the number of services grows, do you think it's a good idea for our frontend or clients to know the address of every single service?
So far, each microservice in our system is exposed through its own URL. While this is fine for development, it doesn't scale well. Clients need to know the address of every service, making the system tightly coupled and harder to maintain.
To solve this, we'll introduce an API Gateway. Instead of communicating with individual services, clients will send all requests to a single entry point. The gateway will handle request routing while also centralizing concerns like authentication, rate limiting, logging, and load balancing.
Let's integrate an API Gateway and simplify our architecture.
we are going to use another Spring related dependancy in order to accomplish this :
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-gateway-server-webflux</artifactId>
</dependency>
Now we can use the RouteLocator Bean to resolve all microservices and now we access all of them using single address.
@Bean
public RouteLocator customRouteLocator(RouteLocatorBuilder builder) {
return builder.routes()
.route("events-stream-java", r -> r
.path("/api/v1/events/stream", "/api/v1/events/stream/**")
.filters(f -> f.setResponseHeader("Cache-Control", "no-cache"))
.uri("http://notification-service:8080"))
.route("customer-service", r -> r
.path("/api/v1/users/**", "/api/v1/orders/**")
.uri("http://customer-service:8080"))
.route("shipment-service", r -> r
.path("/api/v1/shipments/**")
.uri("http://shipment-service:8080"))
.route("delivery-service", r -> r
.path("/api/v1/deliveries/**")
.uri("http://delivery-service:8080"))
.build();
}
This diagram can be the overall overview of what we have done so far with the help of the docker compose we wrap everything.
You can find the docker-compose file in the project folder, here
Now, I think we have covered everything to get started and yeah this not perfect but its good enough to think or it gave lot of space to make us think how we can improve this further.
A Note on Using AI
Throughout this series, I encourage you to use AI as a learning and productivity tool—but not as a replacement for engineering judgment.
AI can help you write code faster, explain concepts, and even suggest implementations. However, designing scalable systems still requires critical thinking. Decisions around architecture, scalability, security, reliability, maintainability, and trade-offs cannot simply be delegated to an AI.
It's also important not to accept every piece of generated code without understanding it. As engineers, we're responsible for keeping our codebase clean, organized, and maintainable. That means following SOLID principles, applying appropriate design patterns, avoiding code duplication, reducing tight coupling, and writing code that's easy to understand and evolve.
Think of AI as a co-worker, not the architect. It performs best when given clear context and well-defined problems, but the responsibility for making sound engineering decisions always remains with us.
So, as we build this project together, let's use AI to accelerate our workflow while continuing to think critically and make deliberate design decisions.
Now that we've put together our event-driven architecture, our services can automatically react to events and publish them to any interested consumers.
To see this in action, let's build a simple Server-Sent Events (SSE) endpoint. This will allow clients to receive real-time updates whenever new events are published.
If you're new to Server-Sent Events (SSE), here's a quick overview before we dive into the implementation.
Server-Sent Events (SSE) is a web technology that allows a server to push continuous, real-time updates to a client over a single, long-lived HTTP connection.
Since our current architecture (which isn't perfect yet 😄—but we'll improve it step by step) already includes a Notification Service, it's the ideal place to expose an SSE endpoint. Any client that wants to receive real-time notifications can simply subscribe to this endpoint, and the server will push updates whenever new events occur.
@Service
public class SseEmitterService {
private static final Logger log = LoggerFactory.getLogger(SseEmitterService.class);
private final CopyOnWriteArrayList<SseEmitter> emitters = new CopyOnWriteArrayList<>();
public SseEmitter createEmitter() {
SseEmitter emitter = new SseEmitter(Long.MAX_VALUE);
emitters.add(emitter);
emitter.onCompletion(() -> emitters.remove(emitter));
emitter.onTimeout(() -> emitters.remove(emitter));
emitter.onError(e -> emitters.remove(emitter));
log.debug("SSE client connected, total clients={}", emitters.size());
return emitter;
}
public void broadcast(String eventType, Object data) {
for (SseEmitter emitter : emitters) {
try {
emitter.send(SseEmitter.event()
.name(eventType)
.data(data));
} catch (IOException e) {
emitters.remove(emitter);
log.debug("SSE client disconnected, total clients={}", emitters.size());
}
}
}
}
We can now call this broadcast method on the every KafkaListener and can send events like this :
sseEmitterService.broadcast("DeliveryStatusChanged", event);
sseEmitterService.broadcast("OrderCreated", event);
sseEmitterService.broadcast("ShipmentStatusChanged", event);
As always, you can explore the implementation in the project repository to see how everything fits together in a real-world example.link
With this, we've made a lot of progress—even if it looks a little unstructured right now. 😄 But that's how every great system begins: a collection of ideas that gradually evolve into something well-designed through consistency, iteration, and continuous improvement.
In the next episode, we'll refine our architecture, improve the design, and take another step toward building a production-ready system.
See you in the next episode! 🚀
Top comments (0)