DEV Community: Jonathan

Java Collections: From Lists to Maps with Time Complexity in Mind

Jonathan — Thu, 07 Aug 2025 09:38:37 +0000

Java’s Collection Framework is powerful, but knowing when to use ArrayList, LinkedList, HashMap, Queue, or Deque can make a big difference in the performance and readability of your code. In this article, I’ll guide you through real world scenarios, time complexities, and examples to help you choose the right data structure depending on your needs.

1. Large lists where random access is frequent. Fast index access? --> ArrayList

Time Complexity:

Access by index: O(1)
Insert/remove at end: O(1) - (except when capacity is exceeded)
Insert/remove in middle or beginning: O(n)

List<String> list = new ArrayList<>();
list.add("Apple");
list.add("Banana");
System.out.println(list.get(1)); // Fast because we know the position of what we are looking for: "Banana"

Don't use ArrayList for frequent insertions or removals at the beginning or middle, it needs to move subsequent elements and reindex them, which can degrade performance.

2. Frequent Insert/Remove at Beginning or End or modifying element during Iteration? --> LinkedList

If your operations involve adding or removing elements at the beginning or end, LinkedList is more efficient.

Time Complexity:

Add/remove at beginning/end: O(1)
Access by index: O(n)

List<String> list = new LinkedList<>();
list.add("first");
list.add(0, "Zero");       // Fast
list.addLast("last");      // Fast
list.removeFirst();        // Fast

Don’t use LinkedList for index-based access, it’s slow compared to ArrayList.

3. Key-based lookup? --> HashMap

Time Complexity:

Get/put by key: O(1)
Iteration over keys/values: O(n)

Map<String, Integer> map = new HashMap<>();
map.put("Alice", 30);
map.put("Bob", 25);
System.out.println(map.get("Bob")); // Fast lookup

3.1 HashMap vs TreeMap vs LinkedHashMap

// HashMap: Fastest, no insertion ordering
Map<String, String> hashMap = new HashMap<>();

// TreeMap: Sorted by keys, slower but natural ordering
Map<String, Integer> treeMap = new TreeMap<>();

        // Adding key-value pairs in random order
        treeMap.put("zebra", 1);
        treeMap.put("apple", 2);
        treeMap.put("monkey", 3);
        treeMap.put("banana", 4);

        System.out.println("TreeMap: " + treeMap);
        // Output: {apple=2, banana=4, monkey=3, zebra=1} (always sorted by key!)

// LinkedHashMap: Maintains insertion order
Map<String, Integer> linkedHashMap = new LinkedHashMap<>();

        linkedHashMap.put("first", 1);
        linkedHashMap.put("second", 2);
        linkedHashMap.put("third", 3);
        linkedHashMap.put("second", 20); // Updates existing key, maintains position
        linkedHashMap.put("fourth", 4);

        System.out.println("LinkedHashMap: " + linkedHashMap);
        // Output: {first=1, second=20, third=3, fourth=4} (insertion order preserved!)

4. Need Unique Elements Only? --> Set

When you need to ensure no duplicates in your collection, use a Set.
Time Complexity (HashSet):

Add/remove/contains: O(1)

Set<String> uniqueEmails = new HashSet<>();
uniqueEmails.add("user@example.com");
uniqueEmails.add("admin@example.com");
uniqueEmails.add("user@example.com"); // Duplicate - won't be added

Variants:

HashSet: Fastest, no ordering
TreeSet: Sorted unique elements
LinkedHashSet: Unique + insertion order preserved

5. Process Items In The Order They Arrived (FIFO) --> Queue

Interface: Queue
Implementations: LinkedList, ArrayDeque

Time Complexity:

Add (offer): O(1)
Remove (poll): O(1)

Queue<String> queue = new LinkedList<>();
queue.offer("Task1");
queue.offer("Task2");
System.out.println(queue.poll()); // "Task1"

6. Double-Ended Queue --> Deque

Interface: Deque
Implementations: ArrayDeque, LinkedList

Time Complexity:

Add/remove from both ends: O(1)

Deque<String> deque = new ArrayDeque<>();
deque.addFirst("First");
deque.addLast("Last");
System.out.println(deque.removeLast()); // "Last"

Virtual Threads in Java: A Lightweight Concurrency Revolution

Jonathan — Fri, 11 Jul 2025 10:31:00 +0000

With the introduction of Virtual Threads in Java 21 as a preview feature (and stable in later versions), the way we think about concurrency in Java is changing dramatically.

What Are Virtual Threads?

In Java, when you want your program to do multiple things at the same time like handling lots of user requests, you typically use threads. But traditional threads (called platform threads) are expensive. Each one uses up a chunk of memory and is tied to a real thread in your computer's operating system (OS).

If you try to create thousands of threads (like in a busy server), the system starts to struggle. That's why developers had to get clever, using tools like asynchronous code, callbacks, or reactive libraries like Reactor or WebFlux. These solutions work, but they’re harder to read, write, and debug.

Virtual threads look and act like regular Java threads, but under the hood, they’re much lighter. Instead of being tied directly to OS threads, they’re managed by the JVM (Java Virtual Machine) itself. This means you can create millions of them without crashing your system.

This is what a basic user service might look like using the classic, readable style:

@GetMapping("/user/{id}")
public ResponseEntity<User> getUser(@PathVariable String id) {
    User user = userService.findById(id); // blocking call
    return ResponseEntity.ok(user);
}

This is simple. It reads top-down, is easy to understand, and you can use try/catch or other flow control as expected.
Below, we have the same code using Spring WebFlux and Project Reactor:

// Inside the userController class
@GetMapping("/user/{id}")
public Mono<ResponseEntity<User>> getUser(@PathVariable String id) {
    return userService.findById(id)
        .map(user -> ResponseEntity.ok(user))
        .onErrorResume(e -> Mono.just(ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).build()));
}

// Inside the userService class
public Mono<User> findById(String id) {
    return webClient.get()
        .uri("/users/" + id)
        .retrieve()
        .bodyToMono(User.class);
}

The previous example becomes harder because you’re no longer returning the result directly, you’re returning a Mono (a future-like wrapper). You need to chain callbacks (map, flatMap, onErrorResume, etc.) to handle logic and the Try/catch doesn’t work the same way, so error handling becomes more abstract.

What Problem Do Virtual Threads Solve?

Traditional Java threads can’t scale well when you need thousands of them.

Let’s say you’re building a web server that handles one request per thread. With just a few hundred users, you're fine. But as traffic grows, the number of threads balloons, memory usage spikes, and performance drops. It’s just not sustainable.

So developers switched to non-blocking, reactive code. This approach uses fewer threads and handles more work, but the code becomes harder to follow as we saw in the previous example.

When Should You Use Virtual Threads?

Virtual Threads shine when:

You need to handle a high number of concurrent tasks (one thread per request in a web server).
You want clean, readable synchronous style code without reactive complexity.
You're dealing with tasks that spend most of their time waiting on I/O (like database calls, API requests, or file operations).

Do Virtual Threads Replace WebFlux?

Probably not entirely, but for many use cases, yes.

Reactive programming (like Spring WebFlux or Reactor) became popular because traditional threads couldn’t handle high loads. But reactive code is harder to write and understand. With virtual threads, you can write simple, blocking code and still scale to thousands of requests.

So if your app is mostly I/O-bound—like calling APIs, querying databases, or reading files, virtual threads can let you drop the complexity of reactive programming.

Still, if you need fine grained control, backpressure, or advanced streaming, reactive might still make sense.

Do Virtual Threads Eliminate Blocking?

No — virtual threads do not magically make blocking go away. They just make it cheaper and more scalable.

For example, if a virtual thread calls a blocking database operation or waits for a slow file read, that thread is still blocked. The difference is that the JVM can easily pause and resume virtual threads, so it can handle many more of them without exhausting OS resources. But the actual blocking, like waiting for a database response, still happens. To avoid blocking, you can use for example CompletableFuture:

Run tasks asynchronously without blocking the main thread.
Chain operations (thenApply, thenCompose, etc.)

You're not blocking on the result — you react when it's ready:

CompletableFuture<String> future = CompletableFuture.supplyAsync(() -> {
    // Simulate long-running task
    return callApiOrDb();
});

future.thenAccept(result -> {
    System.out.println("Got result: " + result);
});

Conclusion

Virtual threads are lightweight threads managed by the JVM, not the OS.
They let you write blocking, synchronous code, without the overhead of traditional threads.
Great for high concurrency, I/O-heavy applications.
Makes complex reactive code unnecessary in many cases.
A big step forward in making Java simpler and more powerful.

From JVM to Native Compilation with Spring Boot: What It Means and Why It Matters

Jonathan — Wed, 04 Jun 2025 23:09:46 +0000

Traditionally, Java developers have followed the same standard flow: write Java code, compile it to bytecode, and run it on the Java Virtual Machine (JVM). While this has worked well, it comes with trade-offs: slow startup, high memory usage, and a dependency on the JVM at runtime.

Now, thanks to tools like Spring Native and GraalVM, we can compile Spring Boot applications into native executables, binaries that start in milliseconds and consume a fraction of the memory. This lets Spring Boot apps run as fast and light as Quarkus, without having to change to a different framework.

In this article, I’ll explain what native compilation means, how it works, and why it makes a big difference for modern applications like microservices and serverless functions.

Traditional Java Compilation (JVM-Based)

Here’s how a traditional Java app works:

You write Java code.
The javac compiler turns it into bytecode (.class files).
You package everything into a .jar.
You run it with the JVM: java -jar myapp.jar

At runtime, the JVM translates bytecode into machine code by either interpreting it or compiling it on the fly using just-in-time (JIT) compilation.

Limitations

Slow startup (seconds)
Higher memory usage
Requires a JVM installed on the host machine

Native Compilation: What’s New?

With GraalVM Native Image, you can take your Java code and compile it into a native binary (like a .exe or Linux ELF file) that:

Runs without the JVM
Starts in milliseconds
Uses significantly less memory

This is called ahead-of-time (AOT) compilation, the code is compiled to native machine instructions before it's run, not during runtime like with the JVM.

Spring Boot + Native Support

Starting with Spring Boot 3.x, Spring officially supports AOT + native image compilation through:

Spring AOT engine
Spring Native (based on GraalVM)

This means you can take your Spring Boot app and compile it into a self contained binary.

JVM vs Native: Key Differences

Feature	JVM-based Spring Boot	Native Spring Boot
Requires JVM	Yes	No
Startup time	Slow (seconds)	Fast (milliseconds)
Memory usage	High (100MB+)	Low (20–40MB)
Packaging	`.jar` file	Native executable (`.exe`, ELF)
Ideal for	Monoliths, legacy apps	Microservices, serverless, CLI

When Should You Use Native Compilation?

Microservices
Serverless functions (e.g., AWS Lambda, Cloud Run)
Applications with short lifecycle and high scalability demands

Not ideal for

Highly dynamic apps using a lot of runtime reflection (though Spring AOT is improving fast)

Conclusion

Native compilation brings a new level of performance and efficiency to Spring Boot applications. For the first time, you can write your app in Java and deploy it:

With zero JVM dependency
With instant startup
Using less memory

Java is no longer just "write once, run anywhere", now it’s also fast, lightweight, and cloud-ready.

How to Handle Millions of Rows in SQL Without Killing Performance

Jonathan — Tue, 03 Jun 2025 15:39:09 +0000

When working with massive datasets, an inefficient SQL query can significantly degrade database performance. One common mistake? Using a giant IN (...) clause to filter a large set of values.

This post shows you why JOIN with a temporary table is a smarter, faster alternative and how to apply it in a real use case.

The Problem: IN with Too Many Values

Imagine you have a huge table transactions with more than 5 billion rows and you're trying to retrieve all the transactions for 10,000 specific users.

The first and easy way that comes to our mind would be doing something like this:

SELECT *
FROM transactions
WHERE user_id IN (101, 102, 103, ..., 10000);

It may look simple, but it can lead to serious performance issues.

The DB might scan the entire table
The IN list might not be optimized well
Performance degrades fast as the list grows

The Solution: JOIN with a Temp Table

Instead of passing a huge list, use a temporary table:

CREATE TEMP TABLE temp_user_ids (user_id BIGINT);

Remember that Temporary tables exist only for the duration of the session or transaction, depending on the database.
Now, insert your 10,000 user IDs:

INSERT INTO temp_user_ids (user_id) VALUES
(101), (102), (103), ..., (10000);

Now query:

SELECT t.*
FROM transactions t
JOIN temp_user_ids u ON t.user_id = u.user_id;

Why is this better?
The database can use an index on transactions.user_id

It's easier to parallelize.
It avoids large memory buffers and complex query plans.
It's faster and more predictable.

Conclusion

If your SQL query is slow and you're using a massive IN (...) list, switch to a JOIN with a temp table. It’s faster, cleaner, and lets your database do what it does best: optimize joins using indexes.

Using a temporary table instead of a long IN (...) list makes your SQL query faster and easier for the database to handle. When you use IN with thousands of values, the database may get slow because it has to check each value one by one and might not use indexes properly. But with a temporary table, you can store all the values in one place, add an index, and let the database use a faster join method to find matches. This makes the query run quicker, use less memory, and work better with large amounts of data.

Apache Kafka: Topics, Partitions, Brokers

Jonathan — Tue, 22 Apr 2025 22:46:23 +0000

Introduction

In this post, we're going to take a deeper dive into Apache Kafka. If you'd like an introduction to Event-Driven Architectures, feel free to check out my previous article.

Kafka is a powerful stream processing tool. It's like a super efficient digital post office for real time data. Instead of sending single letters, it handles streams of messages (like a continuous flow of packages) between different applications or systems. It's designed to be incredibly fast, reliable, and can manage huge amounts of data, making it perfect for things like tracking user activity on a website, processing financial transactions, or collecting data from sensors.

Concept	Description
Producer	Sends data or publishes messages to Kafka.
Consumer	Reads or subscribes to data from Kafka to process it. Multiple consumers can subscribe to the same data.
Topic	a specific channel or a category where different kinds of data are organized(like a queue). It's like having different mailboxes in a post office, each labeled for a particular type of mail. Producers send their messages to a specific topic, and consumers who are interested in that type of data will subscribe to that particular topic to receive the messages.
Partition	A physical division of a Topic. Messages within a Topic are split into Partitions for scalability and parallel processing. This division allows Kafka to handle more data and process it faster because different parts of the data can be processed at the same time.
Broker	A Kafka server responsible for storing topic partitions and handling read and write requests from producers and consumers.
Cluster	A group of Kafka brokers working together to provide fault tolerance and high availability.
Zookeeper	A centralized service used for coordinating the Kafka cluster (configuration, synchronization, and group services; optional in newer versions). Kafka newer versions (2.8+) started moving away from Zookeeper, they introduced KRaft Mode, meaning Kafka can now manage itself without needing Zookeeper.

Example of topic and partition to understand it better.

A topic is like a book (it’s about a single subject — "Orders").
Partitions are chapters of the same book, breaking the content into smaller pieces so people can read/write different parts at the same time.
Multiple topics would be different books ("Orders", "Payments", "Shipments").

Why Partitions Matter?

Scalability: More partitions = more parallelism = faster reading/writing.
Ordering: Kafka guarantees message order within a single partition. If you use a key (like a user ID), all events with the same key are sent to the same partition, ensuring they are processed in order. While multiple user IDs can share the same partition, Kafka still maintains the correct order for each individual key. However, Kafka does not guarantee ordering across different partitions.
Persistance: Messages are kept for a configured time, even after being read. What happens when the events have been consumed in Kafka? Are they removed from the topic? No, Kafka’s Model: Log-Based, Not Queue-Based. Events remain in the topic for a fixed period,even after being consumed. That means:

Events are written to a topic (which is like a log file).
Consumers read from the log and track their own offset. The offset is a pointer to the last message they successfully read.
Events stay in the topic for a fixed duration (based on the retention time), even after they’ve been consumed.

Fault tolerance: Kafka replicates each partition, copies are made and stored on other brokers (servers). How? Each partition has a leader broker that handles writes/reads. Another broker acts as a follower (replica) to back up the partition.

Kafka Consumer Groups

A consumer group is a set of consumers that collaboratively consume messages from Kafka topics. Typically, these are multiple instances of the same application running in parallel, like three instances of a microservice deployed for load balancing. Inside a consumer group, each partition is assigned to only one consumer, so every message from a partition is processed by just one instance. This helps spread the workload evenly across the consumers.

How Does It Work?
Let’s say you have:

A Kafka topic with 3 partitions (P0, P1, P2).
A consumer group with 2 consumers/instances(EC2): C1 and C2.

Kafka will assign partitions to consumers in the group:

C1 might read from P0 and P1.
C2 reads from P2.

Each partition is read by only one consumer in the group at a time.
But the same topic can be consumed by multiple different consumer groups independently.

Why This Matters / What’s the Goal?

The goal is to:

Distribute the load between multiple service instances.
Scale horizontally — more partitions = more consumers = more throughput.
Make your system fault-tolerant, if one EC2 instance fails, Kafka can reassign its partition to the other instance.

Possible Scenario: One Instance Fails

Let’s say:

Your Kafka topic has 2 partitions.
You have 2 EC2 instances running your microservice (both are consumers in the same group).
Instance 1 (Consumer A) → handles Partition 0
Instance 2 (Consumer B) → handles Partition 1

Suddenly, Instance 1 crashes.

What Kafka Does Automatically
1.Detects the Failure

Kafka uses heartbeats to check if consumers are alive.
If a consumer stops sending heartbeats (default timeout is ~10 seconds), Kafka marks it as dead.

2.Triggers a Rebalance

Kafka reassigns the partitions that the dead consumer was handling.
Now, Instance 2 (Consumer B) gets both Partition 0 and Partition 1.

3.Continues from Last Offset

Kafka tracks the offsets for each partition in the consumer group.
Consumer B resumes from the last committed offset of Partition 0.
No messages are lost or reprocessed unless your app is configured to do so.

What Happens When Instance 1 Comes Back?

It rejoins the consumer group.
Kafka rebalances again.
The workload (partitions) may be redistributed:
Instance 1 may take back one of the partitions.

Kafka Retention Time `cleanup.policy=delete`

Means how long Kafka keeps a message in a topic, regardless of whether it's been read or not. The purpose is to allow consumers to re-read messages for a certain period of time (for recovery, auditing, or analytics).

How does it work?
Set via topic configuration:
retention.ms (in milliseconds): retention.ms = 604800000 → 7 days

Kafka will delete old messages (per partition) after they reach this age and the disk space is available for new data.

Note: You can also configure by size, using retention.bytes, but time based is more common.

Kafka Log Compaction `cleanup.policy=compact`

Compaction is a process that ensures only the latest value for each key is retained in a topic. Useful for state stores, configurations, or changelogs.

How does it work?
Compaction is enabled by setting cleanup.policy=compact (the default is delete). You can also combine both policies by setting cleanup.policy=compact,delete. In this case, Kafka will compact messages to keep only the latest value per key, while also deleting older segments based on the configured retention time or size.

Kafka will keep the most recent message per key and remove older ones. Take into account that compaction is asynchronous, it runs in the background and is not instant. Kafka compaction is perfect when each message represents an update to a specific key, and you only care about the most recent value.

Imagine you write:

key1 → value1
key2 → value2
key1 → value3

key2 → value2
key1 → value3   (latest value for key1)

Serialization and Deserialization in Kafka

Serialization: converting data (an object, a record, or a message) into a byte stream. Kafka requires all messages to be in a byte format, so producers must serialize data before sending it to Kafka.
Deserialization: reverse process, converting the byte stream back into the original data format.

Kafka is format agnostic, it doesn’t care what format your data is in, as long as it’s a byte array.

The producer serializes the message into bytes.
Kafka stores it.
The consumer deserializes the message into a usable format (JSON, Avro, etc).

Avro + Schema Registry Is the Preferred Choice

Avro + Schema Registry is currently the most widely adopted serialization strategy in large scale systems with many microservices.

Key Points

Compactness:Avro serializes data in a compact binary format, which significantly reduces message size compared to JSON
Schema validation: Avro enforces a schema, which means producers and consumers both agree on the structure of the data. The schema itself isn't embedded in each message, only a small schema ID is sent — reducing payload size and allowing strong data contracts between services
Schema evolution: Avro is designed to support changes over time, add optional fields, remove unused fields, change default values. This is crucial in microservices, where one team might update a service without breaking another.

At small scale, JSON is easier. But at enterprise or production scale, Avro makes data governance, backward compatibility, and performance manageable.

Example

Producer Side (Serialization)

When a producer sends a message, it:

Serializes the object using the Avro schema.
Registers the schema (if new) in the Schema Registry.
Includes the schema ID in the message payload (usually as a small header or prefix).

Consumer Side (Deserialization)

When the consumer receives a message, it:

Reads the schema ID from the message.
Fetches the schema from the Schema Registry using that ID.
Deserializes the message using the schema (to a GenericRecord or specific java class).

Let's see some code

Avro serialization/deserialization using Confluent Schema Registry.
Producer and consumer of a User object.
Automatic generation of Java classes from .avsc.

Avro Schema: src/main/avro/User.avsc

{
  "type": "record",
  "name": "User",
  "namespace": "com.example.kafka.avro",
  "fields": [
    { "name": "id", "type": "string" },
    { "name": "name", "type": "string" },
    { "name": "email", "type": ["null", "string"], "default": null }
  ]
}

Kafka Configuration: application.yml

spring:
  kafka:
    bootstrap-servers: localhost:9092 # where the Kafka broker(s) are running.
    properties:
      schema.registry.url: http://localhost:8081 # where the Confluent Schema Registry is running 
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer # Converts the value (like an Avro User object) into binary Avro format, with metadata that links to its schema stored in Schema Registry
    consumer:
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer # Converts the binary Avro back into a usable Java object, using the schema from Schema Registry.
      properties:
        specific.avro.reader: true # Kafka uses the generated Avro class, which gives you compile-time type safety. If false, you'd get a GenericRecord
    listener:
      missing-topics-fatal: false # it will not fail if the topic doesn’t exist yet — useful during development or if topics are created dynamically

KafkaProducer.java

import com.example.kafka.avro.User;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Service;

@Component
public class KafkaProducer {

    private final KafkaTemplate<String, User> kafkaTemplate;

    public KafkaProducer(KafkaTemplate<String, User> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void sendUser(User user) {
        kafkaTemplate.send("user-topic", user.getId().toString(), user);
        System.out.println("Produced User: " + user);
    }
}

KafkaConsumer.java

import com.example.kafka.avro.User;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Service;

@Component
public class KafkaConsumer {

    @KafkaListener(topics = "user-topic", groupId = "user-group")
    public void consume(User user) {
        System.out.println("Consumed User: " + user);
    }
}

KafkaAvroApplication.java

@SpringBootApplication
public class KafkaAvroApplication implements CommandLineRunner {

    @Autowired
    private KafkaProducer producer;

    public static void main(String[] args) {
        SpringApplication.run(KafkaAvroApplication.class, args);
    }

    @Override
    public void run(String... args) {
        User user = User.builder()
                .setId("u1")
                .setName("Jhoni")
                .setEmail("jhoni@example.com")
                .build();
        producer.sendUser(user);
    }
}

Understanding concepts in Event Driven Architectures (EDA)

Jonathan — Wed, 02 Apr 2025 10:54:14 +0000

Event Driven Architecture has become a very popular choice in recent times for developing highly scalable distributed systems. Is a way of designing software where different parts of the system communicate by sending and reacting to events.

An event is something that happens, like:

A user signs up
A user uploads a photo
A payment is completed

Instead of making direct requests to each other, different parts of the system listen for these events and react when needed(asynchronously).

For example:

The Payment Service is completed and emits an event: "payment done".
The Inventory Service listens for that event and updates the stock for the new product that has been bought.
The Email Service listens and sends a confirmation email.

Let's see a very simple diagram of the components in Event Driven Architecture

Key Components

1. Event Producer/Publisher
The system or service that creates and emits events when something happens.
Example: A User Service sends an event "User Registered" when a new user signs up.

2. Message Broker
A message broker is a middleware that acts as an intermediary between event producers and event consumers. Its primary role is to receive, store, and route events efficiently, ensuring that the events are delivered to the correct consumer services.

In this context, there is a crucial concept inside message brokers called Channels. These channels act as communication routes that orchestrate the flow of events from producers to the right consumers.

In Kafka, these channels are called Topics. It supports multi subscriber models, allowing multiple consumers to listen to the same publisher.
In RabbitMQ, these channels are called Queues. RabbitMQ is typically point-to-point for queues, meaning that only one consumer can process a message from a queue.

Example technologies: Kafka, RabbitMQ, AWS SNS/SQS, Google Pub/Sub.

3. Event Consumer/Subscriber
The service that listens to events and reacts to them.
Example: A Notification Service listens for "User Registered" and sends a welcome email.

4. Event
The actual message that describes what happened. It usually contains structured data (JSON, XML).
Example event message:

{
  "data": {
    "id": "event-123456",
    "type": "user.event.registered",
    "timestamp": "2025-04-01T12:00", 
    "attributes": {
      "id": "usr-98765",
      "email": "user@example.com",
      "surname": "example",
    }
  }
}

Broker Exchanges

In message brokers, an exchange is a routing mechanism that determines how messages are delivered to queues. Different types of exchanges provide different message distribution patterns.

Direct Exchange (1:1 Routing)
Routes messages to queues based on an exact match between a routing key and a queue binding key (RabbitMQ).
How it works?

The producer sends a message with a routing key.
The exchange forwards the message only to queues with a matching binding key.

Example:
A message with routing key = "order.created" goes only to the queue bound with "order.created".

Fanout Exchange (Broadcast)
Sends messages to all bound queues, ignoring the routing key
(RabbitMQ-Fanout Exchange | Kafka-multiple consumers per topic).

How it works?

The producer sends a message.
The exchange copies the message to all queues bound to it.

Example:
A notification event is broadcasted to multiple services (email, SMS, push notifications).

Topic Exchange (Pattern Matching)
Routes messages based on wildcard patterns in routing keys. Supports *, matches one word and #, matches multiple words. (RabbitMQ-Topic Exchange | Kafka-topics with partition keys).
How it works?

Uses . to separate words in routing keys ("order.payment.failed").

Example:
A queue bound to "order.*" receives "order.created" and "order.cancelled", but not "user.created".

Advantages of EDA - Reactive Manifesto

Scalability: decoupled services can scale independently.
Flexibility: easy to add new event consumers without modifying existing services.
Real-time Processing: enables real-time data streaming and quick responses.
Resilience: failures in one service don’t impact the entire system as long as events and subscriptions are managed properly
Loose Coupling: components interact through events, reducing dependencies.

Disadvantages of EDA:

Eventual consistency: different parts of the system might temporarily have outdated information until all events are processed.
Complex Debugging: harder to trace issues across multiple async events.
Increased Latency: some events may take longer to be processed due to async nature.
Event Duplications: requires idempotency to avoid duplicate processing.

Solution when publishing?
For example, kafka has Idempotent Producer to prevent duplicate writes from retries. You can enable it in the configuration setup for your Kafka producer:

properties.put("enable.idempotence", "true");

Event Ordering Issues: some brokers don’t guarantee message order (RabbitMQ).
Learning Curve: requires expertise in event-driven design patterns.

Conclusion

If you are thinking of implementing a distributed, real-time and highly scalable system, you should definitely consider implementing an event-driven architecture to take advantage of all the benefits it offers.

Domain Driven Design(DDD) - Understanding Main Concepts

Jonathan — Tue, 18 Mar 2025 23:43:08 +0000

Domain Driven Design(DDD) is a form of software design that focuses on understanding and reflecting the business in code. Clean Architectures in general work really well with DDD. This happens because of layered separation, where the domain is at the center, meaning the domain layer does not depend on anything external.

Main Concepts

1. Domain
Is the business problem we are solving. A good domain model captures the essential business rules and behaviours while staying independent of infrastructure concerns. That's why it's important to use the same language as the domain/business team, ensuring that developers (technical) and domain experts (non technical) stay aligned. This is what its called, ubiquitous language.

2. Entities and Value Objects

Entities are defined by their unique identity and not by their attributes. Even if their properties change, they are still the same entity.
For example, a Customer is an entity because it has a unique identity (customer ID or email address) that distinguishes it from other customers. Even if the customer’s name, or other details change, the customer remains the same as long as the identity (ID or email) stays the same.
Value objects(VO), on the other hand, are defined by their attributes and have no unique identity. Two value objects that have the same attributes are considered identical.
For example, Address can be a VO, two identical addresses (same street, same city, same postal code) are interchangeable and don't hold any unique identity beyond their attributes.

In an organization, the structure can be composed of multiple entities, each with its corresponding value objects.

3. Aggregates and Aggregate Roots

An aggregate is a group of related domain objects that should be treated as a single unit (have a look at the image below for a better understanding). This ensures consistency and encapsulate business rules.
The aggregate root is the primary entity that controls access to other objects within the aggregate. The aggregate root is always an entity and never a value object.

Aggregates communicate primarily through domain events. When something happens within one aggregate that needs to affect another aggregate, the first aggregate sends an event, and other aggregates are listening and react to that event.

A more real life example of aggregates. In a library system, a library can manage multiple books, each with its own attributes such as title, author, and location within the library.

Library (Aggregate Root)

The Library is an entity because it has a unique identity (libraryId) and it represents the collection of books within the system.
It is also the aggregate root, meaning it controls access to the other entities within the aggregate (the Book entities in this case).

Book (Entity)

The Book is an entity because it also has a unique identity (the isbn) and it represents a specific book in the library collection.
It is part of the Library aggregate, and its properties are managed through the Library.

BookDetails and Location (VO)

They are value objects because they don’t have an identity and are defined only by their attributes.

4. Domain Services

Sometimes, business logic does not naturally belong to an entity or value object. In such cases, we use domain services to encapsulate that logic.

5. Domain Events

Domain events capture important occurrences within the business domain. For example, when a payment is completed, an event like PaymentCompleted can be published, triggering related actions such as sending an email notification.

6. Bounded Contexts

This concept is a bit abstract. Bounded contexts, represents subdomains with its own models and logic. It's like a "boundary" that separates one part of a system from others. For example, in an online store, you might have one bounded context for order management and another for payment processing, each with its own models and logic.
Don't mix aggregates and bounded contexts, aggregates are smaller groups and exist inside a bounded context, but bounded contexts help separate different business areas within the whole system

Conclusion

DDD is not just about structuring code, it’s a mindset shift that emphasizes deep collaboration with domain experts and models business logic in a way that is both clear and maintainable.

Microservices vs. Monoliths: How to Choose the Right Architecture for Your Project

Jonathan — Mon, 10 Mar 2025 11:18:22 +0000

When designing a software system, one of the most crucial decisions is choosing the right architectural style. Should you build a monolithic application or adopt a microservices architecture? Both approaches have their advantages and drawbacks, and the best choice depends on factors like scalability, complexity, and team structure. In this post, we’ll break down the key differences, benefits, and when to use each approach.

What is a Monolithic Architecture?

A monolithic application is built as a single unified unit where all components (UI, business logic, database operations, etc.) are tightly integrated and deployed together.

✅ Advantages of Monoliths

Easier to Develop and Deploy: Everything is in one place. Easier to build, test, and deploy.
Better Performance: No request calls between different services (less latency).
Easier Debugging: Having everything in one place makes fixing problems easier.
Less Maintenance Overhead: No need for complex service management (load balancing, distributed tracing,...).

❌ Challenges of Monoliths

Limited Scalability: Scaling a monolithic application often means replicating the entire system, which can be inefficient and may not provide the same level of flexibility as scaling individual services in a microservices architecture.
Slower Development in Large Teams: As the codebase grows, conflicts between teams can increase, and the development process can slow down due to bottlenecks, especially during merges and integration.
Technology Lock-In: The entire system is usually built with a single technology stack, limiting flexibility.
Difficult Continuous Deployment: Any change requires redeploying the whole application, increasing the risk of failures.

What is a Microservices Architecture?

A microservices architecture breaks down an application into smaller, loosely coupled services that can be developed, deployed, and scaled independently. Each microservice is responsible for a specific business functionality and communicates with others via APIs.

✅ Advantages of Microservices

Independent Scalability: Scale only the services that need more resources instead of scaling the entire system.
Faster Development and Deployment: Teams can work on different services independently, accelerating the development process.
Technology Flexibility: Each service can use the most suitable technology stack.
Better Fault Isolation: A failure in one service does not necessarily bring down the entire system.

❌ Challenges of Microservices

Increased Complexity: Managing multiple services, databases, and APIs requires more knowledge.
Higher Operational Overhead: Requires orchestration tools, monitoring solutions, service discovery, and API gateways.
Network Latency: Services communication over the network can introduce latency and performance issues, but tools like API Gateways and strategies like asynchronous communication (using message queues, for example) can help mitigate this.
Distributed Data Management: Maintaining data consistency across services is more challenging.

SAGA Pattern

Is a commonly used pattern in microservices architecture to manage distributed transactions. Ensures data consistency using a series of local transactions, each with a compensating(rollback) action in case of failure.

Simple Example: E-commerce Order
You have 3 services:

Order Service: Creates an order.
Payment Service: Charges the customer.
Inventory Service: Reserves the product.

Success Scenario:

Create Order → OK
Charge Payment → OK
Reserve Inventory → OK All done, order successful.

Failure Scenario (Inventory fails):

Create Order → OK
Charge Payment → OK
Reserve Inventory → fails

Trigger compensating actions:

Refund Payment
Cancel Order

Two Saga Types:

Choreography (event-based)

Services listen for events and react.
No central orchestrator.
Good for simple workflows.

Orchestration (command-based)

One central orchestrator service tells others what to do.
Easier to manage complex flows.

Conclusion

Choosing between a monolithic or microservices architecture really comes down to what your application needs, the size of your team, and your long-term goals. Monolithic apps are simpler to start with, but as your project grows, they can become harder to manage. Microservices give you more flexibility and scalability, but they come with more complexity and maintenance. By understanding the pros and cons of each approach, you’ll be in a better position to pick the right one for your project.

Designing a Scalable Architecture - with Some Spring Boot Examples

Jonathan — Wed, 05 Mar 2025 22:55:33 +0000

Scalability is a critical factor when designing modern backend systems. A well-structured Spring Boot application should be able to handle increasing loads efficiently, ensuring reliability and performance. In this article, we’ll explore the best practices for building a scalable architecture.

What Makes an Architecture Scalable?

A scalable system can handle increasing workloads by either scaling up (vertical scaling) or scaling down (horizontal scaling) while maintaining performance. The key factors that contribute to scalability include:

Efficient resource management
Stateless microservices
Proper database scaling strategies
Asynchronous processing
Load balancing and caching

Scaling Approaches

Scaling Up (Vertical Scaling) – Increasing the power of a single machine by adding more CPU, RAM, or storage.
Scaling Down (Horizontal Scaling) – Adding more machines (nodes/instances) to distribute the load.

Key Components of a Scalable Architecture

1. Adopting a Microservices Architecture

Instead of a monolithic approach, designing a microservices-based system allows independent scaling of each service.

Break the application into smaller, independent microservices.
Centralized configuration: use Spring Cloud Config for configuration management. This helps update configurations dynamically without redeploying services.
Deploy services independently (dockerize).
Service discovery for automatically locating and connecting microservices without hardcoding their network locations, enabling dynamic scaling and resilience in distributed systems (EKS or ECS)

2. Stateless and Horizontally Scalable Services

In a stateless architecture, the service doesn't store session data between requests. Each request is treated as an independent transaction. Instead, any necessary state information is passed with each request, typically through parameters, headers, or tokens.

JWT. It enables stateless authentication by embedding user info within the token itself. How does JWT Work in a Stateless API?

Client logs in → The authentication server issues a JWT upon successful login.
Client includes JWT in requests → The token is passed in the Authorization header

Authorization: Bearer xxxxx

Service verifies JWT → The backend validates the token using a secret key or public key.
Request is processed → If valid, the request is processed without needing session lookups.

3. Database Scaling Strategies

A single database can become a bottleneck as traffic grows. Consider the following approaches:

Database Partitioning (Sharding) – Splitting large datasets into smaller, manageable parts.
Read Replicas – Creating read-only database replicas to reduce the load on the primary database.The primary database only handles writes, while multiple read replicas handle read operations, improving performance
NoSQL Databases – Using MongoDB, Cassandra, or DynamoDB for high scalability needs. However, it's important to note that NoSQL databases may favor availability and partition tolerance over strong consistency. The choice between NoSQL and relational databases should depend on your application's needs, whether you prioritize consistent data for critical transactions or need high availability and fault tolerance for large-scale, distributed applications.

4. Asynchronous Processing with Message Queues

Asynchronous services are often preferred over synchronous ones in scalable architectures because of their efficiency, scalability, and ability to handle high traffic volumes without blocking processes. Spring Boot integrates well with popular message brokers like Kafka, RabbitMQ and Amazon SQS. Synchronous requests block the execution until a task is completed (higher latency), so it can be more appropriate when immediate responses are needed, or when the task being performed is lightweight and doesn’t involve waiting on external resources.

@Async, Spring Boot allows methods to be executed asynchronously using this annotation. Spring will run the method in a separate thread, allowing the main thread to continue without waiting for the method to complete.

@Service
public class EmailService {

    @Async
    public void sendEmail(String emailAddress) {
        // Simulate sending an email...
    }
}

...

public void execute() {
    emailService.sendEmail("test@gmail.com");
    // The main thread continues without waiting for the email to be sent
}
...

5. Load Balancing and API Gateway

Distribute traffic efficiently across multiple instances to prevent overloading a single service and ensure high availability.

API Gateway basically acts as a single entry point for all client requests. Lets see some of the main features.

Routing & Load Balancing – Directs requests to the correct microservice based on URL paths.
Authentication & Security – Implements OAuth, JWT, API keys, and rate limiting.
Request Transformation – Modifies request headers, parameters, or payloads before forwarding.
Caching & Compression – Improves performance by reducing redundant API calls.
Circuit Breaker & Retry Mechanism – Prevents cascading failures and enhances fault tolerance.

Using both, Load Balancers and API Gateways ensures a resilient, scalable, and secure architecture

Clients --> API Gateway --> Load Balancer(through EKS) --> service A/service B....

6. Caching for Performance Optimization

Reduce database queries by caching frequently accessed data.

Redis, Amazon ElastiCache, Spring cache Abstraction.

Spring Cache Abstraction allows you to define caching behavior declaratively using annotations. This removes the need to write low-level caching logic manually. Most commonly:
@Cacheable: Caches the result of a method based on the method parameters. If the same method is called again with the same parameters, the cached result is returned instead of executing the method.

@Cacheable("users") // 'users' is the cache name and the method params the key
public User getUserById(Long id) {
    return userRepository.findById(id);
}

@CachePut: Updates the cache with the result of a method, but always executes the method, unlike @Cacheable, which may skip execution if the result is cached.

@CachePut(value = "users", key = "#user.id")
public User updateUser(User user) {
    return userRepository.save(user);
}

@CacheEvict: Removes entries from the cache. It can be used to clear the cache based on conditions or manually evict data when it’s no longer needed.

@CacheEvict(value = "users", key = "#id")
public void deleteUser(Long id) {
    userRepository.deleteById(id);
}

Conclusion

The previous points are some of the key considerations when designing scalable architectures. However, other important factors, such as logging, security, monitoring, error handling, retry mechanisms, etc etc.., should also be considered. Incorporating these aspects ensures that your application remains robust, secure, and easy to maintain as it grows and evolves.

Hexagonal Architecture in Spring Boot: A Practical Guide

Jonathan — Wed, 12 Feb 2025 12:33:36 +0000

Hexagonal Architecture, also known as Ports and Adapters, was introduced by Alistair Cockburn in 2005. Cockburn developed this concept to solve the problem of tight coupling between application logic and external dependencies (like databases, UI frameworks, and APIs). His main goal was to make systems more maintainable, testable, and adaptable by clearly separating core business logic from infrastructure concerns.
Hexagonal Architecture shares similarities with other architectural patterns designed to promote separation of concerns and modularity.

Layered Architecture (N-Tier Architecture)
Clean Architecture (by Robert C. Martin - Uncle Bob)
Onion Architecture (by Jeffrey Palermo)

Why is it Called Hexagonal Architecture?
The term Hexagonal Architecture comes from the hexagon shape used to visually represent the core system and its interactions with the external world.

Why a Hexagon?

The hexagon shape is a metaphor to visually represent the system’s core and its interactions.
Unlike a rectangle or circle, a hexagon does not imply a hierarchical structure (like traditional N-tier architecture).
It has multiple connection points (ports) where you can easily plug in different adapters, like a UI, database, or external APIs.
The number six isn't important, it simply provides enough sides for a clean and organized representation.

Ports: The Entry Points to Your Business Logic
In Hexagonal Architecture, ports represent interfaces that define the use cases or functionalities of the system. They specify what the application can do (e.g., add a product to the cart, remove a product, calculate the total), but not how it's done. Ports are the entry points to your core business logic.

public interface CartService {
    void addProduct(Product product);
    void removeProduct(String productId);
    double calculateTotal();
    Cart getCart();
}

In this example, CartService is a primary port that defines the use cases for managing a shopping cart.

Adapters: Bridging Your Core Logic and the Outside World
Adapters are the implementations that interact with external systems and convert data between the outside world and the core business logic. There are two types of adapters in Hexagonal Architecture:

Primary Adapters: These handle input from external sources, such as user requests, UI, or API calls. They interact with the primary ports (use cases) of the system. For example, a REST controller that receives HTTP requests and delegates the actions to a service (like CartServiceImpl).

Example of a primary adapter (controller)

@RestController
@RequestMapping("/cart")
public class CartController {
    private final CartService cartService;

    @Autowired
    public CartController(CartService cartService) {
        this.cartService = cartService;
    }

    @PostMapping("/add")
    public void addProduct(@RequestBody Product product) {
        cartService.addProduct(product);
    }
}

CartController acts as a primary adapter because it handles HTTP requests (external input) and delegates them to the CartService (the primary port).

Secondary Adapters: These handle output to external systems, such as databases, message queues, or APIs. They implement the secondary ports, which are interfaces for data persistence or external integrations.

Example of a secondary adapter (repository)

public class InMemoryCartRepository implements CartRepository {
    private Cart cart = new Cart();

    @Override
    public void save(Cart cart) {
        this.cart = cart;
    }

    @Override
    public Cart get() {
        return cart;
    }
}

Primary Adapters ➡ Think of them as input handlers (Controllers, CLI, API Gateways) that take external requests and pass them into the system.
Secondary Adapters ➡ Think of them as output handlers (Repositories, API clients, message brokers) that take results from the system and pass them to the outside world.

An Example Package Structure in Java

com.company.cart
│
├── domain/
│   ├── model/
│   │   ├── Product.java
│   │   ├── Cart.java
│   ├── repository/
│   │   ├── CartRepository.java
│   ├── exception/
│   │   ├── CartException.java
│
├── application/
│   ├── service/
│   │   ├── CartService.java
│
├── infrastructure/
│   ├── adapters/
│   │   ├── primary/
│   │   │   ├── controller/
│   │   │   │   ├── CartController.java
│   │   │   ├── error/
│   │   │   │   ├── ControllerExceptionHandler.java
│   │   │   │   ├── ErrorResponse.java
│   │   │   ├── dto/
│   │   │   │   ├── CartRequestDTO.java
│   │   │   │   ├── CartResponseDTO.java
│   │   │   ├── mapper/
│   │   │   │   ├── CartMapper.java
│   │   ├── secondary/ // Adapters that **provide** infrastructure (e.g., persistence, external APIs)
│   │   │   ├── InMemoryCartRepository.java
│
├── configuration/
│   ├── CartServiceConfig.java
│
└── CartApplication.java

How Do These Layers Interact?
When we need to interact with an infrastructure service from an application service, we follow the Dependency Inversion Principle. This principle suggests that rather than directly depending on concrete implementations, we inject the necessary dependencies through the constructor of the class. This approach promotes loose coupling and makes the system more flexible and easier to test.

Ports vs. Adapters: Understanding the Difference
Ports are interfaces that define what the system should do. They represent the use cases of the application (e.g., CartService).
Adapters are the implementations of those ports, responsible for managing the interaction with the outside world (like HTTP requests in the case of primary adapters, or database interactions in the case of secondary adapters).

Does the Implementation of a Primary Port Need to Be a Primary Adapter?
No. The implementation of a primary port (such as CartServiceImpl) does not have to be a primary adapter. While primary ports define the use cases, their implementation is simply the business logic that satisfies those use cases. The primary adapter (like a controller) is the one that handles the external interaction (e.g., receiving requests, calling the service methods).

CartService is a primary port (interface).
CartServiceImpl is the implementation of the primary port, but it is not an adapter.
CartController is a primary adapter, as it handles HTTP requests and delegates to the CartService (the primary port).

Conclusion
In Hexagonal Architecture, ports define what the system does (use cases), while adapters define how the system interacts with the outside world. The implementation of a port does not have to be an adapter itself; rather, it's the business logic that fulfills the requirements set by the port. Adapters, on the other hand, are responsible for connecting the external world to the core logic via the ports.

This separation makes the system easier to maintain, test, and adapt to different external technologies.

Building, testing and publishing your Java project with Github actions.

Jonathan — Sat, 31 Oct 2020 12:11:41 +0000

Introduction

I'm going to create a continuous integration (CI) workflow to build, test and publish the project in Github Packages.

The project:

Spring boot with maven
Postgres database with a table users.
One endpoint /users that returns a list of users. In the example, I just have the user jonathan with ID 1. My integration test will check /users returns a list of size 1.

@RestController
@RequestMapping("/users")
public class UserController {

    private final UserService userService;

    @Autowired
    public UserController(UserService userService) {
        this.userService = userService;
    }

    @GetMapping
    public List<User> findAll() {
        return userService.findAll();
    }
}

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
public class UserEndpointIT {
    @Autowired
    private TestRestTemplate restTemplate;

    @Autowired
    protected DatabaseService databaseService;

    @LocalServerPort
    private int port;

    @BeforeAll
    public void initDatabase(){
        databaseService.fill();
    }

    @Test
    public void user_endpoint_should_return_all_users() {
        HttpHeaders headers = new HttpHeaders();
        HttpEntity entity = new HttpEntity(headers);

        ResponseEntity<List<User>> response = this.restTemplate.exchange(createUrlWith("/users"), HttpMethod.GET, entity, new ParameterizedTypeReference<List<User>>() {
        });
        int EXPECTED_USERS = 1;

        assertEquals(EXPECTED_USERS, response.getBody().size());
    }

    private String createUrlWith(String endpoint) {
        return "http://localhost:" + port + endpoint;
    }
}

Flyway as a database migration tool. (checkout my post about flyway if you don't know about it)
The following Maven plugin (docker-maven-plugin) in my pom.xml to run a postgres docker container for my integration tests.

...
<plugin>
   <groupId>io.fabric8</groupId>
   <artifactId>docker-maven-plugin</artifactId>
   <version>${docker-maven-plugin.version}</version>
   <configuration>
      <images>
         <image>
            <name>${it.postgresql.image}</name>
            <run>
               <ports>
                  <port>${it.postgresql.port}:5432</port>
               </ports>
               <env>
                  <POSTGRES_USER>${it.postgresql.username}</POSTGRES_USER>
                  <POSTGRES_PASSWORD>${it.postgresql.password}</POSTGRES_PASSWORD>
                  <POSTGRES_DB>${it.postgresql.db}</POSTGRES_DB>
               </env>
               <wait>
                  <log>database system is ready to accept connections</log>
               </wait>
            </run>
         </image>
      </images>
   </configuration>
   <executions>
      <execution>
         <id>start</id>
         <phase>pre-integration-test</phase>
         <goals>
            <goal>start</goal>
         </goals>
      </execution>
      <execution>
         <id>stop</id>
         <phase>post-integration-test</phase>
         <goals>
            <goal>stop</goal>
         </goals>
      </execution>
   </executions>
</plugin>
...

Github actions allow us Continuous Integration (CI) and Continuous Deployment (CD) from our Github repositories.
How does it work? An event triggers a workflow, for example a push to master branch or a pull request, and that workflow could have different jobs. We can have a job to run our tests and another job to deploy the app. Each job uses steps that has different tasks or better known as actions. We can create our own actions or use the ones created by the community. We are also allowed to name each action and give a brief description about what it does by using name key. It's important to note that by default, jobs will run in parallel unless we configure them to run sequentially (maybe any job depends on another).

Creating the workflow

First of all, we need to create the workflow file in the .github/workflows directory.

name: my first workflow for a java project
on:
  push:
    branches: [ master ]

jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-java@v1
        with:
          java-version: 11
          settings-path: ".m2/"
      - uses: actions/cache@v2
        with:
          path: ~/.m2/repository
          key: ${{ runner.os }}-${{ hashFiles('**/pom.xml') }}
      - name: Publish package
        run: mvn $MAVEN_CLI_OPTS clean deploy
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode"
      - name: Copying target jar
        run: |
          mkdir myTarget
          cp target/*.jar myTarget
      - name: Uploading jar
        uses: actions/upload-artifact@v2
        with:
          name: myPackage
          path: myTarget

We begin by adding a name for our workflow and setting the event that will trigger it. In this case, the event is a push to my master branch. Now, we need to set the jobs for the workflow. The runs-on configures the job to run on a virtual machine hosted by Github, to be more precise an Ubuntu Linux runner (read more here). In the above example, I just have one job called publish that performs the following steps:

The checkout action downloads a copy of our repository on the runner.
The setup action configures the Java environment. The with key allow us to specify the variables that will be available to the action. In the example, we set up the JDK version and with settings-path, we specify the location for the settings.xml file.
Considering jobs on hosted GitHub runners start in a clean virtual environment, we want to save time so that dependencies don't need to be downloaded each time a job is ran. In the example, I'm caching all the dependency jars. It's important to set a key so that the action will try to restore the cache files based on that key. To create the key, I'm using the runner.os that gives me where the job is currently running, and the hashfile function that returns a hash for the given files (read more here).
Im going to publish my project to Github Packages, that's why you see I run mvn clean deploy. At this point, is important to know that the action actions/setup-java@v1 used previously, is going to set up automatically a Maven settings.xml file generating the authentication for the server like the shown below. To use the GITHUB_TOKEN secret, we must reference it in our workflow file, as you see in my workflow file, I'm passing the token as an input to the action Publish package. If we don't do this, we can see an error in the logs (401 Unauthorized).

<servers>
    <server>
        <id>github</id>
        <username>${{ secrets.GITHUB_ACTOR }}</username>
        <password>${{ secrets.GITHUB_TOKEN }}</password>
    </server>
</servers>

The server created will have an id of github, GITHUB_ACTOR environment variable is our username and GITHUB_TOKEN environment variable is our password (these env. variables exist in our github repository by default). In my pom.xml, I I just need to add the repository configuration where the package will be uploaded.

...
<distributionManagement>
    <repository>
        <id>github</id>
        <name>github packages</name>
        <url>https://maven.pkg.github.com//[your_github_username]/[your_repository_name]</url>
    </repository>
</distributionManagement>
...

If we make a push, on the right side of our repository, we can find our package.

If we click on it, we can see the dependency.

One of the last steps is to upload the artifact. Remember that an artifact is file or a collection of files produced during a workflow run. First, what I do is to create myTarget folder and copy the jar. The last actions used is upload-artifact@v2 that receives two inputs, the name of the artifact and the files that will be uploaded.

If we download myPackage and unzip it, we can see the jar.

When we make a push to the repository, in the Actions tab we can see the workflow running. If everything is fine, we'll see a green tick.

When we click on the workflow, we can see a detailed list of the jobs and their steps. In my example, I just have the publish job.

If we check the log of the publish package action, we can see how my database container is running, the spring context is up, flyway migrations are okay, and the tests passed successfully

Github packages

GitHub Packages is a hosting service that allows us to host our packages privately or publicly and use them as dependencies in our projects. As we saw in the previous section, we have uploaded the project as a dependency to Github Packages. But how can I install a maven dependency stored in github packages registry? apart from adding the dependency in the pom.xml, we must follow the following steps:

In our pom.xml, we need to point to the repository where the dependency we want to install is located.

...
<repositories>
    <repository>
        <id>github</id>
        <url>https://maven.pkg.github.com/[github_username]/[repository_name]</url>
    </repository>
</repositories>
...

The github package registry is available through the GitHub api and needs authorization, so we need to add our github credentials in the settings.xml. In my case I have added them to my global settings (~ / .m2 / settings.xml)

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                          https://maven.apache.org/xsd/settings-1.0.0.xsd">

    <servers>
        <server>
            <id>github</id>
            <username>[your_github_username]</username>
            <password>[your_github_token]</password>
        </server>
    </servers>

</settings>

To create a token, we should go to our github Settings > Developer Settings > Personal access tokens and enable the read: packages permission.

Once these steps have been followed and if everything has gone well, the dependency will be installed correctly 😃.

Testcontainers - dockerize your integration tests in java

Jonathan — Tue, 07 Jul 2020 17:54:04 +0000

Using an in-memory database like H2 in Java have some downsides because the tests could depend on features that in-memory databases can't reproduce and some tests that have passed locally may fail in production. This affects our tests reliability because we won't cover 100% the same scenarios as in our real environment.
Testcontainers comes to the rescue so that we can dockerize our tests. It's a Java library that allows to create any docker instances and manipulate them.

I'm going to be using JUnit5, but if you use JUnit4 is perfectly fine too. Let's start by adding the dependency in the pom.xml. We can add the generic dependency or a more specific one (to have a preconfigured container). When using databases, remember to add their drivers (Testcontainers will not add it for you). Take a look at the list of preconfigured containers in maven repository. You can find for mongoDB, postgresql, cassandra, elasticsearch, rabbitmq, etc.

//Generic dependency
<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>testcontainers</artifactId>
    <version>1.14.3</version>
    <scope>test</scope>
</dependency>

//Specific dependency for mysql
<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>mysql</artifactId>
    <version>1.14.3</version>
    <scope>test</scope>
</dependency>

//driver
<dependency>
    <groupId>mysql</groupId>
    <artifactId>mysql-connector-java</artifactId>
    <version>8.0.20</version>
</dependency>

How does Testcontainers work?

It starts a container with the specific docker image (in my example, I'm using a mysql image).
Another container called Ryuk will also be started and it's main task is to manage the startup and stop of the container.

One of the good things about this library is its integration with JUnit. To be able to use the next annotations, we need to add this dependency.

<dependency>
    <groupId>org.testcontainers</groupId>
    <artifactId>junit-jupiter</artifactId>
    <version>1.14.3</version>
    <scope>test</scope>
</dependency>

From the docs:

@Testcontainers: is a JUnit Jupiter extension to activate automatic startup and stop of containers used in a test case.
@Container: is used in conjunction with the @Testcontainers annotation to mark containers that should be managed by the Testcontainers extension.

Creating a generic container

We can create a generic container from any public docker image or a docker-compose. Since this is a generic approach, we need to configure more things than if we already had a preconfigured container.

 @Container
 private GenericContainer container = new GenericContainer("image_name")
            .withExposedPorts(port_number);

withExposedPorts(port), we expose the default internal port (in mysql for example is 3306) from the container that will be mapped to a random port. To retrieve that random port at runtime, we can use the getMappedPort(original_port) or just getFirstMappedPort() method. If we don't expose the port, we'll get en error saying 'Container doesn't expose any ports'. We can also add environment variables to the container with .withEnv(), execute command inside a container (like docker exec), manage our own waiting and startup strategies, etc. Have a look at the documentation for deeper information.

Creating a mysql container

As I said before, we have many preconfigured containers with some out-of-the-box configuration that makes our lives easier. I'm going to use a mysql container for my tests.
We can decide whether we want to start and stop the container once per tests class or once per tests method (we'll see later how to create a singleton container)

//Once per test class
 @Container
 private static final MySQLContainer mysql = new MySQLContainer("mysql:latest");

// Once per test method
 @Container
 private MySQLContainer mysql = new MySQLContainer("mysql:latest");

Note: If you are using JUnit4, you can use @Rule and @ClassRule annotations.

The following example starts a mysql container and then runs my HelloEndpointIT class. I'm using static final in the container instance, so the container will be shared between all tests methods. The mysql container give me some methods to configure a specific database name, username and password. If we don't specify this, we will use default values (database name: test, password: test, username: test)

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@ActiveProfiles(profiles = "test")
@TestInstance(TestInstance.Lifecycle.PER_CLASS)
@Testcontainers
public class HelloEndpointIT {

    @Autowired
    private TestRestTemplate restTemplate;

    @LocalServerPort
    private int port;

    @Container
    private static final MySQLContainer mysql = new MySQLContainer("mysql:latest")
                .withDatabaseName("demo_db_name")
                .withUsername("any_username")
                .withPassword("any_passw");

    @BeforeAll
    private void initDatabaseProperties() {
        System.setProperty("spring.datasource.url", mysql.getJdbcUrl());
        System.setProperty("spring.datasource.username", mysql.getUsername());
        System.setProperty("spring.datasource.password", mysql.getPassword());
    }

    @Test
    public void hello_endpoint_should_return_hello_world() {
        HttpHeaders headers = new HttpHeaders();
        HttpEntity entity = new HttpEntity(headers);

        ResponseEntity<String> response = this.restTemplate.exchange(createUrlWith("/hello"), HttpMethod.GET, entity, String.class);

        assertThat(response.getStatusCode(), equalTo(HttpStatus.OK));
        assertThat(response.getBody(), equalTo("Hello world"));
    }

    private String createUrlWith(String endpoint) {
        return "http://localhost:" + port + endpoint;
    }
}

When the container is ready, I need to set my datasource configuration. We can get the url with its mapped port using the getJdbcUrl() method, as well as the getUsername() and getPassword(). I'm setting this configuration before running my tests using the @BeforeAll annotation provided by JUnit5. In JUnit4 is @BeforeClass.

Creating a singleton container

So far we have seen how to run our container once per test class or method, but I would like to just create an instance for all my tests classes. So, let's see how to run a singleton container before running all our integration tests.

We are going to use static Initializers to initialize the container only once. We need to do it in an abstract class and extend all our IT classes. We also need to manually start the container in our initializer block and when the tests finish, the Ryuk container will take care of stopping it.

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@ActiveProfiles(profiles = "test")
public abstract class DemoEndpointIT {

    @Autowired
    private TestRestTemplate restTemplate;

    @LocalServerPort
    private int port;

    private static final MySQLContainer mysql;

    static {
        mysql = new MySQLContainer("mysql:latest");
        mysql.start();
        System.setProperty("spring.datasource.url", mysql.getJdbcUrl());
        System.setProperty("spring.datasource.username", mysql.getUsername());
        System.setProperty("spring.datasource.password", mysql.getPassword());
    }

    protected String createUrlWith(String endpoint) {
        return "http://localhost:" + port + endpoint;
    }

    protected TestRestTemplate getRestTemplate() {
        return this.restTemplate;
    }
}

As you saw in the examples, having a mysql container ready for my integrations tests were super easy and almost no configuration was necessary.

DEV Community: Jonathan

Java Collections: From Lists to Maps with Time Complexity in Mind

1. Large lists where random access is frequent. Fast index access? --> ArrayList

2. Frequent Insert/Remove at Beginning or End or modifying element during Iteration? --> LinkedList

3. Key-based lookup? --> HashMap

3.1 HashMap vs TreeMap vs LinkedHashMap

4. Need Unique Elements Only? --> Set

5. Process Items In The Order They Arrived (FIFO) --> Queue

6. Double-Ended Queue --> Deque

Virtual Threads in Java: A Lightweight Concurrency Revolution

What Are Virtual Threads?

What Problem Do Virtual Threads Solve?

When Should You Use Virtual Threads?

Do Virtual Threads Replace WebFlux?

Do Virtual Threads Eliminate Blocking?

Conclusion

From JVM to Native Compilation with Spring Boot: What It Means and Why It Matters

Traditional Java Compilation (JVM-Based)

Native Compilation: What’s New?

Spring Boot + Native Support

JVM vs Native: Key Differences

Conclusion

How to Handle Millions of Rows in SQL Without Killing Performance

The Problem: IN with Too Many Values

The Solution: JOIN with a Temp Table

Conclusion

Apache Kafka: Topics, Partitions, Brokers

Introduction

Why Partitions Matter?

Kafka Consumer Groups

Kafka Retention Time cleanup.policy=delete

Kafka Log Compaction cleanup.policy=compact

Serialization and Deserialization in Kafka

Avro + Schema Registry Is the Preferred Choice

Let's see some code

Understanding concepts in Event Driven Architectures (EDA)

Key Components

Broker Exchanges

Advantages of EDA - Reactive Manifesto

Disadvantages of EDA:

Conclusion

Domain Driven Design(DDD) - Understanding Main Concepts

Main Concepts

Conclusion

Microservices vs. Monoliths: How to Choose the Right Architecture for Your Project

What is a Monolithic Architecture?

What is a Microservices Architecture?

SAGA Pattern

Two Saga Types:

Conclusion

Designing a Scalable Architecture - with Some Spring Boot Examples

What Makes an Architecture Scalable?

Key Components of a Scalable Architecture

1. Adopting a Microservices Architecture

2. Stateless and Horizontally Scalable Services

3. Database Scaling Strategies

4. Asynchronous Processing with Message Queues

5. Load Balancing and API Gateway

6. Caching for Performance Optimization

Conclusion

Hexagonal Architecture in Spring Boot: A Practical Guide

Building, testing and publishing your Java project with Github actions.

Introduction

Creating the workflow

Github packages

Testcontainers - dockerize your integration tests in java

Creating a generic container

Creating a mysql container

Creating a singleton container

Kafka Retention Time `cleanup.policy=delete`

Kafka Log Compaction `cleanup.policy=compact`