DEV Community: Ivelin Yanev

The state of AI code reviews: An 18-month retrospective

Ivelin Yanev — Wed, 31 Dec 2025 16:09:20 +0000

It has been almost a year and a half since our company started using AI intensively in our everyday work. We've tested various tools- such as Windsurf, GitHub Copilot, Cursor, Claude, and LibreChat- and I can now confidently comment on how AI models affect software engineering as a discipline.

In this article, I'll try to present my objective opinion on using AI for code reviews.

First, let me clarify- I'm not an AI specialist. I don't have the deep technical knowledge to discuss models or their architecture, as so many people on the Internet tend to do these days. It feels like everyone thinks they're an expert on LLMs, which obviously isn't realistic.

Code Review: A Critical Step in Software Development

Code review is a key stage in every software project, and neglecting it often leads to unpredictable and critical problems later on. Since the very beginning of the software era, reviewing written code- regardless of programming language- has been seen as a required step before the code is used in production.

Let's look at an example from history- the Apollo spacecraft software.

Hundreds of engineers worked on it during those years, creating the software without which a safe lunar landing and return would have been impossible. The Apollo software, called Luminary, contained over 145,000 lines of code. Every single file included comments (see on GitHub), and the entire codebase went through multiple review and approval iterations.

Today, we have automated tools, compilers, and sophisticated software practices. But back then, every line was written, reviewed, and optimized by hand. Even in the early 1960s, processes like defining requirements, design, coding, testing, and maintenance were strictly followed.

Margaret Hamilton, who led the lab developing the Apollo flight software, once said:

What became apparent with Apollo- though it is not how it worked- is that it is better to define your system up front to minimize errors, rather than producing a bunch of code that then has to be corrected with patches on patches. It's a message that seems to have gone unheeded- in this respect, software today is still built the way it was 50 years ago.

By focusing on finding and fixing errors early, the system was stable enough to handle unexpected CPU overloads just seconds before the lunar landing. No software errors were reported during any of the manned Apollo missions- a remarkable testament to human precision.

The Code Review Pyramid

A well-known concept in software engineering is the Code Review Pyramid, which illustrates the relative importance of different review aspects: Code Style, Documentation, Testing, Implementation, and Functionality & Design.

From my own experience, I've always applied this approach when reviewing others code. But the first review always happens with my own code- by me.

Before you push your changes or ask an AI to check them, revisit the pyramid yourself.

Don't just check for syntax.
Imagine you are reading it for the first time.
What is confusing? What is implicit?

I find it helpful to take a break between writing and reviewing my code. Many developers submit changes at the end of the day, but that's when fatigue leads to overlooked errors. Waiting until morning and reviewing the changes with fresh eyes dramatically improves the result.

Where Does AI Excel?

Several parts of the review process can be automated- checking code style, syntax errors, test coverage, code optimization, and error detection.

AI models are exceptionally precise in finding mistakes and identifying poor coding practices. Their consistency is remarkable: they spot the same problems every single time. When it comes to adherence to conventions or catching minor flaws, AI is laser-focused.

Moreover, AI effortlessly scales- it can review dozens of pull requests per day without fatigue, something almost impossible for a human reviewer.

Here are a few clear advantages of using AI in code review:

Immediate feedback – In traditional reviews, responses might take hours or even days, leaving developers waiting. With AI, the feedback comes instantly.
No cognitive fatigue – Reviewing hundreds of lines of someone else's code can be mentally draining. AI never tires.
Productivity scaling – In large teams, human reviewers spend significant time on reviews, impacting productivity. For AI, team size doesn't matter- it performs equally fast and effectively regardless.

A Cisco study found that reviewing more than 400 lines of code at once reduces one's ability to find bugs, with most defects discovered in the first 200 lines. This insight shaped industry practices- but AI doesn't suffer from this limitation. It can handle large reviews without performance degradation.

The Human Mind: The Unbeatable Factor

While AI performs flawlessly on the tasks mentioned above- and I rely on it heavily. It still lacks something fundamental- understanding.

As experts have pointed out, current AI systems are, in many ways, quite limited. We're fooled into thinking they're intelligent because they handle language so well. But they don't understand the physical world. They lack persistent memory, true reasoning, and long-term planning- all crucial aspects of genuine intelligence.

Machine learning generally operates under three paradigms:

Supervised learning

This is the classical approach. The model is trained using a dataset of examples that have both input and the correct output (labels). For instance, if you train a system to recognize objects, you might show it an image of a table and label it "table". The model then produces its own output and receives feedback: if it's wrong, its internal parameters are adjusted. Repeating this process millions of times helps the system form strong associations between inputs and correct predictions.

Reinforcement learning

This technique more closely resembles certain aspects of human learning. Instead of being told the exact right answer, the AI acts, observes the consequences, and receives a reward or penalty. It adjusts its future actions to maximize the long‑term reward. Think of how you learned to ride a bike- through trial, error, and correction. However, this paradigm has limitations. It’s inefficient and effective only in clearly defined environments (like playing chess, Go, or poker) where success metrics are known and unambiguous. In complex, real‑world settings without clear feedback, reinforcement learning becomes impractical.

Self-supervised learning

This is the foundation of the most recent revolution in AI, including large language models such as ChatGPT. Here, the system learns from unlabeled data by creating its own predictive tasks- for example, trying to predict missing words in a sentence. By training on vast quantities of text, the model builds internal representations of patterns and relationships between words and concepts. This approach allows AI to gain an impressive ability to generate coherent and context‑aware language, but it still does not give the model genuine understanding or reasoning ability- it is pattern recognition, not comprehension.

Even with all this sophistication, AI remains fundamentally limited by its training data and objectives. It does not understand things as humans do; it merely models statistical relationships.

Software, by definition, is a tool that must operate according to human needs and actions- sometimes even playing a role in life-critical systems. This gap is where human consciousness and intuition remain irreplaceable.

Final Thoughts

I'll end with a visual that perfectly captures how the development landscape has shifted in the past two years.

Writing code itself has never truly been the problem. The real challenge has always been delivering error-free and dependable software.

The lesson is clear: AI is a powerful ally—especially for automating repetitive parts of the review process—but the human element remains the final safeguard of wisdom, empathy, and understanding that no algorithm can yet fully emulate.

Java 25: Top 3 Features That Redefine Performance, Startup, and Efficiency

Ivelin Yanev — Fri, 31 Oct 2025 19:25:44 +0000

It’s been a month since the release of the new Long-Term Support (LTS) version of Java. This release brings a host of new language and compiler features, but in this post, I’ll focus on several that stand out as truly transformative for developers and production systems alike.

1. Memory Efficiency: Compact Object Headers

One of the most exciting improvements in JDK 25 is the introduction of Compact Object Headers, now a finalized feature via JEP 519. This enhancement builds upon JEP 450, where Compact Object Headers were first introduced as an experimental feature.

In short, this feature compresses the Java object header from 96 bits down to 64 bits, dramatically reducing heap memory usage for existing applications.

How It Works?

Previously, the Mark Word contained several unused bits — 27 in total. Out of 96 bits, only 69 were actually needed. Java 25 reorganizes and optimizes the structure to fit into a compact 64-bit format.

The new header structure now directly stores:

A compressed class pointer (reduced from 32 bits to 22 bits)
A 31-bit identity hash code (unchanged)
4 bits reserved for Project Valhalla
4 bits for object age (unchanged)
1 bit for the “Self Forwarded Tag”
2 tag bits (unchanged)

Performance results

Early testing shows substantial performance and memory gains:

Up to 22% less heap usage
8% faster execution times
Amazon reports up to 30% CPU reduction across hundreds of production services
15% fewer garbage collection cycles for both G1 and Parallel collectors
JSON parsing benchmarks show 10% faster execution
Throughput overhead stays below 5% even in worst-case scenarios

These improvements make Compact Object Headers a massive win for both cloud and on-premise deployments.

2. Faster Startup: Project Leyden Integration

One of Java’s longest-standing challenges has been slow startup and warmup times. JDK 25 makes a significant leap forward in this area through enhancements connected to Project Leyden.

Project Leyden’s main goal is to improve application startup and warmup performance by introducing profile-guided optimizations and ahead-of-time (AOT) capabilities at the JVM level.

How It Works?

During a typical application run, the JVM collects profiling data — information about which methods are invoked most frequently or consume the most CPU. Traditionally, this profiling happens at runtime during the first execution phase.

With Java 25 and Leyden’s enhancements, these profiles can now be saved during a “training run” and reused later. In subsequent executions, the JVM can skip the early profiling stage and begin optimized compilation immediately, resulting in much faster startup.

Benchmark Example

Recent benchmark results running over the Quarkus framework show measurable improvements in startup times and warmup efficiency.

Source: OpenJDK Project Leyden README

These enhancements align well with modern deployment patterns — especially for containerized or serverless environments — where rapid startup is critical.

3. Stable Values: Safe, On-Demand Initialization

JDK 25 introduces a powerful new construct called Stable Values — a mechanism for defining values that are initialized exactly once and then remain immutable throughout the lifetime of the application.

This feature standardizes lazy initialization patterns that developers often implement manually and allows the JVM to optimize these values in the same way as final fields through constant folding and other compiler optimizations.

A StableValue behaves like a container that holds an object — the so-called “content.”

It starts uninitialized.
It is initialized once, when first accessed.
After initialization, it becomes immutable and thread-safe.

Here’s a simple example:

public class ConfigService {

    // Create a StableValue container for caching configuration
    private final StableValue<AppConfig> config = StableValue.of();

    public AppConfig getConfig() {
        // Initialize lazily on first access
        return config.orElseSet(this::loadConfig);
    }

    private AppConfig loadConfig() {
        System.out.println("Loading configuration from file...");
        // Simulate reading configuration data
        return new AppConfig("prod", 20, true);
    }

    public void printConfig() {
        AppConfig cfg = getConfig();
        System.out.printf("Mode: %s | Workers: %d | Cache: %s%n",
                cfg.mode(), cfg.workerCount(), cfg.cacheEnabled());
    }

    // Sample record representing configuration structure
    private record AppConfig(String mode, int workerCount, boolean cacheEnabled) {}
}

What happens here:

The first call to getConfig() triggers loadConfig() and stores the result in the stable value.
All subsequent accesses return the same cached instance without reloading.
- Even in multithreaded contexts, orElseSet() ensures initialization happens exactly once.

This pattern eliminates the need for manually synchronized lazy initialization blocks. Once populated, the stable value behaves just like a final constant, and the JVM can optimize it accordingly.

JDK 25 also includes related classes such as:

StableList – for lists initialized once
StableMap – for single-initialization maps
StableFunction – for functions that compute a result once, then cache it immutably

Conclusion

Java 25 marks a pivotal moment in the evolution of the Java platform.

With Compact Object Headers, Project Leyden startup optimizations, and Stable Values, the focus is clear: greater performance, efficiency, and modernized application behavior for cloud-native environments.

If you’re maintaining long-lived services or performance-critical applications, upgrading to Java 25 should definitely be on your roadmap.

How to programmatically look up a bean in Quarkus

Ivelin Yanev — Thu, 15 May 2025 17:51:43 +0000

In Quarkus, a modern Java framework optimized for cloud-native and serverless applications, Contexts and Dependency Injection (CDI) is a core feature for managing beans. While dependency injection via @Inject is the standard way to access beans, there are scenarios where you need to programmatically look up beans at runtime. This article explores various methods to programmatically look up beans in Quarkus, with practical examples and best practices.

Why would you need programmatic bean lookup?

You need to resolve beans dynamically based on runtime conditions (e.g., selecting a specific implementation).
You’re working in a non-CDI-managed class (e.g., a static context, utility class or third-party integrations).
You need to iterate over all beans of a given type or select beans with specific qualifiers.
You want to check for bean availability or defer bean resolution for performance reasons.

Quarkus provides several approaches to programmatically look up beans, leveraging its optimized CDI implementation, Arc. The primary methods include using the Arc container, the standard CDI API, Instance<T> with or without @Any.

Methods for programmatic bean lookup

For all examples, we'll use the following bean interface and implementation:

public interface MyBean {
  String getName();
  void doSomething();
}

@ApplicationScoped
public class BeanImpl1 implements MyBean{

  @Override
  public String getName() {
    return "Standard Bean";
  }

  @Override
  public void doSomething() {
    System.out.println("BeanImpl1 is working!");
  }
}

Using the Arc Container(Quarkus-specific)

The Arc container is Quarkus CDI implementation, optimized for fast startup and low memory usage. It provides a straightforward API for programmatic bean lookup via Arc.container().

 InstanceHandle<MyBean> beanHandle = Arc.container().instance(MyBean.class);
    if (beanHandle.isAvailable()) {
      MyBean myBean = beanHandle.get();
      myBean.doSomething();
    } else {
      System.out.println("MyBean not found");
    }

However, when running this code, you might encounter the following error:

================================================================================
CDI: programmatic lookup problem detected
-----------------------------------------
At least one bean matched the required type and qualifiers but was marked as unused and removed during build

Stack frame: 
Required type: interface org.acme.beans.MyBean
Required qualifiers: [@jakarta.enterprise.inject.Default()]
Removed beans:
        - CLASS bean  [types=[interface org.acme.beans.MyBean, class org.acme.beans.BeanImpl1], qualifiers=null]
Solutions:
        - Application developers can eliminate false positives via the @Unremovable annotation
        - Extensions can eliminate false positives via build items, e.g. using the UnremovableBeanBuildItem
        - See also https://quarkus.io/guides/cdi-reference#remove_unused_beans
        - Enable the DEBUG log level to see the full stack trace
================================================================================

Understanding the Problem

Quarkus uses build-time optimization to improve startup time and reduce memory usage. Part of this optimization is unused bean removal, where Quarkus analyzes your application at build time and removes beans that appear to be unused.

When using Arc.container().instance(MyBean.class) for programmatic lookup, you're encountering this error because Quarkus couldn't detect at build time that BeanImpl1 (which implements MyBean) would be needed at runtime. The bean was considered "unused" and was removed during the build process, and when you try to look it up at runtime, it's no longer available.

Solutions to Fix the Problem

1.Mark Your Bean as @Unremovable

@Unremovable
@ApplicationScoped
public class BeanImpl1 implements MyBean {
    // Implementation
}

2.Disable Unused Bean Removal (in application.properties)

quarkus.arc.remove-unused-beans=false

Using the Standard CDI API

    MyBean myBean = CDI.current().select(MyBean.class).get();
    myBean.doSomething();

Using @Inject with Instance<T>

For CDI-managed beans, using Instance<T> offers a flexible approach to dynamic lookup.

  @Inject
  Instance<MyBean> myBeanInstance;

  if (myBeanInstance.isResolvable()) {
      MyBean bean = myBeanInstance.get();
      bean.doSomething();
    }

Using @Any with Instance<T>

The @Any qualifier extends Instance<T> capabilities to access all beans of a type regardless of qualifiers, which is particularly useful for plugins or strategy patterns.

Let's create an additional implementation for our bean:

@ApplicationScoped
public class BeanImpl2 implements MyBean{

  @Override
  public String getName() {
    return "Specific BeanImpl2";
  }

  @Override
  public void doSomething() {
    System.out.println("BeanImpl2 (specificBean) is working!");
  }
}

Now we can look up all beans:

  @Inject
  @Any
  Instance<MyBean> allBeans;

  System.out.println("Found " + allBeans.stream().count() + "implementations");

  allBeans.forEach(bean -> {
    System.out.println("Processing bean: " + bean.getClass().getSimpleName());
      bean.doSomething();
    });

Output:

Found 2 implementations
Processing bean: BeanImpl1_ClientProxy
BeanImpl1 is working!
Processing bean: BeanImpl2_ClientProxy
BeanImpl2 (specificBean) is working!

Conclusion

Programmatic bean lookup in Quarkus provides flexibility for dynamic bean resolution and working with multiple implementations. While @Inject should remain your primary approach for accessing beans in most scenarios, the methods described in this article offer powerful alternatives for specific use cases.

When choosing a lookup method, consider:

Build-time optimization impacts (remember the @Unremovable annotation when needed)
Whether you need to access multiple beans or specific implementations
The context in which you're performing the lookup (CDI-managed vs. non-CDI-managed.

How to Build an AI Application with Quarkus and LangChain4j

Ivelin Yanev — Sun, 16 Mar 2025 19:06:09 +0000

In light of the LLM boom over the past year, we can conclude that virtually everyone has, for one reason or another, used a model from one of the major providers, such as Facebook, Google, Microsoft, Amazon, and others.

From a technical standpoint, it's fascinating to explore how we can develop software that leverages LLMs. In this comprehensive guide, we'll walk through building an AI application using Quarkus and LangChain4j.

Prerequisites

Basic knowledge of Java, Quarkus and LangChain4j
JDK 17 or newer
Maven or Gradle
An API key from an LLM provider (e.g., OpenAI)

What is LangChain4j?

LangChain4j is a powerful Java framework designed to simplify the integration of Large Language Models (LLMs) and other AI capabilities into Java applications. As a Java port of the popular LangChain project, it provides developers with a robust and type safe way to build AI-powered applications while maintaining the reliability and structure that Java is known for.

LangChain4j - Overview

The framework is built around composable components that can be mixed and matched to create custom AI solutions:

Language Models
Memory Systems
Document Loaders
Vector Stores
Embeddings
Tools and Agents

LangChain4j - Components

Core Components Explained

Language Models
Language Models (LLMs) are the AI engines that process and generate text, with LangChain4j primarily supporting OpenAI's GPT models and other compatible LLMs. They handle tasks like text generation, summarization, and structured data extraction through a unified interface.
Memory Systems
Memory Systems maintain conversation history and context between interactions, allowing for coherent multi-turn conversations. The can be configured to store messages in various backends (in-memory, database) and manage conversation context with customizable retention policies.
Document Loaders
Document Loaders provide interfaces to read and process various file formats (PDF, TXT, HTML) into a standardized format for AI processing. They handle the complexities of file reading and initial text extraction, making documents ready for further processing.
Vector Stores
Vector Stores are specialized databases that store and retrieve text embeddings, enabling semantic search capabilities. They support various backends like Pinecone or local storage, making it efficient to find semantically similar content.
Embeddings
Embeddings convert text into numerical vectors that capture semantic meaning, essential for similarity searches and content comparison. They work with various embedding models (like OpenAI's embeddings) to transform text into mathematical representations.
Tools and Agents
Tools and Agents are components that extend LLM capabilities by providing specific functionalities (like calculations or API calls) and autonomous decision-making abilities. They allow LLMs to interact with external systems and perform complex tasks through defined interfaces.

Configuration Setup

Adding LangChain4j extensions:

<dependency>
    <groupId>io.quarkiverse.langchain4j</groupId>
    <artifactId>quarkus-langchain4j</artifactId>
    <version>${langchain4j.version}</version>
</dependency>
<dependency>
    <groupId>dev.langchain4j</groupId>
    <artifactId>langchain4j-core</artifactId>
    <version>${langchain4j-core.version}</version>
</dependency>

OpenAI Configuration

quarkus.langchain4j.openai.api-key=${OPENAI_API_KEY}
quarkus.langchain4j.openai.timeout=60s
quarkus.langchain4j.openai.chat-model.temperature=0.7

Simple AI Service

The Simple AI Service is designed for straightforward AI tasks, such as generating text based on a given prompt. In your project, this is implemented in the SimplePostAiService interface

How It Works

Interface Definition:

The @RegisterAiService annotation marks the interface as an AI service that LangChain4j will implement.
The writeNews method takes a topic and length as input and generates a news article.

Example Code:

@RegisterAiService
public interface SimplePostAiService {
    String writeNews(String topic, int length);
}

Use Case:

This service can be used to generate short news articles or summaries based on a specific topic. For example, calling writeNews("technology", 100) might generate a 100-word article about the latest tech trends.

Integration:

The service is injected into the GreetingResource class and exposed via a REST endpoint:

@GET
@Path("/news/{topic}/{length}")
public String writeNews(@PathParam("topic") String topic, @PathParam("length") int length) {
    return simplePostAiService.writeNews(topic, length);
}

Advanced AI Service with Memory

Memory is a fundamental component in LangChain4j that plays a crucial role in creating meaningful and contextual interactions between users and AI models. Unlike traditional request-response systems where each interaction stands alone, memory enables applications to maintain context and create more natural, flowing conversations.

The primary purpose of memory in LangChain4j is to maintain conversation history and context. Without memory, each interaction with an AI model would be isolated, forcing users to repeatedly provide context or background information. For example, in a conversation about travel planning, memory allows the AI to remember previously mentioned destinations, dates, or preferences without requiring the user to restate this information in each message.

The Advanced AI Service with Memory enhances AI interactions by maintaining context across multiple requests. This is implemented in the AdvancedPostAiService interface.

Interface Definition:

The @RegisterAiService annotation marks the interface as an AI service.
The chatWithMailPostBotWithMemory method uses the @MemoryId annotation to associate a memory with a specific user.

Example Code:

@RegisterAiService
public interface AdvancedPostAiService {
    String chatWithMailPostBotWithMemory(@MemoryId String userId, String text);
}

Use Case:

This service is ideal for applications like chatbots, where maintaining context across conversations is crucial. For example, a user could ask, "What’s the weather today?" and follow up with "What about tomorrow?" without needing to repeat the location.

Integration:

The service is injected into the GreetingResource class and exposed via a REST endpoint

@GET
@Path("/mail-with-memory")
public String writeNewsWithMemory(
    @QueryParam("text") String text, @QueryParam("userId") String userId) {
    return advancedPostAiService.chatWithMailPostBotWithMemory(userId, text);
}

Understanding Message Types in LangChain4j

System Messages (@SystemMessage)
System messages serve as instructions or context-setting
information for the AI model. They help define the behavior, role,
or constraints for the AI's responses. These messages are not
typically visible to end users but are crucial for controlling how
the AI behaves.
User Messages (@UserMessage)
User messages represent the actual input from users interacting with the AI system. These messages contain the queries, statements, or requests that the AI needs to respond to. They form the core of the interaction between users and the AI.

Understanding Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation (RAG) is an advanced approach that combines the power of large language models (LLMs) with a knowledge base of specific information. Instead of relying solely on an LLM's pre-trained knowledge, RAG retrieves relevant information from a custom document collection before generating responses. This approach significantly improves the accuracy and reliability of AI generated responses by grounding them in specific, verified information.

Why RAG is Needed?

Traditional LLMs face several limitations that RAG helps to overcome:

LLMs have knowledge limited to their training data cutoff date. RAG allows them to access current information.
LLMs might generate plausible but incorrect information. RAG reduces this by providing specific, accurate source material.
Companies often need responses based on their specific documentation, policies, or knowledge base. RAG enables this customization.
With RAG, responses can be traced back to source documents, making the system more transparent and trustworthy.

Key Components of RAG: A Step-by-Step Process

Document Processing
The first crucial step in implementing RAG involves preparing and processing your documents. This starts with loading documents from various sources such as files, databases, or web content. Documents are then split into smaller, manageable chunks that are optimal for processing and retrieval. The splitting process must balance chunk size with context preservation, ensuring that each segment maintains meaningful information while staying within the model's token limits. This stage also includes cleaning and standardizing the text, removing irrelevant information, and structuring the content appropriately.
Embedding Generation
Once documents are processed and split, the next step is converting these text chunks into embeddings. Embeddings are numerical representations of text that capture semantic meaning, allowing for efficient similarity searches. This process uses embedding models (such as those from OpenAI or other providers) to transform text into high-dimensional vectors. These embeddings serve as the foundation for semantic search capabilities, enabling the system to find relevant information based on meaning rather than just keywords.
Vector Storage
The third key component involves storing and organizing these embeddings in a vector database. Vector stores are specialized databases designed to efficiently store and retrieve high-dimensional vectors. They index the embeddings in a way that enables fast similarity searches, which is crucial for retrieving relevant context during query processing. The choice of vector store depends on factors like data volume, performance requirements, and scaling needs. Popular options include Pinecone, Weaviate, or local vector stores for smaller applications.
Retrieval Mechanism
When a query is received, the system converts it into an embedding and searches the vector store for similar vectors. This retrieval process identifies the most relevant document chunks based on semantic similarity. The mechanism can be enhanced with filters, metadata search, and hybrid approaches combining semantic and keyword search for better accuracy.
Context Assembly
After retrieving relevant chunks, the system assembles them into a coherent context. This involves selecting the most pertinent information, ordering it logically, and ensuring it fits within the model's context window. The assembly process might include deduplication, relevance ranking, and context summarization to optimize the input for the language model.
Response Generation
The final step involves combining the assembled context with the original query to create a prompt for the language model. This prompt engineering ensures the model understands how to use the provided context to generate accurate, relevant responses. The system then processes this through the LLM to generate the final response, which should be grounded in the retrieved information while maintaining natural language fluency.

Retrieval-Augmented Generation (RAG)

The Document Retrieval Service leverages LangChain4j’s embedding models and document storage to retrieve relevant information.

This is implemented in the DocsAiService interface.

Interface Definition:

The @RegisterAiService annotation marks the interface as an AI service.
The chatWithQuarkusDocsBot method takes a user prompt and retrieves relevant information from a document store.

Example Code:

@RegisterAiService
public interface DocsAiService {
    String chatWithQuarkusDocsBot(@UserMessage String prompt);
}

Embedding Ingesters:

Embedding ingesters are responsible for processing and storing documents in a way that makes them searchable. They split documents into chunks, generate embeddings, and store them in a vector database.

Document Splitting:

DocumentSplitter splitter = DocumentSplitters.recursive(500, 0);

Embedding Generation:

List<Embedding> embeddings = embeddingModel.embed(chunks);

Storage in Vector Store:

EmbeddingStoreIngestor ingestor = EmbeddingStoreIngestor.builder()
    .embeddingStore(embeddingStore)
    .embeddingModel(embeddingModel)
    .documentSplitter(splitter)
    .build();
ingestor.ingest(documents);

The DocumentsRetriever class is a Quarkus-managed bean that provides a RetrievalAugmentor for integrating retrieval-augmented generation (RAG) into your application. It is annotated with @ApplicationScoped, meaning it is instantiated once and shared across the application. The class constructs an EmbeddingStoreContentRetriever using a provided RedisEmbeddingStore and EmbeddingModel. This retriever fetches relevant documents based on semantic similarity. It configures a DefaultRetrievalAugmentor with the retriever, which can be used to augment LLM responses with contextually relevant information. The maxResults parameter is set to 20, limiting the number
of documents retrieved per query. The get() method returns the configured RetrievalAugmentor, which can be injected into other components to enhance LLM interactions.

@ApplicationScoped
public class DocumentsRetriever implements Supplier<RetrievalAugmentor> {

  private final RetrievalAugmentor augmentor;

  public DocumentsRetriever(final RedisEmbeddingStore store, final EmbeddingModel model) {
    EmbeddingStoreContentRetriever contentRetriever =
        EmbeddingStoreContentRetriever.builder()
            .embeddingModel(model)
            .embeddingStore(store)
            .maxResults(20)
            .build();

    augmentor = DefaultRetrievalAugmentor.builder().contentRetriever(contentRetriever).build();
  }

  @Override
  public RetrievalAugmentor get() {
    return augmentor;
  }
}

You can explore the complete source code for this project in the GitHub repository.

Conclusion

Building AI applications with Quarkus and LangChain4j provides a powerful combination for creating intelligent, production-ready solutions. Through this guide, we've explored the essential components and implementation patterns, from basic AI services to advanced features like RAG and embedding ingesters.

A Deep Dive into Graceful Shutdown in Go

Ivelin Yanev — Mon, 10 Feb 2025 19:02:17 +0000

When developing web services in Go, implementing proper shutdown handling is crucial for maintaining system reliability and data integrity. A graceful shutdown ensures that your service can terminate safely while completing existing operations, rather than abruptly stopping and potentially leaving tasks unfinished.

Let's understand the core mechanism of graceful shutdown through a practical example.

The Core Concept

Graceful shutdown involves coordinating between incoming termination signals, ongoing operations, and resource cleanup. In Go, this is typically achieved using signal handling, goroutines, and context-based cancellation. Here's a minimal implementation that demonstrates these concepts:

package main

import (
    "context"
    "errors"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"
)

func main() {
    mux := http.NewServeMux()

    mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("Hello, World!"))
    })

    server := &http.Server{
        Addr:         ":8787",
        Handler:      mux,
        ReadTimeout:  15 * time.Second,
        WriteTimeout: 15 * time.Second,
        IdleTimeout:  60 * time.Second,
    }

    serverError := make(chan error, 1)

    go func() {
        log.Printf("Server is running on http://localhost%s", server.Addr)
        if err := server.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
            serverError <- err
        }
    }()

    stop := make(chan os.Signal, 1)
    signal.Notify(stop, os.Interrupt, syscall.SIGTERM)

    select {
        case err := <-serverError:
            log.Printf("Server error: %v", err)
        case sig := <-stop:
            log.Printf("Received shutdown signal: %v", sig)
    }

    log.Println("Server is shutting down...")

    ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
    defer cancel()

    if err := server.Shutdown(ctx); err != nil {
        log.Printf("Server shutdown error: %v", err)
        return
    }

    log.Println("Server exited properly")
}

How It Works

The implementation starts by running the HTTP server in a separate goroutine. This allows the main goroutine to handle shutdown signals without blocking the server's operation. We create two channels: one for server errors and another for OS signals.

The signal handling channel (stop) is buffered to ensure we don't miss the shutdown signal even if we're not immediately ready to process it. We register this channel to receive both SIGINT (Ctrl+C) and SIGTERM signals, which are common ways to request server shutdown.

We use a select statement to wait for either a server error or a shutdown signal. This creates a blocking point where our program waits for something to happen. When either occurs, we begin the shutdown process.

The shutdown itself uses a context with timeout to ensure we don't hang indefinitely. The server.Shutdown method stops accepting new connections immediately while allowing existing requests to complete. If ongoing requests take longer than our timeout (15 seconds in this case), the shutdown will be forced.

The Shutdown Sequence

When a shutdown signal is received, the following sequence occurs:

First, the server stops accepting new connections. This prevents new requests from starting while we're trying to shut down.

Next, the server waits for all existing connections to complete their current requests. This is the "graceful" part of the shutdown - we're giving ongoing operations a chance to finish properly.

If any requests are still running when the context timeout expires, they will be forcefully terminated. This prevents the server from hanging indefinitely if some requests take too long.

Finally, once all requests are complete or the timeout is reached, the server exits and the program can terminate cleanly.

This visual representation helps understand how different components interact during the shutdown process and how the program coordinates between normal operation and graceful termination.

Real-World Considerations

In production environments, you might need to handle additional cleanup tasks during shutdown. This could include closing database connections, saving state, or cleaning up temporary files. These operations should be added before the server.Shutdown call and should respect the same context timeout.

It's also worth noting that the 15-second timeout we've used is arbitrary. In practice, you should choose a timeout value that makes sense for your application's typical request duration and cleanup needs.

Testing the Implementation

You can test this implementation by running the server and sending it a shutdown signal. For example, pressing Ctrl+C in the terminal will trigger the graceful shutdown process. You should see the shutdown messages in the logs, and any in-flight requests should complete before the server exits.

Key Benefits of This Implementation

Non-blocking Operation
- Server runs in a separate goroutine
- Main goroutine remains responsive to signals
Controlled Shutdown
- Existing requests are allowed to complete
- New requests are rejected
- Timeout prevents hanging
Error Handling
- Captures both server errors and shutdown errors
- Distinguishes between normal shutdown and errors

Conclusion

Graceful shutdown is a crucial pattern for production-ready Go services. This implementation provides a solid foundation that ensures your service can handle termination requests properly, complete ongoing work, and exit cleanly. While the concept is straightforward, proper implementation ensures reliability and maintains a good user experience even during service termination.

The key to understanding graceful shutdown is recognizing that it's about managing the transition from running to stopped states in a way that preserves system integrity and user experience. This implementation demonstrates how Go's concurrency primitives and standard library make this possible in a clean and efficient way.

Mastering Pointers in Go: Enhancing Safety, Performance, and Code Maintainability

Ivelin Yanev — Sun, 12 Jan 2025 19:00:13 +0000

Pointers play a crucial role in Go programming, offering a powerful tool for developers to directly access and manipulate the memory addresses of variables. Unlike traditional variables that store actual data values, pointers store the memory locations where these values are kept. This unique capability enables pointers to modify the original data in memory, providing an efficient way to handle data and optimize program performance.

Memory addresses, represented in hexadecimal format (e.g., 0xAFFFF), serve as the foundation for pointers. When you declare a pointer variable, it is essentially a special type of variable designed to hold a memory address of another variable, not the data itself.

Take, for example, a pointer p in Go, which contains the reference 0x0001, pointing directly to the memory address of another variable x. This relationship allows p to directly interact with a's value, showcasing the power and utility of pointers in Go.

Here's a visual representation of how pointers work:

Deep Dive into Pointers in Go

To declare a pointer in Go, the syntax var p *T is used, where T represents the type of the variable to which the pointer will refer. Consider the following example where p is a pointer to an int variable:

1. var a int = 10
2. var p *int = &a

Here, p stores the address of a, and through pointer dereferencing (*p), it can access or modify the value of a. This mechanism is fundamental in Go for efficient data manipulation and memory management.

Let's see a basic example:

func main() {
    x := 42
    p := &x
    fmt.Printf("x: %v\n", x)
    fmt.Printf("&x: %v\n", &x)
    fmt.Printf("p: %v\n", p)
    fmt.Printf("*p: %v\n", *p)

    pp := &p
    fmt.Printf("**pp: %v\n", **pp)
}

Output

Value of x: 42
Address of x: 0xc000012120
Value stored in p: 0xc000012120
Value at the address p: 42
**pp: 42

Pointers in Go don’t function the way they do in C/C++

A common misconception about when to use pointers in Go stems from comparing pointers in Go directly with pointers in C. It's crucial to understand the distinctions between the two to grasp how pointers function within each language's ecosystem. Let's delve into these differences:

No Pointer Arithmetic

Unlike C, where pointer arithmetic allows direct manipulation of memory addresses, Go does not support pointer arithmetic. This intentional design choice by Go leads to several significant advantages:

Prevents Buffer Overflow Vulnerabilities: By eliminating pointer arithmetic, Go inherently reduces the risk of buffer overflow vulnerabilities, a common security issue in C programs that allows attackers to execute arbitrary code.
Makes Code Safer and More Maintainable: Without the complexity introduced by direct memory manipulation, Go code becomes easier to understand, safer to work with, and inherently more maintainable. Developers can focus on the logic of their applications rather than the intricacies of memory management.
Reduces Memory-Related Bugs: The elimination of pointer arithmetic minimizes common pitfalls such as memory leaks and segmentation faults, making Go programs more robust and stable.
Simplifies Garbage Collection: Go's approach to pointers and memory management streamlines garbage collection, as the compiler and runtime have a clearer understanding of object lifecycles and memory usage patterns. This simplification leads to more efficient garbage collection and, subsequently, better performance.

By eliminating pointer arithmetic, Go safeguards against the misuse of pointers, resulting in more reliable and maintainable code.

Memory Management and Dangling Pointers

In Go, memory management is significantly simplified compared to languages like C, thanks to its garbage collector.

// In Go - No manual memory management is needed
func createPerson() *Person {
    p := &Person{Name: "John"}
    return p // Safe: Go's garbage collector handles cleanup automatically
}

// In C
struct Person* createPerson() {
    struct Person* p = malloc(sizeof(struct Person));
    // It's necessary to remember to free(p) later to avoid memory leaks
    return p;
}

No Manual Memory Allocation/Deallocation: Go abstracts away the complexities of memory allocation and deallocation through its garbage collector, simplifying programming and minimizing errors.
Absence of Dangling Pointers: Dangling pointers, which occur when a memory address referenced by a pointer is freed or reallocated without updating the pointer, are a common source of bugs in manual memory management systems. Go's garbage collector ensures that objects are only cleaned up when there are no existing references to them, effectively preventing dangling pointers.
Prevention of Memory Leaks: Memory leaks, often caused by forgetting to deallocate memory that is no longer needed, are significantly mitigated in Go. While objects with reachable pointers are not freed in Go, thereby preventing leaks due to lost references, in C, programmers must diligently manage memory manually to avoid such issues.

Nil Pointer Behavior

In Go, attempting to dereference a nil pointer results in a panic. This behavior places an additional responsibility on developers to carefully handle all possible cases of nil references and avoid unintended modifications. While this can increase the overhead of code maintenance and debugging, it also serves as a safeguard against some types of bugs:

type Student struct {
    Name string
    Age  int
}

func main() {
    var student *Student
    fmt.Println(student.Name)
    fmt.Println(student.Age)
}

The output indicates a panic due to an invalid memory address or nil pointer dereference:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x48f6fa]

Because student is a nil pointer, not associated with any valid memory address, attempting to access its fields (Name and Age) leads to a runtime panic.

In contrast, in C, dereferencing a null pointer is considered unsafe. Uninitialized pointers in C point to random parts of memory (undefined), making them even more hazardous. Dereferencing such an undefined pointer could mean the program continues operating with corrupt data, leading to unpredictable behavior, data corruption, or even worse outcomes.

This approach does come with its trade-offs - it results in a more complex Go compiler compared to the C compiler, which is relatively simpler. Consequently, this complexity can sometimes make Go programs appear slower in execution compared to their C counterparts.

Common misconception: "Pointers are always faster"

A prevalent notion suggests that utilizing pointers can enhance your application's speed by minimizing data copying. This concept stems from Go's architecture as a garbage-collected language. When a pointer is passed to a function, Go conducts Escape Analysis to determine whether the variable in question should reside on the stack or be allocated on the heap. While essential, this process introduces a certain level of overhead. Moreover, if the outcome of the analysis dictates heap allocation for the variable, further time is consumed during garbage collection (GC) cycles. This dynamic illustrates that while pointers reduce direct data copying, their impact on performance is nuanced, influenced by the underlying mechanisms of memory management and garbage collection in Go.

Escape analysis

Go employs escape analysis to determine the dynamic scope of values within its environment. This process is integral to how Go manages memory allocation and optimization. At its core, the goal is to allocate Go values within the function stack frame whenever possible. The Go compiler takes on the task of predetermining which memory allocations can be safely deallocated, subsequently emitting machine instructions to handle this cleanup process efficiently.

The compiler conducts a static code analysis to ascertain whether a value should be allocated on the stack frame of the function that constructs it, or if it must "escape" to the heap. It's important to note that Go does not provide any specific keywords or functions that allow developers to explicitly direct this behavior. Rather, it is the conventions and patterns in how the code is written that influence this decision-making process.

A value may escape to the heap for several reasons. If the compiler cannot determine the size of a variable, if the variable is too large for the stack, or if the compiler cannot reliably discern whether the variable will be used after the function concludes, the value will likely be allocated on the heap. Additionally, if the function stack frame becomes obsolete, this can also trigger a value to escape to the heap.

However, can we definitively know whether a value is stored on the heap or the stack? The reality is that only the compiler has full visibility into where a value is ultimately stored at any given time.

Whenever a value is shared beyond the immediate scope of a function's stack frame, it will be allocated on the heap. This is where escape analysis algorithms come into play, identifying these scenarios to ensure the program remains integral. This integrity is crucial for maintaining accurate, consistent, and efficient access to any value within the program. Escape analysis is, therefore, a fundamental aspect of Go's approach to memory management, optimizing for performance and safety of the executed code.

Look at this example to learn the basic mechanics behind escape analysis:

type student struct {
    name  string
    email string
}

func main() {
    st1 := createStudent1()
    st2 := createStident2()

    println("st1", &st1, "st2", &st2)
}

//go:noinline
func createStudent1() student {
    st := student{
        name:  "Name1",
        email: "name1@mail.com",
    }

    println("V1", &st)
    return st
}

//go:noinline
func createStudent2() *student {
    st := student{
        name:  "Name2",
        email: "anme2@mail.com",
    }

    println("V2", &st)
    return &st
}

The //go:noinline directive prevents inlining these functions, ensuring that our examples show clear calls for escape analysis illustration purposes.

We've defined two functions, createStudent1 and createStudent2, to demonstrate different outcomes of escape analysis. Both versions attempt to create a user instance but differ in their return types and how they handle memory.

createStudent1: Value Semantics

In createStudent1, a student instance is created and returned by value. This means a copy of st is made and passed up the call stack when the function returns. The Go compiler determines that &st does not escape to the heap in this scenario. The value exists on the stack frame of createStudent1 and a copy is made for the stack frame of main.

Figure 1 – Value Semantics
createStudent2: Pointer Semantics

In contrast, createStudent2 returns a pointer to a student instance, designed to share the student value across stack frames. This scenario underscores the crucial role of escape analysis. Sharing a pointer risks accessing invalid memory if not managed correctly.

If the situation described in Figure 2 actually occurred, it would present a significant integrity issue. The pointer would be targeting memory down the call stack that has been invalidated. The subsequent function call by main would result in this previously pointed-to memory being re-allocated and re-initialized.

Figure 2 – Pointer Semantics

Here, escape analysis steps in to preserve the integrity of the system. Given this scenario, the compiler determines that it is unsafe to allocate the student value within the stack frame of createStudent2. Consequently, it opts to allocate this value on the heap instead, a decision made at the moment of construction.

Functions have direct access to memory within their own frames via the frame pointer. However, accessing memory outside their frame requires indirect access through a pointer. This implies that values destined to escape to the heap are accessed indirectly as well.

In Go, the process of constructing a value does not inherently indicate where in memory that value resides. It is only upon executing the return statement that it becomes apparent a value must escape to the heap.

Thus, following the execution of such a function, one could conceptualize the stack in a manner reflecting these dynamics.

You could visualize the stack looking like this after the function call.

The st variable on the stack frame for createStudent2, represents a value that is on the heap, not the stack. This means using st to access the value, requires pointer access and not the direct access the syntax is suggesting.

To understand the compiler's decisions regarding memory allocation, you can request a detailed report. This is achieved by using the -gcflags switch with the -m option in a go build command.

$ go build -gcflags "-m -m"

Consider the output from this command:

./main.go:16:6: cannot inline createStudent1: marked go:noinline
./main.go:27:6: cannot inline createStudent2: marked go:noinline
./main.go:8:6: cannot inline main: function too complex: cost 133 exceeds budget 80
./main.go:28:2: st escapes to heap:
./main.go:28:2:   flow: ~r0 = &st:
./main.go:28:2:     from &st (address-of) at ./main.go:34:9
./main.go:28:2:     from return &st (return) at ./main.go:34:2
./main.go:28:2: moved to heap: st

This output showcases the compiler's escape analysis results. Here's a breakdown:

The compiler reports that it cannot inline certain functions (createUser1, createUser2, and main) due to specific directives (go:noinline) or because they are non-leaf functions.
For createUser1, the output indicates the reference to st within the function doesn't escape to the heap. This means the object's lifetime is confined to the stack frame of the function. In contrast, during createUser2, it’s noted that &st escapes to the heap. This is explicitly linked to the return statement, which causes the variable u, assigned inside the function, to be moved to heap memory. This is essential as the function is returning a reference to st, necessitating its existence beyond the function scope.

Garbage collection

Go incorporates a built-in garbage collection mechanism that automatically handles memory allocation and deallocation, contrasting sharply with languages like C/C++ which require manual management of memory. While garbage collection alleviates developers from the complexities of memory management, it introduces latency as a trade-off.

One notable characteristic in Go is that passing pointers might be slower than passing values directly. This behavior is attributed to Go's nature as a garbage-collected language. Whenever a pointer is passed to a function, Go performs an escape analysis to determine whether the variable should reside on the heap or the stack. This process incurs overhead, and variables allocated on the heap further exacerbate latency during garbage collection cycles. Conversely, variables constrained to the stack bypass the garbage collector entirely, benefiting from the simple and efficient push/pop operations associated with stack memory management.

Memory management on the stack is inherently faster due to its straightforward access pattern, where memory allocation and deallocation are accomplished by merely incrementing or decrementing a pointer or an integer. In contrast, heap memory management involves more complex bookkeeping for allocations and deallocations.

When to use pointers in Go

Copying Large Structs
While it may seem that pointers are less performant due to the overhead of garbage collection, they prove advantageous with large structs. In such cases, the efficiency gained by avoiding the duplication of large datasets can outweigh the overhead introduced by garbage collection.
Mutability
To mutate a variable passed to a function, passing a pointer is essential. The default pass-by-value approach means any modifications are made on a copy, thus not affecting the original variable in the calling function.
API Consistency
Employing pointer receivers consistently across your API can maintain its uniformity, especially beneficial if at least one method requires a pointer receiver to mutate the struct.

Why I prefer values?

My preference for passing values over pointers is rooted in several key arguments, detailed below:

Fixed-sized Types
We're considering types such as integers, floats, small structs, and arrays here. These types maintain a consistent memory footprint, often comparable to or smaller than the size of a pointer on many systems. Utilizing values for these smaller, fixed size data types is not only memory efficient but also in line with best practices for minimizing overhead.
Immutability
Passing by value ensures the receiving function gets an independent copy of the data. This characteristic is crucial for avoiding unintended side effects; any modifications made within the function remain localized, preserving the original data outside the function's scope. The call-by-value mechanism thus serves as a protective barrier, ensuring data integrity.
Performance Benefits of Passing Values
Despite potential concerns, passing values is frequently fast and, in many scenarios, can outpace the use of pointers:
- Efficiency in Data Copying: For small-sized data, the act of copying can be more efficient than dealing with pointer indirection. The direct access to data mitigates the delays introduced by additional layers of memory dereferencing typical of pointer usage.
- Reduced Garbage Collector Workload: Directly passing values lessens the burden on the garbage collector. With fewer pointers to track, the garbage collection process becomes more streamlined, contributing to overall performance gains.
- Memory Locality: Data passed by value is often stored contiguously in memory. This arrangement favors the processor's caching mechanisms, allowing faster data access due to improved cache hits. The spatial locality of value-based data access lends itself to significant performance enhancements, particularly in computational-heavy operations.

Conclusion

In summary, pointers in Go provide direct memory address access, facilitating data manipulation and optimization not just for efficiency but also for programming pattern flexibility. Unlike pointer arithmetic in C, Go's approach to pointers is designed to enhance safety and maintainability, crucially supported by its built-in garbage collection system. While the understanding and use of pointers versus values in Go can deeply impact application performance and safety, Go's design inherently guides developers towards making informed and effective choices. Through mechanisms like escape analysis, Go ensures optimal memory management, balancing the power of pointers with the safety and simplicity of value semantics. This careful balance allows developers to craft robust, efficient Go applications with a clear understanding of when and how to use pointers to their advantage.

Building Robust REST Client with Quarkus: A Comprehensive Guide

Ivelin Yanev — Fri, 27 Dec 2024 17:00:59 +0000

Introduction

When building modern microservices applications, consuming external REST services is a common requirement. Quarkus provides powerful built-in support for REST clients, offering both traditional Java EE approaches and modern MicroProfile solutions. In this article, we'll focus on the MicroProfile REST Client approach, as it provides a more modern, maintainable, and feature-rich way to build REST clients in Quarkus applications. This approach aligns better with microservices architectures and provides superior integration with other MicroProfile features like Config, Fault Tolerance, and Metrics.

It offers:

Declarative, interface-based client definitions;
Type-safe request/response handling;
Automatic client generation;
Built-in integration with MicroProfile Configl
Better integration with CDI and other MicroProfile features;
Simplified error handling and resilience pattern.

Getting Started

Adding Required Dependencies

<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-rest-client-jackson</artifactId>
</dependency>

Creating Your First REST Client

    @RegisterRestClient(configKey = "simple-api")
     public interface SimpleRestClient {
     @GET
     @Path("/simple")
     Response simple();
  }

Configuration

Configure your REST client in application.properties:

quarkus.rest-client.simple-api.url=http://simple-api:8080

@RegisterRestClient: Registers the interface for CDI injection as a REST Client;
@Path, @GET: Standard Jakarta REST annotations defining service access;
configKey: Allows using a custom configuration key instead of the fully qualified interface name.

Real-World Example: Cryptocurrency API Client

Let's build a practical example using the CoinCap API to fetch cryptocurrency data.

@RegisterRestClient(configKey = "crypto-currency")
public interface CryptoCurrencyRestClient {
    @GET
    @Path("/assets")
    AssetData getAssets();

    @GET
    @Path("/assets/{id}/history")
    AssetHistoryData getAssetHistory(
        @PathParam("id") String id,
        @QueryParam("interval") String interval
    );
}

and the record classes:

public record AssetData(List<Crypto> data) {
  public record Crypto(String id,
                       String rank,
                       String symbol,
                       String name,
                       String supply,
                       String maxSupply,
                       String marketCapUsd,
                       String volumeUsd24Hr,
                       String priceUsd,
                       String changePercent24Hr,
                       String vwap24Hr,
                       String explorer) {

  }
}

public record AssetHistoryData(List<DataPoint> data) {
  public record DataPoint(
      String priceUsd,
      long time,
      String circulatingSupply,
      String date
  ) {}
}

You can easily inject and use the REST client within your Quarkus application. Implementation in a Resource:

@Path("/crypto")
public class CryptoCurrencyResource {
    private final CryptoCurrencyRestClient cryptoCurrencyRestClient;

    public CryptoCurrencyResource(@RestClient CryptoCurrencyRestClient cryptoCurrencyRestClient) {
        this.cryptoCurrencyRestClient = cryptoCurrencyRestClient;
    }

    @GET
    public AssetData getCurrency() {
        return cryptoCurrencyRestClient.getAssets();
    }
}

Advanced Features

Fault Tolerance

In distributed systems, network calls can fail. Quarkus provides robust fault tolerance features through SmallRye Fault Tolerance:

Add the dependency:

<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-smallrye-fault-tolerance</artifactId>
</dependency>

Implement fault tolerance patterns:

Retry Pattern

The @Retry annotation provides sophisticated retry capabilities when service calls fail. Here are the key configuration options:

@GET
@Path("/assets/{id}/history")
@Retry(
    maxRetries = 5,                    // Maximum number of retry attempts
    delay = 200,                       // Delay between retries in milliseconds
    jitter = 100,                      // Random variation added to delay
    retryOn = {                        // Specify which exceptions trigger retry
        IOException.class,
        TimeoutException.class
    },
    abortOn = {                        // Exceptions that immediately abort retry
        SecurityException.class,
        IllegalArgumentException.class
    },
    maxDuration = 2000                 // Maximum duration for all retry attempts
)
AssetHistoryData getAssetHistory(
    @PathParam("id") String id,
    @QueryParam("interval") String interval
);

You can also implement exponential backoff:

@Retry(
    maxRetries = 3,
    delay = 1000,
    delayUnit = ChronoUnit.MILLIS,
    exponential = true                 // Enable exponential backoff
)

Fallback Pattern

Fallbacks provide alternative behavior when all retry attempts fail. You can implement fallbacks in multiple ways:

@GET
@Path("/assets")
@Retry(maxRetries = 2)
@Fallback(fallbackMethod = "getDefaultAssets")
AssetData getAssets();

// Fallback method must have the same return type and parameters
private AssetData getDefaultAssets() {
    log.warn("Using fallback method for getAssets");
    return AssetData.builder()
        .timestamp(System.currentTimeMillis())
        .status("FALLBACK")
        .build();
}

Circuit Breaker Pattern

The Circuit Breaker pattern prevents cascade failures in distributed systems. Here's a detailed configuration:

@GET
@Path("/assets")
@CircuitBreaker(
    requestVolumeThreshold = 4,       // Minimum requests before CB can trip
    failureRatio = 0.5,               // Failure ratio to trip CB (50%)
    delay = 1000,                     // Time CB stays open before half-open
    successThreshold = 2,             // Successes needed to close CB
    failOn = {                        // Exceptions counted as failures
        IOException.class,
        TimeoutException.class
    },
    skipOn = {                        // Exceptions not counted as failures
        BusinessException.class
    }
)
@Fallback(fallbackMethod = "getDefaultAssets")  // Combine with fallback
AssetData getAssets();

Circuit Breaker States:

CLOSED: Normal operation, calls pass through
OPEN: Calls fail fast without executing
HALF-OPEN: Limited calls allowed to test service recovery

Dynamic Base URL Configuration

The @Url annotation provides flexible runtime URL configuration:

Basic Usage:

 @GET
 @Path("/assets")
 AssetData getAssets(@Url String url);
 // Usage in code
 client.getAssets("https://alternate-api.example.com/v2");

Combining with Path Parameters:

 @GET
 @Path("/assets/{id}")
 AssetData getAssetById(@Url String baseUrl, @PathParam("id") String id);
 // Usage
 client.getAssetById("https://api-backup.example.com", "bitcoin");

URL Priority:

 // Configuration priority (highest to lowest):
 // 1. @Url parameter
 // 2. Programmatic configuration
 // 3. application.properties
 // 4. Default URL (if specified in @RegisterRestClient)

 @RegisterRestClient(baseUri = "https://default.api.example.com")
 public interface DynamicRestClient {
    @GET
    @Path("/resource")
     Response getResource(@Url String url);
   }

Custom Headers Management

Header management is crucial for authentication, tracking, and protocol compliance:

Static Headers

@RegisterRestClient(configKey = "api-client")
@ClientHeaderParam(name = "API-Version", value = "1.0")
@ClientHeaderParam(name = "Client-ID", value = "${app.client.id}")
public interface ApiClient {
   @GET
   @Path("/data")
   Response getData();
}

Dynamic Headers using Methods:

@RegisterRestClient(configKey = "secure-api")
public interface SecureApiClient {
    @GET
    @Path("/protected-resource")
    @ClientHeaderParam(name = "Authorization", value = "{generateAuthHeader}")
    @ClientHeaderParam(name = "Request-Time", value = "{generateTimestamp}")
    Response getProtectedResource();

    default String generateAuthHeader() {
        return "Bearer " + TokenGenerator.generateToken();
    }

    default String generateTimestamp() {
        return String.valueOf(System.currentTimeMillis());
    }
}

Real-World Example: Integrating with Freesound API

Let's explore a practical example of building a REST client for the Freesound API, which demonstrates header-based authentication and parameter handling.

Creating the REST Client Interface

   @RegisterRestClient(configKey = "free-sound")
   @ClientHeaderParam(name = "Authorization", value ={getAuthorizationHeader}")
   public interface FreeSoundRestClient {

      default String getAuthorizationHeader() {
        Config config = ConfigProvider.getConfig();
        String apiKey = config.getConfigValue("free.sound.api.key").getValue();
        return "Token " + apiKey;
    }

    @GET
    @Path("/users/{username}")
    UserResponse getUsers(@PathParam("username") String username);    
}

Configuration Setup

 # Base URL configuration
 quarkus.rest-client.free-sound.url=https://freesound.org/apiv2

 # API Key configuration
 free.sound.api.key=${FREESOUND_API_KEY}

Response Data Models

  public record UserResponse(String url,
                       String username,
                       String about,
                       String home_page,
                       Avatar avatar,
                       LocalDateTime date_joined,
                       int num_sounds,
                       String sounds,
                       int num_packs,
                       String packs,
                       int num_posts,
                       int num_comments) {

 public record Avatar(
  String small,
  String medium,
  String large
  ) {}
}

Using the Client in a Resource

@Path("/freesound")
@Produces(MediaType.APPLICATION_JSON)
public class FreeSoundResource {
private final FreeSoundRestClient freeSoundClient;

public FreeSoundResource(@RestClient FreeSoundRestClient freeSoundClient) {
    this.freeSoundClient = freeSoundClient;
}

@GET
@Path("/users/{username}")
public UserResponse getUserProfile(@PathParam("username") String username) {
    return freeSoundClient.getUsers(username);
}
}

Conclusion

Quarkus REST Client provides a powerful and flexible way to consume REST services in your microservices architecture. By combining MicroProfile REST Client with Quarkus's fault tolerance features, you can build robust and reliable service communications.
The full source code for this article is available on GitHub

Optimizing Memory Usage in Go: Mastering Data Structure Alignment

Ivelin Yanev — Sun, 10 Nov 2024 16:18:03 +0000

Memory optimization is crucial for writing performant software systems. When a software has a finite amount of memory to work with, many issues can arise when that memory isn't used efficiently. That's why memory optimization is critical for better overall performance.

Go inherits many of C's advantageous features, but what I notice is that a large part of people who use it do not know the full power of this language. One of the reasons may be a lack of knowledge about how it works at a low level, or a lack of experience with languages like C or C++. I mention C and C++ because the foundations of Go are pretty much built on the wonderful features of C/C++. It's no coincidence that I will quote an interview of Ken Thompson at Google I/O 2012:

For me, the reason I was enthusiastic about Go is because just about the same time that we were starting on Go, I read (or tried to read) the C++0x proposed standard, and that was a convincer for me.

Today, we're going to talk about how we can optimize our Go program, and more specifically, how it would be good to use structs in Go. Let's first say what a structure is:

A struct is a user-defined data type that groups related variables of different types under a single name.

To fully understand where the problem lies, we will mention that modern processors do not read 1 byte at a time from the memory. How the CPU fetches the data or instructions which are stored in the memory?

In computer architecture, a word is a unit of data that a processor can handle in a single operation - generally the smallest addressable unit of memory. It's a fixed-size group of bits (binary digits). The word size of a processor determines its ability to handle data efficiently. Common word sizes include 8, 16, 32, and 64 bits. Some computer processor architectures support a half word, which is half the number of bits in a word, and a double word, which is two contiguous words.

In nowday the most common architectures is 32 bit and 64 bit. If you have 32 bit processor then it means it can access 4 bytes at a time which means word size is 4 bytes. If you have 64 bit processor the it can access 8 bytes at time which means word size is 8 bytes.

When we store the data in memory, each 32-bit data word has a unique address, as shown below.

Figure. 1 ‑ Word-Addressable Memory

We can read the data in memory and load it to one register using the load word (lw) instruction.

After know the above theoria let's see what is the practise. For the descripte the cases with structure data structure, I will demonstrate with C languge. A struct in C is a composite data type that allows you to group multiple variables together and store them in the same block of memory. As we said early the CPU access the data depend of the givemn architecture. Every data type in C will have alignment requirements.

So let's we have the following as simple structure:

// structure 1
typedef struct example_1 {
    char c;
    short int s;
} struct1_t;


// structure 2
typedef struct example_2 {
    double d;
    int s;
    char c;
} struct2_t;

And now try to calculate the size of the following structures:

Size of Structure 1 = Size of (char + short int) = 1 + 2 = 3.

Size of Structure 2 = Size of (double + int + char) = 8 + 4 + 1= 13.

The real sizes using a C program might surprise you.

#include <stdio.h>


// structure 1
typedef struct example_1 {
    char c;
    short int s;
} struct1_t;

// structure 2
typedef struct example_2 {
    double d;
    int s;
    char c;
} struct2_t;

int main()
{
    printf("sizeof(struct1_t) = %lu\n", sizeof(struct1_t));
    printf("sizeof(struct2_t) = %lu\n", sizeof(struct2_t));

    return 0;
}

Output

sizeof(struct1_t) = 4
sizeof(struct2_t) = 16

As we can see, the size of the structures is different from those we calculated.

What is the reason for that?

C and Go employ a technique known as "struct padding" to ensure data is appropriately aligned in memory, which can significantly affect performance due to hardware and architectural constraints. Data padding and alignment conform to the requirements of the system's architecture, mainly to optimize CPU access times by ensuring data boundaries align with word sizes.

Let's go through an example to illustrate how Go handles padding and alignment, consider the following struct:

type Employee struct {
  IsAdmin  bool
  Id       int64
  Age      int32
  Salary   float32
}

A bool is 1 byte, int64 is 8 bytes, int32 is 4 bytes and float32 is 4 bytes = 17 bytes(total).

Let's validate the struct size by examining the compiled Go program:

package main

import (
    "fmt"
    "unsafe"
)

type Employee struct {
    IsAdmin bool
    Id      int64
    Age     int32
    Salary  float32
}

func main() {

    var emp Employee

    fmt.Printf("Size of Employee: %d\n", unsafe.Sizeof(emp))
}

Output

Size of Employee: 24

The reported size is 24 bytes, not 17. This discrepancy is due to memory alignment. To understand how alignment works, we need to inspect the structure and visualize the memory which it ocupate.

Figure 2 - Unoptimized Memory Layout

The struct Employee will consume 8*3 = 24 bytes. Yo see the problem now, there are a lot of empty holes in the layout of Employee (those gaps created by the alignment rules are called “padding”).

Padding Optimization and Performance Impact

Understanding how memory alignment and padding can affect the performance of an application is crucial. Specifically, data alignment impacts the number of CPU cycles required to access fields within a struct. This influence arises mostly from CPU cache effects, rather than raw clock cycles themselves, as cache behavior depends heavily on data locality and alignment within memory blocks.

Modern CPUs fetch data from memory into a faster intermediary called cache, organized in fixed-size blocks (commonly 64 bytes). When data is well-aligned and localized within the same or fewer cache lines, the CPU can access it more quickly due to reduced cache loading operations.

Consider the following Go structures to illustrate poor versus optimal alignment:

// Poorly aligned struct
type Misaligned struct {
    Age        uint8  // Uses 1 byte, followed by 7 bytes of padding to align the next field
    PassportId uint64 // 8-byte aligned uint64 for the passport ID
    Children   uint16 //2-byte aligned uint16

// Well-aligned struct
type Aligned struct {
    Age        uint8  // Starting with 1 byte
    Children   uint16 // Next, 2 bytes; all these combine into a 3-byte sequence
    PassportId uint64 // Finally, an 8-byte aligned uint64 without needing additional padding
}

How Alignment Affects Performance

CPU reads data in word-size instead of byte-size. As I described in the begining a word in a 64-bit system is 8 bytes, while a word in a 32-bit system is 4 bytes. In short, CPU reads address in the multiple of its word size. To fetch the variable passportId, our CPU takes two cycles to access the data instead of one. The first cycle will fetch memory 0 to 7 and the subsequent cycle will fetch the rest. And this is inefficient- we need of data structure alignment.By simply aligning the data, computers ensure that the var passportId can be retrieved in ONE CPU cycle.

Figure 3 - Comparing Memory Access Efficiency

Padding is the key to achieving data alignment. Padding occurs because modern CPUs are optimized to read data from memory at aligned addresses. This alignment allows the CPU to read the data in a single operation.

Figure 4 - Simply aligning the data

Without padding, data may be misaligned, leading to multiple memory accesses and slower performance. Therefore, while padding might waste some memory, it ensures that your program runs efficiently.

Padding Optimization Strategies

Aligned struct consumes lesser memory simply because it possesses a better struct fields order compared to Misaligned. Because of padding, two 13 bytes data structures turn out to be 16 bytes and 24 bytes respectively. Hence, you save up extra memory by simply reordering your struct fields.

Figure 5 - Optimizing Field Order

Improperly aligned data can slow performance as the CPU might need multiple cycles to access misaligned fields. Conversely, correctly aligned data minimizes cache line loads, which is crucial for performance, especially in systems where memory speed is a bottleneck.

Let’s do a simple benchmark to prove it:

var AlignedArr []Aligned
var MisalignedArr []Misaligned

func init() {
    const sampleSize = 1000
    AlignedArr = make([]Aligned, sampleSize)
    MisalignedArr = make([]Misaligned, sampleSize)
    for i := 0; i < sampleSize; i++ {
        AlignedArr[i] = Aligned{Age: uint8(i % 256), Siblings: uint16(i), Children: uint64(i)}
        MisalignedArr[i] = Misaligned{Age: uint8(i % 256), PassportId: uint64(i), Children: uint16(i)}
    }
}

func traverseAligned() uint16 {
    var arbitraryNum uint16
    for _, item := range AlignedArr {
        arbitraryNum += item.Siblings
    }
    return arbitraryNum
}

func traverseMisaligned() uint16 {
    var arbitraryNum uint16
    for _, item := range MisalignedArr {
        arbitraryNum += item.Children
    }
    return arbitraryNum
}

func BenchmarkTraverseAligned(b *testing.B) {
    for n := 0; n < b.N; n++ {
        traverseAligned()
    }
}

func BenchmarkTraverseMisaligned(b *testing.B) {
    for n := 0; n < b.N; n++ {
        traverseMisaligned()
    }
}

Output

go test -bench=.
goos: linux
goarch: amd64
pkg: test-project
cpu: 11th Gen Intel(R) Core(TM) i9-11950H @ 2.60GHz
BenchmarkTraverseAligned-16              3022234               403.7 ns/op
BenchmarkTraverseMisaligned-16           4300167               299.1 ns/op
PASS
ok      test-project    3.195s

As you can see the traversing the Aligned indeed takes lesser time than its counterpart.

Padding’s added to make sure each struct field lines up properly in memory based on its needs, like we saw earlier. But while it enables efficient access, padding can also waste space if the fields ain’t ordered well.

Understanding how to properly align struct fields to minimize memory waste due to padding is important for efficient memory usage, especially in performance-critical applications. Below, I will provide an example with a poorly aligned structure and then show an optimized version of the same structure.

In a poorly aligned struct, fields are ordered without considering their sizes and alignment requirements, which can lead to added padding and increased memory usage:

// Badly aligned structure
type Person struct {
    Active   bool      // 1 byte + 7 bytes padding
    Salary   float64   // 8 bytes
    Age      int32     // 4 bytes + 4 bytes padding
    Nickname string    // 16 bytes (string is typically 16 bytes on a 64-bit system)
}

The total memory could hence be 1 (bool) + 7 (padding) + 8 (float64) + 4 (int32) + 4 (padding) + 16 (string) = 40 bytes.

An optimized structure arranges fields from largest to smallest size, significantly reducing or eliminating the need for additional padding:

// Well-aligned structure
type Person struct {
    Salary   float64   // 8 bytes
    Nickname string    // 16 bytes
    Age      int32     // 4 bytes
    Active   bool      // 1 byte + 3 bytes padding
}

The total memory would then neatly comprise 8 (float64) + 16 (string) + 4 (int32) + 1 (bool) + 3 (padding) = 32 bytes.

Let's proof the above:

package main

import (
    "fmt"
    "unsafe"
)

type PoorlyAlignedPerson struct {
    Active   bool
    Salary   float64
    Age      int32
    Nickname string
}

type WellAlignedPerson struct {
    Salary   float64
    Nickname string
    Age      int32
    Active   bool
}

func main() {
    poorlyAligned := PoorlyAlignedPerson{}
    wellAligned := WellAlignedPerson{}

    fmt.Printf("Size of PoorlyAlignedPerson: %d bytes\n", unsafe.Sizeof(poorlyAligned))
    fmt.Printf("Size of WellAlignedPerson: %d bytes\n", unsafe.Sizeof(wellAligned))
}

Output

Size of PoorlyAlignedPerson: 40 bytes
Size of WellAlignedPerson: 32 bytes

Reducing the structure size from 40 bytes to 32 bytes means a 20% reduction in memory usage per instance of Person. This can lead to considerable savings in applications where many such instances are created or stored, improving cache efficiency and potentially reducing the number of cache misses.

Conclusion

Data alignment is a pivotal factor in optimizing memory utilization and enhancing system performance. By arranging struct data correctly, memory usage becomes not only more efficient but also faster in terms of CPU read times, contributing significantly to overall system efficiency.

Effective Project Structuring for Microservices with Quarkus

Ivelin Yanev — Wed, 18 Sep 2024 18:12:31 +0000

In the ever-evolving landscape of software development, the adoption of microservices architecture is rapidly gaining traction due to its scalability, flexibility, and maintainability. Quarkus, as a Java-Native Kubernetes stack tailored for GraalVM and HotSpot, optimizes Java specifically for containers and enables it to become an effective platform for serverless, cloud, and Kubernetes environments.

Key to leveraging Quarkus effectively is understanding how to structure your project correctly. Proper project structuring not only enhances the manageability of the code but also plays a crucial role in the success of deploying microservices architecture.

Microservices architecture distinguishes itself by several defining characteristics that Quarkus enhances:

Small and Focused - each service in a microservices architecture is designed to perform a single, well-defined task, promoting simplicity and focus. Quarkus supports this by facilitating lightweight, bootable jars and native compilations, which are ideal for specific tasks.
Independent- independence allows services to be developed, deployed, and even scaled without reliance on other services. Quarkus's container-first philosophy ensures that each microservice can run in its isolated environment.
Loosely Coupled- services communicate via well-defined APIs, reducing the dependency on internal implementations. Quarkus encourages decoupling through standards like MicroProfile and JAX-RS, making services interaction seamless and efficient.
Scalable- efficiently managing increasing workloads is crucial. Quarkus's reactive programming capabilities enable services to be responsive and scalable under varying loads.
Flexible- the ability to adapt to changing conditions is facilitated by the dynamic scaling and resilience features of Quarkus, supporting on-the-fly adjustments in a cloud-native environment.
Resilient- Quarkus enhances the robustness of microservices with built-in fault tolerance and health-check capabilities, ensuring the services are resistant to failures and recover swiftly from interruptions.

Adopting Maven multi-module project structures has proven effective in managing large-scale Java applications, and Quarkus fully supports this modality. Let's detail the Maven multi-module structure:

A Maven multi-module project consists of an aggregator POM that orchestrates several submodules. This structure is beneficial for dividing a large application into manageable, independently deployable units that can be maintained and updated with ease. Each module typically has a focused responsibility and can be deployed independently from others.

Here’s a simplified layout based on a real-world application:

parent (aggregator pom.xml)
├── api (API interfaces and DTOs)
│   ├── pom.xml
│   └── src
├── core (Business logic)
│   ├── pom.xml
│   └── src
├── data-access (Data layer integration)
│   ├── pom.xml
│   └── src

I've tailored an optimized project structure specifically for Quarkus that enhances scalability, maintainability, and coherence across diverse development teams. The design, which I originally developed in 2018, has seen several refinements to better suit evolving technological trends. Initially, the structure wasn't specifically crafted with Quarkus in mind; however, throughout its application, it has proven exceptionally compatible with Quarkus's philosophy and coding practices. Let us examine this structure as implemented in an IDE like IntelliJ IDEA - though it could be effectively adapted to any other modern IDE.

You can find the full project structure in the GitHub repository

Starting from the top-level, the project is divided into several modules, each with a specific role:

api-dtos- this module is crucial for de-coupling the internal representations of data from what is exposed externally through APIs. By separating Data Transfer Objects (DTOs) from domain models, the module ensures that any changes- be they in database schema or business logic- do not directly affect the clients who consume these APIs. This separation enhances API stability and versioning.
commons- this utility module acts as a repository for shared logic, constants, and helpers that are used across multiple other modules in the project. Centralizing common functionalities in this way reduces duplication and fosters consistency throughout the application. It’s particularly helpful for storing utilities like data format converters, standard validators, or common business rules.
configuration- configuration management is streamlined in this module, which consolidates all configuration settings for the entire application in one place, simplifying access and management. This central repository of configurations ensures that settings are consistently applied and managed, simplifying environment shifts (such as from development to production) and minimizing the potential for errors due to misconfiguration. Beyond simple configuration files, this module is also the ideal location for defining custom managed beans that are essential across the application, such as ObjectMapper for JSON serializing and deserializing, custom deserializers, and other specialized beans.
db-entities- dedicated exclusively to database interactions, this module encapsulates all ORM classes or database schemas, ensuring they are separated from the business logic and API layers. Segregating entities in this manner helps maintain clean architecture principles by decoupling data access mechanisms from business rules processing, hence promoting a cleaner and more traceable codebase.
services- here lies the core business logic, encapsulated in service classes that operate on domain models and use db-entities for data persistence. Each service focuses on a specific business task, enhancing modularity and reusability. This separation also aids in isolating business operations from direct client interactions, which are handled by controller classes in other modules, thus adhering to the single responsibility principle.
resources- amid some debate, this module was named to reflect its alignment with JAX-RS, where classes annotated with JAX-RS are referred to as resources. This module handles the creation of REST endpoints, acting as the communication layer that interfaces with external clients. The name 'resources' was chosen over alternatives like 'rest' or 'controllers' to emphasize standardization and adherence to established RESTful principles as described in the Red Hat resource for building REST APIs with Quarkus. By aligning with standard terminology, the project aims to be intuitive for developers familiar with JAX-RS and REST conventions.
application- serves as an umbrella module that brings all the pieces together, ensuring they are correctly wired and ready to deploy.

Each of these modules plays a strategic role tailored to foster ease of maintenance, scalability, and robustness in a Cloud-Native microservice environment using Quarkus.

The above structure is based on the following Architecture layers which is mentioned in the RedHat resources

Much Abstraction is Too Much

When discussing project structures, a common question arises about the need for the layered approach I previously described. Here, I evoke the words of David J. Wheeler:

All problems in computer science can be solved by another level of indirection, except for the problem of too many layers of indirection. - [David J. Wheeler from The C++ Programming Language 4th edition.]

Mastering Data Routing in Apache Camel: Leveraging the Splitter Pattern

Ivelin Yanev — Sun, 15 Sep 2024 07:05:23 +0000

Hello again! In my upcoming articles, I plan to explore several key patterns provided by Apache Camel that are invaluable for your ETL processes. This current article focuses on the Splitter Enterprise Integration Pattern (EIP).

Overview

Messages passed to integration applications often arrive in a format that is less than ideal for immediate processing. Frequently, these are composite messages containing multiple elements, each requiring individual processing. This is where the Splitter pattern proves beneficial by dividing incoming messages into a sequence of more manageable messages.

The diagram below (Figure 1) illustrates the simplicity of the Splitter pattern- it takes a single message and splits it into multiple distinct messages.

Figure 1 – Splitter Pattern (enterpriseintegrationpatterns.com)

Using Splitter in Apache Camel

Implementing the Splitter pattern in Apache Camel is straightforward. Incorporate the split method in the route definition. This method requires an expression to dictate how the message should be split.

Commonly, you might need to split a list, set, map, array, or any other collection. For such cases, use the body method to retrieve the message body before splitting.

The Splitter behaves akin to a comprehensive iterator that processes each entry consecutively. The sequence diagram in (Figure 2) offers detailed insights into this iterator's operation.

Figure 2 – A sequence diagram

Upon receiving a message, the Splitter evaluates an Expression which returns the message body. This result is then utilized to create a java.util.Iterator.

By default, the split method divides the message body based on its value type, as illustrated below:

java.util.Collection: Splits each element from the collection.
java.util.Map: Splits each map entry.
Object[]: Splits each array element.
Iterator: Splits each iteration.
Iterable: Splits each iterable component.
String: Splits each string character.

The Splitter then utilizes the iterator till all data is processed. Each resultant message is a copy of the original with its body replaced by the portion from the iterator.

Let's explore how to employ the Splitter pattern in Apache Camel with practical examples.

Example 1: Splitting an ArrayList of Strings

Consider a route that splits an ArrayList of String and logs each element. Here's how you define this route:

@ApplicationScoped
public class SplitterRoute extends RouteBuilder {

    @Override
    public void configure() throws Exception {
        from("direct:split_start_1")
        .log("Before Split line ${body}")
        .split(body())
        .log("Split line ${body}");
    }
}

Call the above route by passing an ArrayList of Strings as the message body:

List<String> body = List.of("A", "B", "C");
producerTemplate.sendBody("direct:split_start_1", body);

Running the above code yields the following output:

2023-07-14 20:10:03,389 INFO  Before Split line [A, B, C]
2023-07-14 20:10:03,391 INFO  Split line A
2023-07-14 20:10:03,392 INFO  Split line B
2023-07-14 20:10:03,392 INFO  Split line C

Example 2: Splitting a String by Custom Delimiter

In this example, we split a String using a custom delimiter #:

@ApplicationScoped
public class SplitterRoute extends RouteBuilder {

    @Override
    public void configure() throws Exception {
        from("direct:split_start_2")
        .log("Before Split line ${body}")
        .split(body(), "#")
        .log("Split line ${body}");
    }
}

To call the route and pass the String as a message body:

producerTemplate.sendBody("direct:split_start_2", "A#B#C");

Running this code, you see the following in the log:

2024-07-14 20:19:19,229 INFO  Before Split line A#B#C
2024-07-14 20:19:19,229 INFO  Split line A
2024-07-14 20:19:19,229 INFO  Split line B
2024-07-14 20:19:19,230 INFO  Split line C

These examples demonstrate the versatility of the Splitter pattern in handling various types of message formats.

As you can see, the Splitter pattern is very useful when you need to split a message into multiple messages. But let's see another example where we are going to split an Object by its fields. For the demo purpose, we are going to provide the User object with the List of Address objects.

The User class is:

public record UserDetails(String userId, List<Address> addresses) {}

and the Address class is:

public record Address(String addressId, String address) {}

The route definition is:

@ApplicationScoped
public class SplitterRoute extends RouteBuilder {

  @Override
  public void configure() throws Exception {
    from("direct:userDetails")
        .log("User Id: ${body.userId}")
        .split(method(UserService.class))
        .log("Split line ${body}");
  }

  static class UserService {
    public static List<Address> getDetails(UserDetails userDetails) {
      return userDetails.addresses();
    }
  }
}

and to call the above route as pass the User object as a message body:

final UserDetails userDetails = new UserDetails(
    "user-1",
    List.of(
        new Address("1", "Address 1"),
        new Address("2", "Address 2")
    )
);

producerTemplate.sendBody("direct:userDetails", userDetails);

When you run the above code, you will see the following output in the log:

2024-07-14 21:09:02,225 INFO  [route36] (Quarkus Main Thread) User Id: user-1
2024-07-14 21:09:02,226 INFO  [route36] (Quarkus Main Thread) Split line Address[addressId=1, address=Address 1]
2024-07-14 21:09:02,226 INFO  [route36] (Quarkus Main Thread) Split line Address[addressId=2, address=Address 2]

The method component in Apache Camel is a versatile and powerful tool that allows routes to directly invoke methods on beans. This capability is particularly useful in enterprise applications where business logic needs to be cleanly encapsulated within service classes, yet seamlessly integrated within Camel routes for orchestration of data flows.

The route invokes a method from the UserService class. Apache Camel looks for this method in the UserService class to handle how the message should be split. The method should be designed to accept the incoming message body and return a collection, such as a List or Array, which Camel then iterates over. Each element of the array or list becomes a new exchange in Camel, which is processed independently for the remainder of the route.

By integrating dynamic splitting with conditional routing using choice() and when() constructs, Apache Camel provides a robust framework for bespoke data processing workflows.

Understanding `choice()` and `when()` in Apache Camel

In Apache Camel, choice() and when are powerful constructs that enable content-based routing. This paradigm mimics the if-else or switch-caseconstructs familiar in many programming languages, allowing routes to make decisions based on the content of messages.

The choice() construct serves as a container for one or more when() clauses. Each when() clause specifies a condition. When a condition evaluates to true, Apache Camel routes the message to the processing logic defined within that when() block. If none of the conditions in any of the when() clauses match, an optional otherwise() clause can be executed, akin to an else statement in traditional programming.

Detailed Example with choice() and when().

Imagine a scenario where an application processes orders and sends them to different processing streams based on their category, such as "books", "electronics", or "clothing". This setup allows for highly specialized processing for different categories of items.

Here's how you might set up such a route in Apache Camel:

Route Configuration: Define a route that starts from an endpoint (for example, direct:startOrderRoute). This route will examine the type of order and route the message accordingly.
Using choice() and when(): Inside the route, a choice() component examines the message. Based on the content, it routes the message to different endpoints using when().
Logging and Processing: Each category has a dedicated log that records the action taken, ensuring traceability.

Here's a sample implementation:

public class OrderRouteBuilder extends RouteBuilder {

    @Override
    public void configure() throws Exception {
        from("direct:start")
            .choice()
                .when(body().contains("books"))
                    .log("Routing to Books processing: ${body}")
                    .to("direct:books")
                .when(body().contains("electronics"))
                    .log("Routing to Electronics processing: ${body}")
                    .to("direct:electronics")
                .when(body().contains("clothing"))
                    .log("Routing to Clothing processing: ${body}")
                    .to("direct:clothing")
                .otherwise()
                    .log("Unknown order type: ${body}")
            .end();

        from("direct:books")
            .log("Processing Books Order: ${body}");

        from("direct:electronics")
            .log("Processing Electronics Order: ${body}");

        from("direct:clothing")
            .log("Processing Clothing Order: ${body}");
    }
}

Now you can run the route by sending different order types to the direct:start endpoint:

producerTemplate.sendBody("direct:startOrderRoute", "An order for books");

and you will see the following output in the log:

2024-07-16 19:16:12,357 INFO  [route5] (Quarkus Main Thread) Received Order: An order for books
2024-07-16 19:16:12,357 INFO  [route5] (Quarkus Main Thread) Routing to Books processing: An order for books
2024-07-16 19:16:12,358 INFO  [route6] (Quarkus Main Thread) Processing Books Order: An order for books

As you can see, the message is routed to the books endpoint and processed accordingly. You can test the other categories in a similar manner.

Practical Usage and Tips

Complex Conditions: when() can handle complex expressions combining multiple conditions.
Using .endChoice(): For readability and to prevent misrouting, especially in a route with nested .choice() constructs, use .endChoice() to clearly indicate the end of a choice block.
Flexibility: This approach is highly flexible, allowing the addition of new conditions and endpoints as business requirements evolve.

Conclusion

The Splitter pattern stands out as a powerful tool in Apache Camel's arsenal, enabling developers to effectively handle complex, bulky data structures by breaking them down into more manageable pieces. This pattern not only simplifies data processing workflows but also enhances maintainability and scalability within integration solutions.

In this article, we've explored the practical implementation of the Splitter pattern in a range of contexts - from handling lists and custom delimited strings to operating on more complex structures like Java objects. Each example showcased how Apache Camel facilitates straightforward data manipulations, making seemingly complex integration tasks much simpler.

Additionally, the integration of choice() and when() constructs further refines the data routing capabilities in Apache Camel, offering precise control over message flow based on specific content criteria. This ability to route data conditionally mimics conventional programming logic, bringing a familiar and intuitive approach to route definitions.

As businesses continue to deal with increasingly diverse and voluminous data, understanding and implementing these patterns will become crucial. The flexibility and power of Apache Camel in managing data flows allow developers to build robust, efficient data ingestion and processing pipelines that are crucial for today's data-driven applications.

And last but not least, the all above examples are implemented in the Quarkus framework as using the Camel Quarkus extension.

Exploring Core Features and Components of Apache Camel

Ivelin Yanev — Sun, 01 Sep 2024 12:11:42 +0000

Hello friends! In our previous discussion, we delved into the integration of Apache Camel with Quarkus, demonstrating how to craft real-world applications using Apache Camel. As we continue our series, we aim to take a deep dive into the crucial components and the intrinsic details of Apache Camel.

Enterprise Integration Patterns

At its core, Apache Camel is structured around the concepts introduced in the Enterprise Integration Patterns (EIP) book by Gregor Hohpe and Bobby Woolf. This book outlines numerous patterns that have become standardizations for designing and documenting robust integration solutions across enterprise applications
or systems. Here's an overview of some pivotal patterns utilised within Apache Camel:

Aggregator

The Aggregator pattern(Figure 1) is essential for collecting and consolidating related messages into a cohesive single message, facilitating comprehensive processing. It acts as a specialized filter, accumulating correlated messages until a complete set of data is received, at which point it publishes an aggregated output
for further processing.

Figure 1 – Aggregator Pattern (enterpriseintegrationpatterns.com)

Content-Based Router

This pattern(Figure 2) dynamically routes messages based on their content to appropriate receivers. Routing decisions can depend on various message attributes such as field presence or specific field values.

Figure 2 – Content-Based Router pattern (enterpriseintegrationpatterns.com)

Dynamic Router

The Dynamic Router(Figure 3) facilitates routing decisions made at runtime, adapting dynamically based on rules defined externally or through user input, supporting modern service-oriented architectures.

Figure 3 – Dynamic Router pattern (enterpriseintegrationpatterns.com)

Message Filter

A Message Filter(Figure 4) directs messages to an output channel or discards them based on specified criteria, ensuring that only messages meeting certain conditions are processed further.

Figure 4 – Message Filter pattern (enterpriseintegrationpatterns.com)

Process Manager

This pattern(Figure 5) orchestrates the sequence of steps in a business process, handling both the execution order and any occurring exceptions. It underpins complex workflows where sequential processing and error management are critical.

Figure 5 – Process Manager pattern (enterpriseintegrationpatterns.com)

Normalizer

The Normalizer pattern(Figure 6) is a critical tool in Apache Camel that addresses the challenges of message format discrepancies among different systems. It takes incoming messages in various formats and converts them into a standardized format before further processing, ensuring consistency across the data handling pipeline. This pattern is particularly beneficial in environments where messages originate from diverse sources with varying formats.

Figure 6 – Normalizer pattern (enterpriseintegrationpatterns.com)

Splitter

Handling complex messages composed of multiple data items is streamlined by the Splitter pattern(Figure 7). This pattern efficiently divides a compound message into its constituent elements, allowing each element to be processed independently. This is immensely useful in scenarios where different parts of a message need to be routed or processed differently based on their individual characteristics.

Figure 7 – Splitter pattern (enterpriseintegrationpatterns.com)

I have to mention that these are just a few of the patterns that are used in Apache Camel. There are many more patterns that are used in Apache Camel. But I consider these patterns to be the most important ones.

Camel Fundamentals

At its essence, the Apache Camel framework is centered around a powerful routing engine, or more accurately, a routing-engine builder. This engine empowers developers to devise bespoke routing rules, determine which sources to accept messages from, and define how those messages should be processed and dispatched to various destinations. Apache Camel supports the definition of complex routing rules through an integration language similar to those found in intricate business processes.

One of the cardinal principles of Camel is its data-agnostic nature. This flexibility is crucial as it allows developers to interact with any type of system without the strict requirement of transforming data into a predefined canonical format. The ability to handle diverse data forms seamlessly is what makes Camel a versatile tool in the toolkit of any system integrator.

Message

In the realm of Apache Camel, messages are the fundamental entities that facilitate communication between systems via messaging channels. These components are illustrated in (Figure 8). During the course of a route's execution, messages can undergo various transformations—they can be altered, duplicated,
or entirely replaced depending on the specific needs of the process. Messages inherently flow uni-directionally from a sender to a receiver and comprise several components:

Body (Payload): The main content of the message.
Headers: Metadata associated with the message which can include keys and their respective values.
Attachments: Optional files that can be sent along with the message.

Figure 8 – Apache Camel Message structure

Messages in Apache Camel are uniquely identified by an identifier of type java.lang.String. The uniqueness of this identifier is enforced by the message creator and is dependent on the protocol used, though the format itself is not standardized. For protocols lacking a unique message identification scheme, Camel employs its own ID generator.

Headers in a Camel message serve as key-value pairs containing metadata such as sender identifiers, hints about content encoding, authentication data, and more. Each header name is a unique, case-insensitive string, while the value can be any object (java.lang.Object), reflecting Camel's flexible handling of header types. All headers are stored within the message as a map.

Additionally, messages may include optional attachments, commonly utilized in contexts involving web services and email transactions. The body of the message, also of type java.lang.Object, is versatile, accommodating any form of content. This flexibility mandates that the application designer ensures content comprehensibility across different systems. To aid in this, Camel provides various mechanisms, including automatic type conversion when necessary, to transform data into a compatible format for both sender and receiver, facilitating seamless data integration across diverse environments.

Exchange

In Apache Camel, an Exchange is a pivotal message container that navigates data through a Camel route. As illustrated in (Figure 9), it encapsulates a message, supporting its transformation and processing through a series of predefined steps within a Camel route. The Exchange implements the following org.apache.camel.Exchange interface.

Figure 9 – An Apache Camel exchange

The Exchange is designed to accommodate different styles of messaging, particularly emphasizing the request-reply pattern. It is robust enough to carry information about faults or errors, should exceptions arise during the processing of a message.

Exchange ID: This is a unique identifier for the exchange, automatically generated by Camel to ensure traceability..
Message Exchange Pattern MEP: Pecifies the messaging style, either InOnly or InOut. For InOnly, the transaction involves only the incoming message. For InOut, an additional outgoing message (Out Message) exists to relay responses back to the initiator.
Exception - The Exchange captures exceptions occurring during routing, centralizing error management for easy handling and mitigation.
Body: Each message (In and Out) contains a payload of type java.lang.Object, allowing for diverse content types.
Headers: Stored as a map, headers include key-value pairs associated with the message, carrying metadata such as routing cues, authentication keys, and other contextual information.
Properties: Similar to headers but enduring for the entirety of the exchange, properties hold global-level data pertinent throughout the message processing lifecycle.
In message: The foundational component, this mandatory element encapsulates incoming request data from inbound channels.
Out message: An optional component that exists in InOut exchanges, carrying the response data to an outbound channel.

In Apache Camel, an Exchange is a message container which carries the data through a Camel route. It encapsulates a message and allows it to be transformed and processed across a series of processing steps defined in a Camel route. An Exchange also facilitates the pattern of request-reply messaging and might carry fault or error information if exceptions occur during message processing.

Camel Context

The Apache Camel Context is an essential element within Apache Camel, serving as the core framework that orchestrates the integration framework's functionality. It is where the routing rules, configurations, components, and additional integration elements converge. The Camel Context(Figure 10) initializes, configures, and oversees the lifecycle of all components and routes it contains.

Figure 10 – Apache Camel context

Within the Camel Context, the following critical operations are facilitated:

Loading Components and Data Formats: This involves the initialization and availability management of components and data formats used across various routes.
Configuring Routes: It provides a mechanism to define the paths that messages follow, including the rules for how messages are processed and mediated across different endpoints.
Starting and Stopping Routes: The Camel Context manages the activation and deactivation of routes, ensuring these operations are performed in a thread-safe manner.
Error Handling: Implements centralized error-handling mechanisms that can be utilised across all routes within the context.
Managing Resources: It ensures efficient management of resources like thread pools or connections, releasing them appropriately when not required anymore.

The Camel Context can be configured either programmatically or declaratively. For instance, in a Java-based setup:

import org.apache.camel.CamelContext;
import org.apache.camel.impl.DefaultCamelContext;

public class MainApp {
    public static void main(String[] args) {
        CamelContext camelContext = new DefaultCamelContext();
        try {
            // Add routes, components, etc.
            camelContext.start();
            Thread.sleep(10000);
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            try {
                camelContext.stop();
            } catch (Exception e) {
               // Handle exception
            }
        }
    }
}

For environments like Quarkus, the Camel Context is typically retrieved and managed as follows:

@Inject CamelContext context;

if (context.isStarted()) {
  context.getRouteController().startRoute("start_route");
}

When leveraging Quarkus, the Camel Context is automagically provisioned and managed:

@ApplicationScoped
public class DemoRoute extends RouteBuilder {

  @Override
  public void configure() throws Exception {

    from("direct:start_route")
        .log("Starting route: ${routeId}, headers: ${headers}")
        .setHeader("header_abc", constant("header_value"))
        .to("direct:route_1")
        .to("direct:route_3");
  }
}

Endpoints

In Apache Camel, endpoints represent the interfaces for connecting the Camel application with external systems or services. They are the points at which routes either begin (consume) or end (produce).

Some common types of endpoints include:

A Camel file endpoint can be used to read from and write to files in a directory.

    // Route to read files from the directory "input" and move 
    processed files to "output"
    from("file:input?move=processed")
   .to("file:output");

HTTP endpoints are used for integrating with HTTP services.

    // Route to consume data from an HTTP service
    from("timer:foo?period=60000")
   .to("http://example.com/api/data")
   .to("log:result");

Direct and SEDA Both direct and seda are used for in-memory synchronous and asynchronous message queuing respectively within Camel.

// Using direct for synchronous call
from("direct:start")
    .to("log:info");

Routes

Routes in Camel define the message flow between endpoints, incorporating a series of processors or transformations. They are crucial for constructing the processing logic within a Camel application.

Here’s an example demonstrating a series of interconnected routes:

@ApplicationScoped
public class DemoRoute extends RouteBuilder {

  @Override
  public void configure() throws Exception {

    from("direct:start_route")
        .log("Starting route: ${routeId}, headers: ${headers}")
        .setHeader("header_abc", constant("header_value"))
        .to("direct:route_1")
        .to("direct:route_3");

    from("direct:route_1")
        .log("Starting route_1: ${routeId}, headers: ${headers}, thread: ${threadName}")
        .process(
            exchange -> {
              exchange.getIn().setHeader("header_abc", "UPDATED_HEADER_VALUE");
            })
        .to("direct:route_2");

    from("direct:route_2")
        .log("Starting route_2: ${routeId}, headers: ${headers}, thread: ${threadName}");
  }
}

Here the first route starts from the direct:start_route endpoint, logs the routeId and headers, set the new header with key: header_abc, and then forwards the message to the next route direct:route_1. The second route logs the routeId, headers, and the thread name, and then forwards the message to the next route direct:route_2. The third route logs the routeId, headers, and the thread name.

Conclusion

In this detailed exploration of Apache Camel, we have traversed the core concepts and essential components that make it an indispensable tool in the realm of enterprise integration. Beginning with a thorough examination of Enterprise Integration Patterns (EIPs), we understood how Camel utilizes patterns like Aggregators, Splitters, and Normalizers to address common integration challenges effectively.

Further, we delved into the architectural fundamentals of Camel, highlighting its versatile routing capabilities, flexible message model, and the pivotal role of the Camel Context in managing and orchestrating these elements. We also covered practical aspects, demonstrating how routes are defined and managed, along with a look at various endpoint types that facilitate communication with external systems.

Practical Guide to Apache Camel with Quarkus: Building an ETL Application

Ivelin Yanev — Sun, 01 Sep 2024 11:43:20 +0000

I am excited to introduce a series of articles about Apache Camel. In this first post, rather than delving deeply into the complexities of Apache Camel, I will present a practical use case to showcase its capabilities. Specifically, you'll learn how to create a simple Extract, Transform, and Load (ETL) application between two databases using Apache Camel.

Introduction to Apache Camel - Brief Overview

Before we dive into the practical use case, let's briefly introduce Apache Camel. Apache Camel is an open-source integration framework that leverages Enterprise Integration Patterns (EIP) to facilitate the integration of various systems.

In today's world, numerous systems of different types coexist. Some may be legacy systems, while others are new. These systems often need to interact and integrate with each other, which can be challenging due to differing implementations and message formats. One solution is to write custom code to bridge these differences, but this can lead to tight coupling and maintenance difficulties.

Instead, Apache Camel offers an additional layer to mediate the differences between systems, resulting in loose coupling and easier maintenance. Camel uses an API (or declarative Java Domain Specific Language) to configure routing and mediation rules based on EIP.

Enterprise Integration Patterns (EIP)

To understand Apache Camel, it's important to grasp "Enterprise Integration Patterns" (EIP). The book "Enterprise Integration Patterns" describes a set of patterns for designing large, component-based systems where components can run in the same process or on different machines. The key idea is that systems should be message-oriented, with components communicating via messages. The patterns provide a toolkit for implementing these communications (Figure 1).

Figure 1 – Basic Elements of an Integration Solution (enterpriseintegrationpatterns.com)

Key Terminologies in Apache Camel

EndPoint: An endpoint is a channel through which messages are sent and received. It serves as the interface between a component and the outside world.
Message: A message is a data structure used for communication between systems, consisting of a header and a body. The header contains metadata, and the body contains the actual data.
Channel: A channel connects two endpoints, facilitating the sending and receiving of messages.
Router: A router directs messages from one endpoint to another, determining the message path.
Translator: A translator converts messages from one format to another.

I consider to stop here for now about the introduction to Apache Camel. Now let's show you how to create a simple ETL application between two databases using Apache Camel.

Problem Statement

Let's assume we have a highly loaded system where one critical component is the database. At some point, we need to deal with this data outside of the usual operational cases - training ML models, generating exports, graphs, or we simply need some part of the data. Of course, this would burden our operational database even more, and for this purpose, it would be optimal to have a mechanism with which we can extract the necessary data, transform it into the form we need, and store it in another database - other than the operational one. With this strategy, we solve the problems of excessive potential overloading of our operational base. Moreover, with such a mechanism, we can perform this operation at a time when our system is not too loaded (e.g., nighttime).

Solution Overview

The solution is shown in the diagram below (Figure 2). We will use Apache Camel to create a simple ETL application between two databases. The application will extract data from a source database, transform it, and load it into a target database. We can introduce different strategies to implement this solution, focusing on how to extract the data from the source database. I assume that the criteria for selecting the data will be based on the modification date of the record. This strategy provides an opportunity to extract the data that has been modified, too.

Figure 2 – Syncing Data Between Two Databases Using Apache Camel

Source and target databases will have the following table structure:

CREATE TABLE IF NOT EXISTS user
(
    id            serial PRIMARY KEY,
    username      VARCHAR(50)  NOT NULL,
    password      VARCHAR(50)  NOT NULL,
    email         VARCHAR(255) NOT NULL,
    created_at    timestamp default now()::timestamp without time zone,
    last_modified TIMESTAMP DEFAULT now()::timestamp without time zone
);

In the target database, we'll transform the username to uppercase before inserting it.

We'll use Camel Quarkus extensions for various Camel components. Specifically, we'll use the Camel SQL component to interact with the databases. The SQL component supports performing SQL queries, inserts, updates, and deletes.

First, create a class that extends RouteBuilder and override the configure method:

@ApplicationScoped
public class UserRoute extends RouteBuilder {
    @Override
    public void configure() throws Exception {
        // your code here
    }
}

The used of @ApplicationScoped annotation is not mandatory here, but I prefer to indicate that the class is a CDI bean and should be managed by the CDI container.

As I mentioned above, we will use the Camel SQL component to interact with the databases. We need to configure the Camel SQL component to connect to the source and target databases. We will use the Quarkus Agroal extension to configure the data sources. The Agroal extension provides a connection pool for the data sources. We will configure the data sources in the application.properties file.

#
# Source Database Configuration
quarkus.datasource.source_db.db-kind=postgresql
quarkus.datasource.source_db.jdbc.url=jdbc:postgresql://localhost:5001/demo
quarkus.datasource.source_db.username=test
quarkus.datasource.source_db.password=password1
#
#
# Target Database Configuration
quarkus.datasource.target_db.db-kind=postgresql
quarkus.datasource.target_db.jdbc.url=jdbc:postgresql://localhost:6001/demo
quarkus.datasource.target_db.username=test
quarkus.datasource.target_db.password=password1
#

Now we can configure the Camel SQL component to connect to the source and target databases. We will use the sql component to create a SQL endpoint for the source and target databases.

The SQL component uses the following endpoint URI notation:

sql:select * from table where id=# order by name[?options]

But we need mechanism to run the operation automatically. We will use the timer component to trigger the ETL process every seconds. The timer component is used to generate message exchanges when a timer fires. The timer component uses the following endpoint URI notation:

timer:name[?options]

In our route we use configuration as follows:

 from("timer://userSync?delay={{etl.timer.delay}}&period={{etl.timer.period}}")

The {{etl.timer.delay}} and {{etl.timer.period}} are the configuration values that we will define in the application.properties file.

etl.timer.period=10000
etl.timer.delay=1000

In order to transform the data before inserting it into the target database, we need to provide our translator:

.process(exchange -> {
    final Map<String, Object> rows = exchange.getIn().getBody(Map.class);

    final String userName = (String) rows.get("username");

    final String userNameToUpperCase = userName.toUpperCase();

    log.info("User name: {} converted to upper case: {}", userName, userNameToUpperCase);

    rows.put("username", userNameToUpperCase);
})

The Processor interface is used to implement consumers of message exchanges or to implement a Message Translator and other use-cases.

And voila, we have a simple ETL application between two databases using Apache Camel.

When you run the application, you should see the following output in the logs:

2024-06-09 13:15:49,257 INFO  [route1] (Camel (camel-1) thread #1 - timer://userSync) Extracting Max last_modified value from source database
2024-06-09 13:15:49,258 INFO  [route1] (Camel (camel-1) thread #1 - timer://userSync) No record found in target database
2024-06-09 13:15:49,258 INFO  [route2] (Camel (camel-1) thread #1 - timer://userSync) The last_modified from source DB: 
2024-06-09 13:15:49,274 INFO  [route2] (Camel (camel-1) thread #1 - timer://userSync) Extracting records from source database
2024-06-09 13:15:49,277 INFO  [org.iqn.cam.rou.UserRoute] (Camel (camel-1) thread #1 - timer://userSync) User name: john_doe converted to upper case: JOHN_DOE
2024-06-09 13:15:49,282 INFO  [org.iqn.cam.rou.UserRoute] (Camel (camel-1) thread #1 - timer://userSync) User name: jane_smith converted to upper case: JANE_SMITH
2024-06-09 13:15:49,283 INFO  [org.iqn.cam.rou.UserRoute] (Camel (camel-1) thread #1 - timer://userSync) User name: alice_miller converted to upper case: ALICE_MILLER

You can find the full source code of the application in the GitHub repository.

Conclusion

With this setup, we've created a simple ETL application using Apache Camel that extracts data from a source database, transforms it, and loads it into a target database. This approach helps reduce the load on the operational database and allows us to perform data extraction during off-peak times.

DEV Community: Ivelin Yanev

The state of AI code reviews: An 18-month retrospective

Code Review: A Critical Step in Software Development

The Code Review Pyramid

Where Does AI Excel?

The Human Mind: The Unbeatable Factor

Java 25: Top 3 Features That Redefine Performance, Startup, and Efficiency

1. Memory Efficiency: Compact Object Headers

2. Faster Startup: Project Leyden Integration

3. Stable Values: Safe, On-Demand Initialization

Conclusion

How to programmatically look up a bean in Quarkus

Why would you need programmatic bean lookup?

Methods for programmatic bean lookup

Conclusion

How to Build an AI Application with Quarkus and LangChain4j

Prerequisites

What is LangChain4j?

Core Components Explained

Configuration Setup

Simple AI Service

Advanced AI Service with Memory

Understanding Message Types in LangChain4j

Understanding Retrieval-Augmented Generation (RAG)

Why RAG is Needed?

Key Components of RAG: A Step-by-Step Process

Conclusion

A Deep Dive into Graceful Shutdown in Go

The Core Concept

How It Works

The Shutdown Sequence

Real-World Considerations

Testing the Implementation

Key Benefits of This Implementation

Conclusion

Mastering Pointers in Go: Enhancing Safety, Performance, and Code Maintainability

Deep Dive into Pointers in Go

Pointers in Go don’t function the way they do in C/C++

Escape analysis

Garbage collection

When to use pointers in Go

Why I prefer values?

Conclusion

Building Robust REST Client with Quarkus: A Comprehensive Guide

Introduction

Getting Started

Real-World Example: Cryptocurrency API Client

Advanced Features

Fault Tolerance

Retry Pattern

Fallback Pattern

Circuit Breaker Pattern

Dynamic Base URL Configuration

Custom Headers Management

Real-World Example: Integrating with Freesound API

Conclusion

Optimizing Memory Usage in Go: Mastering Data Structure Alignment

What is the reason for that?

Padding Optimization and Performance Impact

How Alignment Affects Performance

Padding Optimization Strategies

Conclusion

Effective Project Structuring for Microservices with Quarkus

Much Abstraction is Too Much

Mastering Data Routing in Apache Camel: Leveraging the Splitter Pattern

Overview

Using Splitter in Apache Camel

Understanding choice() and when() in Apache Camel

Practical Usage and Tips

Conclusion

Exploring Core Features and Components of Apache Camel

Enterprise Integration Patterns

Camel Fundamentals

Message

Exchange

Camel Context

Endpoints

Routes

Conclusion

Practical Guide to Apache Camel with Quarkus: Building an ETL Application

Understanding `choice()` and `when()` in Apache Camel