DEV Community: hiclab

Message Throttling with Apache Camel Master Component

hiclab — Sat, 22 Nov 2025 18:00:31 +0000

The Problem

When managing events, it's common to encounter a rapid succession of events within a brief period. For instance, within an application driven by events, creating an entity can trigger multiple events like EntityAdded, EntityChanged, SubEntityAdded, etc. However, if the primary objective is to monitor changes in a specific entity item, processing all these events to dispatch a notification or outbound message to downstream systems might be unnecessary.

The Solution

Based on the problem described above, we are often interested in only knowing that an update has occurred at a certain moment, rather than communicating every single change. This can be effectively managed within a short predefined timeframe, typically 5 to 10 seconds.

This pattern is particularly common when dealing with primary business objects. While each change can generate multiple events, downstream systems don’t need to process all updates individually, they only need to know that an object was modified and should fetch the latest state.

The goal is to inform interested parties that an update has occurred while avoiding unnecessary system load, especially to external systems. One approach is to implement a throttling mechanism leveraging Apache Camel components.

Apache Camel's Master component provides an elegant solution for implementing this throttling mechanism, especially in clustered environments. The Master component ensures that only one node in a cluster actively consumes from an endpoint, preventing duplicate processing when multiple application instances are running. However, the Master component alone is not sufficient and it requires a transport mechanism that supports distributed messaging. This is where message queue services like Amazon SQS, JMS-compliant brokers (such as ActiveMQ), or other similar messaging platforms or distributed cache systems become essential to enable proper coordination across the cluster.

Why SQS Over Direct Component?

While it might be logical to use Camel's direct component for internal routing, this approach has significant limitations in clustered environments. This component only works within a single JVM, making it unsuitable for distributed deployments. When using direct with the Master component, only the elected leader route will work properly while the other routes in other nodes will fail when attempting to consume messages from that endpoint, as the direct endpoint simply doesn't exist in their JVM context.
Using Amazon SQS solves this problem by providing a centralized message store where all cluster nodes can access the same message queue.

Implementation Pattern

Here is how to implement throttling with the Master component and SQS:

from("master:notification:aws2-sqs://queue-1")
  .routeId(“throttlingRoute”)
  .aggregate().exchangeProperty(ENTITY_ID_PROPERTY)
  .completionTimeout(“5000”)
  .aggregationStrategy(new UseLatestAggregationStrategy())
  .to(“direct:notify”);

This works as follows:

Multiple messages are sent to the SQS queue.
Those messages arrive at the SQS queue in an interval of a few seconds.
The Master component ensures only the elected leader route consumes the messages.
The aggregation strategy discards all but the latest message.
After the completion timeout is reached, only the latest message proceeds to the notification process.
If the leader route fails, another route in another node is automatically elected and continues processing.

The pattern described above is particularly valuable for status update handlers and any scenario where notifying about one state among various that occurred at almost the same time, or the most recent state, is more important than notifying every intermediate change.

Push vs Pull in Task Assignment

hiclab — Fri, 14 Apr 2023 09:35:27 +0000

In this post, we will explore the differences between push and pull approaches in task assignment, the advantages and disadvantages of each approach, and when each approach is most appropriate.
In the push model, an authority figure such as a team lead or manager assigns tasks to individual team members. On the other hand, the pull model is more decentralized and empowers individual team members to take on tasks based on priorities, skills and their own interests.

Push Model

In traditional software development methodologies, it is more common to use the push approach to task assignment where tasks are assigned to each team member. This approach is often hierarchical in nature, with a clear chain of command and centralized decision-making.
There are several scenarios in which a task may be assigned, including situations that require specialization, experience, or when it is related to other tasks.
This model has several negative impacts on team dynamics:

Team members often tend to focus on their individual assignments over the shared goals of the team. When individuals focus solely on their own work, they may fail to see the bigger picture and miss opportunities for collaboration and collective problem-solving because of communication barriers.
Team members may become disinterested and demotivated due to the absence of ownership and autonomy. When team members feel they have little control over the project's direction and outcomes, they may feel undervalued and unengaged. This can lead to a decrease in productivity and lower quality work.
Someone needs to track and optimize each team member's individual queue of work. This is needed to maintain a well balanced distribution of tasks and an appropriate workload.
Teams may face difficulties when trying to adapt to changing priorities, especially when new tasks with higher priorities are added to the existing workload. When team members are already focused on their assigned tasks and have a queue of work ahead of them, it can be difficult to shift their attention and resources towards the new high-priority tasks.

Pull Model

This model is often used in agile methodologies to promote flexibility and collaboration among team members where they proactively select high-priority tasks from the to-do list or backlog. Following this approach, the focus is on working and completing high priority work rather than keeping team members constantly occupied.
This approach has the following benefits:

Encouraging self-organizing teams can help optimize task management and allocation. Instead of assigning tasks to individuals, the team collaboratively manages the backlog. This approach not only ensures that tasks are distributed fairly but also creates a culture of open communication and collaboration within the team.
Team members work on new tasks when they have capacity while ensuring that the existing work is completed. This approach allows team members to be more flexible and responsive to changes in the project scope or priorities. If a new high-priority task arises, team members who have capacity can quickly shift their focus to the new task.
Each team member manages and takes responsibility for their tasks, including updating task status and relevant information as needed. This promotes a sense of autonomy and responsibility within the team which can lead to greater engagement and job satisfaction.
The team lead and team members are focused on high value tasks instead of wasting efforts on assigning and monitoring tasks.

While both push and pull models for task assignment have their advantages and disadvantages, the pull model is generally considered the default approach particularly in agile methodologies because it aligns with agile principles. For example, Kanban has one essential principle called Pull System which emphasizes the importance of pulling work from a prioritized backlog based on capacity. However this is not enough without limiting the work in progress (WIP). By limiting the number of tasks in progress, teams can focus on completing high-priority tasks, avoid multitasking and detect bottlenecks.

Check messages in Problems view with RCPTT

hiclab — Tue, 28 Apr 2020 11:51:24 +0000

When testing with RCPTT, it is common to check messages in Problems view to make sure that only the expected ones are shown.

The Problems view in Eclipse groups messages by severity but we can also group them by type or other criteria or not at all. In our case, we will use the default grouping that is by severity.

Find a message in Problems view

The following procedure basically constructs a path based on a regular expression in this format $severity "s $.*item(s?)$/" $message and checks whether an item with this path exists.

/* Checks whether the Problems view contains a message with a given severity. */
proc problemExists [val message] [val severity Error] {
    with [get-view Problems | get-tree] {
        if [itemExists [concat $severity "s \\(.*item(s?)\\)/" $message]] {
            bool true
        }
    } | length | gt 0
}

Notice that itemExists is a procedure we created in the past to check whether an item is visible in a tree structure. For more details, see this post.
The argument Severity is optional since it has Error as default value. This is because in most cases, we look for error messages.

Here are some examples of verifications done in Problems view.

// check an error message
problemExists -message "The local variable company may not have been initialized" | verify-true

// error message not expected
problemExists -message "The value of the field Company.name is not used" | verify-false

// check a warning message
problemExists -message "The value of the field Company.name is not used" -severity Warning | verify-true

Count messages in Problems view

Another interesting verification we can do is to count the number of messages that appear in Problems view for a particular severity.

Similarly to the procedure described above, we first check if an item exists for the provided severity then we go through its child items (messages) to count them. If we want to count the occurrences of a concrete message then we have to iterate over all those messages and consider only matches.

/* 
Counts how many times a message with a given severity appears in the Problems view.
If no message is specified then it counts all messages with the specified severity. 
*/
proc problemCount [val severity] [val message ""] {
    let [val severityNode [concat $severity "s \\(.*item(s?)\\)"]] {
        with [get-view Problems | get-tree] {
            if [itemExists $severityNode] {
                with [get-item $severityNode] {
                    expand
                    if [$message | eq ""] {
                        // count all messages of the specified severity
                        get-items | length
                    } -else {
                        // count only the specified message
                        get-items | foreach [val item] {
                            if [$item | get-property text -raw | eq $message] {
                                // message found
                                emit true
                            }
                        } | length
                    }
                }
            } -else {
                // no item is found for the specified severity
                int 0
            }
        }
    }
}

Expand ECL command is necessary to make sure that child items of severity items are visible, otherwise get-items returns nothing.

Here are some usage examples:

problemCount Error | eq 1 | verify-true

problemCount Warning | eq 2 | verify-true

problemCount Warning "The value of the field Company.name is not used" | eq 1 | verify-true

Help the team to define the sprint goal

hiclab — Mon, 27 Apr 2020 13:26:45 +0000

We all know that having a good sprint goal is very important for any team, but do we all know how to define it?

In some cases, the team is not able to define a sprint goal at all because they basically consider it optional or they don’t know how to do it and as result they don’t see how to benefit from it.

Working without a sprint goal

I admit that in the past, I worked in different teams who were used to work without a sprint goal. Here are some of the related issues we faced in our daily work:

The team is not able to deliver working software at the end of a sprint
Team members focus separately on their own goals and don’t work toward the same goal
The team doesn’t focus always on implementing the most important features
Bottlenecks occur frequently in testing and code review which prevent work from being completed
Lack of collaboration between team members
Team members remain specialized in a particular area of product
The daily meeting becomes a simple status report

Bad sprint goal

In one occasion we tried to define a sprint goal to tackle some of the problems mentioned above but we failed. At that moment, we thought that we did well but much later we discovered that we defined a bad goal. Here are some examples:

Complete user stories A and B
Fix high-priority bugs

As you have noticed, those goals do not describe any special purpose for the sprint; they are generic and encourage the team to focus on completing tasks instead of delivering value to customers.

Good sprint goal?

With a good sprint goal, we enforce the importance of having a real objective that is aligned with customer expectations. Now the question is, how do we actually define a good sprint goal?

Create a custom report for RCPTT

hiclab — Sun, 26 Apr 2020 12:13:52 +0000

RCPTT (RCP Testing Tool) includes a functionality that allows to generate reports in multiple formats based on the results obtained from executing test cases. However, the default reports might be insufficient for our needs but fortunately; RCPTT is flexible because it supports generating custom reports.

In this post, we illustrate the basic steps needed to create a custom report renderer so it can be used in RCPTT whether IDE or Test Runner. We use Maven to configure and build the plugin and other related artifacts. By the way, we use a specific project to build an update site for our plugin.

The full code is available at the following repository rcptt-reporting.

Create a report renderer

RCPTT provides the interface IReportRenderer which is the entry point for report generation. Renderers implementing this interface receive a Report iterable object to iterate over all internal reports generated for a given test execution.

public class XMLCustomReportRenderer implements IReportRenderer {

    @Override
    public IStatus generateReport(IContentFactory factory, String reportName, Iterable<Report> reports) {

        // iterate over reports, process their information and create a file in your preferred format ...
        return Status.OK_STATUS;
    }

    @Override
    public String[] getGeneratedFileNames(String reportName) {
        return new String[] { reportName };
    }
}

To access to the results of executed tests, we have to iterate over a collection of Report objects and retrieve the information as follows:

Iterator<Report> reportIterator = reports.iterator();
while (reportIterator.hasNext()) {
    Report report = reportIterator.next();
    Node item = report.getRoot();
    Q7Info info = (Q7Info) item.getProperties().get(IQ7ReportConstants.ROOT);

    // Q7Info object contains the information of an executed test case
}

Add an extension

Once the report renderer is implemented, we have to add an extension for this plugin to be able to use it. The following entry must be added to plugin.xml.

<extension point="org.eclipse.rcptt.reporting.reportRenderer">
    <reportRenderer
        class="com.hiclab.rcptt.reporting.XMLCustomReportRenderer"
        description="XML Custom Report Renderer"
        extension="xml"
        id="XmlReport"
        name="Xml Custom Report">
    </reportRenderer>
</extension>

The id is needed especially when using Test Runner. For more details, check the description of the argument report in Test Runner documentation.

Zero Bug Policy, is it really worth it?

hiclab — Tue, 14 Apr 2020 08:37:57 +0000

Years ago, I was on a project where the number of bugs was considerably growing day after day. This situation led to a poor backlog management because we had to spend a lot of effort to triage and prioritize hundred of bugs every few weeks, almost all them remained in the backlog for several months. As you have probably experienced, bugs tend to become obsolete as time passes.

As a team, we were aware that most of reported bugs actually were not real bugs but enhancements and they were completely ignored because the priority was to implement new features and of course fix only critical defects. As you know, customers always want value.

It took a time to figure out how to solve the problem. I remember that at the first opportunity, I proposed the Zero Bug Policy based on what I read in articles and books. In my opinion, this was the optimal and effective way to handle the issue but for the team, it sounded a little bit extreme.

The concept of Zero Bug Policy is very simple. There is basically one rule to follow: any identified bug should be either fixed immediately or rejected. In other words, an issue is either a high priority bug or it is not a bug at all.

Here are some benefits of applying this rule:

Reduce the effort spent in bug tracking
Eliminate the need of triage and reprioritization of bugs
Focus on fixing most relevant issues first
Keep the backlog as small as possible

In our case, finally we adopted a conservative approach that consisted in fixing as soon as possible any critical bug and leave the others to be fixed later in the near future, as long as they are considered relevant.
This rule contributes considerably to the better management of the backlog. However, an effort was still needed to review the existing bugs since their number could not be reduced to zero or close to zero.

I was pretty sure that this could be an important step to apply the Zero Bug Policy in the future.

How to access HTML elements in RCPTT using Javascript

hiclab — Sun, 12 Apr 2020 11:01:57 +0000

Did you ever have to write an RCPTT test to do some verification in an HTML?

Suppose that our Eclipse application under test generates HTML files and we want to check that links, buttons, fields and other elements are valid. Here is where Javascript comes to the rescue.

Javascript provides many functions that can be used to access elements on a web page through the browser. We assume that the page is opened in an internal browser so we can have access to it when testing with rcptt. This can be done using the ECL command invoke and the Javascript function evaluate executed on the control object Browser.

Accessing object properties

In the following example, we want to check that a given link with its corresponding text exist at least once in a web page.

proc linkExists [val href] [val text] {
    let [val script [concat
        "var elements = document.querySelectorAll(\"a[href='" $href "']\");"
        "for (var i = 0; i < elements.length; i++) {"
        "   if (elements[i].innerHTML != '" $text "') {"
        "       return false;"
        "   }"
        "}"
        "return true;"
        ]] {

        $script | log
        get-control Browser | get-object | invoke evaluate $script
    }
}
with [get-editor "Index"] {
    linkExists "/new" "Write a post" | verify-true
    linkExists "/new" "Write a post?" | verify-false
}

Performing actions

In this example, we just wanto to simulate a click on a given link identified by its HTML id.

proc clickOnLink [val id] {
    get-control Browser | get-object | invoke evaluate [concat "document.getElementById('" $id "').click()"]
}
get-editor "Index" | clickOnLink "connect-link"

How to run a Shell command in RCPTT

hiclab — Fri, 10 Apr 2020 10:56:20 +0000

There are several reasons to run an external process when writing automated test cases. Those are some examples:

Unzip files
Run a script or batch file
Access to the processes that are running on the system
Create folders and files

Basically, to execute a system command, we use exec-process ECL command provided by RCPTT and pass the required parameters. Once executed, the command provides the following information:

Exit code
Stdout
Stderr

The executed command depends on the operating system we are using. In our case, it is Windows so we have to pass the command as argument to cmd.exe shell.

Create a new folder

exec-process cmd "/c" md "C:\\temp\\new_folder" -ignoreExitCode –ignoreStderr

The parameters ignoreExitCode and ignoreStderr are added in this case to ignore any failure if for example, there is already a folder with the same name.

Unzip files

let [val unzipCmd [concat "\"C:\\Program Files\\7-Zip\\7z\"" " x " "C:\\temp\\output.zip" " -aoa -oC:\\temp\\new_folder"]] {
    exec-process cmd "/c" $unzipCmd
}

Windows processes

// check if a given windows process is running
proc processExists [val name] {
    exec-process cmd "/c" tasklist "/FI" [concat "IMAGENAME eq " $name] | get stdout | invoke contains $name
}
processExists "rcptt.exe" | verify-true

Invoke a Java static method from RCPTT

hiclab — Thu, 09 Apr 2020 10:11:34 +0000

It makes no sense to spend lot of time writing complex procedures when we could reuse all the functions available in our application. We don’t want to reinvent the wheel; instead we want to focus on the main goal to test the application.
The invoke-static ECL command can be used for this purpose, to invoke any static method in a java class. Any class available in the plugins provided by the application under test can be used depending on argument types.
Here are some situations where we can save time reusing utility functions provided by Java SE or Apache Commons.

Apache Commons StringUtils

/*
Returns true if the specified value is numeric.
*/
proc isNumeric [val value] {
    invoke-static -pluginId "org.apache.commons.lang3" -className "org.apache.commons.lang3.StringUtils" 
        -methodName isNumeric -args [str $value]
}

/*
Reverses a string.
*/
proc reverse [val string] {
    invoke-static -pluginId "org.apache.commons.lang3" -className "org.apache.commons.lang3.StringUtils" 
        -methodName reverse -args $string
}

Some verifications

isNumeric "abc" | verify-false
isNumeric "123" | verify-true

reverse "abc" | equals "cba" | verify-true

Apache Commons FileUtils

// get the user directory path
invoke-static -pluginId "org.apache.commons.io" -className "org.apache.commons.io.FileUtils" 
        -methodName getUserDirectoryPath | log

Java System

// get the value of a environment variable
invoke-static -pluginId "org.eclipse.rcptt.ecl.core" -className "java.lang.System" 
    -methodName getenv -args "JAVA_HOME" | log

How to check if an item exists in a tree or table in RCPTT

hiclab — Wed, 08 Apr 2020 10:29:11 +0000

When working with trees and tables, have you ever faced a case where you want to check whether a given item is visible?

In RCPTT, an item is obtained using the get-item ECL command. If you try to get an item that does not exist then an error is returned saying that the item could not be found and the test case fails immediately.

In general, invoking get-item can be enough to verify that an given item exists but what to do if we want to continue the execution of the test case to perform other verifications even if the item does not exist.

We create a procedure to handle the case where an item is not found so the error can be captured using try and catch commands as follows:

proc itemExists [val parent -input] [val path] {
    try {
        $parent | get-item $path
        bool true
    } -catch {
        bool false
    }
}

As you can see, it is simple and lets us have control over the execution of any command that can fail. In this case, the procedure returns true if the item exists or false if an error occurs when getting the item.

Example of general usage

get-preferences-menu | click
with [get-window Preferences] {
    get-tree | itemExists Help | verify-true
    get-tree | itemExists HelpNotFound | verify-false
}

Example of specific usage with conditions

if [itemExists Help] {
    // item is found, do more verifications
} -else {
    // item is not found, do more verifications if you want
}

Check if a list contains a value in RCPTT

hiclab — Tue, 07 Apr 2020 10:54:25 +0000

This post will focus on how to iterate over a list to check whether it contains a given element.

Using each

The ECL command each is the one of most used commands in RCPTT. It is very useful because it makes the code clean, simple and much more readable in comparison with the command loop. However, we have to take into account that it doesn’t support breaks. That is, we are forced to go through all elements of the list even if it is not necessary, for example when we have found the target element before reaching the end of the list.

In the following example, we return a value if the list contains the specified element. Afterwards, we check whether the number of returned values is greater than 0 in order to return true or false.

proc itemExists [val items -input] [val targetItem] {
    $items | each [val it] {
        if [$it | eq $targetItem] {
            bool true
        }
    } | length | gt 0
}

Using loop

The command loop supports a kind of break using recur which is another command. Basically, we invoke recur to continue the execution of the loop. We are allowed to use variables and modify them when iterating.

proc itemExists [val items -input] [val targetItem] {
    loop [val index 0] [val max [$items | invoke elements | invoke size]] {
        if [$index | lt $max] {
            if [$items | get $index | not-eq $targetItem] {
                recur [$index | plus 1][$max]
            } -else {
                bool true
            }
        } -else {
            bool false
        }
    } | equals true
}

As you can see above, the code is too complex in comparison with the previous one and make sure that the list is not empty since it fails when running $items | invoke elements.

Count occurrences of an element in a list in RCPTT

hiclab — Mon, 06 Apr 2020 11:57:44 +0000

For those who don’t know, RCPTT (RCP Testing Tool) originally known as Q7 IDE, is an open source project with a focus to test Eclipse plugins and RCP applications. For more details, visit the RCPTT website.

Let’s assume we have a list of strings and we want to count the frequency of a given string. We will use the command each to iterate over the list and check whether it contains the string.

Let’s create a procedure in ECL, the scripting language used by RCPTT.

proc countOccurrences [val items -input] [val targetItem] {
    $items | each [val it] {
        if [$it | eq $targetItem] {
            bool true
        }
    } | length
}

The trick here is to return some value in the if section every time the string is found. In our example, we return a boolean but actually we can return any other value. This is important in order to count the number of returned values at the end.

Run some tests.

global [val items [list a b c a d e a a]]

$items | countOccurrences a | log
$items | countOccurrences p | log

The output.

!MESSAGE 4
!MESSAGE 0