Installing a Boomi Molecule on AWS EC2, Part 2: Unicast Clustering and the Web Server

This is part 2 of a two-part series. If you haven't done it yet, read Part 1 first. Part 1 covers building the EFS and EC2 infrastructure and running the Molecule installer. This part picks up once both nodes are installed and mounted on the same shared folder.

Part 5: Switch the cluster to unicast

Why this step exists

When a Molecule node starts, it needs to find the other nodes. Boomi's default is multicast. The node sends a "who's out there?" message to a special group address and port, and any other node on the same network replies. That works on an office LAN. AWS doesn't carry multicast traffic, though. The message goes nowhere, each node decides it's alone, and each one declares itself the head of its own one-node cluster.

Unicast replaces the broadcast with an explicit list: these are the nodes in this cluster, at these addresses, listening on TCP port 7800. Two settings control it:

Setting in the platform	Property name in the config file	Value
Cluster Network Transport Type (older name: Cluster Communication Method)	com.boomi.container.cluster.type	UNICAST
Initial Hosts for Unicast	com.boomi.container.cluster.initialHosts	<NODE_1_PRIVATE_IP>[7800],<NODE_2_PRIVATE_IP>[7800]

The list uses the private IPs, each followed by [7800], separated by commas with no spaces. It doesn't have to include every node, just enough that at least one is always running. I listed both for simplicity.

Set it in the platform, not in the file

The reliable way is through the platform UI, and it needs the Molecule to be online first. That creates a small chicken-and-egg sequence:

1. Start node 1 only. Wait until it shows green in Runtime Management. On its own it will run in multicast mode, which is fine for now because nothing else is trying to join.
2. In Runtime Management, click the Molecule, then open Properties and the Advanced tab.
3. The Advanced tab has a long alphabetical dropdown. Scroll to the C section, select Cluster Network Transport Type, and set it to UNICAST.
4. Open the dropdown again, scroll to the I section, select Initial Hosts for Unicast, and enter the list of private IPs.
5. Save. The platform pushes the values to the runtime and writes them to the config file on EFS.
6. Restart node 1 with ./atom restart. The cluster layer only switches transport after a restart.

Common mistake: editing the config file by hand while the Molecule is offline. That's exactly what I tried first, adding the two lines to container.properties with echo. The lines were there when I checked. But when node 1 started, it registered with the platform, the platform pushed its own copy of the properties (which didn't have my lines), and my edits were overwritten. The node came up in multicast mode. The platform is the source of truth for runtime properties, so set them in the UI.

Common mistake: looking for the old property name. Older guides call this setting "Cluster Communication Method." In current versions it's listed as Cluster Network Transport Type. The dropdown is alphabetical, so scroll rather than searching.

Leave Cluster Network Bind Address unset. The node binds to its private IP automatically, and the log confirms that with a line about jgroups.bind_addr.

Check which mode a node is really in

Don't rely on the settings page alone. Look at the ports the Java process has open:

sudo ss -tulnp | grep java

ss lists sockets. -t is TCP, -u is UDP, -l shows listening ports only, -n shows port numbers instead of names, and -p shows which process owns each one.

What you see	What it means
A UDP line for 228.10.10.10:45588	Still in multicast mode. The unicast settings haven't taken effect
A TCP LISTEN line for <private-ip>:7800	Unicast. The node is listening for the other nodes on the cluster port

This check is what finally showed me the real problem, which I describe in Part 6.

Part 6: Start the nodes and form the cluster

Start node 1

On node 1:

cd /mnt/boomi/Molecule_<YOUR_NAME>/bin
./atom start

The bin folder holds the control script. Despite the name, atom is also the script for Molecules. Wait about a minute, then check:

./atom status

It prints atom is running. or atom is not running. In Runtime Management, the Molecule should turn green.

Start node 2

Node 2 never runs the installer, because the installation is already on the shared disk. It just starts the same script:

cd /mnt/boomi/Molecule_<YOUR_NAME>/bin
./atom start

Do this only after node 1 is green, the unicast settings from Part 5 are in place, and node 1 has been restarted with them. Then wait two minutes and open Runtime Management, then the Molecule, then Cluster Status.

What success looks like

You should see two rows, both Online, with a star next to exactly one Node ID. That star marks the head node. There should be no Cluster Issues tab. Take a screenshot for your records.

Problem: MULTIPLE_HEAD_NODES

This is the problem most people hit on their first AWS Molecule, and it's exactly what happened to me. The Cluster Issues tab showed a severe warning, MULTIPLE_HEAD_NODES, with each node reporting a different head node.

It means each node wrote a small "view snapshot" file into the bin/views folder on the shared disk, recording who it thinks the cluster members are and who the head is. The platform compares those files. In my case each file said "I'm the head, and I'm the only member." So the two nodes weren't talking to each other at all. They were two separate one-node clusters sharing a disk. That's dangerous, because both will try to run the same scheduled work and write to the same execution folders.

There are only three possible causes, so check them in this order, from cheapest to most involved:

1. The nodes are still in multicast mode, so discovery messages never arrive. This was my actual cause.
2. The network between the nodes is blocked, either because one instance is in the wrong security group or the self-referencing rule is missing.
3. The nodes started too close together, or node 2 started before node 1 finished forming the cluster, and stale snapshot files kept the wrong state alive.

Start with a clean stop. Stop node 2, then node 1, so nothing changes while you investigate:

cd /mnt/boomi/Molecule_<YOUR_NAME>/bin
./atom stop

Check that the settings reached the runtime. The config file is on the shared disk, so either node will show the same thing:

grep -i cluster /mnt/boomi/Molecule_<YOUR_NAME>/conf/container.properties

You want to see cluster.type=UNICAST and an initialHosts line. If they're missing, go back to Part 5 and set them in the UI. In my case they were there, which was misleading, and the next check showed why.

Check the network in both directions. From node 2, run ping -c 3 <NODE_1_PRIVATE_IP>, and from node 1, run ping -c 3 <NODE_2_PRIVATE_IP>. The -c 3 option sends three packets and stops. If you see 0% packet loss, the security group is letting the nodes reach each other. If you see 100% loss, one instance isn't in the shared security group. Fix it under EC2, then Instances, then Actions, then Security, then Change security groups.

Check what the running node is actually doing. Start node 1 and wait a minute, then run sudo ss -tulnp | grep java. In my case the output showed a UDP line for the multicast address and no TCP 7800 at all. So the node was still in multicast mode, even though the file said UNICAST. The platform sync had overwritten my hand edits, and setting the properties through the UI fixed it. Once I'd done that and restarted node 1, ss showed a TCP LISTEN on port 7800, and node 2 joined.

You can also test the cluster port directly from the other node:

sudo yum install nmap-ncat -y
nc -zv <NODE_1_PRIVATE_IP> 7800

nc -z opens a connection and closes it without sending data. -v tells you what happened. A "connection refused" while ss shows nothing on port 7800 means the service isn't listening, which is a configuration problem. A "connection refused" or timeout while ss does show 7800 means a firewall problem.

Finish with an ordered restart, so no stale state survives:

# on node 2, then node 1
./atom stop

# once, from either node: clear the old snapshot files
rm -f /mnt/boomi/Molecule_<YOUR_NAME>/bin/views/*.dat

# on node 1
./atom start
# wait until green in Runtime Management, then wait one more full minute
sudo ss -tlnp | grep 7800   # must show java on 7800 before you continue

# on node 2
./atom start

Deleting the .dat files is safe, because they're regenerated within a minute. Skipping the wait between node 1 and node 2 is what creates the race condition in cause 3.

Problem: NODE_JAVA_HOME_MISMATCH

Right after the cluster formed, another warning appeared: NODE_JAVA_HOME_MISMATCH, saying node 2 was out of sync with the current Java home.

Read the bracket in the message first. It says node 2 is reporting that node 1 is the head. That's good news, because it means MULTIPLE_HEAD_NODES is resolved.

The remaining issue comes from the "store runtime assets locally" choice during installation. Each node keeps its own copy of the Java runtime. Node 1's copy was created by the installer, but node 2's was created when it first started and hadn't been marked current yet. Restarting node 2 refreshes it:

# on node 2
./atom stop
./atom start

If the warning comes back, force node 2 to rebuild its local copy from the shared installation:

./atom stop
rm -rf /home/ec2-user/boomi_local/*
./atom start

This deletes only node 2's local cache. It never touches the shared installation.

Part 7: Configure the Shared Web Server

A Molecule doesn't listen on any inbound HTTP port until you configure one. Until you do, the API Management page warns that no listener ports are set, and curl http://localhost:9090 on a node says it can't connect.

Set the port in the platform

In Runtime Management, open the Molecule, then Shared Web Server. Check that the breadcrumb at the top names the Molecule, not an individual Atom. Under Listening Port Configuration, add a row with port 9090, SSL unchecked, and Authentication Type set to None. That's fine for testing only. Scroll to the bottom and save.

Restart the nodes

A new listener port only opens at startup, so both nodes have to restart. In a cluster, restart one node at a time so the Molecule stays online:

# on node 2
./atom stop

# on node 1
./atom restart
# wait until green, plus one more minute

# on node 2
./atom start

A Molecule node takes longer to come up than an Atom, so be patient.

Verify on each node

sudo ss -tlnp | grep 9090
curl http://localhost:9090

You want a LISTEN line owned by java on port 9090. The curl should return an HTML page reading Error 404 NOT_FOUND. That 404 is the correct result. It proves the web server is running. There's just nothing deployed at the root path yet.

A note on the Base URL column: it shows node 1's hostname, but every node opens the port, so requests to node 2's IP on 9090 work the same way. A load balancer in front of the nodes is what gives clients one address. You don't need it for testing.

Problem: the port didn't open after the restart

On my first restart, ss only showed port 7800 and a random port, and curl couldn't connect. The setting was in the config file, though.

The container log explained why. The web server libraries loaded at one time, and about five minutes later the log printed the full property set, including sharedServer.http.connector.0.port=9090. The property arrived after the restart, because I'd saved the change in the UI after restarting the node, not before. The runtime accepted the new value, but a listener port only binds at startup.

The fix was another restart on each node, followed by a 90-second wait. The rule is: save first, restart second, then wait a full 90 seconds before you check.

If the port still doesn't show up after a correctly ordered save and restart, try deploying a listener process to the Molecule and check again. On some versions the shared web server doesn't bind until at least one Web Services Server process is deployed.

Part 8: Next steps

With the cluster online and the web server listening, the Molecule is ready for real work. These are the next things I plan to do.

1. Deploy a test listener. A simple one is a Start step set to Web Services Server with a REST resource and POST, a Message step, and Return Documents. Deploy it to the Molecule's environment. Because the Molecule lives in a different account from my original Atom, I had to rebuild the process there.
2. Load-test each node separately. Run Autocannon from inside a node, pointing it at each node's private IP one at a time. Then check Process Reporting. Executions should show both node IDs, which proves the cluster is actually sharing work.
3. Put a load balancer in front. An Application Load Balancer with a target group containing both nodes on port 9090 gives you one stable URL and spreads requests across the nodes. That's the production shape of a Molecule. I cover this in the next post.
4. Turn on HTTPS and authentication on the Shared Web Server before exposing anything beyond a test network.

Day-to-day operation

Stopping for the day

Order matters in a cluster. Stop the member node first and the head node last, so the head doesn't spend time re-electing itself. On node 2, run ./atom stop and then ./atom status to confirm it's down. Then do the same on node 1.

Then, in the AWS Console, go to EC2, then Instances, select both nodes, and choose Instance state, then Stop instance. Stopping ends the compute charge, which is almost all of the cost. EFS, the security group, and the key pair are left alone.

Resuming

1. Start both instances in the console. Their public IPs will have changed, so copy the new ones for SSH. The private IPs are the same, so the unicast list still works.
2. On each node, run df -h /mnt/boomi to confirm EFS came back through /etc/fstab. If it didn't, run sudo mount -a.
3. On node 1, run ./atom start and wait for green plus one minute.
4. On node 2, run ./atom start.
5. Check Cluster Status.

Pause versus tear down

	Stop instances	Terminate instances and delete EFS
Compute charge	None	None
Storage charge	Root disks of stopped instances plus EFS, a few rupees per day	None
Molecule preserved?	Yes, exactly as you left it	No, you'll need a full reinstall
When to choose it	Pausing for days or weeks	When the experiment is finished

If you terminate, also release any Elastic IPs, delete leftover EBS volumes and snapshots, and remove the Molecule from Runtime Management so it doesn't stay behind as an offline runtime.

Common mistakes, in the order they happen

Stage	Mistake	Symptom	Fix
Before you start	Not checking the Molecule licence count	Installer fails at "Retrieving Container ID" and rolls back	Check Licensing first, free a slot, or use another account
Before you start	Downloading from the wrong Boomi account	Molecule shows up in the wrong account, or the token is rejected	Switch accounts, generate the token there, reuse the installer
Security group	Leaving the description blank	The form won't submit	Type any text
Security group	Adding self-referencing rules during creation	"CIDR block, a security group ID or a prefix list has to be specified"	Create with SSH and 9090 first, then add the two self-referencing rules by editing
Security group	Typing the group name in the Description column	Rule saves but does nothing	Put the group in the Source search box
EFS	Mounting before the targets are Available or before attaching the security group	Mount hangs, then "Connection timed out"	Wait for Available, then assign the group under Network, then Manage
EFS	Missing .efs in the DNS name, or stray backticks	"Failed to resolve server"	Copy the DNS name from the EFS console exactly
SSH	Using the private IP from your laptop	Connection reset by peer	Use the public IP
SSH	Your laptop's IP changed since the rule was created	Operation timed out on port 22	Edit the SSH rule's source to My IP again
SSH	Running ssh or scp while already on a server	Key file not accessible, or cannot stat the local file	Read the prompt, and run ssh and scp from your laptop
Terminal tabs	Losing track of which tab is which node	Commands run on the wrong machine	Quit the app, reconnect deliberately, and check each prompt
scp	Leaving off :~/ at the end of the destination	Copy fails or goes to the wrong place	Use user@host:~/
Installer	Typing an email or token at a [1] or [2] prompt	The prompt repeats	Type the digit
Installer	Choosing option 1 when you meant a token	Asked for a username and password	Press Ctrl+C, run again, answer 2
Installer	Typing the environment name or ID instead of its number	The list reprints	Type the number in brackets
Installer	Typing y at a [1] or [2] prompt	The prompt repeats	Type 1 or 2
Installer	Pressing Enter on a required field with empty brackets	"... is required," then loops back	Type the value
Installer	Accepting the default installation directory	Node 2 has nothing to start	Install under /mnt/boomi
Installer	Node 2's boomi_local folder not created	Node 2 fails to start or reports a missing local directory	mkdir -p /home/ec2-user/boomi_local on every node
Unicast	Leaving multicast on	MULTIPLE_HEAD_NODES, and the nodes never join	Set Cluster Network Transport Type to UNICAST and set the initial hosts
Unicast	Editing container.properties by hand	The lines are there, but the node is still in multicast mode	The platform overwrites the file on sync. Set the properties in the UI, then restart
Unicast	Searching for the old property name	Can't find "Cluster Communication Method"	It's now called Cluster Network Transport Type
Unicast	Using public IPs in the initial hosts list	Nodes can't connect on 7800	Use private IPs
Start-up	Starting node 2 too soon after node 1	MULTIPLE_HEAD_NODES from stale snapshots	Wait for green plus a minute, delete bin/views/*.dat, and restart in order
Cluster	NODE_JAVA_HOME_MISMATCH after joining	Severe issue on node 2	Restart node 2, and if needed clear its boomi_local folder
Web server	Restarting before saving the port	9090 never opens	Save first, restart, then wait 90 seconds
Web server	Treating the root-path 404 as a failure	Confusion	A 404 from / means the web server is up

Command reference

On your laptop

Command	What it does
chmod 400 ~/Downloads/<KEY_FILE>.pem	Makes the key file readable only by you. SSH refuses keys with looser permissions. Run this once
ssh -i ~/Downloads/<KEY_FILE>.pem ec2-user@<PUBLIC_IP>	Opens a terminal on a node. -i selects the key file
scp -i ~/Downloads/<KEY_FILE>.pem <local-file> ec2-user@<PUBLIC_IP>:~/	Copies a file from your laptop to the node's home folder

On the nodes: setup

Command	What it does
sudo yum update -y	Updates the operating system packages. -y answers yes to prompts
sudo yum install java-17-amazon-corretto nfs-utils -y	Installs a system Java and the NFS client tools
sudo mkdir -p /mnt/boomi	Creates the folder the shared disk attaches to. -p creates parent folders and doesn't complain if the folder exists
sudo mount -t nfs4 -o nfsvers=4.1 <EFS_DNS_NAME>:/ /mnt/boomi	Attaches the root of the EFS file system to /mnt/boomi using NFS 4.1
df -h /mnt/boomi	Shows what's mounted at that path and how much space it has
`echo ":/ /mnt/boomi nfs4 nfsvers=4.1,_netdev 0 0" \	sudo tee -a /etc/fstab`
sudo mount -a	Mounts everything listed in /etc/fstab that isn't already mounted
sudo chown ec2-user:ec2-user /mnt/boomi	Makes ec2-user the owner of the shared folder
touch <file>, ls <dir>, rm <file>	Create an empty file, list a folder, or delete a file
mkdir -p /home/ec2-user/boomi_local	Creates the per-node local assets folder

On the nodes: installer

Command	What it does
chmod u+x molecule_install64.sh	Gives the downloaded installer permission to run
./molecule_install64.sh	Runs the installer. The ./ is needed because the script isn't on the PATH
Ctrl+C	Cancels a running command. It's safe to use before the installer's summary screen

On the nodes: Molecule control

Run these from /mnt/boomi/Molecule_<YOUR_NAME>/bin.

Command	What it does
./atom start	Starts this node
./atom stop	Stops this node cleanly
./atom restart	Stops, then starts
./atom status	Prints whether this node is running

On the nodes: diagnostics

Command	What it does
sudo ss -tlnp	Lists TCP ports that are listening, and which process owns each
`sudo ss -tulnp \	grep java`
ping -c 3 <private-ip>	Sends three packets to the other node to test basic reachability
nc -zv <private-ip> 7800	Tests whether a TCP connection to the cluster port succeeds. Install it with sudo yum install nmap-ncat -y
grep -i cluster .../conf/container.properties	Shows the cluster settings the runtime currently holds
grep -r 9090 .../conf/	Finds any file under conf that mentions the port
`grep -i -E "pattern1\	pattern2" \
{% raw %}tail -50 <logfile>	Shows the last 50 lines of a file
rm -f .../bin/views/*.dat	Deletes the cluster snapshot files. They regenerate on start. Use this to clear stale head-node state

Appendix A: How to read installer prompts

Every question the console installer asks follows one of three patterns. Once you recognize them, most of the installer mistakes go away.

Pattern	Example	What to type
Options with characters in brackets	User Name and Password [1, Enter], Token [2], or Yes [y, Enter], No [n]	Exactly the character in the brackets. Nothing else is accepted. The option marked Enter is the default
A default value in brackets	Choose an installation directory [/home/ec2-user/...]	Enter to accept the default, or type a replacement
Empty brackets	Token [] or Working Data Local Storage Directory (required) []	There's no default. Type the value. Enter on its own gets rejected

If a prompt keeps repeating, you've given it something it doesn't recognize. Look at the brackets again.

Appendix B: Where the logs are

All of these paths are under the shared installation, so you can read them from either node.

Path	Contents
/mnt/boomi/Molecule_<YOUR_NAME>/logs/YYYY_MM_DD.container.<node-id>.log	One container log per node per day. The node ID is the private IP with underscores, for example <NODE_ID>. This is the first place to look for cluster and web server problems
/mnt/boomi/Molecule_<YOUR_NAME>/conf/container.properties	The runtime properties as last synced from the platform
/mnt/boomi/Molecule_<YOUR_NAME>/bin/views/*.dat	Cluster snapshot files, one per node
/home/ec2-user/boomi_local/ (on each node)	Per-node local copies of the Java runtime, libraries, and working data

Useful search terms for the container log are cluster, unicast, multicast, MULTIPLE_HEAD_NODES, jgroups.bind_addr, sharedServer, jetty, and 9090. Searching for Container Properties for account shows the full property set at the moment the platform pushed it, with a timestamp you can compare to your restart time.

Closing

Getting the cluster to form was the hardest part. Most of the problems came from one thing: the nodes couldn't see each other, and the cause wasn't obvious from the error messages. The fix was switching to unicast through the UI, checking the actual open ports rather than trusting the settings page, and restarting in the right order.

In the next post I'll put an Application Load Balancer in front of these two nodes, so clients get one address and keep working when a node goes down.