So, if you have read my intro post when I started this blog like forever ago (Feb 7, 2021 to be exact), then you know that I’m super excited about a project called Atomizer. I’ve been slowly stringing you (my readers) along for the last few posts to build up some background so that I can reference it when I start talking about MPDC (Massively Parallel Distributed Computing).
When I built out Atomizer with a team of students (shout out to Matthew, Megan, and Nick) at the University of Illinois Springfield, we tested it with what is known as a Monte Carlo π simulation. The Monto Carlo π simulation showed that the system functioned as a distributed computing framework using a mix of node- and cluster-level parallelization. Monte Carlo π simulations are commonly used to explain how algorithms can be parallelized and distributed, so it worked well for those purposes.
Assume that you had a square dartboard, and there was a circle inside that square that met the edges of the square (See Figure 1).
If you throw darts at the dart board then calculate the number of darts that landed in the circle compared to the total number of darts thrown, then you can estimate the value of π.
π = 4 x ( # darts in circle / total # of tosses)
The equation for this simulation can be parallelized by making each throw of the dart its own process. It 1) calculates its “toss” or random x/y coordinates, 2) determines if the toss is in the circle, and 3) reports back to the original process. This algorithm allows for each toss to happen in its own “simulation.”
Once the parent process receives the results of all processes, it can then calculate the final π value.
So, in honor of π day 2021, I’ve set up Atomizer so you can try a Monte Carlo π simulation yourself!
Once you have the above applications installed on your system it’s super easy to run your own simulation using an Atomizer cluster.
From your terminal or git UI clone the following repo and move into the
git clone https://github.com/devnw/atomizer-test-agent cd ./atomizer-test-agent
Execute the following code in a terminal window from the
atomizer-test-agent directory. Figure 2 shows what this cluster looks like graphically.
NOTE: You can adjust the number of nodes
docker-compose starts by adjusting the
--scale atomizer=3 argument from 3 to whatever number you want.
# Pull down the docker images from docker hub for this compose file docker-compose pull # Spin up the rabbitmq instance and 3 node Atomizer cluster (pre-loaded with the Monte Carlo π simulation) docker-compose up --scale atomizer=3
docker-compose did not work, click here to visit the
docker-compose trouble shooting section.
NOTE: You may see messages similar to those below when executing this command. These can be ignored as it indicates that the RabbitMQ instance has not yet fully started listening on the network for the Atomizer nodes to connect to. If you see these kinds of messages, wait 30 seconds or so before moving on to the next step.
# These are normal log messages to see which indicate # the Atomizer nodes are working on connecting to the # RabbitMQ instance waiting for host [atomrabbit:5672], attempt 3 failed waiting for host [atomrabbit:5672], attempt 4 failed waiting for host [atomrabbit:5672], attempt 5 failed
From _ another _ terminal window install the
atomizer-test-console using the following command specific to your version of
# For Go 1.16 go install atomizer.io/test-console@latest # For Other Go Versions go get atomizer.io/test-console@latest
After executing these commands the
test-console should have been installed by the
Go tool. From here execute the following command, which will connect to the cluster:
NOTE: If this doesn’t work, click here to visit the troubleshooting section.
The cluster uses default connection information for RabbitMQ so the
test-console should immediately connect and you should see something similar to this:
Enter Tosses: _
Enter the number of tosses you want the simulation to run and hit enter.
The results of the simulation should be displayed fairly quickly depending on the number of tosses you entered. Look at the
test-console once the calculation finishes, and it should show something similar to the example below:
Enter Tosses: 500000 Pi Estimation: 3.151984
Since Monte Carlo π is a simulation, it is important to point out that the π estimation that results are not going to be exact and will not be the same every time – even if the number of tosses is the same. Also, the π estimation will be more accurate the higher the number of tosses.
Fair warning though, the Atomizer system is intended to run on multiple systems not a singular system, and it will very quickly consume a large portion of your system’s CPU if the number of tosses is high.
docker-compose file that is included in the
atomizer-test-agent repository has disabled logging. This was done to maximize the throughput of the Atomizer cluster, since
IO can greatly impede performance if it’s unnecessary to the functionality of the application.
If you want to see what the cluster is doing, though, you can re-enable logging by updating the
docker-compose.yml file and adding
- LOGLEVEL=INFO in the
environment section of the file like this:
atomizer: image: 'atomizer/test-agent:latest' depends_on: - "rabbit-mq" environment: - CONNECTIONSTRING=... - QUEUE=... - LOGLEVEL=INFO
This setting creates several listeners in the
atomizer-test-agent which will display events as they pass through the communication elements of the AMQP system as well as the Atomizer engine itself. Once you’ve made this change, then re-run the
docker-compose command to stand up your cluster and re-run the simulation with the
NOTE: If you stop and re-start the cluster, you will need to stop and re-start the
test-console in order to re-connect to the new RabbitMQ instance.
docker-compose.yml file is set up to use a specific version of RabbitMQ which exposes the RabbitMQ management console. If you want to view the management console and see the messages as they go through RabbitMQ while running a simulation, then navigate to http://localhost:8080/
At the login screen enter the default credentials
guest and click Login.
Under the Queues section you will see that there is an
atomizer queue, and, once you start a simulation, you will see a couple more queues show up in the list, similar to what you see in Figure 4.
Each of the three queues are important. The primary queue – where requests are pulled from for each node – is the
atomizer queue. Then, one of the other two queues is the queue which the
test-console listens to, and the third is the one the
montecarlopi atom listens to for the results of the toss simulations.
The coordination between these queues allows for messages to flow to the appropriate destinations so that results can be calculated and viewed by the user.
After you are done running the simulation, don’t forget to cleanup your environment by running the following commands:
docker-compose stop docker-compose rm
docker-compose command is not working as expected, here is a set of docker commands you can run instead to stand up a three node cluster.
NOTE: The following commands add logging to
stdout. If you want to disable logging and improve performance, remove
-e LOGLEVEL=INFO from the command.
# Setup a Docker network for the cluster to communicate on docker network create atomizer_nw # Setup the RabbitMQ container instance docker run -d --rm=true --hostname my-rabbit --name some-rabbit -p 8080:15672 -p 5672:5672 --network=atomizer_nw rabbitmq:3-management # Setup each Atomizer node in a different terminal, or run each command with the `-d` flag to ignore it's output docker run --rm=true -e CONNECTIONSTRING=amqp://guest:guest@some-rabbit:5672/ -e QUEUE=atomizer -e LOGLEVEL=INFO --network=atomizer_nw atomizer/test-agent:latest docker run --rm=true -e CONNECTIONSTRING=amqp://guest:guest@some-rabbit:5672/ -e QUEUE=atomizer -e LOGLEVEL=INFO --network=atomizer_nw atomizer/test-agent:latest docker run --rm=true -e CONNECTIONSTRING=amqp://guest:guest@some-rabbit:5672/ -e QUEUE=atomizer -e LOGLEVEL=INFO --network=atomizer_nw atomizer/test-agent:latest
If the atomizer
test-console did not work when you tried to run it, then pull the repository down and build it directly using the following commands:
git clone https://github.com/devnw/atomizer-test-console cd ./atomizer-test-console go build . ./test-console
This will build the
test-console and allow for direct execution.