Orchestrating Application Workloads in Distributed Embedded Systems: Writing and Scaling a Pub Application - Part 2

Introduction

This blog covers the second part of Orchestrating Application Workloads in Distributed Embedded Systems. We will go over how to expose Greengrass IPC in a Nomad cluster and have containerized applications publishing metrics to AWS IoT Core using the Greengrass IPC.

Prerequisites

It is essential to follow the first part of the blog, which covers bootstrapping devices with AWS IoT Greengrass and HashiCorp Nomad. Once that is done, we can jump into the application part and the required configuration. As a reminder, the source code is located here: https://github.com/aws-iot-builder-tools/greengrass-nomad-demo

Greengrass IPC Proxy

In order for applications to access Greengrass IPC, we need to create a proxy. We will use socat to forward the ipc.socket via the network (TCP) and then use socat on the application side to create an ipc.socket file. The example can be found under ggv2-nomad-setup/ggv2-proxy/ipc/recipe.yaml. Here we deploy the Nomad job:

job "ggv2-server-ipc" {
    datacenters = ["dc1"]
    type = "system"
    group "server-ipc-group" {

        constraint {
            attribute = "\${meta.greengrass_ipc}"
            operator  = "="
            value     = "server"
        }

        network {
            port "ipc_socat" {
                static = 3307
            }
        }
        service {
            name = "ggv2-server-ipc"
            port = "ipc_socat"
            provider = "nomad"
        }

        task "server-ipc-task" {
            driver = "raw_exec"
            config {
                command = "socat"
                args = [
                    "TCP-LISTEN:3307,fork,nonblock",
                    "UNIX-CONNECT:$AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT,nonblock"
                ]
            }
        }
    }
}

This job will use the AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT provided by the Greengrass Component deployment and run a socat command by connecting to the defined socket and exposing it over TCP on a reserved port 3307. Note that the deployment of this job will have constraints and will only target the devices tagged as greengrass_ipc=server, as this is intended to be deployed only on a client where Greengrass is running.

To deploy this to our Greengrass device, we will use the same methods from the previous blog post. Which should look something like this:

Start with building and publishing the component by doing gdk build and gdk publish, making sure you are in the ggv2-nomad-setup/ggv2-proxy/ipc/ directory.

Additionally, to deploy this to the targets, we will need to add this to a deployment.json:

        "ggv2.nomad.proxy.ipc": {
            "componentVersion": "1.0.0",
            "runWith": {}
        }

respecting the name of the component and the version provided by the GDK.

After that executing the command below will deploy it to our target:

aws greengrassv2 create-deployment \
    --cli-input-json file://deployment.json\
    --region ${AWS_REGION}

Once the command executes successfully, we will be ready to move forward with our application.

Application Overview

We will have a simple application written in python that publishes information about used memory and CPU and publishes this information using the Greengrass IPC to AWS IoT Core. The topic here is constructed by having NOMAD_SHORT_ALLOC_ID as a prefix followed by /iot/telemetry. We will use this info later once we scale the application across the cluster and start receiving messages on multiple MQTT topics.

Here is the Python code for the application:

import json
import time
import os

import awsiot.greengrasscoreipc
import awsiot.greengrasscoreipc.model as model


NOMAD_SHORT_ALLOC_ID = os.getenv('NOMAD_SHORT_ALLOC_ID')

def get_used_mem():
    with open('/proc/meminfo', 'r') as f:
        for line in f:
            if line.startswith('MemTotal:'):
                total_mem = int(line.split()[1]) * 1024  # convert to bytes
            elif line.startswith('MemAvailable:'):
                available_mem = int(line.split()[1]) * 1024  # convert to bytes
                break

    return total_mem - available_mem

def get_cpu_usage():
    with open('/proc/stat', 'r') as f:
        line = f.readline()
        cpu_time = sum(map(int, line.split()[1:]))
        idle_time = int(line.split()[4])

    return (cpu_time - idle_time) / cpu_time

if __name__ == '__main__':
    ipc_client = awsiot.greengrasscoreipc.connect()

    while True:
        telemetry_data = {
            "timestamp": int(round(time.time() * 1000)),
            "used_memory": get_used_mem(),
            "cpu_usage": get_cpu_usage()
        }

        op = ipc_client.new_publish_to_iot_core()
        op.activate(model.PublishToIoTCoreRequest(
            topic_name=f"{NOMAD_SHORT_ALLOC_ID}/iot/telemetry",
            qos=model.QOS.AT_LEAST_ONCE,
            payload=json.dumps(telemetry_data).encode(),
        ))
        try:
            result = op.get_response().result(timeout=5.0)
            print("successfully published message:", result)
        except Exception as e:
            print("failed to publish message:", e)

        time.sleep(5)

The application can be found under examples/nomad/nomad-docker-pub/app.py. On top of this we will be using a Dokerfile to containerized. In order for this to work with GDK, we will be using build_system: "custom" and specify the script for building and publishing the image to ECR:

{
  "component": {
    "nomad.docker.pub": {
      "author": "Nenad Ilic",
      "version": "NEXT_PATCH",
      "build": {
        "build_system": "custom",
        "custom_build_command": [
          "./build.sh"
         ]
      },
      "publish": {
        "bucket": "greengrass-component-artifacts",
        "region": "eu-west-1"
      }
    }
  },
  "gdk_version": "1.1.0"
}

Where build.sh will look like this:

set -e
AWS_ACCOUNT_ID=$(aws sts get-caller-identity |  jq -r '.Account')
AWS_REGION=$(jq -r '.component | to_entries[0] | .value.publish.region' gdk-config.json)
COMPONENT_NAME=$(jq -r '.component | keys | .[0]' gdk-config.json)
COMPONENT_AUTHOR=$(jq -r '.component | to_entries[0] | .value.author' gdk-config.json)
COMPONENT_NAME_DIR=$(echo $COMPONENT_NAME | tr '.' '-')

rm -rf greengrass-build
mkdir -p greengrass-build/artifacts/$COMPONENT_NAME/NEXT_PATCH
mkdir -p greengrass-build/recipes
cp recipe.yaml greengrass-build/recipes/recipe.yaml
sed -i "s/{COMPONENT_NAME}/$COMPONENT_NAME/" greengrass-build/recipes/recipe.yaml
sed -i "s/{COMPONENT_AUTHOR}/$COMPONENT_AUTHOR/" greengrass-build/recipes/recipe.yaml
sed -i "s/{AWS_ACCOUNT_ID}/$AWS_ACCOUNT_ID/" greengrass-build/recipes/recipe.yaml
sed -i "s/{AWS_REGION}/$AWS_REGION/" greengrass-build/recipes/recipe.yaml
sed -i "s/{COMPONENT_NAME_DIR}/$COMPONENT_NAME_DIR/" greengrass-build/recipes/recipe.yaml

docker build -t $COMPONENT_NAME_DIR .
docker tag $COMPONENT_NAME_DIR:latest $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$COMPONENT_NAME_DIR:latest

if aws ecr describe-repositories --region $AWS_REGION --repository-names $COMPONENT_NAME_DIR > /dev/null 2>&1
then
    echo "Repository $COMPONENT_NAME_DIR already exists."
else
    # Create the repository if it does not exist
    aws ecr create-repository --region $AWS_REGION --repository-name $COMPONENT_NAME_DIR
    echo "Repository $COMPONENT_NAME_DIR created."
fi
aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com
docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/$COMPONENT_NAME_DIR:latest

The script assumes the AWS CLI is installed and gets all the necessary configuration from the gdk-config.json . The build script will create the appropriate recipe, build the docker image, login to ECR and push it referencing the component name set in the gdk-config.json.

Finally the Nomad Job for deploying the application will look like this:

job "nomad-docker-pub-example" {
    datacenters = ["dc1"]
    type = "service"
    group "pub-example-group" {
        count = 1
        constraint {
            attribute = "\${meta.greengrass_ipc}"
            operator  = "="
            value     = "client"
        }
        task "pub-example-task" {
            driver = "docker"
            config {
                image = "{AWS_ACCOUNT_ID}.dkr.ecr.{AWS_REGION}.amazonaws.com/{COMPONENT_NAME_DIR}:latest"
                command = "/bin/bash"
                args = ["-c", "socat UNIX-LISTEN:\$AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT,fork,nonblock TCP-CONNECT:\$GGV2_SERVER_IPC_ADDRESS,nonblock & python3 -u /pyfiles/app.py "]
            }
            env {
                AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT = "/tmp/ipc.socket"
                SVCUID="$SVCUID"
            }
            template {
                data = <<EOF
# Get all services and add them to env variables with their names
{{ range nomadServices }}
    {{- range nomadService .Name }}
    {{ .Name | toUpper | replaceAll "-" "_" }}_ADDRESS={{ .Address}}:{{ .Port }}{{- end }}
{{ end -}}
EOF
                destination = "local/env"
                env = true
            }
        }
    }
 }

1. Constraint - We are starting with our constraint where this application should be deployed. In this scenario, it would be targeting only Nomad clients where greengrass_ipc=client.
2. Task - Next, we have our task with the Docker driver. Here, we get the image from the ECR, where the variables AWS_ACCOUNT_ID, AWS_ACCOUNT_ID, and COMPONENT_NAME_DIR will be replaced by the build script with the appropriate values. Finally, we come to our command and args. These values would override what is already defined by the Dockerfile. In this scenario, we first create the ipc.socket required by the application using socat. The SVCUID will be then provided by the Greengrass component at the time of running the job, thus provided as an environment variable inside the Docker container.
3. Template - After that, we have a template section that we require to obtain the IP address of our ggv2-server-ipc service that we created earlier. We do this by listing all the services and getting the IP addresses and exporting them as environment variables by also converting their names to uppercase letters and appending _ADDRESS at the end. This provides our env variable GGV2_SERVER_IPC_ADDRESS for our socat command that then looks like this:

socat UNIX-LISTEN:$AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT,fork,nonblock TCP-CONNECT:$GGV2_SERVER_IPC_ADDRESS,nonblock

Which provides the ipc.socket before running the app:

python3 -u /pyfiles/app.py

Once we have this set, we can then go build and publish the component by doing gdk build and gdk publish.

Additionally in order to deploy this to the targets we will need to add this to a deployment.json:

        "nomad.docker.pub": {
            "componentVersion": "1.0.0",
            "runWith": {}
        }

respecting the name of the component and the version provided by the GDK.
After that executing the command below will deploy it to our target:

aws greengrassv2 create-deployment \
    --cli-input-json file://deployment.json\
    --region ${AWS_REGION}

Now we will be ready to scale our application.

Scaling the Application

As of now we should have our application running and publishing the data from a single client, however if we require to spread this application across the cluster, in this scenario second client, and have it report the memory and CPU usage, we can do this by simply changing the count=1 to count=2 in our job file:

--- a/examples/nomad/nomad-docker-pub/recipe.yaml
+++ b/examples/nomad/nomad-docker-pub/recipe.yaml
@@ -31,7 +31,7 @@ Manifests:
             type = "service"

             group "pub-example-group" {
-              count = 1
+              count = 2
               constraint {
                 attribute = "\${meta.greengrass_ipc}"
                 operator  = "="

And use the same method to redeploy.

Now if we go to AWS console and under AWS IoT → MQTT test client we can subscribe to topics <NOMAD_SHORT_ALLOC_ID>/iot/telemetry and should be able to see messages coming. In order to get this ID we can simply run the following command on our device where the nomad server is running:

nomad status nomad-docker-pub-example

And we will find the ID int the Allocations section that looks something like this:

Allocations
ID        Node ID   Task Group         Version  Desired  Status  Created     Modified
5dce9e1a  6ad1a15c  pub-example-group  10       run      running 1m28s ago   32s ago
8c517389  53c96910  pub-example-group  10       run      running 1m28s ago   49s ago

This will then allow us to construct those MQTT topics and start seeing the messages coming from those two instances of our application:

In the next part we will take a look on how to access AWS services from the device using Token Exchange Service (TES).

Conclusion

In this blog post, we covered how to expose the Greengrass IPC in a Nomad cluster, allowing a containerized application to publish metrics to AWS IoT Core using Greengrass IPC. We demonstrated how to create a proxy using socat to forward the ipc.socket via network (TCP) and how to set up an application that reports memory and CPU usage. We also showed how to scale the application across multiple clients in the cluster.

By using AWS IoT Greengrass and HashiCorp Nomad, you can effectively manage, scale and monitor distributed embedded systems, making it easier to deploy and maintain complex IoT applications.

If you have any feedback about this post, or you would like to see more related content, please reach out to me here, or on Twitter or LinkedIn.