DEV Community: Nathan Glover

Let's Try - AWS Glue Automatic Compaction for Apache Iceberg

Nathan Glover — Sat, 18 Nov 2023 14:15:00 +0000

Please reach out to me on Twitter @nathangloverAUS if you have follow up questions!

This post was originally written on DevOpStar. Check it out here

Introduction

Apache Iceberg has been a table format I've been diving deeper and deeper into as of late. Specifically, I've been using the AWS variant supported by AWS Glue and Amazon Athena. The table format has several powerful features that make it an excellent choice for data lake storage - however, one of those features, compaction, was not supported by AWS Glue until recently.

Before the release of automatic compaction of Apache Iceberg tables in AWS Glue, you had to run a compaction job to optimize your tables manually. This was a bit of a pain, given it was a manual query you had to run against the table - This query had a timeout as well, and you would receive the dreaded ICEBERG_OPTIMIZE_MORE_RUNS_NEEDED error after a timeout. This meant you had to run the query again and again and again until it was finally completed.

In this post, we'll look at how to use the new automatic compaction feature in AWS Glue and how it can help you optimize your Iceberg tables.

Why Compaction?

The Apache Iceberg documentation has a great section on why compaction is important. In short, compaction is merging small files into larger files. This is important for several reasons, but the most important are performance and cost. Small files are inefficient to read from and can cause performance issues - especially when reading from S3, where you are charged per request.

When you run a compaction job, you merge small files into larger ones. This means you are reducing the number of files that need to be read from and the number of requests that need to be made to S3. This can have a significant impact on performance and cost.

Pre-requisites

To follow along with this post, you must do some minor setup with Athena, S3 and Glue. If you have used Athena before, there's a good chance you already have this setup (possibly with a different bucket). Feel free to skip this section if you feel confident you have the required setup.

S3

Navigate to the AWS S3 console to start with and create an Athena query bucket

Give the bucket a name such as athena-<region>-<account-id> and click Create bucket. (For example, athena-us-west-2-123456789012.)

Note: for this post, I'll be using the us-west-2 region as it is one of the regions that supports the new automatic compaction feature.

Athena

Navigate to the AWS Athena settings console and click Manage under the Query result and encryption settings section.

Change the Location of query results to the bucket you created in the previous step, and click Save.

Glue

Navigate to the AWS Glue databases console and check to see if you have a database called default. If you do not, create one.

Helper Script

To help demonstrate the new automatic compaction feature, I've created a script to do much of the tedious parts of creating an Iceberg table for you. You can find the script here.

At a high level, the script does the following:

Create an S3 bucket that Iceberg will use to store data and metadata
Generating 1 million rows of sample data and uploading it to S3
Create SQL files for:
- Creating an Iceberg table
- Creating a temporary table for loading sample data
- Loading sample data into Iceberg from the temporary table
- Deleting the tables when we're done
Creating an IAM role for Glue to use for compaction

Let's grab the script and run it so you can follow along.

Note: You must set up AWS credentials on your machine for this to work. If you don't have them set up, you can follow the AWS CLI configuration guide.

# Download the script, make it executable
curl https://gist.githubusercontent.com/t04glovern/04f6f2934353eb1d0fffd487e9b9b6a3/raw \
    > lets-try-iceberg.py \
    && chmod +x lets-try-iceberg.py

# Create a virtual env (optional)
python3 -m venv .venv
source .venv/bin/activate

# Install the dependencies
pip3 install boto3

# Run the script
./lets-try-iceberg.py

Have a look at the output, and you should see something like the following:

$ ./lets-try-iceberg.py
# INFO:root:Bucket iceberg-sample-data-477196 does not exist, creating it...
# INFO:root:Uploaded samples.jsonl.gz to s3://iceberg-sample-data-477196/sample-data/samples.jsonl.gz
# INFO:root:Created IAM role lets-try-iceberg-compaction-role
# INFO:root:Created IAM policy lets-try-iceberg-compaction-policy

If you check the directory you ran the script from, you should see several files created:

$ ls -l
# -rw-r--r-- 1 nathan nathan      366 Nov 18 17:26 1-athena-iceberg-create-table.sql
# -rw-r--r-- 1 nathan nathan      403 Nov 18 17:26 2-athena-create-temp-table.sql
# -rw-r--r-- 1 nathan nathan       94 Nov 18 17:26 3-insert-into-iceberg-from-temp-table.sql
# -rw-r--r-- 1 nathan nathan       62 Nov 18 17:26 4-cleanup-temp-table.sql
# -rw-r--r-- 1 nathan nathan       50 Nov 18 17:26 5-cleanup-iceberg-table.sql

The numbered SQL files are the ones we'll be using to create our Iceberg table and load the sample data into it.

Create & Load the Iceberg Table

Head over to the AWS Athena console and ensure that the default database is selected.

Take the contents of the 1-athena-iceberg-create-table.sql file and paste it into the query editor. Click Run to create the table.

You should see a new table called lets_try_iceberg_compaction under the Tables section of the default database.

We'll create a temporary table to load the sample data into. Copy the 2-athena-create-temp-table.sql file and paste it into the query editor. Click Run to create the table.

Now, we'll load the sample data from the temporary table into the Iceberg table. Copy the 3-insert-into-iceberg-from-temp-table.sql file and paste it into the query editor. Click Run to load the data.

Note: This query will take 15-30 seconds to run.

Finally, let's verify that the data was loaded into the table. Run the following query:

SELECT * FROM lets_try_iceberg_compaction LIMIT 10;

You should see something like the following:

Compaction in Action

Now that we have our Iceberg table created and our sample data loaded into it let's enable the new automatic compaction feature and see it in action.

Navigate to the AWS Glue tables console and select the lets_try_iceberg_compaction table.

At the bottom of the table details, you should see a tab called Table optimization. Click it, then click Enable compaction.

You will be prompted to select a role to use for compaction. Select the role created by the script we ran earlier (lets-try-iceberg-compaction-role), and click Enable compaction.

Navigate back to the AWS Glue tables console and select the lets_try_iceberg_compaction table again. Get a coffee, and when you come back, refresh the page. You should see something like the following:

At this point, your table should start automatically compacting based on the rate of changes made. According to the blog post AWS Glue Data Catalog now supports automatic compaction of Apache Iceberg tables this rate of change is based on the following:

As Iceberg tables can have multiple partitions, the service calculates this change rate for each partition and schedules managed jobs to compact the partitions where this rate of change breaches a threshold value.

You can force another compaction job to verify this by navigating back to the Athena console and rerunning the following query:

INSERT INTO lets_try_iceberg_compaction
SELECT * FROM lets_try_iceberg_compaction_sample_data

This will insert the sample data into the table again, and you should see another compaction job start in the Glue console.

Cleanup

Once you're done, you can use the 4-cleanup-temp-table.sql and 5-cleanup-iceberg-table.sql files to clean up the temporary and Iceberg tables.

-- 4-cleanup-temp-table.sql
DROP TABLE IF EXISTS lets_try_iceberg_compaction_sample_data;

-- 5-cleanup-iceberg-table.sql
DROP TABLE IF EXISTS lets_try_iceberg_compaction;

Then, navigate to the AWS S3 console and Empty then Delete the bucket that was created by the script.

Finally, navigate to the AWS IAM Roles console and delete the role (lets-try-iceberg-compaction-role) and policy (lets-try-iceberg-compaction-policy) that were created by the script.

Summary

This post looked at the new automatic compaction feature for Apache Iceberg tables in AWS Glue. We saw how to enable it and how it works in action. We also saw how to force a compaction job to run and how to clean up the resources we created.

Something to keep in mind is that this feature's pricing is based on DPU hours. According to the AWS Glue pricing page:

If you compact Apache Iceberg tables, and the compaction run for 30 minutes and consume 2 DPUs, you will be billed 2 DPUs * 1/2 hour * $0.44/DPU-Hour, which equals $0.44.

This pricing could add up quickly if you have a high rate of change on your tables - before this feature, you would have to manually run the compaction job, which, although tedious, would incur minimal costs under the Athena pricing model. You will likely want to keep this feature in mind when deciding whether or not to enable this feature. An alternative would be to run the compaction job manually, using a pattern similar to how I did automatic VACUUM jobs in my Vacuuming Amazon Athena Iceberg with AWS Step Functions post.

I'm excited to see this feature released and looking forward to seeing what other features are added to the Glue variant of Iceberg in the future.

If you have any questions, comments or feedback, please get in touch with me on Twitter @nathangloverAUS, LinkedIn or leave a comment below!

Automatic API Key rotation for Amazon Managed Grafana

Nathan Glover — Sun, 26 Feb 2023 04:15:00 +0000

Please reach out to me on Twitter @nathangloverAUS if you have follow up questions!

This post was originally written on DevOpStar. Check it out here

Amazon Managed Grafana has an unfortunate limitation where API keys created have a maximum expiration of 30 days. This limitation is quite frustrating if you were trying to automate the deployment of Grafana dashboards and datasources as part of your CI/CD pipeline - as you would need to manually update the API key every 30 days or your deployments would fail.

This problem is exacerbated by the fact that Amazon Managed Grafana API keys are billed out at the cost of a full user license - so you cannot simply create a new API key every time you deploy your dashboards either. I found this out the expensive way when I didn't read the pricing guide properly and created an API key for each deployment ($8 a key).

Hopefully, Amazon will address this limitation in the future - but in the meantime, I've written a simple pattern that can be used to automatically rotate an API key every 30 days and store it for use in AWS Secrets Manager. At the end of this post, I outline my plea to Amazon to address this limitation along with some suggestions on how they could do it.

Overview of the solution

I've opted to build this example in terraform as it is most likely you will be wanting to deploy Grafana dashboards as part of your CI/CD pipeline. Terraform is arguably the best tool for this - however, there is no reason why the code in this post couldn't be adapted to work with other tools such as CloudFormation or AWS CDK.

The solution is made up of two components:

AWS Secret is created with a rotation lifecycle policy that will trigger a Lambda function every 30 days
AWS Lambda Function that will create a new API key in Amazon Managed Grafana and update the AWS Secret with the new key

It is expected that you will have already created an Amazon Managed Grafana instance (though you can copy the terraform from this post side by side with your existing terraform and it will work).

The source code for this example can be found at t04glovern/amazon-managed-grafana-api-key-rotation-terraform.

Solution Walkthrough

We'll begin by looking at the python code that is in charge of rotating the Managed Grafana API keys - as understanding how that works will help when we look at the terraform code.

Look at src/rotate.py in the source code for this example

The function is going to expect three environment variables to be present

grafana_secret_arn = os.environ['GRAFANA_API_SECRET_ARN']
grafana_api_key_name = os.environ['GRAFANA_API_KEY_NAME']
grafana_workspace_id = os.environ['GRAFANA_WORKSPACE_ID']

NOTE: While you don't technically need to pass the secret ARN as it is available in the Lambda context when the function is invoked by secrets manager, I've opted to do so as it makes it easier to understand.

Next, we attempt to delete any existing API keys with the same name as the one we are about to create. This is to ensure that we clean up old API keys and we don't get billed for duplicates (even though you cannot have multiple API keys with the same name).

try:
    grafana_client.delete_workspace_api_key(
        keyName=grafana_api_key_name,
        workspaceId=grafana_workspace_id
    )
except grafana_client.exceptions.ResourceNotFoundException:
    pass

Following cleanup, we create a new API key with a 30-day expiration.

try:
    new_api_key = grafana_client.create_workspace_api_key(
        keyName=grafana_api_key_name,
        keyRole='ADMIN',
        secondsToLive=2592000,
        workspaceId=grafana_workspace_id
    )['key']
except botocore.exceptions.ClientError as error:
    logger.error(error)
    return {
        'statusCode': 500,
        'message': 'Error: Failed to generate new API key'
    }

The last step is to update the AWS Secret with the new API key.

try:
    secretmanager_client.update_secret(
        SecretId=grafana_secret_arn,
        SecretString=new_api_key
    )
except botocore.exceptions.ClientError as error:
    logger.error(error)
    return {
        'statusCode': 500,
        'message': 'Error: Failed to update secret'
    }

Now that we've seen how the Lambda function works, let's look at the terraform code that will deploy it.

variables.tf

There are two expected variables for this solution to function - You can however substitute these with hardcoded values or references to your own terraform resources instead.

variable "name" {
  type        = string
  description = "Named identifier for the workspace and related resources"
}

variable "grafana_workspace_id" {
  type        = string
  description = "The ID of the Grafana workspace to manage"
}

main.tf

The main.tf file is where the bulk of the solution is defined. The first thing we do is create an AWS Secret that will store the API key.

resource "aws_secretsmanager_secret" "api_key" {
  name = "${var.name}-api-key"
}

The next part looks complicated but is required to bundle the python code into a zip file that can be deployed to Lambda. The code is zipped up and stored in a zip directory alongside src and is not checked into source control.

resource "random_uuid" "lambda_src_hash" {
  keepers = {
    for filename in setunion(
      fileset("${path.module}/src/", "*.py"),
      fileset("${path.module}/src/", "requirements.txt"),
    ) :
    filename => filemd5("${path.module}/src/${filename}")
  }
}

data "archive_file" "lambda_zip" {
  depends_on = [
    null_resource.install_dependencies
  ]

  type       = "zip"
  source_dir = "${path.module}/src/"
  excludes = [
    "__pycache__"
  ]
  output_path = "${path.module}/zip/${random_uuid.lambda_src_hash.result}.zip"
}

Unfortunately, because AWS Lambda python runtime is not running the most up-to-date boto3 core version - we must force the terraform to install and zip a more recent version of boto3 which complicates this solution quite a lot. If you are reading this post in the future, check out the AWS Lambda Python Runtime page to see if this is still required. As of writing this, version boto3-1.20.32 is the latest version available and boto3-1.26.65 is needed.

resource "null_resource" "install_dependencies" {
  provisioner "local-exec" {
    command = "pip install -r ${path.module}/src/requirements.txt -t ${path.module}/src/  --upgrade"
  }

  triggers = {
    dependencies_versions = filemd5("${path.module}/src/requirements.txt")
  }
}

Skipping over the IAM role and policy terraform (which I won't explain in this post, but you can find in the source code on GitHub), we can see that the Lambda function is created and provided the environment variables we defined earlier.

resource "aws_lambda_function" "api_key_rotation" {
  function_name = "${var.name}-api-key-rotation"

  filename         = data.archive_file.lambda_zip.output_path
  source_code_hash = data.archive_file.lambda_zip.output_base64sha256

  handler = "rotate.lambda_handler"
  runtime = "python3.9"
  environment {
    variables = {
      GRAFANA_API_SECRET_ARN = aws_secretsmanager_secret.api_key.arn
      GRAFANA_API_KEY_NAME   = "${var.name}-mangement-api-key"
      GRAFANA_WORKSPACE_ID   = var.grafana_workspace_id
    }
  }

  role = aws_iam_role.api_key_rotation_lambda_role.arn
}

With both the lambda and secret created, a Secret manager rotation schedule is created to invoke the lambda function every 29 days.

resource "aws_lambda_permission" "secrets_manager_api_key_rotation" {
  statement_id  = "AllowExecutionFromSecretsManager"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.api_key_rotation.function_name
  principal     = "secretsmanager.amazonaws.com"
}

resource "aws_secretsmanager_secret_rotation" "api_key" {
  secret_id           = aws_secretsmanager_secret.api_key.id
  rotation_lambda_arn = aws_lambda_function.api_key_rotation.arn

  rotation_rules {
    automatically_after_days = 29
  }
}

If you only need the terraform to deploy an API key - but do not necessarily use it straight away, then you could get away with not including the final part of the terraform code. However, if there is a requirement to use the API key immediately after deployment in the same Terraform stack, then you will need to add a null_resource to delay the terraform execution until the secret has been rotated.

I set an arbitrary 20 second delay, but if you wanted to be safe you could increase that.

resource "null_resource" "api_key_delay" {
  provisioner "local-exec" {
    command = "sleep 20"
  }
  triggers = {
    after = aws_secretsmanager_secret_rotation.api_key.id
  }
}

The following terraform can be used to retrieve the API key from the newly created and updated secret.

data "aws_secretsmanager_secret" "api_key" {
  depends_on = [
    null_resource.api_key_delay
  ]
  arn = aws_secretsmanager_secret.api_key.arn
}

data "aws_secretsmanager_secret_version" "api_key" {
  secret_id = data.aws_secretsmanager_secret.api_key.id
}

Plea to Amazon

In this post, we have seen how to use Terraform to deploy an AWS Secret and Lambda function that will rotate the API key for an Amazon Managed Grafana workspace. We did this to get around some frustrating limitations with the way that Amazon Managed Grafana provides API keys that hopefully will be addressed in the future.

My request to Amazon is for them to make this solution I outlined redundant by doing the following:

Provide a way to vend short-lived session tokens for API access against Managed Grafanas API for use in CI/CD pipelines.
If this is not possible, provide a way to create API keys that are not tied to a specific user account - but manage the storage and rotation of these keys in Secrets manager similar to RDS: https://aws.amazon.com/about-aws/whats-new/2022/12/amazon-rds-integration-aws-secrets-manager/
If this is not possible, reduce the price of API keys to $0.01 per month so we can use them in CI/CD pipelines without worrying about the cost.

If you've had this same problem and can think of a better way to solve it, please let me know on Twitter @nathangloverAUS or in the comments below.

Sort AWS Config and Security Hub emails with Power Automate

Nathan Glover — Sun, 05 Feb 2023 15:39:00 +0000

Please reach out to me on Twitter @nathangloverAUS if you have follow up questions!

This post was originally written on DevOpStar. Check it out here

If you have ever enabled AWS Config or AWS Security Hub before you probably remember getting spammed with compliance emails the day after enabling the functionality. While this is good for audibility it can quickly become a burden to parse through the daily inbox of compliance events.

This isn't helped by the format of the emails, as they come in as JSON blobs and can be difficult for humans to parse them anyway! Computers, however, are very good at reading JSON so it makes sense that there must be a way to parse these incoming compliance emails and highlight only the important compliance alerting.

That's where Microsoft Power Automate comes in. This powerful tool can help you automate the process of sorting, categorizing and prioritizing alerts, freeing up your time to focus on more important tasks. In this blog post, we will take a deep dive into how to use Microsoft Power Automate to sort incoming emails from AWS Config and AWS Security Hub.

Seem below is the final flow that we will be creating!

Overview of the problem

The goal is for us to be able to sort these emails into folders based on the configRuleName and then flag any emails that have a complianceType of NON_COMPLIANT as a high priority. This will make it easier for us to find the important alerts and prioritize them for remediation.

Below is an example of the email that comes in from AWS when a config rule is violated (if you are using AWS Control Tower). Things like the configRuleName and complianceType are the fields that are important to us, as they tell us what rule was violated and whether it was a violation or not.

This example email contains the following JSON block, which we will use later on when constructing the Power Automate flow, so keep it handy.

{
  "version": "0",
  "id": "xxxxxxxx-fc96-3fac-9c63-xxxxxxxxxxxx",
  "detail-type": "Config Rules Compliance Change",
  "source": "aws.config",
  "account": "012345678901",
  "time": "2023-01-31T13:56:17Z",
  "region": "ap-southeast-2",
  "resources": [],
  "detail": {
    "resourceId": "sg-xxxxxxxxxxxxxxxxx",
    "awsRegion": "ap-southeast-2",
    "awsAccountId": "012345678901",
    "configRuleName": "AWSControlTower_AWS-GR_RESTRICTED_COMMON_PORTS",
    "recordVersion": "1.0",
    "configRuleARN": "arn:aws:config:ap-southeast-2:012345678901:config-rule/config-rule-qlw9ls",
    "messageType": "ComplianceChangeNotification",
    "newEvaluationResult": {
      "evaluationResultIdentifier": {
        "evaluationResultQualifier": {
          "configRuleName": "AWSControlTower_AWS-GR_RESTRICTED_COMMON_PORTS",
          "resourceType": "AWS::EC2::SecurityGroup",
          "resourceId": "sg-xxxxxxxxxxxxxxxxx",
          "evaluationMode": "DETECTIVE"
        },
        "orderingTimestamp": "2023-01-31T13:56:01.897Z"
      },
      "complianceType": "NON_COMPLIANT",
      "resultRecordedTime": "2023-01-31T13:56:16.785Z",
      "configRuleInvokedTime": "2023-01-31T13:56:16.403Z"
    },
    "notificationCreationTime": "2023-01-31T13:56:17.035Z",
    "resourceType": "AWS::EC2::SecurityGroup"
  }
}

We also want to be able to handle emails from AWS Security Hub, which are formatted the same however have a configRuleName that is prefixed with "securityhub" instead of "AWSControlTower".

Creating the Flow

Start by heading over to https://make.powerautomate.com/ and sign in with your Microsoft account. Once you are logged in, click on "My flows" in the left-hand menu and then click the create button.

As the emails come into a shared outlook mailbox, we want to create a new flow that triggers when an email arrives in a shared mailbox.

Rename the rule to something that makes more sense, such as "When a Security Rule Compliance Change email arrives", then modify the advanced settings to specify the following:

From: no-reply@sns.amazonaws.com
Subject Filter: Config Rules Compliance Change

Now that we have the rule to capture the emails we'd like to sort, we need to somehow extract the JSON block that is included in the email - while ignoring all the extra text outside of the JSON block. To do this we must first convert the email HTML to text using the Html to text block.

In the block settings, select Body as the content to convert.

The next step is probably the most complicated part as we need to do a couple of operations on the text output to extract just the JSON body - and ensure it's in a legal JSON format.

To do this we will use a Compose block with an expression that will perform the following steps:

Split the text output from the email on the character --, as these characters are directly after the JSON block in the email
Use the concat function to only operate on the content before the --, by selecting just index 0
Replace any newline encoding strings (\n) with an empty string, which just strips them out of the data.
Replace delimited double quotes with just a single quote, removing the delimiter from the string

These four steps can be done by pasting the following into the Expression section of the compose block.

replace(
  replace(
    concat(
      split(
        outputs('Html_to_text')?['body'],'--'
      )[0]
    ),'\n',''
  ),'\"','"'
)

I would also recommend renaming the block to "Split on -- to retain just JSON from email" to make it clear what the step is doing.

Now create a Parse JSON block where the Content field is set to the output of the previous block. For the Schema field, select Generate from sample and paste in the JSON block from the email. This will generate a schema that we can use to parse the JSON.

I also recommend renaming the block to "Parse Config Rule JSON" for clarity.

Now that we have the JSON block parsed, we can begin to set up the logic to sort the emails into folders based on the configRuleName. To do this however we need to set a couple of variables that we can use later on for dealing with some complicated formatting.

These two variables are:

quote: A single quote character that can be referenced as a variable - this is to get around needing to escape single quote characters in expressions.
folderName: This variable will store the folder name for the config email to go into. We will use this later on when creating the folder if it doesn't exist.

Create two Initialize variable blocks like so - making sure to also rename the blocks to something more meaningful.

Now we can set the folderName variable based on the configRuleName. To do this we use an if statement to check if the configRuleName starts with "AWSControlTower_AWS-", and if it does, we set the folderName variable to the suffix of the configRuleName after the "AWSControlTower_AWS-" prefix.

If it is not prefixed with "AWSControlTower_AWS-", we know it is a Security Hub rule and we can set the folderName variable to the configRuleName after the "securityhub-" prefix.

The expression used for the AWSControlTower rule is:

split(body('Parse_Config_Rule_JSON')?['detail']?['configRuleName'], '-')[sub(length(split(body('Parse_Config_Rule_JSON')?['detail']?['configRuleName'], '-')), 1)]

The expression used for the securityhub rule is:

slice(body('Parse_Config_Rule_JSON')?['detail']?['configRuleName'], 0, lastIndexOf(body('Parse_Config_Rule_JSON')?['detail']?['configRuleName'], '-'))

Now we need to prepare a variable to store the folder ID of the folder we are going to move the email to. We will also need to set a couple of static variables that are specific to our setup

Create three new Initialize variable blocks and set the variable names to the following

folderId: This will store the folder ID of the folder we are going to move the email into.
mailboxEmail: This is the email address of the shared mailbox we are going to move the email into.
parentFolderId: This is the folder ID of the parent folder in which we are going to create the new folders.

To get the parentFolderId - it is easiest to head over to https://developer.microsoft.com/en-us/graph/graph-explorer and run the following query to get the folder ID of the parent folder the given folder - replace aws@devopstar.com with your shared mailbox email address, and Inbox with the name of the parent folder.

https://graph.microsoft.com/v1.0/users/aws@devopstar.com/mailFolders?filter=displayName eq 'Inbox'

If you get an error, it is likely that you have not authenticated with the Microsoft Graph API - to do this click your profile in the top right corner and select Permission consent.

Make sure that Mail.ReadWrite.Shared is consented to, and try re-running the query.

From the output of the query, you can see the id of the folder - copy this and paste it into the parentFolderId variable and you should be good to move onto the next step.

After the variables are set, create a "Send an HTTP request" for Outlook365 block - which allows us to make Microsoft Graph requests within the context of Outlook.

Set the Method to GET and the URI to the following expression:

concat('https://graph.microsoft.com/v1.0/users/', variables('mailboxEmail'), '/mailFolders/', variables('parentFolderId'),'/childFolders?$filter=displayName eq ', variables('quote'), variables('folderName'), variables('quote'))

Rename the block to "Check If Folder Exists" for clarity.

To parse the response, we need to create a Parse JSON block and set the Content field to the output Body of the previous block. Change the name of this new block to "Parse Folder Exist Check JSON" and then in the Schema set the following JSON:

{
  "type": "object",
  "properties": {
    "@@odata.context": {
      "type": "string"
    },
    "value": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "id": {
            "type": "string"
          },
          "displayName": {
            "type": "string"
          },
          "parentFolderId": {
            "type": "string"
          },
          "childFolderCount": {
            "type": "integer"
          },
          "unreadItemCount": {
            "type": "integer"
          },
          "totalItemCount": {
            "type": "integer"
          },
          "isHidden": {
            "type": "boolean"
          }
        },
        "required": [
          "id",
          "displayName",
          "parentFolderId",
          "childFolderCount",
          "unreadItemCount",
          "totalItemCount",
          "isHidden"
        ]
      }
    }
  }
}

The next step is to check if the folder exists. To do this we need to check the length of the value array in the JSON response. If the length is greater than 0, then we know the folder exists and we can set the folderId variable to the id of the first item in the value array.

If the length is 0, then we know the folder doesn't exist and we need to create it. I'm going to break this down into two separate steps - but to start with create an if statement block called "Check If Folder exists by checking the length of response value" and set the condition to the following expression:

length(body('Parse_Folder_Exist_Check_JSON')?['value'])

Starting with the true branch, create a new Set variable block and set the folderId variable to the id variable from the "Parse Folder Exist Check JSON" outputs.

It should automatically create an "Apply to each" expression for you when you click on "id" in the dropdown.

Now we need to create the false branch of the if statement. This is where we will create the folder if it doesn't exist. To do this we need to create a new "Send an HTTP request" for Outlook365 block and set the Method to POST and the URI to the following expression:

concat('https://graph.microsoft.com/v1.0/users/', variables('mailboxEmail'), '/mailFolders/', variables('parentFolderId'),'/childFolders')

Set the Body to the following expression as well

concat('{"displayName": "', variables('folderName'), '"}')

Set the name of this block to "Create Folder" then create a new Parse JSON block and set the Content field to the output body of the previous block. Change the name of this new block to "Parse JSON for Create Folder" and then in the Schema set the following JSON:

{
  "type": "object",
  "properties": {
    "@@odata.context": {
      "type": "string"
    },
    "id": {
      "type": "string"
    },
    "displayName": {
      "type": "string"
    },
    "parentFolderId": {
      "type": "string"
    },
    "childFolderCount": {
      "type": "integer"
    },
    "unreadItemCount": {
      "type": "integer"
    },
    "totalItemCount": {
      "type": "integer"
    },
    "sizeInBytes": {
      "type": "integer"
    },
    "isHidden": {
      "type": "boolean"
    }
  }
}

Finally set the folderId variable to the id variable from the "Parse JSON for Create Folder" output.

We're on the home stretch now. We now create a final if statement block called "Check if New Compliance Result is NOT COMPLIANT" and perform a is not equal to check on the complianceResult variable and the string "COMPLIANT".

If the result is not compliant, then we need to flag the email by using the Flag Email block

Otherwise, we should mark the COMPLIANT emails as read by using the Mark Email as Read block.

The last step is to move the email to the appropriate folder. To do this we use the Move Email block and set the Folder variable to be in the following format where it is prefixed by Id::

Testing the flow

Now we have the flow built, we can test it. To do this, you will need to forward yourself a copy of the email that AWS SNS sends you - before you do this however, change the "When a Security Rule Compliance Change email arrives" block to trigger on receiving an email from your email address - this is just temporary so you can test the flow.

Once you have forwarded yourself a copy of the email, if the email is NON_COMPLIANT you should see it flagged and moved to a folder named after the configRuleName.

Summary

In this post, we have built a Power Automate flow that will automatically flag and moves emails from AWS Config and Security hub. This is a great way to keep on top of your AWS Security Hub compliance results and ensure that you are only having to deal with the emails that require your attention.

If you have any questions or if there's anything in this guide that doesn't make sense, please reach out to me on Twitter @nathangloverAUS or via the contact page.

Setup GitHub Codespaces with AWS IAM Roles Anywhere

Nathan Glover — Thu, 22 Sep 2022 14:50:35 +0000

In this blog post, I'm going to try out AWS IAM Roles Anywhere by setting it up for use inside GitHub Codespaces.

The [offical documentation for IAM Roles Anywhere is okay, however, it is written to only provide two options for users who are getting started:

Create your private CA - and we're (AWS) not going to show you how to do that.
Use ACM PCA - a service that costs $400 a month upfront the moment you click the create button.

If you're a beginner, you'll probably just give up here... This is what I nearly did until I stumbled on Aidan Steele's open-source client for interacting with IAM Roles Anywhere is a much friendlier way.

I also came across Björn Wenzel's example of using the offical aws_signing_helper along with HashiCorp's vault to manage a certificate authority. This process could also be used to generate the certificates.

Codespaces .devcontainer

By far the easiest way to get started with a custom environment in GitHub Codespaces is to define a .devcontainer/ directory and populate it with a Development Container configuration. I won't go into heaps of detail in this post on the inner workings of the configuration, but at the very least I'll get you started with a configuration that will work for setting up a Codespace environment with IAM Roles Anywhere.

In the repository that you want to use with GitHub Codespaces, create the following files.

.devcontainer/Dockerfile

# [Choice] Ubuntu version (use ubuntu-22.04 or ubuntu-18.04 on local arm64/Apple Silicon): ubuntu-22.04, ubuntu-20.04, ubuntu-18.04
ARG VARIANT="jammy"
FROM mcr.microsoft.com/vscode/devcontainers/base:0-${VARIANT}

.devcontainer/devcontainer.json

// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
// https://github.com/microsoft/vscode-dev-containers/tree/v0.241.1/containers/ubuntu
{
    "name": "Ubuntu",
    "build": {
        "dockerfile": "Dockerfile",
        "args": { "VARIANT": "ubuntu-22.04" }
    },
    "postStartCommand": ".devcontainer/env.sh",
    "remoteUser": "vscode",
    "features": {
        "git": "os-provided",
        "aws-cli": "latest",
        "golang": "latest",
        "sshd": "latest"
    }
}

.devcontainer/env.sh

#!/bin/bash

set -x
set -e

# Install openrolesanywhere client
if [ -e /tmp/openrolesanywhere ];
  then rm -rf /tmp/openrolesanywhere;
fi
git clone https://github.com/aidansteele/openrolesanywhere.git /tmp/openrolesanywhere
cd /tmp/openrolesanywhere/cmd/openrolesanywhere
go install .

Create Codespace

Commit the changes above to your selected repository then open it up in GitHub Codespaces.

Within the example .devcontainer above I've done the hard work of installing the AWS CLI and dependencies needed to run the openrolesanywhere client.

Roles Anywhere Configuration

Authenticate with AWS manually (one last time)

I know the whole point of this exercise was to remove the need to get temporary credentials/access tokens, but for the last time (hopefully ever), you'll need to set up your AWS CLI credentials within this Codespace just so we're able to deploy all the resources needed.

Follow the quick configuration guide and ensure that whatever user you are deploying this with has at least permissions to interact fully with KMS and IAM.

For this guide, we'll be deploying to us-east-1 as well, so I've added the following line to my ~/.aws/config file as well just to ensure I'm consistent.

[default]
region = us-east-1

Certificate Authority & KMS Key

Start by deploying a KMS key that will be used as the private key for our certificate authority. An example KMS key can be deployed through the kms.yml template by running the following:

aws cloudformation deploy \
    --region us-east-1 \
    --template-file ./kms.yml \
    --stack-name openrolesanywhere-kms

Output from this stack is the KMS Key ID that will be needed when creating the certificate authority with the openrolesanywhere CLI. Go ahead now and create the certificate authority and pass it to the KMS Key ID output. This can be done all in one step with the following

Feel free to tweak other CA options by referencing Aidan Steele's project directly.

KMS_KEY_ID=$(aws cloudformation describe-stacks \
    --stack-name "openrolesanywhere-kms" \
    --region us-east-1 \
    --query 'Stacks[0].Outputs[?OutputKey==`KeyId`].OutputValue' \
    --output text)

openrolesanywhere admin create-ca \
    --name openrolesanywhere-trust \
    --kms-key-id $KMS_KEY_ID \
    --validity-duration 8760h \
    --serial-number 1 \
    --common-name Codespaces \
    --organization DevOpStar

The above creates whats called an IAM Roles Anywhere trust anchor and sets it up to use our certificate authority all in one step.

Roles Anywhere Profile

A Roles Anywhere profile now has to be created which maps the trust anchor created prior and a given role that will be assumed by this Codespace eventually.

You can see an example in the role.yml template which provides basic S3 access to a bucket called roles-anywhere-codespaces-example. I recommend changing this role template to contain a different set of resources based on your requirements before deploying.

Deploy by running the following commands:

aws cloudformation deploy \
    --region us-east-1 \
    --template-file ./role.yml \
    --stack-name openrolesanywhere-role \
    --capabilities CAPABILITY_NAMED_IAM

# arn:aws:iam::012345678912:role/roles-anywhere-codespaces-example
S3_EXAMPLE_ROLE=$(aws cloudformation describe-stacks \
    --region us-east-1 \
    --stack-name "openrolesanywhere-role" \
    --query 'Stacks[0].Outputs[?OutputKey==`RoleArn`].OutputValue' \
    --output text)

openrolesanywhere admin create-profile \
    --name codespaces-example \
    --session-duration 3600s \
    --role-arn $S3_EXAMPLE_ROLE

Backup CA

IMPORTANT: At this point you have a CA configured within your codespaces that should be backed up and stored somewhere safe/offline. Copy the following files to a safe location for when you want to generate other certificates using the same CA:

~/.config/openrolesanywhere/ca.pem
~/.config/openrolesanywhere/profile-arn.txt
~/.config/openrolesanywhere/trust-anchor-arn.txt

The ca.pem is a very important file as it cannot be recovered if you lose it. The profile-arn.txt and trust-anchor-arn.txt files are just mappings to ARNs of resources that can be retrieved from the AWS console if needed.

Private Key for signing

Next, we must create a private key that will be used to sign our requests to AWS IAM Roles Anywhere.

Ideally this key should only be used for this Codespace instances and shouldn't be shared across devices (even if it is possible to).

Create a new SSH private key to use for this example

ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa_codespaces -C "your_email@example.com"
# Press ENTER a bunch
ssh-add ~/.ssh/id_rsa_codespaces

Run ssh-add -l to get a list of fingerprints for the keys stored in your SSH agent. In my example, I have this output.

$ ssh-add -l
# 4096 SHA256:dxzQKbZvcaQkOpJ55YbZ+1/aWENrgBb8zZIkxl5BRGE your_email@example.com (RSA)

Now we need to generate a certificate request using the openrolesanywhere client. This is done by running the following command:

openrolesanywhere request-certificate \
    --ssh-fingerprint SHA256:dxzQKbZvcaQkOpJ55YbZ+1/aWENrgBb8zZIkxl5BRGE > ./key.csr

Next, the CSR has to be accepted, and a certificate generated by running the following command:

openrolesanywhere admin accept-request \
    --request-file ./key.csr \
    --validity-duration 8760h \
    --serial-number 2 \
    --common-name Codespaces \
    --organization DevOpStar > codespaces.pem

This creates a file called codespaces.pem, which is our certificate used alongside our private key to make calls to AWS IAM Roles Anywhere.

Testing Certificates

Copy the contents of the codespaces.pem file to a new file in ~/.config/openrolesanywhere/codespaces.pem.

Retrieve the output ARN from our role.yml stack by running the following command again:

aws cloudformation describe-stacks \
    --region us-east-1 \
    --stack-name "openrolesanywhere-role" \
    --query 'Stacks[0].Outputs[?OutputKey==`RoleArn`].OutputValue' \
    --output text

Then using this role ARN, update your ~/.aws/config:

[profile default]
credential_process = openrolesanywhere credential-process --name codespaces --role-arn arn:aws:iam::012345678912:role/roles-anywhere-codespaces-example
region = us-east-1

Test that the Roles Anywhere profile works by running the following command; the output should look similar.

aws sts get-caller-identity
# {
#     "UserId": "AROAUBLxxxxxxxxxxxxx:02",
#     "Account": "012345678901",
#     "Arn": "arn:aws:sts::012345678901:assumed-role/roles-anywhere-codespaces-example/02"
# }

Configuring Codespaces Secrets

Create the following GitHub repository secrets by navigating to:

https://github.com/{GithubUser}/{RepoName}/settings/secrets/codespaces

The following secrets are required:

ROLES_ANYWHERE_CERTIFICATE: The contents of the codespaces.pem file.
ROLES_ANYWHERE_ROLE: The role output from the role.yml stack
SSH_PRIVATE_SIGNING_KEY: Private key contents from the ~/.ssh/id_rsa_codespaces file

Once those secrets are created, clear out all the ancillary files that were created during this setup process that might be in your repository structure .pem, .csr, and possibly the .yml files (safe to delete).

Modify .devcontainer/env.sh

Finally, we can modify the .devcontainer/env.sh file to now contain the following script that is also available at https://github.com/t04glovern/roles-anywhere-codespaces/blob/main/.devcontainer/env.sh

The script does the following:

Installed openrolesanywhere client
Imports private key for signing from GitHub secrets into codespaces
Imports the codespaces.pem certificate into the openrolesanywhere client directory
Configures AWS CLI to use the openrolesanywhere client for the credentials_process

#!/bin/bash

set -x
set -e

# Install openrolesanywhere client
if [ -e /tmp/openrolesanywhere ];
  then rm -rf /tmp/openrolesanywhere;
fi
git clone https://github.com/aidansteele/openrolesanywhere.git /tmp/openrolesanywhere
cd /tmp/openrolesanywhere/cmd/openrolesanywhere
go install .

if ([ -z "${ROLES_ANYWHERE_CERTIFICATE}" ] && [ -z "${ROLES_ANYWHERE_ROLE}" ] && [ -z "${SSH_PRIVATE_SIGNING_KEY}" ]); then
  echo "ROLES_ANYWHERE_CERTIFICATE, ROLES_ANYWHERE_ROLE or SSH_PRIVATE_SIGNING_KEY are undefined - skipping AWS auth setup within Codespaces"
else
  # Setup SSH Signing key
  mkdir -p ~/.ssh
  if [ -e ~/.ssh/id_rsa_codespaces ];
    then rm -rf ~/.ssh/id_rsa_codespaces;
  fi
  printenv 'SSH_PRIVATE_SIGNING_KEY' > ~/.ssh/id_rsa_codespaces
  chmod 400 ~/.ssh/id_rsa_codespaces
  ssh-keygen -y -f ~/.ssh/id_rsa_codespaces > ~/.ssh/id_rsa_codespaces.pub

  # Setup openrolesanywhere config
  mkdir -p ~/.config/openrolesanywhere
  printenv 'ROLES_ANYWHERE_CERTIFICATE' > ~/.config/openrolesanywhere/codespaces.pem

  # Create credential handler for AWS credential_process
  sudo tee /opt/roles-anywhere-handler << END
#!/bin/bash
eval "\$(ssh-agent -s)" > /dev/null
ssh-add ~/.ssh/id_rsa_codespaces > /dev/null
openrolesanywhere credential-process --name codespaces --role-arn $ROLES_ANYWHERE_ROLE
END

  # Setup AWS config
  mkdir -p ~/.aws
  tee ~/.aws/config << END
[profile default]
credential_process = /opt/roles-anywhere-handler
region = us-east-1
END
fi

Commit the changes above to your repository and give it a try by rebuilding your Codespace from scratch - Delete it from https://github.com/codespaces and re-create!

Once booted up you should find that running the following command still returns an authenticated response

aws sts get-caller-identity
# {
#     "UserId": "AROAUBLxxxxxxxxxxxxx:02",
#     "Account": "012345678901",
#     "Arn": "arn:aws:sts::012345678901:assumed-role/roles-anywhere-codespaces-example/02"
# }

Cleanup

To remove all the resources we've created, you'll need to delete:

The Trust anchor and profile from the Roles Anywhere console: https://us-east-1.console.aws.amazon.com/rolesanywhere/home?region=us-east-1#
The CloudFormation stacks openrolesanywhere-role and openrolesanywhere-kms from https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1
GitHub Secrets for your codespace.

Summary

GitHub Codespaces and AWS IAM Roles Anywhere are now configured to work together, but not without some fairly complex configuration. Ultimately I'm hoping to one day see GitHub Codespaces surfacing OIDC/JWT authentication that can be used with how GitHub actions currently work with Identity providers to deploy into accounts.

For now, this is a fairly Ok solution.

I'm interested to hear your thoughts on this, and I suppose how it could be improved. Please reach out to me on Twitter if you have any further queries or if you have other ways of dealing with this process!

⚡ Supercharge your command line experience

Nathan Glover — Sun, 24 Apr 2022 12:48:43 +0000

.bashrc and .bash_profile organization

Something that can't be overstated is how handy it can be to split up your dotfile/configuration files that are loaded in when your shell session is kicked off. If you know your way around the shell a little then you've probably seen the .bashrc and .bash_profile files.

The .bashrc is run everytime a new non-login shell is opened (a non-login shell would be like when you've logged into a server already, and then opened a new shell). .bash_profile is executed once on login; so when you ssh into a system or login to a computer.

Typically you will find the following in your .bashrc so that all your shell configurtion can live in .bash_profile and everything is loaded in when a new shell is opened.

[ -n "$PS1" ] && source ~/.bash_profile;

Doing the above is fine, however I'm a firm believer that it's better to split up the configuration in your .bashrc file into modular chunks. Adding the following block near the top of your .bash_profile lets you split up configuration for aliases, exports and any other separation you might want to create.

# Load the shell dotfiles, and then some:
for file in ~/.{exports,aliases}; do
    [ -r "$file" ] && [ -f "$file" ] && source "$file";
done;
unset file;

The above then allows you too create the files .aliases and .exports and populate them with any configuration that will be loaded in on shell launch as well.

.exports

exports expose envirnoment settings to your system and the software that runs on it. You'd be surprised by the level of customization available for most commonly use tools that can be tweaked by changing specific environment variables.

Below are a couple fantastic options you might not know about that can be set in your .exports or .bash_profile.

Note: the following configuration can live in your .bash_profile as well if you didn't setup a separate .exports file.

# Increase Bash history size. Allow 32³ entries; the default is 500.
export HISTSIZE='32768';
export HISTFILESIZE="${HISTSIZE}";
# Omit duplicates and commands that begin with a space from history.
export HISTCONTROL='ignoreboth';

# Prefer AU English and use UTF-8. - universal way of setting language
export LANG='en_AU.UTF-8';
export LC_ALL='en_AU.UTF-8';

.inputrc

The .inputrc file provides different customization options that affect interactive shell programs (such as Bash and Python). Though any libraries that leverage the GNU Realine library can adopt the customiations contained by the .inputrc file.

Surprisingly, much of this functionality is disabled by default! Below are some of the options I use and a description of what they do.

# When pressing Tab autocomplete, match regardless of case
set completion-ignore-case on

# Use the text that has already been typed as the prefix for searching through
# commands (i.e. more intelligent Up/Down behavior)
"\e[B": history-search-forward
"\e[A": history-search-backward

# List all matches in case multiple possible completions are possible
set show-all-if-ambiguous on

.aliases

Aliases are another highly underrated way you can customize your shell experience that allows you to alias a set of commands to another word/command.

Below are a couple of great examples of how I use aliases to speed up my shell use experience.

Note: the following configuration can live in your .bash_profile as well if you didn't setup a separate .aliases file.

# Easier navigation: .., ..., ...., ....., ~ and -
alias ..="cd .."
alias ...="cd ../.."
alias ....="cd ../../.."
alias .....="cd ../../../.."
alias ~="cd ~" # `cd` is probably faster to type though
alias -- -="cd -"

# Reload the shell (i.e. invoke as a login shell)
alias reload="exec ${SHELL} -l"

# Print each PATH entry on a separate line
alias path='echo -e ${PATH//:/\\n}'

# Enable aliases to be sudo’ed
alias sudo='sudo '

Another colourful change you can apply! adding the following config can allow you to customize the color of the output from ls and grep

# Detect which `ls` flavor is in use
if ls --color > /dev/null 2>&1; then # GNU `ls`
    colorflag="--color"
    export LS_COLORS='no=00:fi=00:di=01;31:ln=01;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.avi=01;35:*.fli=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.ogg=01;35:*.mp3=01;35:*.wav=01;35:'
else # macOS `ls`
    colorflag="-G"
    export LSCOLORS='BxBxhxDxfxhxhxhxhxcxcx'
fi

# List all files colorized in long format
alias l="ls -lF ${colorflag}"

# List all files colorized in long format, excluding . and ..
alias la="ls -lAF ${colorflag}"

# List only directories
alias lsd="ls -lF ${colorflag} | grep --color=never '^d'"

# Always use color output for `ls`
alias ls="command ls ${colorflag}"

# Always enable colored `grep` output
# Note: `GREP_OPTIONS="--color=auto"` is deprecated, hence the alias usage.
alias grep='grep --color=auto'
alias fgrep='fgrep --color=auto'
alias egrep='egrep --color=auto'

The above config can be seen below, where:

folders are coloured red, files with execute permissions are green and whereas standard files come up as grey.

Summary

Do you have any other awesome command line hacks that work really well for you? Let me know by reaching out to me on twitter and show me @nathangloverAUS.

Building The Ultimate Cat Watch

Nathan Glover — Wed, 08 Jul 2020 14:46:52 +0000

If you've ever met me you'll know that I love my cats. You'll also know that a fair few of my past my tech projects have revolved around my furry family members. So when I got my hands on some brand new LilyGo T-Watch 2020's recently I knew I had to build the ultimate cat loving accessory for myself.

What Watch?

The LilyGo T-Watch 2020 is the recent revision of the LilyGo T-Watch line. These aren't your typical smart watches, as they are completely open source and fully programmable meaning that you have total control over the software that runs on the device.

Not only that but the watch has really awesome spec, especially given the ~$30 price tag. Some of the highlights are:

ESP32 - Regular readers will note how much I love this chip
1.54 inch LCD - Color screen!
FT6236U Touch controller - Because a smart watch without a touchscreen isn't really a smart watch
WiFi & Bluetooth - Connectivity to keep us... errr... connected
Triaxial accelerometer - Step counter
Vibration motor, Speaker, Infrared signal sensor and a Real Time Clock (obviously)

All this compressed down into a nice wearable form factor that makes this unit truly awesome as a platform for wearable projects.

The Plan

I don't start any project without a clear vision of what I want the achieve. In this case it was simple; Two steps:

Take a picture of my cat and put it on the watch (Critical)
Display the current time (Less Critical)

Programming The Watch

Getting started with programming on the T-Watch is done easiest through the use of PlatformIO. PlatformIO is a truly awesome echo-system for working with embedded devices as it handles all the annoying project configurations and dependency loading for you.

Setting up Platform IO

PlatformIO works best with VSCode, so if you haven't already got the IDE installed go ahead and do that first. Then once installed, head over to the Extension manager and install PlatformIO IDE

Importing the Project

Now for you guys following along at home, I'll cut to the chase and give you the code that you can pull down and follow along with. Clone the repo and open it up in VSCode.

git clone https://github.com/t04glovern/ultimate-cat-watch.git

The magic of PlatformIO means that everything else is handled for you! But lets take a look behind the curtains to see how it works. Checking out the platformio.ini file you can see that everything that is needed to setup your environment for flashing code to the watch is seen there:

[env:ttgo-t-watch]
platform = espressif32
board = ttgo-t-watch
framework = arduino

monitor_speed = 115200

lib_deps =
  TTGO TWatch Library

Have a look in the src folder for all the code related to this project that can be flashed to your watch. Lets dive into what the code does and how it can be changed to display my cat!

cat_font.c

cat_font.c is a C (the programming language) formatted font file. Since the watch has no concept of what a font is, we need to convert a font to a format that is basically a list of the pixels that need to be written to the screen to display each letter.

Luckily there's a fantastic tool that does this for us on the LVGL website called fontconverter. Find a font that you like, I personally picked Kurri Island - Måns Grebäck and downloaded a copy of the .ttf file.

Then navigate back to the fontconverter tool and upload the file. Copy down similar settings to what I've used, and define all the symbols you want converted (I've just used the numbers 0-9).

Note: The settings you use might need some tweaking, you'll be able to adjust the settings later once you flash the code and see how it looks

When you hit convert you'll be given an output file called cat_font.c which can be placed to override the existing file I had in the ultimate-cat-watch project.

cat_png.c

cat_png.c is similar to the font file from above, however instead we're dealing with an entire image that needs to be converted to a C (the programming language) Array.

Luckily there's a tool for doing this as well on the LVGL website called imageconverter. Find an image you want to display on the watch; making sure it has a resolution of roughly 240x240. You can use sites like picresize.com to change the dimensions of your image.

Here's a picture of my cat if instead you just want to use her!

Upload the image and use similar settings to what I have used below:

Click convert and you'll be given an output file called cat_png.c which, like before can be placed to override the existing file I had in the ultimate-cat-watch project.

main.cpp

main.cpp is the heart of our code and is what brings everything together. Most of the code is copied from the excellent BatmanDial example in the offical TTGO_TWatch_Library but I've added my own spin.

We start out in the setup() function by initializing the watch, clock, and enable the backlight and set it to 150

    Serial.begin(115200);
    watch = TTGOClass::getWatch();
    watch->begin();
    watch->lvgl_begin();
    rtc = watch->rtc;

    // Use compile time
    rtc->check();

    watch->openBL();

    //Lower the brightness
    watch->bl->adjust(150);

Then using the LVGL functions we can set the background source image to be the cat_png C file that was generated

    lv_obj_t *img1 = lv_img_create(lv_scr_act(), NULL);
    lv_img_set_src(img1, &cat_png);
    lv_obj_align(img1, NULL, LV_ALIGN_CENTER, 0, 0);

The last important line that may change for you depending on if you changed the font (or the font name) is just below where the image background is set. We create a LVGL style and set the font C file and font color.

    static lv_style_t style;
    lv_style_init(&style);
    lv_style_set_text_color(&style, LV_STATE_DEFAULT, LV_COLOR_WHITE);
    lv_style_set_text_font(&style, LV_STATE_DEFAULT, &cat_font);

The rest of the code is simply laying out the format of the on-screen clock, and telling the watch how regularly to update the display

Flashing the Code

We're finally ready to flash out code! You'll need to plug in the LilyGo T-Watch into your computer using the provided USB cable first obviously.

Head to the bottom of the VSCode screen and click the -> arrow to begin compiling and uploading the firmware to the connected device

Once the upload is complete, the watch should automatically start up and display the image of my (or your) cat.

Conclusion

The LilyGo T-Watch 2020 is a fun little hobby device that hits the mark perfectly with the price point. Paired up with PlatformIO and you have a match made in heaven for beginners looking for something fun to do in the embedded space

I'm interested to hear what you think about the LilyGo T-Watch, hit me up on Twitter @nathangloverAUS if you have any thoughts or questions

Internet of Things - Recap (June 2020)

Nathan Glover — Fri, 03 Jul 2020 15:06:43 +0000

Please reach out to me on Twitter @nathangloverAUS if you have follow up questions!

This post was originally written on DevOpStar. Check it out here

Welcome to the June 2020 issue of the yet to be officially named Internet of Things monthly recap. Join me as I take you through some of the interesting things I've seen in the world of IoT this month.

Community Picks

Meshtastic

Meshtastic is an open source mesh communication platform that just released its first beta code this month. I've personally fallen for a failed Kickstarter in the past that promised to do exactly what Meshtastic is doing for 5 times the price, so I was extremely excited to see a project that uses easy to find embedded micro controllers (ESP32 in this case)

Not only is the code open source, but the project uses PlatformIO to build and test, so you know it won't be a pain to do deployments!

I've personally put down an order for 2 boards, so once I receive them look forward to some blogs.

Greengrass Core v1.10.2

Who doesn't like a new Greengrass Core release? The release cycle for Greengrass has slowed down, so I'll take anything I can get. This month we got Greengrass Core v1.10.2 which aside from some general performance improvement and bug fixes gave us a new parameter for the Greengrass core config.json

The mqttOperationTimeout configuration option lets you define the amount of time to allow Greengrass Core to complete a publish, subscribe, or unsubscribe operation with its MQTT connection. By default this setting is 5 seconds.

NVIDIA Jetson Xavier NX Review

As a proud owner of one of the original NVIDIA Jetson Nano boards I was really excited to hear about the launch of the NVIDIA Jetson Xavier NX. This month we started to see review boards surfacing and ETA PRIME put up a great review of the board

Some of the standout specs for this board are:

hexa-core CPU (NVIDIA custom Carmel ARM-based cores)
384-core Volta-based GPU
8GB of LPDDR4x RAM @51.2 GB/s
HDMI, DisplayPort, 4x USB3.1 ports, WiFi, Bluetooth and Ethernet

I was going to put in an order for one of these boards from an Australian distributor but the cost is a little bit outside of my budget for now. When I run into a project where the original NVIDIA Jetson Nano doesn't suffice, I'll definitely make the upgrade though.

ROS 2 Foxy Fitzroy

Do I have any roboticists out there? Well this month marked the release of ROS 2 Foxy Fitzroy. There was a great post put up on the AWS Blog; ROS 2 Foxy Fitzroy: Setting a new standard for production robot development that summarized the update beautifully. Some choice cuts include:

ROS 2 Foxy topics now automatically publish performance metrics and statistics to help users tune their applications
Subscription failures due to QoS incompatibility are automatically reported as errors
Ability to compress and decompress rosbag files while recording and playing data
MoveIt2 included with ROS 2 Foxy

It's been a while since I touched ROS (through RoboMaker) so this update gets me excited to jump back in.

DevOpStar Picks

AWS IoT Greengrass - Jetson Nano

Have you got a NVIDIA Jetson Nano lying around that you want to put to good use? Why not try out AWS IoT Greengrass on it! This month I wrote a Brick on getting started with AWS IoT Greengrass specifically for the NVIDIA Jetson Nano. Check out the post to learn more.

It's targeted specifically at total beginners too, so give it a shot and let me know what you think on Twitter @nathangloverAUS

AWS DeepRacer - Training a Model

Along with Greengrass, I also took a look at what it takes to train a DeepRacer model from scratch, and decided I'd share a simple guide for beginners who are also interested in kick starting their machine learning interests with something very practical.

By the end of the simple tutorial you'll have trained your very own personal model from scratch and customized a fancy car as well. Check out the post to learn more.

AWS DeepRacer - Training a Model

Nathan Glover — Sat, 20 Jun 2020 05:36:18 +0000

Welcome to the AWS DeepRacer - Training a Model brick. In this guild we'll go through setting training your very own DeepRacer model from scratch. You don't need to be an expert either!

Prerequisites

An AWS Account is all you should need for this tutorial.

Note: There will be some costs associated with training. These costs are very minimal (less then $2.00) though, and will likely be covered under the Free-tier if you have a new account.

Setting up your Account

The very first thing I recommend doing is to navigate to the AWS DeepRacer console in the us-east1 region and click on Get started under Reinforcement learning.

We will now work down the list of requirements that will help us train a new model. Step 0 is have AWS reset/create some background resources that will be needed to allow AWS DeepRacer to function.

If this is not the first time working with DeepRacer, you might have ❌'s next to the resources. If this is the case then click Reset resources.

Once the resources have been reset, you will be able to click Create resources next which should eventually turn to ✅'s when completed

Note: This process can take up to 5 minutes to complete

Create a model

We're up to the fun bit in the tutorial where we can train our very first model. To get started, click Create model under the Get started section.

Start by giving your new model a name and description. The name of the model must be unique to your account.

Under environment simulation, select The 2019 DeepRacer Championship Cup

Note: If you have a reason to train your model for any of the other tracks, definitely select the one that seems most appropriate for you.

Click Next at the bottom of the page to move onto Step 2.

In this step we now need to define what kind of race type we would like to train for. Given we're training for just a normal timed race, select Time trail.

Selecting your Agent (Car type) is next, and this is where the guide can start to become more interesting if you want it to be. By default you can choose to train with The Original DeepRacer agent which has the following specs

If you are interested in creating a brand new agent (vehicle) from scratch, then move through the next section Creating an Agent. Otherwise, feel free to select The Original DeepRacer and skip to Reward function.

Creating an Agent [Optional]

We're going to move away from the guide AWS provides us for a second to create our very own Agent. An agent in the context of DeepRacer is the car that drives around the track.

Navigate to the Your garage menu under Reinforcement learning and you should be able to see the The Original DeepRacer agent that is created for you.

This default agent has all the default settings applied, and only uses one camera as its input for information. To understand what I mean by inputs, lets walk through creating a new vehicle by clicking Build new vehicle*.

Note: It is best to understand what kind of track you will be racing on before you create your agent. This is because the track type will heavily influence what types of input data might be considered important when training your model.

You will be presented with a number of options for modifying your agent. Since we picked The 2019 DeepRacer Championship Cup track, lets use it as a foundation for deciding how to build our vehicle.

The first part of designing our vehicle is to select the modifications. We have options for a variety of different camera setups and even the addition of LIDAR.

To keep things simple for now though we're going to stick with the single Camera setup.

Having more information from two cameras can be useful, but only if you write a fantastic reward function to use that data in a meaningful way.

For the Action space we're able to select a variety of different steering & speed optimizations. These variables give the vehicle more or less control over how far it is able to turn, and the speed in which it can move.

Lets breakdown the information we have into some assumptions:

Taking a look at the type of track we are racing on, note that there aren't any hairpin turns or anything that could benefit from having a high degree for steering angle.
Long stretches of track that could benefit from a higher then normal top speed.
8 opportunities for turns and none of them are particularly wide.

Note: The action list is a set of actions the vehicle can take at any point in time. Sometimes having less available actions can be more beneficial for simpler courses.

Here is an example of my desired Action space. Feel free to tinker with these settings, and try something different yourself.

Finally, click Next and you'll need to name your vehicle something unique, along with giving it some cool colors

Click done when you are happy with the car and then head back to the Create model part of the guide. Select your brand new Agent from the list and hit Next.

Reward function

The Reward function is the core of your model; It makes the decisions about what actions to take and when based on a set of (potentially complex) parameters. By default your reward function will look something like the following

def reward_function(params):
    '''
    Example of rewarding the agent to follow center line
    '''

    # Read input parameters
    track_width = params['track_width']
    distance_from_center = params['distance_from_center']

    # Calculate 3 markers that are at varying distances away from the center line
    marker_1 = 0.1 * track_width
    marker_2 = 0.25 * track_width
    marker_3 = 0.5 * track_width

    # Give higher reward if the car is closer to center line and vice versa
    if distance_from_center <= marker_1:
        reward = 1.0
    elif distance_from_center <= marker_2:
        reward = 0.5
    elif distance_from_center <= marker_3:
        reward = 0.1
    else:
        reward = 1e-3  # likely crashed/ close to off track

    return float(reward)

This is the simplest way to approach training a model, and for beginners I would recommend just training with it before trying to dive in too deep too fast.

Note: Reward functions & Hyperparameter tweaking is what makes any model GREAT. Once you have a good understanding of how things work, I recommend exploring these advanced settings in more detail. Watch out for future Bricks/Blog posts on these topics!

When you are ready to begin training, click Create model. The training process will take 60 minutes by default and you can watch how progression is going in the Simulation video stream.

Model evaluation

Once the training process is finished you will see a ✅ indicating its completion. We are now able to run an evaluation by clicking Start evaluation under the model.

When prompted for the Evaluate criteria, select The 2019 DeepRacer Championship Cup.

Under Race type select Time trial

For Virtual race submission, you can opt into competing in the 2020 June Qualifier Time trial. For this example however we will be opting out.

Click Start evaluation to kick off the ~4 minute job. While it is running, you'll be able to preview how your model is performing in the simulation video stream. This view also includes your lap results.

Summary

Congratulations on training your very first DeepRacer model! You can now start to build on your model and make adjustments. Once you have a winning model, you can even submit them to free community events and win prizes.

If you had any issues setting things up, or you have other questions, please let me know by reaching out on Twitter @nathangloverAUS or dropping a comment below.

AWS IoT Greengrass - Jetson Nano

Nathan Glover — Fri, 05 Jun 2020 14:33:55 +0000

Please reach out to me on Twitter @nathangloverAUS if you have follow up questions!

This post was originally written on DevOpStar. Check it out here

Welcome to the AWS IoT Greengrass - Jetson Nano brick. In this overview we'll go through setting up an NVIDIA Jetson Nano with AWS IoT Greengrass. The specific parts that we will cover can be seen below:

Install NVIDIA Jetson Nano Image
Create Greengrass Group
Install AWS IoT Greengrass on Jetson Nano
Deploy Greengrass Group

Prerequisites

AWS Account (Free Tier is acceptable)
Etcher - Used for flashing SD card images
NVIDIA Jetson Nano Developer Kit, although the second half of this guide will work for any Linux operating system.
- 16GB+ microSD card - Anything smaller and you'll have trouble with the uncompressed Jetson Nano image
- Power, Ethernet, Mouse, Keyboard & HDMI cable for hooking up the Jetson Nano

Install NVIDIA Jetson Nano Image

Download the Jetson Nano SD card image from the resources website. The image is ~6GB compressed so be aware that this can take a while to download.

Open up Etcher and complete the following steps:

Select the .zip file you just downloaded
Select the microSD card that you want to flash the image to
Click Flash!

Note: this flashing process can take quite a while (10-15 minutes)

Once the microSD card has been flashed, remove it from your computer and insert it into the Jetson Nano.

Note: It is worth plugging in an ethernet cable at this stage too if you would like to do a system update on first boot.

Proceed to power on the board and run through the setup process using the baked in wizard. The only screen where you should have to focus your time is when you name the device and pick a username/password

Create Greengrass Group

In this set of steps we will begin defining out Greengrass group and export a set of certificates that can be used for your Jetson Nano to talk to AWS IoT

To begin with, navigate to the AWS IoT portal and open up the Greengrass option on the left. From here we are going to run through the Create a Group guided setup by clicking the button

Greeted with two boxes, you are asked which type of setup you would like to run through. For the purpose of this getting started guide, we will be selecting Use default creation

Time to name the Greengrass group, this name needs to be unique within the region of your account. In this guide we've gone with demo-group.

The Core function is used by your Greengrass device to communicate with AWS IoT and other IoT devices on your local network. For this guide we will go with the name that is provided to us by default demo-group_Core.

Finally, you will be prompted to understand that the creation task with perform a number of actions for you under the hood. Click Create Group and Core to have all the actions taken.

Note: If you are interested in learning how you can deploy this resources with AWS CloudFormation, check out t04glovern/aws-greengrass-cfn.

Time to celebrate as you have created your Greengrass group and core! Simply download the tar.gz files and keep them somewhere safe for the next steps.

Click Finish to complete this section. In the next section we will take our Greengrass resources and install them on our Jetson Nano.

Install AWS IoT Greengrass on Jetson Nano

Jumping over to your Jetson Nano device, you will need some way to work on the system. You can either:

SSH to the Jetson Nano: ssh <username>@<ipaddress>
Plug in keyboard/mouse and work directly

The first thing you will need to do is copy the tar.gz files from the previous steps over. This can either be accomplished by a USB, or SCP (if you feel confident with the command line)

An example of using SCP to copy the file across:

scp xxxxxxxxx-setup.tar.gz <username>@<ipaddress>:/home/<username>

# Example of my command
scp 551aca43d9-setup.tar.gz devopstar@devopstar.local:/home/devopstar

With the files onboard, its time to install Greengrass on the Jetson Nano. To do this run the following commands in order.

# create user ggc_user and group ggc_group
sudo adduser --system ggc_user
sudo addgroup --system ggc_group

# download the Greengrass core package
# this is version 1.10.1 at time of writing
wget https://d1onfpft10uf5o.cloudfront.net/greengrass-core/downloads/1.10.1/greengrass-linux-aarch64-1.10.1.tar.gz

# extract the greengrass files to root
sudo tar -xzvf greengrass-linux-aarch64-1.10.1.tar.gz -C /

# install the rootCA
sudo wget -O /greengrass/certs/root.ca.pem https://www.amazontrust.com/repository/AmazonRootCA1.pem

Next we need to extract those Greengrass configuration files we generated in the previous steps. To do this extract the tar.gz (make sure to change the filename based on your file)

sudo tar -xzvf <certificate_id>-setup.tar.gz -C /greengrass

# Example of my command
sudo tar -xzvf 551aca43d9-setup.tar.gz -C /greengrass

Optionally you can also run the following commands to install some extra dependencies

sudo apt install openjdk-8-jdk

We're now setup and ready to start the Greengrass core service.

# start greengrass
cd /greengrass/ggc/core/
sudo ./greengrassd start
# Setting up greengrass daemon
# Validating hardlink/softlink protection
# Waiting for up to 1m10s for Daemon to start

# Greengrass successfully started with PID: 12319

Finally, confirm that the service is working properly by running the following (use the PID from the command output above)

ps aux | grep <PID_FROM_ABOVE>
# root     12319  1.3  0.5 1008696 23308 pts/1   Sl   21:14   0:00 /greengrass/ggc/packages/1.10.1/bin/daemon -core-dir /greengrass/ggc/packages/1.10.1 -greengrassdPid 12314

Deploy Greengrass Group

The final step is to run the deploy for the Greengrass group that was just setup. To do this, navigate to the Greengrass Groups section of the AWS IoT portal.

Open up the Greengrass group that we just deployed by clicking its name.

To run the deployment for the device, simply click Deploy under the Actions menu.

You will be prompted how you would like to deploy, go with Automatic detection.

After a minute or so you should see the Greengrass group has been deployed by a green indicator.

Summary

Congratulations! You created an AWS IoT Greengrass group that is ready for use on your edge based IoT project. With this Greengrass Core setup on the Jetson Nano you can now start looking at more complex tasks such as:

Lambda Functions on the Edge
Machine Learning on the Edge
Docker containers on Greengrass

If you had any issues setting things up, or you have other questions, please let me know by reaching out on Twitter @nathangloverAUS or dropping a comment below.

Escaping from technical debt when it seems impossible

Nathan Glover — Mon, 01 Jun 2020 15:49:38 +0000

Please reach out to me on Twitter @nathangloverAUS if you have follow up questions!

This post was originally written on DevOpStar. Check it out here

On August 16th, 2019 Selfie2Anime went live to unexpected success. Unfortunately, the code deployed wasn't anywhere near ready for the load we experienced and some bad decisions were made in the heat of the moment to get everything up and running. 8 months on and we're dealing with the consequences of those decisions every day.

In this post, I tell the tale of how we came to terms with our problems and learnt to deal with them. I will also give provide some advice about how we might do things differently next time around.

Note that this post was written from the perspective of a small, part-time team; so it might not translate to large scale organizations. However, I would still advocate that most of the recommendations below are at least worth exploring no matter the circumstance.

Context

Before I get into the weeds it will help if you understand what our service is and why we believe we might have been doomed to fail from the get go.

Selfie2Anime uses unsupervised generative networks (UGATIT) to generate anime images from human selfies.

The fundamental premise: An image is uploaded to our service, processed, then sent back to the user. Pretty simple huh? From a distance this might seem like a pretty standard pattern in terms of hosting, the following architecture is probably what comes to your head.

This is more or less exactly what I had in my head too when I prototyped out a very simple concept back at the start of August 2019.

But it wasn't that simple, and if you want to know more about the problems and how we solved them, I recommend checking out Building Selfie2Anime - Iterating on an Idea.

Problems

With the pleasantries out of the way, let's dig into the list of things that ended up being major problems for Selfie2Anime.

I'll fix it later syndrome

When working on software projects, especially when they are experiments or proof of concepts; it's not uncommon to write quick and dirty code to prove out functionality. Normally the rationale is along the lines of "I know this could be better, but I'll fix it up later".

Unsurprisingly this only works if you actually go back and fix it up later.

Here's a great example of some dud code I wrote when building out the image processing loop. If you've worked with Python before, you'll probably notice a few really bad patterns in the code below.

def runner(args, email, image):
    try:
        try:
            fake_img = gan.test_endpoint(image)
            s3 = boto3.client('s3')
            file_name = 'outgoing/' + image
            image_url = s3.put_object(Body=fake_img, Bucket=bucket, Key=file_name)
            email = EmailService()
            email.send_email(email, image_url)
        except Exception as e:
            raise e

    except Exception as e:
        print('FATAL ERROR')
        print(e)

Single broad Exception being caught for all errors
No caching of service clients (email, s3, etc.)
Errors aren't particularly useful.

A more useful way to write the above code could have been what is shown below (however even this isn't necessarily the best way).

# Cache clients in global scope
s3 = boto3.client('s3')
email = EmailService()

def runner(args, email, image):
    try:
        try:
            fake_img = gan.test_endpoint(crop_img)
        except Exception as e:
            print("ERROR: Processing image with GAN: {}".format(e))
            raise e
        try:
            file_name = 'outgoing/' + image
            image_url = s3.put_object(Body=image, Bucket=bucket, Key=file_name)
        except Exception as e:
            print("ERROR: Uploading image to S3: {}".format(e))
            raise e
        try:
            email.send_email(email, image_url)
        except Exception as e:
            print("ERROR: Failed to send email: {}".format(e))
            raise e

    except Exception as e:
        print('FATAL ERROR')
        print(e)

The question is, why wouldn't I write better code the first time around?

There's nothing wrong with writing bad code; but hold yourself accountable for fixing it later.

I found flagging code I wasn't particularly proud of with #TODO's helped a lot as a quick and easy way to acknowledge a problem. Then I would make use of IDE extensions like Todo Tree which keeps a running list of code smells and future improvements right in your IDE.

If you are working with other people who are also touching the code base, then It might also be worthwhile setting up a Trello board and jot down the problems there. This helps the rest of your teammates to understand where you might be cutting corners so they are more understanding when you ask for time to clean up code smells at at later date.

Have a Backlog

Overtime in both my professional and personal careers I've grown to rely on having a documented backlog of work to work from. During my more junior years, I didn't fully understand why it was important and how it can be a very helpful tool for managing technical debt.

A backlog exists to help you and your teammates better understand your inflight and future workload.

Write some dud code? create a task in the backlog to clean it up later
Running into friction when trying to do something in the codebase? throw a card in the backlog to remind you to automate a process when you have some downtime
Noticing users complaining about a bug? Backlog!

I've found that having visibility over tasks and TODO's in a central place helps keep you accountable, and lets your teammates know what kind of time might need to be invested in the future to retroactively improve the project.

CI/CD is more important then you think (most of the time)

During my day job, I pride myself on the level of work that goes into making sure CI/CD is a first-class citizen. Good CI/CD allows you to move faster and get changes into production more frequently, so it seems like a no-brainer to include it in every project you work on.

There is an argument that trying to setup end to end CI/CD at the start of a project can be a form of procrastination. If this is a very early stage personal project or real experimentation where you are trying to prove something out as quick as possible, then I would argue that an end to end deployment strategy is overkill.

You'll usually know when it's time to start setting up a proper deployment pipeline when you are no longer able to make changes regularly, and instead schedule in patches because the process of deploying code has become a hassle.

Automated Builds

I advocate that having an automated build for a new project is always worth setting up, provided it isn't a job that takes a few days. Having some simple way to confirm builds succeed when pushing to production not only helps you test code more frequently, but it allows other members working on the same codebase to be able to test things themselves without needing your involvement.

A good example of this on Selfie2Anime is our docker container builds. It was taking up to 50 minutes to do a full deploy to production as each container build had to include the 7.6gb selfie2anime model. For the first few weeks, I powered through this inconvenience because I didn't see it as an important piece of work.

Over time these builds meant I was terrified of making changes because I knew if I messed something up it would be an hour of downtime.

It took me 8 months to implement these automated docker builds. I'm not proud to admit this, and the moment I finally got around to adding them I realized how badly the fear of failed builds was affecting the release cycles. Below is the actual code change I had to make to enable the automated builds, note how few lines it ended up taking verses the months of putting up with slow releases

version: 0.2
phases:
  pre_build:
    commands:
      - mkdir -p checkpoint
      - echo Downloading s2a model...
      - aws s3 sync s3://devopstar/resources/ugatit/selfie2anime checkpoint/
      - echo Logging in to Amazon ECR...
      - $(aws ecr get-login --no-include-email --region $AWS_DEFAULT_REGION)
  build:
    commands:
      - echo Building the Docker image...
      - docker build -t $IMAGE_REPO_NAME:$CODEBUILD_RESOLVED_SOURCE_VERSION .
      - docker tag $IMAGE_REPO_NAME:$CODEBUILD_RESOLVED_SOURCE_VERSION $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$CODEBUILD_RESOLVED_SOURCE_VERSION
  post_build:
    commands:
      - echo Pushing the Docker image...
      - docker push $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/$IMAGE_REPO_NAME:$CODEBUILD_RESOLVED_SOURCE_VERSION

Another good outcome of setting up this simple automated build was that it gave Rico some context of how the codebase was being built and pushed to production.

A second opinion / Code reviews

Code reviews are another practice that outside of my professional career I didn't properly see the value in. A large part of why I didn't see code reviews as important on Selfie2Anime was that I thought full context would be required by the reviewer in order for them to give me meaningful advice.

For example, Rico hadn't worked on the backend yet so I assumed he wouldn't have the context to make meaningful criticism.

In reality, his distance from the codebase meant he was able to notice things I had become immune from noticing. A good example of this in action was around the following lines of code that pull messages from a queue and return them for processing

while True:
    resp = sqs_client.receive_message(
        QueueUrl=queue_url,
        AttributeNames=['All'],
        MaxNumberOfMessages=10
    )
    try:
        messages.extend(resp['Messages'])
    except KeyError:
        break
    entries = [
        {'Id': msg['MessageId'], 'ReceiptHandle': msg['ReceiptHandle']}
        for msg in resp['Messages']
    ]
    resp = sqs_client.delete_message_batch(
        QueueUrl=queue_url, Entries=entries
    )
    if len(resp['Successful']) != len(entries):
        raise RuntimeError(
            "Failed to delete messages: entries={entries} resp={resp}"
        )
return messages

Rico asked why there was a while True around this code, as the sqs_client.receive_message function call handled the retrieval of 10 messages for me. The while loop was making the code much more susceptible to error; whereas I looked at it as a way to ensure the application was resilient.

In the end, Rico was right, and having him challenge my perspective means that we now have a much more stable system in place.

Recommendation for solo projects

Talk to your co-workers and friends about your code, chances are they'll revel at the opportunity to help. I spoke with @jgunnink a lot when I was working on DynamoDB problems (relating to optimally querying information). He made some good comments about why I wasn't using GSI's (global secondary indexes). This sparked me to go investigate what they were and how they worked.

This simple conversation helped boost my query performance from >1 minute to under a fraction of a second.

Sharing access

Selfie2Anime is deployed into an AWS account that I own myself, it unfortunately ended up sharing the estate with some other personal projects as well. This has the annoying side-effect of being a little difficult to give my team access to all the resources, while also keeping the data safe and private.

We opted to have myself do all the deployments, which initially wasn't a problem because I didn't mind running builds and taking responsibility. However, the outcome of me doing this was that the rest of the team weren't forced to have exposure to the stack and its complexities. This meant that deploying fell entirely on my shoulders, and there was no incentive for my teammate to learn this for themselves.

Sharing responsibility through equal access, especially on small projects like this one helps alleviate strain on one person

The result was to create a tightly scoped AWS IAM Role that would allow Rico into specific resources within my account.

Summary

Technical debt has many different faces, not all of them can be hidden within code. Some are simply just bad takes by individuals who think they have the best intentions but need some external clarity.

I found working on this project helped me better understand when I was digging a deeper hole instead of solving the root cause of my problems, and moving forward I'm excited to kick off new projects with these lessons learnt in mind.

I would LOVE to hear from anyone who has had to deal with slow growing technical debt, especially on smaller projects. It is difficult to find people wanting to talk about bad experiences; so I encourage you to drop a comment below or hit me up on twitter @nathangloverAUS

Internet of Things - Recap (May 2020)

Nathan Glover — Thu, 28 May 2020 05:58:30 +0000

Please reach out to me on Twitter @nathangloverAUS if you have follow up questions!

This post was originally written on DevOpStar. Check it out here

Welcome to the May 2020 issue of the yet to be officially named Internet of Things monthly recap. Join me as I take you through some of the interesting things I've seen in the world of IoT this month.

Community Picks

Project Golden Gate

Golden Gate is an open source project released by Fitbit that hopes to simplify communication between embedded devices by providing an IP-based protocol stacks over the top of Bluetooth low energy (BLE).

The documentation is a bit lacking currently, however there is example code available for Android, iOS and the ESP32. There are comments that this framework will be expanded to support other transport layers besides BLE, so perhaps this is a good repo to star and come back to at some point in the future.

The Maisken Homelay - ESP32 power up board

A lot of people would agree that the ESP32 has become the king in terms of home automation. Look at projects like ESPHome that is entirely based around the use of this incredible board. However there are a couple common patterns where the ESP32 often needs extra components for projects; and this is where The Maisken Homelay could come in handy.

Currently the post outlines that the following features are available in the designs:

4 dry-contact relays
Breakouts to access eight GPIOs, two of which can be analog inputs
Two input-only pins
SDA/SCL pins for I2C

Overall this design will hopefully be useful for home automation systems like sprinklers or appliance controllers that need relays.

Espressif’s ESP32-WROOM-32SE AWS IoT Multi-Account Registration

I should just rename this newsletter, the ESP32 newsletter because we have so much of it this month. The ESP32-WROOM-32SE has been qualited for use with AWS IoT Core Multi-Account Registration.

The chip can integrate with a ATECC608A secure element to store device certificates that could be used for provisioning; simplifying the process of provisioning IoT products into different accounts.

LilyGO - Smartwatch with touch, Wi-Fi and Arduino support

LilyGO unveiled its new TTGO T-Watch-2020, a highly customizable smartwatch using an ESP32. The device has the following features according to the product page:

1.54 inch capacitive screen (ST7789V)
350ma LiPo battery
Triaxial accelerometer (BMA423)
Vibration motor, speaker and infrared signal sensor

The main selling point of this product is that it's an open platform with integration with Scratch, micropython, Arduino and pretty much any other methods you'd normally use to communicate with ESP32 chips.

While the product itself seems to be totally sold out already you can join a queue on the tindie product page

90%+ price reduction for AWS IoT Jobs

If you are a user of AWS IoT Jobs then good news! The cost of running these are ~90% reduced now.

In the past you might have seen me talk about AWS IoT Jobs being used with OTA (over-the-air) updates for devices or just running small blocks of code across huge fleets of managed IoT devices.

Either way, this is great news if you're using the service at scale and have noticed the costs.

DevOpStar Picks

AWS IoT - ESP32-CAM Setup

Occasionally while looking for electronics to play around with I come across something that is so cheap I have to assume it's a scam. For me the ESP-32 CAM was one of such devices; sporting Bluetooth, WiFi, SD Card support and a Camera all for ~$7, it really does seem too good to be true. Lucky for us, this outstanding little SoC is definitely real, and is capable of doing great things (if you are willing to dig into code a little).

In this post we look at the ESP32-CAM and specifically work to get it running on AWS IoT.

Dog Bark Detector - Frontend

Nathan Glover — Sat, 16 May 2020 11:16:51 +0000

Please reach out to me on Twitter @nathangloverAUS if you have follow up questions!

This post was originally written on DevOpStar. Check it out here

I've always been curious as to what my pets get up to when I'm out of the house. This curiosity finally got the best of me this week and I decided to build I wanted to build a dog bark detection system in order to keep track of how often our two dogs bark.

This guide covers the second of three guides around how the Dog Bark Detector was built; specifically focusing on the Frontend. You can see the components highlighted below that are core to this section of the guide.

It covers the following concepts at a high level along with providing all the source code and resources you will need to create your own version.

AppSync data sourcing from DynamoDB
Vue Frontend

Research

You're probably noticing a trend now, but I will always start a project by doing some research for existing code bases or samples available that might have already solved what I am trying to do. This was no exception and I went hunting around for good ways to present the data I've started to accrue.

Turns out I didn't need to look very far this time though...

DynamoDB AppSync data source - Fantastic service that allows you to simply connect an AppSync (GraphQL) API up to an existing DynamoDB database and have all your access patterns created for you.
Amplify frontend - AWS Amplify is an easy way to integrate your frontend with backend AWS services
Vue.js - Frontend framework that the amazing Rico Beti knows really well and offered to help me with

I felt really confident based on the quick research that we'd be able to achieve a simple proof of concept fairly easily based on the tech stack above. In the next section we start implementing it all.

Creating AppSync API

In this section we will start creating our AppSync API. Start by navigating to the AppSync console and click the Create API button. This will start up a launch wizard where you will be selecting Import DynamoDB Table.

Select the region you table is located in and then select your existing table (should auto-fill). You can either use the auto generator for role or if you know what you're doing; select an existing role that will be used by AppSync to connect to the DynamoDB table.

Once you click import you will be prompted to define the extra schema attributes. For the sake of completeness we will add all the required fields to the list so they are generated in the API for us.

Hit complete and you'll be presented back your new AppSync API ready for use. Before we go using it however, we're going make a couple small changes to the schema so that it only allows read access. Navigate to the Schema tab under your new API and comment out the followin sections

// COMMENT THESE
input CreateBarksInput {
    deviceId: String!
    timestamp: Int!
    probability: String!
    camera: String!
    wav_file: String!
}

input UpdateBarksInput {
    deviceId: String!
    timestamp: Int!
    probability: String!
    camera: String!
    wav_file: String!
}

input DeleteBarksInput {
    deviceId: String!
    timestamp: Int!
}

type Mutation {
    createBarks(input: CreateBarksInput!): Barks
    updateBarks(input: UpdateBarksInput!): Barks
    deleteBarks(input: DeleteBarksInput!): Barks
}

Click save and validate that you weren't given any syntax errors when changing the Schema. Next we'll move onto connecting up our API in the Vue.js app.

Connect AppSync API

The next step is to connect the AppSync API to our frontend app by using AWS Amplify. You might have noticed that when you first imported the DynamoDB instance to AppSync you were given a set of commands to run to integrate AWS Amplify. We're going to be following these steps now in order to do just that.

I recommend working with our existing repository t04glover/dog-bark-detector by pulling it down locally

git clone https://github.com/t04glovern/dog-bark-detection.git
cd dog-bark-detection/app

Next remove any existing configuration that we have currently added (this is using our personal API).

rm -rf amplify
rm -rf src/graphql
rm src/API.ts
rm src/aws-exports.js

Next follow the instructions from the AppSync page in regards to configuring this Vue.js app with Amplify. Run the following commands within the app folder of the project.

npm install -g @aws-amplify/cli
amplify init
# ? Enter a name for the project bark-detector
# ? Enter a name for the environment dev
# ? Choose your default editor: Visual Studio Code
# ? Choose the type of app that you're building javascript
# Please tell us about your project
# ? What javascript framework are you using vue
# ? Source Directory Path:  src
# ? Distribution Directory Path: dist
# ? Build Command:  npm run-script build
# ? Start Command: npm run-script serve

amplify add codegen --apiId <API_ID_FROM_YOUR_OWN_PROJECT>

NOTE: When prompted, allow it to generate all the possible API's for you.

Then within the actual Vue.js code in app/src/store/app.ts you are able to see where we are importing the Amplify specific code to setup authentication

...
import Amplify, { API, graphqlOperation } from "aws-amplify";
import { listBarks } from "@/graphql/queries";

const awsconfig = require("@/aws-exports").default;

Amplify.configure(awsconfig);
...

Later on in that same file we call the listBarks GraphQL operation like so in order to pull 500 records and push them into the list of items on the frontend

const allBarks = await API.graphql(graphqlOperation(listBarks, {limit: 500}));
const detections: IDetection[] = (allBarks as any)
    .data.listBarks.items.map((b: any): IDetection => ({
        id: b.deviceId,
        camera: b.camera,
        timestamp: new Date(b.timestamp * 1000),
        certainty: parseFloat(b.probability),
        audioUrl: b.wav_file,
        rating: "unknown",
    }));

this.setDetections({detections});

Run the app by running the following commands within the app folder of the project.

npm install
npm run dev

Open up your browser to http://localhost:8080/ to bask in the glory of the working app

Bonus Graphs

You might have also noticed that there is a Dashboard tab on the left. If you open this up you should be able to see a trend of the last weeks worth of data displayed.

Conclusion

This concludes the final part of this guide where we've achieved our goal of creating a simple Dog Bark frontend using Vue.js, Amazon AppSync and AWS Amplify.

If you have any questions want to show off how you've changed the frontend to be more interesting; please hit me up on Twitter @nathangloverAUS