DEV Community: Oguzhan Ozdemir

Playing with AWS Greengrass

Oguzhan Ozdemir — Thu, 22 Oct 2020 17:48:51 +0000

Backstory

For the last couple of weeks, I've been playing with AWS Greengrass for work. At first, I did some manual setups, but eventually, I needed to automate the deployments and subscription setups. After diving into the documentation and a few chats with AWS Support, I've managed to do a proper CloudFormation stack and do a deployment.

In this post, I'll try to give a worthy walkthrough if you ever want to play with AWS Greengrass. If this is your first time hearing Greengrass, I'll kindly point out the official documentation, here, which obviously does a pretty good job of covering what it is.

Prerequisite

As usual, we would require some setup before we start writing some code. Some of these steps are pretty usual, such as an AWS Account. However, I'll try to give you some instruction about others, as they are unique to this tutorial.

An AWS account, which you can open easily at aws.amazon.com.
Install and configure AWS CLI on your computer.
Install NodeJS. Version 12.X will suffice.
Install Serverless framework. See serverless.com.
Install Docker.

After step 5, we need to fetch the Docker image of Greengrass as we will run the core on our laptop as if it's an IoT device.

Let's pull the Docker image of Greengrass. First, we need to log in to the AWS IoT Greengrass registry in Amazon ECR. If this runs correctly, we'll see Login Succeeded output in our terminal.

$ aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin https://216483018798.dkr.ecr.us-west-2.amazonaws.com

Then, we can pull the Docker image.

$ docker pull 216483018798.dkr.ecr.us-west-2.amazonaws.com/aws-iot-greengrass:latest

Once it's finished, we can move on to the code. Don't worry, we'll get back to this Docker image eventually, but first, we need to get some credentials and do some work.

Coding

Now, before we start working on all the connections, we need a simple serverless function to send a message to a Greengrass topic. So, let's start building it.

Lambda Function

OK. To keep things simple, we are once again going to use the Serverless framework.

$ mkdir greengrassLambda && cd $_
$ serverless create --template aws-nodejs

Let's also init npm and install AWS Greengrass SDK.

$ npm init -y
$ npm install aws-greengrass-core-sdk --save

We are going to update our serverless.yml file as well.

service: greengrasslambda
frameworkVersion: "2"

provider:
  name: aws
  runtime: nodejs12.x
  stage: dev
  region: eu-west-1

functions:
  counter:
    handler: handler.handler

Now, we can write our function in handler.js which is going to be a simple counter that will push messages to our choice of a topic, sample/counter.

'use-strict';

const ggSdk = require('aws-greengrass-core-sdk');

const iotClient = new ggSdk.IotData();
const util = require('util');

let counter = 0;

module.exports.handler = (event, context, callback) => {
  counter++;

  try {
    iotClient.publish({
      topic: 'sample/counter',
      payload: JSON.stringify({
        message: util.format('Sent from Greengrass Core. Invocation Count: %s', counter)
      }),
      queueFullPolicy: 'AllOrError',
    }, () => {
      callback(null, true);
    });
  } catch (error) {
    console.log(error);
  }

  return;
};

Alright. Looks like this part is done, so let's deploy this and start our CloudFormation template for our AWS Greengrass group.

$ sls deploy

AWS Greengrass Group

This part is a bit tricky. There is an example CloudFormation template online for Greengrass, but, to understand what we need to create, you might need to play a bit on AWS Console. I did many manual setups, breaking stuff and more to understand the needs here. I'll try to give you as much insight as possible and break the CloudFormation template into pieces before giving you the whole file.

Also, there is a step which needed to be done manually here. That is, creating a certificate. As far as I can tell, CloudFormation currently doesn't support this. There is an API to handle this but, we'll do this part manually for now.

First, we'll go to the IoT Core service on AWS Console. Navigate to the Certificates menu and click create.

Then, progress as recommended and create the certificates. After it's done, download all three files, activate the certificates and click done. We'll attach the policy using CloudFormation later in the post.

Now we can start writing our CloudFormation template. Let's keep it in the same repository as our lambda function, so, we should create a templates folder and in it, a file named greengrass.json. We'll take three parameters. LambdaName, LambdaVersion, and CertificateHash. Since Greengrass doesn't support $LATEST tag, this will make our job a lot easier. For CertificateHash, we'll take the value as cert/xxxxx....

{
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "AWS IoT Greengrass template for RustyNeuron tutorial.",
  "Parameters": {
    "LambdaName": {
      "Type": "String"
    },
    "LambdaVersion": {
      "Type": "String"
    },
    "CertificateHash": {
      "Type": "String"
    }
  },
  "Resources": {},
  "Outputs": {}
}

Above is the basis of our template. Let's start filling up our Resources. Now, in order this to work, we need several resources in order, as listed below.

1.  AWS::IoT::Thing
2.  AWS::Greengrass::CoreDefinition
3.  AWS::Greengrass::CoreDefinitionVersion
4.  AWS::Greengrass::FunctionDefinition
5.  AWS::Greengrass::FunctionDefinitionVersion
6.  AWS::Greengrass::LoggerDefinition
7.  AWS::Greengrass::LoggerDefinitionVersion
8.  AWS::Greengrass::SubscriptionDefinition
9.  AWS::Greengrass::SubscriptionDefinitionVersion
10. AWS::Greengrass::Group
11. AWS::IoT::Policy
12. AWS::IoT::PolicyPrincipalAttachment
13. AWS::IoT::ThingPrincipalAttachment

Let's start with the top three. We'll name our thing, Neuron.

{
  ...
  "Resources": {

    // COPY FROM HERE

    "NeuronCore": {
      "Type": "AWS::IoT::Thing",
      "Properties": {
        "ThingName": "NeuronCore"
      }
    },
    "NeuronCoreDefinition": {
      "Type": "AWS::Greengrass::CoreDefinition",
      "Properties": {
        "Name": "NeuronCoreDefinition"
      }
    },
    "NeuronCoreDefinitionVersion": {
      "Type": "AWS::Greengrass::CoreDefinitionVersion",
      "Properties": {
        "CoreDefinitionId": {
          "Ref": "NeuronCoreDefinition"
        },
        "Cores": [
          {
            "Id": "NeuronCore",
            "CertificateArn": {
              "Fn::Join": [
                ":",
                [
                  "arn:aws:iot",
                  {
                    "Ref": "AWS::Region"
                  },
                  {
                    "Ref": "AWS::AccountId"
                  },
                  {
                    "Ref": "CertificateHash"
                  }
                ]
              ]
            },
            "ThingArn": {
              "Fn::Join": [
                ":",
                [
                  "arn:aws:iot",
                  {
                    "Ref": "AWS::Region"
                  },
                  {
                    "Ref": "AWS::AccountId"
                  },
                  "thing/NeuronCore"
                ]
              ]
            }
          }
        ]
      }
    },

    // TO HERE

  },
  ...
}

OK. This looks promising. Let's add the Function definitions. Here, we need to be careful about a couple of things. First, IsolationMode must be NoContainer as Greengrass is running on our computer. Second, in the FunctionConfiguration, we mark the Pinned value as false. This is because we designed our lambda function to run on-demand.

{
  ...
  "Resources": {
    ...

    // COPY FROM HERE

    "NeuronFunctionDefinition": {
      "Type": "AWS::Greengrass::FunctionDefinition",
      "Properties": {
        "Name": "NeuronFunctionDefinition"
      }
    },
    "NeuronFunctionDefinitionVersion": {
      "Type": "AWS::Greengrass::FunctionDefinitionVersion",
      "Properties": {
        "FunctionDefinitionId": {
          "Fn::GetAtt": [
            "NeuronFunctionDefinition",
            "Id"
          ]
        },
        "DefaultConfig": {
          "Execution": {
            "IsolationMode": "NoContainer"
          }
        },
        "Functions": [
          {
            "Id": "NeuronLambda",
            "FunctionArn": {
              "Fn::Join": [
                ":",
                [
                  "arn:aws:lambda",
                  {
                    "Ref": "AWS::Region"
                  },
                  {
                    "Ref": "AWS::AccountId"
                  },
                  "function",
                  {
                    "Ref": "LambdaName"
                  },
                  {
                    "Ref": "LambdaVersion"
                  }
                ]
              ]
            },
            "FunctionConfiguration": {
              "Pinned": false,
              "Timeout": 5,
              "EncodingType": "json",
              "Environment": {
                "Execution": {
                  "IsolationMode": "NoContainer",
                  "RunAs": {
                    "Uid": "1",
                    "Gid": "10"
                  }
                }
              }
            }
          }
        ]
      }
    },

    // TO HERE

  },
  ...
}

For Logger definitions, we need two types of loggers. One for CloudWatch and one for the local logs. If you have a problem of seeing logs on CloudWatch, we might need to alter the Greengrass service policy to allow logging. We'll get to that at the end of our article.

{
  ...
  "Resources": {
    ...

    // COPY FROM HERE

    "NeuronLoggerDefinition": {
      "Type": "AWS::Greengrass::LoggerDefinition",
      "Properties": {
        "Name": "NeuronLoggerDefinition"
      }
    },
    "NeuronLoggerDefinitionVersion": {
      "Type": "AWS::Greengrass::LoggerDefinitionVersion",
      "Properties": {
        "LoggerDefinitionId": {
          "Ref": "NeuronLoggerDefinition"
        },
        "Loggers": [
          {
            "Id": "NeuronLoggerCWSystem",
            "Type": "AWSCloudWatch",
            "Component": "GreengrassSystem",
            "Level": "INFO"
          },
          {
            "Id": "NeuronLoggerCWLambda",
            "Type": "AWSCloudWatch",
            "Component": "Lambda",
            "Level": "INFO"
          },
          {
            "Id": "NeuronLoggerLocalSystem",
            "Type": "FileSystem",
            "Component": "GreengrassSystem",
            "Level": "INFO",
            "Space": 25600
          },
          {
            "Id": "NeuronLoggerLocalLambda",
            "Type": "FileSystem",
            "Component": "Lambda",
            "Level": "INFO",
            "Space": 25600
          }
        ]
      }
    },

    // TO HERE

  }
  ...
}

Next, we have Subscriptions. Here, we need two subscriptions.

From lambda to IoT cloud. This is to listen to the messages that are being published by our lambda function.
From IoT cloud to our lambda. This is to easily trigger our lambda function, using the MQTT client on IoT Console.

{
  ...
  "Resources": {
    ...

    // COPY FROM HERE

    "NeuronSubscriptionDefinition": {
      "Type": "AWS::Greengrass::SubscriptionDefinition",
      "Properties": {
        "Name": "NeuronSubscriptionDefinition"
      }
    },
    "NeuronSubscriptionDefinitionVersion": {
      "Type": "AWS::Greengrass::SubscriptionDefinitionVersion",
      "Properties": {
        "SubscriptionDefinitionId": {
          "Ref": "NeuronSubscriptionDefinition"
        },
        "Subscriptions": [
          {
            "Id": "Subscription1",
            "Source": {
              "Fn::Join": [
                ":",
                [
                  "arn:aws:lambda",
                  {
                    "Ref": "AWS::Region"
                  },
                  {
                    "Ref": "AWS::AccountId"
                  },
                  "function",
                  {
                    "Ref": "LambdaName"
                  },
                  {
                    "Ref": "LambdaVersion"
                  }
                ]
              ]
            },
            "Subject": "sample/counter",
            "Target": "cloud"
          },
          {
            "Id": "Subscription2",
            "Source": "cloud",
            "Subject": "sample/trigger",
            "Target": {
              "Fn::Join": [
                ":",
                [
                  "arn:aws:lambda",
                  {
                    "Ref": "AWS::Region"
                  },
                  {
                    "Ref": "AWS::AccountId"
                  },
                  "function",
                  {
                    "Ref": "LambdaName"
                  },
                  {
                    "Ref": "LambdaVersion"
                  }
                ]
              ]
            }
          }
        ]
      }
    },

    // TO HERE

  }
  ...
}

We are very close. Now that every component definition is done, we can write the group's definition and link the components we wrote.

{
  ...
  "Resources": {
    ...

    // COPY FROM HERE

    "NeuronGroup": {
      "Type": "AWS::Greengrass::Group",
      "Properties": {
        "Name": "NeuronGroup",
        "RoleArn": {
          "Fn::Join": [
            ":",
            [
              "arn:aws:iam:",
              {
                "Ref": "AWS::AccountId"
              },
              "role/service-role/Greengrass_ServiceRole"
            ]
          ]
        },
        "InitialVersion": {
          "CoreDefinitionVersionArn": {
            "Ref": "NeuronCoreDefinitionVersion"
          },
          "FunctionDefinitionVersionArn": {
            "Ref": "NeuronFunctionDefinitionVersion"
          },
          "SubscriptionDefinitionVersionArn": {
            "Ref": "NeuronSubscriptionDefinitionVersion"
          },
          "LoggerDefinitionVersionArn": {
            "Ref": "NeuronLoggerDefinitionVersion"
          }
        }
      }
    },

    // TO HERE

  }
  ...
}

All good. Let's add the policy definitions now.

{
  ...
  "Resources": {
    ...

    // COPY FROM HERE

    "NeuronPolicy": {
      "Type": "AWS::IoT::Policy",
      "Properties": {
        "PolicyName": "NeuronPolicy",
        "PolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Action": [
                "iot:Publish",
                "iot:Subscribe",
                "iot:Connect",
                "iot:Receive"
              ],
              "Resource": [
                "*"
              ]
            },
            {
              "Effect": "Allow",
              "Action": [
                "iot:GetThingShadow",
                "iot:UpdateThingShadow",
                "iot:DeleteThingShadow"
              ],
              "Resource": [
                "*"
              ]
            },
            {
              "Effect": "Allow",
              "Action": [
                "greengrass:*"
              ],
              "Resource": [
                "*"
              ]
            }
          ]
        }
      }
    },
    "NeuronPolicyAttachmet": {
      "Type": "AWS::IoT::PolicyPrincipalAttachment",
      "Properties": {
        "PolicyName": {
          "Ref": "NeuronPolicy"
        },
        "Principal": {
          "Fn::Join": [
            ":",
            [
              "arn:aws:iot",
              {
                "Ref": "AWS::Region"
              },
              {
                "Ref": "AWS::AccountId"
              },
              {
                "Ref": "CertificateHash"
              }
            ]
          ]
        }
      },
      "DependsOn": "NeuronPolicy"
    },
    "NeuronBaseCertificateAttachment": {
      "Type": "AWS::IoT::ThingPrincipalAttachment",
      "Properties": {
        "ThingName": "NeuronCore",
        "Principal": {
          "Fn::Join": [
            ":",
            [
              "arn:aws:iot",
              {
                "Ref": "AWS::Region"
              },
              {
                "Ref": "AWS::AccountId"
              },
              {
                "Ref": "CertificateHash"
              }
            ]
          ]
        }
      },
      "DependsOn": "NeuronCore"
    }

    // TO HERE

  }
  ...
}

At this point, our Resources are done. However, we need one more piece to make it right. We need the command to make a new deployment of our group. To do this, we are going to define an output.

{
  ...
  "Resources": {
    ...
  },
  "Outputs": {

    // COPY FROM HERE

    "CommandToDeployGroup": {
      "Value": {
        "Fn::Join": [
          " ",
          [
            "groupVersion=$(cut -d'/' -f6 <<<",
            {
              "Fn::GetAtt": [
                "NeuronGroup",
                "LatestVersionArn"
              ]
            },
            ");",
            "aws --region",
            {
              "Ref": "AWS::Region"
            },
            "greengrass create-deployment --group-id",
            {
              "Ref": "NeuronGroup"
            },
            "--deployment-type NewDeployment --group-version-id",
            "$groupVersion"
          ]
        ]
      }
    }

    // TO HERE

  }
}

Alright. Whew... Let's apply this, shall we? First, we validate the template we created.

$ aws cloudformation validate-template --template-body file://templates/greengrass.json --region eu-west-1

It should give us the following output.

{
    "Parameters": [
        {
            "ParameterKey": "LambdaName",
            "NoEcho": false
        },
        {
            "ParameterKey": "CertificateHash",
            "NoEcho": false
        },
        {
            "ParameterKey": "LambdaVersion",
            "NoEcho": false
        }
    ],
    "Description": "AWS IoT Greengrass template for RustyNeuron tutorial."
}
(END)

Right now, we are ready to create our stack on AWS. So, let's go. This will take a while to complete, but hopefully, it will succeed. Make sure to replace the xxxxx... part with your certificate hash.

$ aws cloudformation create-stack --stack-name RustyNeuronTutorial \
  --template-body file://templates/greengrass.json --region eu-west-1 \
  --parameters ParameterKey=LambdaName,ParameterValue=greengrasslambda-dev-counter \
               ParameterKey=LambdaVersion,ParameterValue=1 \
               ParameterKey=CertificateHash,ParameterValue=cert/xxxxx...

Running IoT Core on Docker

Do you remember that we downloaded some keys and certificates when we started this tutorial? Right, we need them now. To keep things in one place, let's create a greengrass folder in the home root, the ~. However, this is up to you. Just remember to change the paths when we run docker.

$ cd ~
$ mkdir -p Development/greengrass && cd $_
$ mkdir certs
$ mkdir config

Now, we'll copy the three files we got when we created the certificate in the certs folder. Additionally, we need the root.ca.pem file, which is distributed by AWS. To get that, the following command must be run in the certs folder.

$ cd certs                              # /Users/$USER/Development/greengrass/certs
$ sudo wget -O root.ca.pem https://www.amazontrust.com/repository/AmazonRootCA1.pem

Now, we'll write our config.json file. This file, if you create your Greengrass group, will be included in the certificate tar file. However, it's not present if you create your certificate stand-alone.

In the config folder, we create a config.json file with the following content.

{
  "coreThing": {
    "caPath": "root.ca.pem",
    "certPath": "xxxxxxxxxx-certificate.pem.crt",
    "keyPath": "xxxxxxxxxx-private.pem.key",
    "thingArn": "arn:aws:iot:eu-west-1:AWS_ACCOUNT_ID:thing/NeuronCore",
    "iotHost": "xxxxxxxxxxxxxx-ats.iot.eu-west-1.amazonaws.com",
    "ggHost": "greengrass-ats.iot.eu-west-1.amazonaws.com",
    "keepAlive": 600
  },
  "runtime": {
    "cgroup": {
      "useSystemd": "yes"
    }
  },
  "managedRespawn": false,
  "crypto": {
    "principals": {
      "SecretsManager": {
        "privateKeyPath": "file:///greengrass/certs/xxxxxxxxxx-private.pem.key"
      },
      "IoTCertificate": {
        "privateKeyPath": "file:///greengrass/certs/xxxxxxxxxx-private.pem.key",
        "certificatePath": "file:///greengrass/certs/xxxxxxxxxx-certificate.pem.crt"
      }
    },
    "caPath": "file:///greengrass/certs/root.ca.pem"
  }
}

You can get your AWS_ACCOUNT_ID by running aws sts get-caller-identity in your terminal. The masked parts of certPath, keyPath, privateKeyPath, and certificatePath are the hash of the files we downloaded. You can find the iotHost in the thing's Interract menu as shown in the image below.

In the end, your folder structure should look like this.

$ tree .
.
├── certs
│   ├── xxxxxxxxxx-certificate.pem.crt
│   ├── xxxxxxxxxx-private.pem.key
│   ├── xxxxxxxxxx-public.pem.key
│   └── root.ca.pem
└── config
    └── config.json

2 directories, 5 files

We seem to be ready. Let's start our IoT core using Docker. Make sure to change the $USER in the command.

$ docker run --rm --init -it --name aws-iot-greengrass \
    --entrypoint /greengrass-entrypoint.sh \
    -v /Users/$USER/Development/greengrass/certs:/greengrass/certs \
    -v /Users/$USER/Development/greengrass/config:/greengrass/config \
    -p 8883:8883 \
    216483018798.dkr.ecr.us-west-2.amazonaws.com/aws-iot-greengrass:latest

We should get something like this.

Greengrass successfully started with PID: 13

Once we see that our container is running, we can make a group deployment and test it.

AWS Greengrass Group Deployment

We have two options here. One, we can go to the IoT console and click deploy or we can use AWS CLI. Remember that we defined an Output in our CloudFormation template. Let's get that command.

Navigate to the AWS CloudFormation and select RustyNeuronTutorial stack. Go to the Outputs tab.

$ groupVersion=$(cut -d'/' -f6 <<< arn:aws:greengrass:eu-west-1:AWS_ACCOUNT_ID:/greengrass/groups/GREENGRASS_GROUP_ID/versions/GREENGRASS_GROUP_VERSION_ID ); \
    aws --region eu-west-1 greengrass create-deployment \
    --group-id GREENGRASS_GROUP_ID \
    --deployment-type NewDeployment \
    --group-version-id $groupVersion

This should be done in no time and we should see the successful deployment on IoT Console.

Testing

Alright. We are in the home stretch. Let's test this whole setup. Go to the Test page in the IoT Console. Then, subscribe to sample/counter topic to listen to our lambda function.

And now, publish to sample/trigger topic. Remember, we set two subscriptions for both ways. We should successfully get the invocation count message published by our lambda function.

Logging

Logs will stream to two places. CloudWatch and local file system. We configured our group for both, but probably you won't be able to see the logs on CloudWatch. So, let's see the logs in the local file system first.

First, we should connect to our Docker container.

$ docker exec -it aws-iot-greengrass /bin/bash

Logs are kept under /greengrass/ggc/var/log/ as two parts: system and user. To follow the user logs, you can run the following command in your container.

$ tail -f /greengrass/ggc/var/log/user/eu-west-1/AWS_ACCOUNT_ID/LAMBDA_NAME.log

When it comes to CloudWatch logs, we need to give the Greengrass_ServiceRole access to CloudWatch. You could give full access or if you want more granular access, you can attach the following policy to the service role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:PutMetricFilter",
        "logs:PutRetentionPolicy",
        "logs:DescribeLogStreams"
      ],
      "Resource": [
        "arn:aws:logs:*:*:*"
      ]
    }
  ]
}

Go to IAM console and find the Greengrass_ServiceRole. Then, in the Permissions tab, click the Attach policies. Here, you can either select CloudWatchLogsFullAccess or click Create policy and paste the above summary. It's up to you. I'll choose CloudWatchLogsFullAccess. When you trigger your function, like in the Testing section, you'll be able to see the log groups with the /aws/greengrass/ prefix in CloudWatch.

Deleting

To remove the CloudFormation stack properly, first, you need to reset the deployments of your group. You can do that with the following command.

$ aws greengrass reset-deployments --group-id GREENGRASS_GROUP_ID --region eu-west-1 --force

Then, to remove the CloudFormation stack, you'd need the following command.

$ aws cloudformation delete-stack --stack-name RustyNeuronTutorial --region eu-west-1

Source Code and Troubleshooting

The whole thing, except for the certificates and configs, is in this repository; aoozdemir/greengrass-example.

If you encounter any error or problem, I suggest you go to the AWS Documentation. If you find any errata or any suggestion to do things more efficiently, please contact me.

See you in another post!

Building A Simple Serverless Application

Oguzhan Ozdemir — Sun, 27 Sep 2020 12:44:01 +0000

Backstory

Quite recently¹, I joined Thundra² as a Solutions Engineer. Although my title seems non-technical, Thundra and its customers are; and so am I. During this time, I'll also be responsible for all the infrastructure need Thundra has. To do this, I must get used to the serverless world. Hence this post.

I haven't actively worked on serverless architectures so far or with NodeJS. I know, I'm a bit late to the party. So, in my first week at Thundra, I started to play with all that and built myself a simple serverless flow using NodeJS and AWS services.

Prerequisite

We need a couple of things as we build this flow.

An AWS account, which you can open easily at aws.amazon.com.
Install and configure AWS CLI on your computer.
Install NodeJS. Version 12.X will suffice.
Install Serverless framework. See serverless.com.

If all these pieces are installed and working fine on your computer, you are good to go.

The Application

Now, let's talk about what we are going to build. I didn't want my first serverless application to be difficult, but I also wanted to use AWS services other than AWS Lambda. So, I've decided to use SQS and S3 along with it.

The application flow is also quite simple and it is something like this;

Lambda #1 has a POST endpoint to take a payload.
Lambda #1 then sends this payload to an SQS Queue.
Whenever our queue receives a message, it then triggers Lambda #2.
Once Lambda #2 triggered, it prepares a document with the message and uploads it to an S3 Bucket.
That's it. It's up to you what to do with all the documents in your bucket.

As shown in the diagram above, it's not challenging. But that's ok. That's what I wanted.

Coding

This is the fun part. As I mentioned in the prerequisite, we are going to use the Serverless framework to handle all the deployment and resources on AWS. Let's break our application into pieces.

For Lambda #1 we need;
1. A simple lambda function.
2. An SQS queue.
3. The necessary permissions for our function to push a message into our queue.
4. The code.
For Lambda #2 we need;
1. Another lambda function.
2. An S3 Bucket.
3. Again, necessary permissions for our function to upload a document to our bucket.
4. The code.

Lambda #1

First, we need to create a project using the Serverless framework. Let's run the following commands to create a project.

$ mkdir sampleLambda
$ cd sampleLambda
$ serverless create --template aws-nodejs

This will give us the following files.

.
├── .gitignore
├── handler.js
└── serverless.yml

0 directory, 3 files

OK, that's good. But, we should see a few steps ahead, so let's update the files as such.

.
├── .gitignore
├── api
│   └── sqsWrite.js
└── serverless.yml

1 directory, 3 files

What we did is to create an api folder and move our handler.js file into it and rename it to sqsWrite.js. At this point I also highly suggest you use git, so, just run git init and commit now and then.

Now, it's time to update the serverless.yml file according to our needs. You'll see comments in each section in the yaml to give you the idea of what we are doing.

service: samplelambda
frameworkVersion: "2"

# Add some variables to use later on.
custom:
  stage: dev
  region: eu-west-1

# Let's use the variables above in the provider.
provider:
  name: aws
  runtime: nodejs12.x
  stage: ${self:custom.stage}
  region: ${self:custom.region}

  # Lambda #1 needs the `sqs:SendMessage` permission
  # to send data to our queue
  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - "sqs:SendMessage"
      Resource:
        Fn::GetAtt:
          - lambdaPayload         # This is the name I choose for our queue. See the resources.
          - Arn

functions:
  # This is the sqsWrite function definition.
  sqsWrite:
    handler: api/sqsWrite.push    # We're going to name the function `push`.
    memorySize: 128
    description: Send the payload to the SQS Queue

    # We need the SQS URL to use in our code. So, setting it to an env variable.
    environment:
      SQS_URL:
        Ref: lambdaPayload

    # We said that we accept a POST request.
    events:
      - http:
          path: /sqs
          method: post

resources:
  Resources:
    # Here, we defined the SQS Queue.
    lambdaPayload:
      Type: AWS::SQS::Queue
      Properties:
        QueueName: lambdaPayload

Before we apply all this, let's go to our sqsWrite.js file and update it as such to see if it's working properly. The code might not be the best. Again, remember that I'm fairly new to the NodeJS. However, it'll get things done. It also doesn't have the best error handling, but let's move on for now.

'use strict';

const AWS = require('aws-sdk');

const sqsQueue = new AWS.SQS();
const sqsUrl = process.env['SQS_URL'];

module.exports.push = (event, context, callback) => {
  const params = {
    MessageBody: event.body,
    QueueUrl: sqsUrl,
  };

  sqsQueue.sendMessage(params, (err, data) => {
    if (err) {
      console.error(err);
      callback(new Error('Couldn\'t send the message to SQS.'));
      return;
    } else {
      console.log('Successfully sent the message to SQS.');

      callback(null, {
        statusCode: 200,
        body: JSON.stringify({
          message: 'Successfully sent the message to SQS.'
        })
      });
      return;
    }
  });
}

Let's apply all this with the following command.

# sls is short for serverless
$ sls deploy

This will take a short amount of time but in the end, it should give us a URL similar to the following to trigger our lambda.

Service Information
service: samplelambda
stage: dev
region: eu-west-1
stack: samplelambda-dev
resources: 12
api keys:
  None
endpoints:
  POST - https://XXXXXXXXXX.execute-api.eu-west-1.amazonaws.com/dev/sqs
functions:
  sqsWrite: samplelambda-dev-sqsWrite
layers:
  None

Now, let's check things on AWS Console. If we go to AWS Lambda and SQS respectively, we should see our resources created and ready for action.

And, if we go in our lambda function by clicking on it, we should see that our permissions are all clear and our environment variables are set to our queue's URL.

It's time to test the function. You can use curl or Postman to send an HTTP request. Here's the request.

$ curl -L -X POST 'https://XXXXXXXXXX.execute-api.eu-west-1.amazonaws.com/dev/sqs' -H 'Content-Type: application/json' --data-raw '{
    "message": "Sent using curl!"
}'

You should get the following message as a response. If it's not the message you are getting, you might need to do some debugging.

{"message":"Successfully sent the message to SQS."}%

If so, Hurray! You should see your message number go up in the AWS Console as well.

This is a good place to make a git commit :)

Lambda #2

OK, now it's time to start with the next function which will get triggered automatically when the SQS queue receives a message and upload the received document to S3.

Let's create a file called s3Upload.js in our api folder first. We'll fill it right after we finish writing the new definitions in serverless.yml file. This yaml file, with all the things in itself, should look like this. I'll comment the parts I added.

service: samplelambda
frameworkVersion: "2"

custom:
  stage: dev
  region: eu-west-1

  # We need a `globally` unique bucket name. You can name it anything you want.
  # I used my name.
  s3Bucket: sample-lambda-sqs

provider:
  name: aws
  runtime: nodejs12.x
  stage: ${self:custom.stage}
  region: ${self:custom.region}

  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - "sqs:SendMessage"
      Resource:
        Fn::GetAtt:
          - lambdaPayload
          - Arn

    # We need permission, one that allows us to put an object into S3.
    - Effect: "Allow"
      Action:
        - "s3:Put*"
      Resource:
        Fn::Join:
          - ""
          - - "arn:aws:s3:::"
            - "Ref": "lambdaBucket"
            - "/*"

functions:
  sqsWrite:
    handler: api/sqsWrite.push
    memorySize: 128
    description: Send the payload to the SQS Queue
    environment:
      SQS_URL:
        Ref: lambdaPayload
    events:
      - http:
          path: /sqs
          method: post

  # This is the s3Upload function definition.
  s3Upload:
    handler: api/s3Upload.push
    memorySize: 128
    description: Upload the message to S3

    # Again, we need the S3 Bucket name in our code.
    environment:
      S3_BUCKET:
        Ref: lambdaBucket

    # This is the catch.
    # This event will add the ability to
    # get triggered by a new message in our queue.
    events:
      - sqs:
          arn:
            Fn::GetAtt:
              - lambdaPayload
              - Arn
          batchSize: 1
resources:
  Resources:
    lambdaPayload:
      Type: AWS::SQS::Queue
      Properties:
        QueueName: lambdaPayload

    # Here, we defined the S3 Bucket.
    lambdaBucket:
      Type: AWS::S3::Bucket
      Properties:
        AccessControl: BucketOwnerFullControl
        BucketName: ${self:custom.s3Bucket}-${self:service}

Again, before we apply this, let's write the s3Upload function.

'use strict';

const AWS = require('aws-sdk');

const s3 = new AWS.S3();
const s3Bucket = process.env['S3_BUCKET'];

module.exports.push = (event, _, callback) => {
  const object = {
    MessageId: event.Records[0].messageId,
    Attributes: event.Records[0].attributes,
    Body: JSON.parse(event.Records[0].body),
  };

  const buffer = Buffer.from(JSON.stringify(object));

  const params = {
    Bucket: s3Bucket,
    Key: `${event.Records[0].messageId}.json`,
    Body: buffer,
    ContentEncoding: 'base64',
    ContentType: 'application/json',
    ACL: 'public-read',
  };

  s3.putObject(params, function (err, _) {
    if (err) {
      console.log(err, err.stack);
      callback(new Error('Couldn\'t send the document to S3.'));
      return;
    } else {
      console.log('Successfully sent the document to S3.');

      callback(null, {
        statusCode: 200,
        body: JSON.stringify({
          message: 'Successfully sent the document to S3.'
        })
      });
      return;
    }
  });
}

OK, we are ready to apply this. Let's run sls deploy. Once it's done, we should see the second function and our S3 bucket on AWS Console.

If we go into our new function's details, we'll see that SQS trigger is there and ready.

It looks like everything is ready to work all together, so let's test it.

$ curl -L -X POST 'https://XXXXXXXXXX.execute-api.eu-west-1.amazonaws.com/dev/sqs' -H 'Content-Type: application/json' --data-raw '{
    "message": "Really important message!"
}'

And when we got back a success message saying that our message is sent to SQS, we can check our bucket to see if our message is in there.

And if we view the document, we'll see the really important message and a few details we added in our code is all there.

Et voilà!

This was a success in my opinion and I do hope that this was helpful. Of course, if you are experienced and already knew all this, I'd love you to hear your suggestions.

Thundra Integration

For this part, I want to do another post. I mean, I already did this part, but I'm quite new to Thundra. So, I don't have enough information yet or have a scenario in my mind to write a post. Plus, this post already has gotten too long.

However, if you want to do the integration part yourself and discover Thundra, I suggest you go to our website and play with it.

Let's wrap it up here. See you in another post!

Just this week. Really. ↩
Thundra is an end-to-end observability and debugging service for your serverless architecture. See more at thundra.io. ↩