DEV Community: Stephen Borsay

Asynchronous Serverless IoT on AWS featuring WebSockets

Stephen Borsay — Mon, 10 Jan 2022 19:47:06 +0000

Part 2 of a 4 part series on AWS Serverless IoT

In part one of this series of hands-on labs covering AWS Serverless IoT I showed you how to develop the "Worlds Simplest Synchronous Serverless IoT Dashboard on AWS." The design was fairly simple:

We ingest IoT data from our device into AWS IoT Core.
That JSON IoT payload is then put into a S3 bucket using an IoT Core Action/Rule.
Our static website, in that same public S3 bucket as our IoT data, then consumes our IoT data on an adjustable time interval.
The web host then graphs the IoT data in a line chart using Highcharts.js.

While this IoT implementation was both simple and effective, in fact the "World’s Simplest" as determined by extensive scientific review, it did have some problems.

The main problem with the previous synchronous IoT design was that IoT data was only extracted on interval, which led to over-fetching of stale data as well as under-fetching of new IoT data that could easily have been missed if synchronous polling didn’t align with data replacement in our S3 bucket. It should be remembered that this is an inherent ‘issue’ with serverless design. When we have a “serverful” design, like with a Linux partition using AWS EC2, we have a live server instance that can “serve up” data on demand. With a “serverless” model AWS Lambda runs server code on demand, thus the web host and the server can't implicitly execute coordinated “CRUD” operations without special provisions.

So how could we fix these issues? Well the obvious answer is to move our IoT design flow from a synchronous polling model to an asynchronous model. This can be also re-framed as moving from a “client pull” of data from our website host to a "server push" of data from AWS Lambda. There are various ways to do this in a serverless model, here are three :

AWS WebSockets in the browser.
MQTT over WebSockets using AWS IoT device SDK for JavaScript in the browser.
GraphQL with AWS AppSync/Amplify in the browser.

AWS WebSockets to the rescue

In this tutorial we will focus on the AWS WebSockets solution for asynchronous serverless design. I have a more generic WebSockets solution tailored for IoT which requires a manual input of the connection ID which simplifies both of the lambda functions I use in this tutorial. You can see this more standard solution in my Udemy Class or on my YouTube channel. However, for this written walk-through lab I’m providing my “improved” WebSockets solution which is more complicated but also eliminates the need for manual input of the connection ID in an otherwise automated design flow.

Just like MQTT, which most of you are familiar with, WebSockets allows bi-directional communication so our lambda function can send and receive data directly from our website host. This allows us to move from our synchronous client polling to asynchronous server pushing. AWS WebSockets requires AWS API Gateway to provide an external and internal WebSockets endpoint URL to accomplish this.

Synchronous – Our website pulling IoT data via polling from a data repository in S3 on interval (Client pull)

Asynchronous – Our Lambda function pushing IoT Data via WebSockets to our Website host as soon as the data is available (Server push)

Now that we have a basic understanding of the advantages of WebSockets for Asynchronous IoT we can proceed to build our design flow on AWS to achieve these aforementioned advantages.

Table of contents:

Step 1 - Create a public Bucket in S3 and then enable static Web hosting
Step 2 - Set up a variable in the Systems Manager Parameter Store
Step 3 - Creating our Lambda functions
Step 4 – Creating WebSocket Endpoints in AWS API Gateway
Step 5 – Creating an AWS IoT Core Action and Rule
Step 6 – Uploading your HTML and JavaScript code to create a asynchronous visualization for your IoT data.
Step 7 – Populating and visualizing your IoT data using an automated IoT data producer

All the code posted in this tutorial can also be found at:

https://github.com/sborsay/Serverless-IoT-on-AWS/tree/master/Level4_design/3_%20Asynchronous_IoT_Intermediate

✅Step 1 - Create a public Bucket in S3 and then enable static Web hosting

For those of you who read my first article "World's Simplest Synchronous Serverless AWS IoT Dashboard" I already explained this process step by step. For those that didn't read it I'll iterate the process again here. You can skip this step if you remember the procedure from the first tutorial in this series. After we make the public bucket we are going to add additional web hosting capabilities to our S3 bucket.

Whenever we create a public bucket the first caveat is to confirm the bucket will only store data that we don’t mind sharing with the world. For our example we are just using the S3 bucket to hold IoT JSON data showing temperature, humidity, and timestamps. I think sharing basic environmental data from an unknown location is not too much of a privacy risk. The advantage of using a public bucket for our static web host, with an open bucket policy and permissive CORS rule, is that it makes the website easily accessible from anywhere in the world without having to use a paid service like AWS CloudFront and Route 53.

Since re:Invent 2021 AWS has changed the process in which to make a S3 bucket public. They have added one extra default permission which must be proactively changed to ensure you are not declaring a public bucket by mistake. AWS is especially concerned with people making buckets public unintentionally, the danger being that they will hold sensitive or personal data, and in the past unethical hackers have used search tools to find private data in S3 public buckets to exploit them. Fortunately for our use case, we don’t care about outsiders viewing our environmental data.

Many of you already know how to make a S3 public bucket for a static webhost on AWS. For those that don’t know how to do this in 2022, I will document it below.

Making a Public S3 Bucket

The process of creating a public S3 bucket for website hosting:

Go to AWS S3 and then select “Create bucket”

A) Give your bucket a globally unique name, here I call mine a catchy name: mybucket034975

B) Keep your S3 bucket in the same region as the rest of your AWS services for this lab.

C) Switch “Object Ownership” to “ACL’s enabled”, this is new for late 2021! We now must first enable our Access Control Lists to make them public.

D) Unblock your S3 bucket and acknowledge that you really want to do this. Scary anti-exculpatory stuff! 😧

F) Finally, select the “Create bucket” button at the bottom of the screen. That's all you have to do for this page, but don’t worry, we are going to have more opportunities to make sure we really, really, and truly want to create a public bucket soon. 👍

G) Now go back into your newly created bucket and click on the “Permissions” tab.

F) Go to Bucket Policy and choose “Edit.” We will paste and save a basic read-only policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicRea2411145d",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion"
            ],
            "Resource": "arn:aws:s3:::<Paste-Your-Bucket-Name-Here>/*"
        }
    ]
}

You must paste the name of your bucket into the policy then follow it by ‘/*’ to allow access to all Get/Read partitions within the bucket. Also it's a good idea to change the “Sid” to something unique within your account.

G) Now we get a chance to visit that ACL we enabled earlier in this process. Click “Edit” and then make the changes as shown below:

We are giving “Everyone,” or at least those who know or can discover our unique bucket URL, permission to read our bucket info. Click on the 'List' and 'Read' buttons where shown and then acknowledge again that you are extra special certain that you want to do this 😏. Then click “Save changes.”

CORS rules used to be in XML only format and then AWS decided to keep everything consistent and switch the CORS format to JSON. Even though this change caused some legacy conflict issues with existing XML CORS rules it was the right choice as JSON is clearly better than XML despite what the SOAP fans on social media will tell you 👍. Below is a generic CORS JSON document you can use in your own S3 bucket:

[
    {
        "AllowedHeaders": [
            "Authorization"
        ],
        "AllowedMethods": [
            "GET"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [],
        "MaxAgeSeconds": 6000
    }
]

That’s it for making your cheap and easily accessible public bucket but now it's time to effortlessly turn our public bucket into a static webhost.

As I said before AWS makes it so the same S3 bucket can be enabled to both hold IoT data and to host a static website with a static IP address for pennies a month.

We are now ready to convert our public bucket so that it can facilitate hosting a static website.

Go to your new S3 public bucket, select the "Properties" tab, then scroll down to the bottom where we can edit "Static website hosting" and select "Edit."

Now enable website hosting and name your index document “index.html”, this will be our landing page for our visualization website. Click “Save changes” at the bottom of the page and you are good to go.

That’s it! Now your open public bucket is configured as a webhost with a unique URL address that is statically available worldwide. You have just changed your uber cheap and accessible public bucket into a uber cheap and accessible public bucket that can also host a website with a static IP address. 😲

In my Udemy course I speak more about inexpensive ways to add security while avoiding paying for CloudFront or Route 53 for accessible public buckets and static websites in S3. However I will tacitly reveal “one weird trick” that I find very effective for pretty good protection regarding free S3 public bucket security: Simply google “restrict IP range in a S3 public bucket policy.”

✅Step 2 - Set up a variable in the Systems Manager Parameter Store

We need to keep a hold of the unique session connection ID as a requirement of WebSockets. This is similar to unique client ID’s are a requirement of the MQTT protocol to keep track of device clients. I will be using the AWS Systems Manager Parameter Store rather than DynamoDB (as is typical) to store our connection ID. I can get away with this massive simplification and cost saving mechanism over using a database because unlike a generic and overused “chatroom” websocket examples, which requires us to keep track of an unknown number of connection ID’s (chatroom participants), for our use case we can be assured that we only need one connection ID. This connection ID denotes our connection between our browser client and AWS as our server.

Now navigate in the AWS console to the “Systems Manager” service and select the “Parameter Store” on the panel on the left hand side of the screen. On the upper right of the screen select the “Create parameter” button.

Choose a parameter name like “connection_id”, choose Type: String, and then put some dummy string value into the box below (we don’t care about the string value here because it will be overwritten by our connection Lambda). That’s it, super easy, and for all effective purposes Parameter Store is free for 10,000 strings; compare this cost to DynamoDB once you are off the free tier!

Finally hit, "Create parameter" and you are done.

✅ Step 3 – Creating our Lambda functions

We will need two lambda functions for this lab.

A) Lambda function one – ‘connection’ Lambda: This lambda function receives the connection ID from the website host via the $connect route and then stores the ID into the AWS Parameter Store as a string variable. The $connect route on API Gateway is mapped to this lambda.

B) Lambda function two – ‘sendIoTdata’: This lambda function will forward our IoT data to our static website host through the 'message' route in API Gateway. We will need both the lambda mapped 'message' route and the ability to execute an API Gateway 'post_to_connection()' function to achieve bi-directional data transfer capabilities with API Gateway mediating the data pipe between our lambda function and our website. We will also use an API from the AWS Parameter Store to retrieve the connection ID that was written from our 'connection' lambda function.

1. Connection Lambda

Navigate to Lambda and create a new Lambda function in Node.js. Call it something like “myConnection”, you can also choose your own name. We will be using the Node.js V. 14 runtime.

After you create your lambda paste the following code into the function.

//Extra Permission required: SSM (search for 'system' in inline policies)

const AWS = require('aws-sdk') 


var mySSM_Client = new AWS.SSM();   //create new client object of System Manager Class

exports.handler = async (event, context) => {

 console.log(event); 
 let connectionId = event.requestContext.connectionId;
 console.log("myConnectionID is: ", connectionId)

//-----------------Begin SSM Code

//https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SSM.html#putParameter-property
var params = {
  Name: '<Insert-Your-SSM-Parameter-Name-Here>',
  Value: connectionId,
  Overwrite: true  //not required but default is False
};

//await and promise() stub are not documented but necessary for //function to work -  UNFORTUNATELY

var mySSM_request = await mySSM_Client.putParameter(params, function(err, data) {
  if (err) console.log(err, err.stack); // an error occurred
  else     console.log("success: ", data);           // successful response
}).promise()
//var mySSM_request = await mySSM_Client.putParameter(params).promise(); //should also work; sans padantic if err else

console.log("My request: ", mySSM_request)

//------------------End SSM Code

     const response = {
        statusCode: 200,
        body: JSON.stringify('Hello from Lambda!'),
    };
    return response;  //need a response or we get disconnected immediately

};

At this point you should paste the name of your parameter variable string you made in the Parameter store earlier into the Lambda function on line 22 where it reads:

Before:

 Name: '<Insert-Your-SSM-Parameter-Name-Here>'

After:

Name: connection_Id,

As you can see the function has two main routines.

A) The lambda function receives the connection ID by extracting this necessary parameter by digging into the blob of browser data that is returned from our static website host as soon as we open our custom visualization website. This will be done by integrating this connection lambda with our $connect route key that we will set up in API Gateway soon. You may not know but the code below:

 console.log(event); 
 let connectionId = event.requestContext.connectionId;
 console.log("myConnectionID is: ", connectionId)

logged your event info to AWS CloudWatch. In CloudWatch you can view the big blob of data your browser sent to your lambda. Deep within this blob is the connection ID that will be saved in the AWS System Manager Parameter Store.

B) After the connection ID is successfully extracted, the lambda function uses the “put” API from the Systems Manager Parameter Store to save the connection ID to a string variable. That string variable in the Parameter Store (xxxxx…) is now overwritten with the real variable and is made available for use when called by our "sendIoTdata" lambda function that we will create next. Whenever a new connection is made the new connection ID is returned from the website, via the $connect route key.

A few notes about this function are in order:

You may notice that I have put a link to the put Parameter API code documentation within the lambda function as a comment. Notice that I had to modify the original API code or it would not work as documented by AWS. I received no errors as to the problem with the 'put' api, but the parameter string value would not be stored. However I have dealt with this issue before when using Node.js in Lambda and it seems to be an ongoing challenge. To fix this problem I had to add the additional await and promise() stub to the function. I will only make two opinionated points about this; One, this issue should be addressed by the Lambda tools team to optimize the Lambda runtime compiler for Node.js to ameliorate this issue when there are no external calls, and thus indeterminate waits required. Two, the longer I work with lambda in Python and Node the more I prefer using Python for Lambda when appropriate. Assured sequential instructional operation just makes dev life easier😎.

Adding permissions to our connection Lambda Function

No, we aren't done with our connection lambda yet, we still need to add the necessary permission so that our lambda function has the ability to write to the Parameter Store in the AWS Systems Manager. To accomplish this navigate to IAM by going to:

Configuration tab → Permissions tab→ Role name → and then open your role up to get to AWS IAM.

Within IAM we need to add an inline policy. Click the “Add inline policy” on the right hand side of the screen.

We will add our Parameter Store permission which is in the “Systems Manager” service. To add this permission search for it by typing in "sys" in the policy search box until "System Manager" comes up and then grant it all Access and all Resources permissions. Now click “Review Policy” and give your policy a name like “SysManager4connection” and then create policy. Your screen should look like this:

Press “Create Policy and now you are done.

Now that we are done assigning the needed Systems Manager permission for our policy we can create our second Lambda function.

2. sendIoTData Lambda

Navigate back to Lambda and create a new Lambda function in Python 3.8. Call it something like “sendIoTData”, you can also choose your own name.

“Create function” and then paste the following code into the function:

import json
import boto3
Websocket_HTTPS_URL = "https://<Insert-Websocket-Endpoint_Here>"
client = boto3.client("apigatewaymanagementapi", endpoint_url = Websocket_HTTPS_URL)
ssm_Client = boto3.client('ssm')


def lambda_handler(event, context):
    print(event) 
    response_ssm = ssm_Client.get_parameter(Name='<Insert_ConnectionId-Parameter-Name-Here>')
    print("my stored connection id: ", response_ssm['Parameter']['Value'] )
    connectionId =  response_ssm['Parameter']['Value']  #dig into the response blob to get our string cvalue
    Test_Message = json.dumps({ "message": "Hello from lambda, hardcoded test message"})
    IoT_Message = json.dumps(event)
    #AWS API Gateway API's require 'key=value' arguments
    response = client.post_to_connection(ConnectionId = connectionId, Data = IoT_Message)

This Lambda function accomplishes two main tasks.

A) The lambda function retrieves the stored connection ID we saved in the previous function, and then adds it as a key value parameter to our “post to connection” API. Lambda uses the "get_parameter()" API to retrieve the connection ID we saved in the Systems Manager Parameter Store with the connection lambda.

B) The lambda function receives our incoming IoT data from AWS IoT Core as an event, and then dispatches the IoT JSON payload along with the connection ID and the internal WebSockets https endpoint via the API Gateway client's "post_to_connection()" function and on to our website via the external WebSocket endpoint.

At this point you should paste the name of your parameter variable from the Parameter Store into the lambda function:

Before:

response_ssm = ssm_Client.get_parameter(Name='<Insert-Your-SSM-Parameter-Name-Here>')

After:

response_ssm = ssm_Client.get_parameter(Name='connection_Id')

Now deploy the sendIoTdata function.

At this point we still need the internal WebSocket HTTPS endpoint which we won’t get until we create a WebSocket API in AWS API Gateway. We will accomplish this in the next section.

A few interesting notes to help explain the operation of this sendIoTdata function. To retrieve our connection ID we have to dig into the blob of data returned as a response when invoking the client object of type SSM. This is a different blob of information than what we received from our website host. If you want to see this information, or any other information in lambda, simply 'print' out the response and go to AWS CloudWatch to examine the blob info. CloudWatch is a necessary tool for any lambda debugging. For convenience CloudWatch permissions are always included in basic lambda execution roles which are created automatically for you when you created your function.

A second thing to note is that the API:

post_to_connection(ConnectionId = connectionId, Data = IoT_Message)

requires a key-value pair as function parameters. The IoT message has the option of being any hardcoded message we add to the lambda function or just a JSON payload from a test event. Of course we will be using a real IoT JSON payload as a message when we add in the AWS IoT Core service in a coming step. This function is part of the API Gateway client ("apigatewaymanagementapi") and thus needs the related permission to execute the API.

Adding permissions to our sendIoTdata Lambda function

Bad news, we have to add two extra permissions for this Lambda function to integrate it with our complete design flow. Good news, you already know how to do this from our previous step. The two permissions we will add are “Systems Manager” and “ExecuteAPI.”

To add the "Systems Manager" permission policy to give access to the parameter store, simply follow the same instructions from the previous lambda permissions.

To find the "ExecuteAPI" permissions policy just search for "execute" in the policies search box. Duplicating the same process as before, you can give access to all resources to make things easy or narrow the permissions down to your lambdas ARN if you have to be pedantic or are on a shared AWS account. Now you are ready to move to the next step. More good news, you have just completed the most difficult part of this tutorial.

You should now have three policies for this lambda: default execution role, Systems Manager, and ExecuteAPI

Finally, note that we need "ExecuteAPI" in the 'sendIoTdata' lambda function because the "post_to_connection()" API needs to pass our IoT payloads through API Gateway before it can be sent to our webpage. The ExecuteAPI permission policy allows our lambda function to 'execute' the internal https endpoint via the API Gateway 'client' using the "post_to_connection()" API. We will be creating our internal and external WebSocket endpoints in the next section in AWS API Gateway.

✅ Step 4 – Creating WebSocket Endpoints with AWS API Gateway

WebSockets form a direct connection with your website host and thus they need a URL which directs both the website and the lambda function as to where to direct their data exchange. API Gateway provides these endpoints for this exchange just as it would for a normal REST API.

We will be obtaining two WebSocket endpoints from our creation process in API Gateway. One internal endpoint (https) for use in the lambda function and one external endpoint (wss) for use on our webpage.

External WebSocket endpoint: this is the URL with a 'wss' prefix. This is the AWS WebSockets endpoint needed in our JavaScript code on our website to communicate with API Gateway, and in turn, communicate with our lambda functions.
Internal WebSocket Endpoint: this is the URL with a 'https' prefix. This is the AWS WebSocket endpoint needed in our 'sendIoTdata' lambda function to communicate with API Gateway through the post_to_connection() function, and in turn, communicate with our web page.

You should now create your own API Gateway WebSocket endpoints and routes by navigating to API Gateway –> Create API and choosing to build a Websocket API.

Select a name for your WebSocket API and then type “request.body.action” in the box for Route Selection Expression. This is the standard path to designate a WebSocket action like “message”, “join”, or “send”.

Choose the “Next” button and then on the next screen we will select one pre-made macro route and one custom route. The one route that we need is the connection route. I'm not adding optional routes just to keep the tutorial as simple as possible while still maintaining reasonable functionality.

Select to add a connection route with the macro called $connect to route our "connection" lambda function. The second route we will designate is a custom route we will call “message”. This is the custom route which forms a bi-directional pipe or "socket" between our website and AWS allowing for incoming and outgoing data.

Select “Next"

Now we have to link our WebSockets API routes with our two previously created lambdas. You may see why I created the lambdas first even though it is counter intuitive to our design. This is typical for IoT design flows as we often have to work backwards. To link our two lambdas to our two routes just select the lambda functions you just created and link them to the appropriate route key of similar names.

Select “Next” then leave the stage name as “production” and select “Next” again, and then finally “create and deploy.”
Now to view the WebSocket endpoints go to the tab “stages” on the left of your screen. Click your only stage which should be in blue and called “production.” After selecting it you should now see your two WebSocket endpoints on the top of your screen. Leave this screen open as we will need both endpoints.

As an aside, this single external "wss" socket URL can work with any static webhost external to AWS to connect to AWS. Thus you can also host this Highcharts visualization website on JSbin, JSFiddle, playcode.io, or any other web based host you like if all you want to do is test the IoT design and don't care about retaining your website after testing.

OK, now we can complete our "SendIoTdata" Lambda function with the new internal https WebSocket URL and then re-deploy the lambda function. To do this open a new tab and navigate back to your lambda "sendIoTdata" lambda function. The format for pasting the endpoint to your lambda follows next.

Adding the WebSockets internal endpoint to sendIoTMessage Lambda

B) Insert HTTP Endpoint:

Before:

Websocket_HTTPS_URL = 'https://<Insert-Websocket-Endpoint_Here>'

After:

Websocket_HTTPS_URL = 'https://4astring7.execute-api.us-east-1.amazonaws.com/production'

For this endpoint use the 'https://' prefix

Don’t forget to 're-Deploy' your lambda or your changes won’t be made.

✅Step 5 – Creating an AWS IoT Core Action and Rule

In this step we will create an AWS IoT Core Action and Rule that will send our IoT data from the MQTT broker in IoT Core to our 'sendIoTData' lambda function. Once the IoT payload is in our lambda function with the connection ID being set, the JSON payloads can then be forwarded to our static website via the internal WebSocket endpoint.

In AWS IoT Core select: Action→ Rules→ Create

Select a name for your Rule and then change the Rules Query Statement to the following:

SELECT *, timestamp() as timestamps FROM 'iot/#'

This query does two important things that differ from the default RQS. First, it adds a field to our incoming JSON IoT payload called “timestamps.” This "timestamps" field is a literal match for a variable extant in our JavaScript webcode. Using the premade AWS function timestamp() also adds a Unix/Epoch timestamp to our payload. We will need this for indexing the X-axis on our line chart. Of course adding the timestamp can be done on the device, or in lambda depending on your needs or your preferences. Often on non-application level MCU’s on embedded devices, UNIX timestamps cannot be programmed without additional RTC hardware or extra library's, in these cases I often use “uptime” as a relative time index rather than an absolute time format like the timestamp() function provides.

The second important thing the RQS does is that it uses a topic from which the hash/pound allows a fungible extension to the incoming base topic. So for instance, if I was using MQTT messaging to communicate between lambda functions (standard inter-lambda is SNS) I could have one topic published as 'iot/lambdaTopic' and another as 'iot/deviceTopic'. Using the 'iot/#' means that both formats will be picked up by my RQS and then I could discriminate how I handle multiple topics coming into the same or different lambda function by topic extension. The exciting possibilities are endless!

Your Rule should now look something like this:

Let’s select “Add Action” next.

Choose to add a “send message to a Lambda function” action. Can you guess what Lambda we will add?

Select “add action” which sends us back to our Rules page where we must select “Create Rule."

The final thing we must do is make sure our new rule is 'enabled.' On the next page by selecting the breadcrumbs next to your new rule. You can find your new Rule at the bottom of any Rules you may have created previously. Once found make sure your Rule is "enabled".

✅ Step 6 – Uploading your HTML and JavaScript code to create a asynchronous visualization for your IoT data

We have two files to upload to our public bucket and our newly created webhost. The files are called 'index.html' and 'main.js'. While the index.html is the exact same as the one listed in the previous article in this series, the main.js file is different. The main.js now has WebSockets compatible asynchronous code as shown here:

const socket = new WebSocket('<Insert-Your-WSS-Endpoint-With-Prefix-Here>')

socket.addEventListener('open', event => {
  console.log('WebSocket is connected, now check for your new Connection ID in Cloudwatch on AWS')
})


socket.addEventListener('message', event => {

    console.log('Your iot payload is:', event.data);
    drawChart(event.data);
    })

By these few lines of code we can now take advantage of WebSockets to enable the "sendIoTdata" lambda function to perform a "server push" and bring event changes (incoming IoT payloads) directly into our static web host. This is possible through the socket event listener for 'message' which forms a bidirectional link with API Gateway given the external WebSocket address and the route key of 'message'.

The index.html is our launch page. Copy the following code and save it locally as "index.html":

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <title>Dashboard</title>
</head>

<body>
    <div class="container">
        <h1>Asynchronous Weather Data with AWS Websockets</h1>

        <div class="panel panel-info">
            <div class="panel-heading">
                <h3 class="panel-title"><strong>Line Chart</strong></h3>
            </div>
            <div class="panel-body">
                <div id="container1"></div>
            </div>
        </div>
    </div>

    <script src="https://code.jquery.com/jquery-3.1.1.min.js"></script>
    <script src="https://code.highcharts.com/highcharts.js"></script>
    <script src="./main.js"></script>
</body>

</html>

The main.js is our JavaScript page (below). Copy the following code then open it in the editor of your choice. We have one change to this code we need to make before you can upload it to S3:

let humArr = [], tempArr = [], upArr = [];
let myChart = Highcharts.chart('container1', {

    title: {
        text: 'Line chart'
    },

    subtitle: {
        text: 'subtitle'
    },

    yAxis: {
        title: {
            text: 'Value'
        }
    },

    xAxis: {
        categories: upArr
    },

    legend: {
        layout: 'vertical',
        align: 'right',
        verticalAlign: 'middle'
    },

    plotOptions: {
        series: {
            label: {
                connectorAllowed: false
            }
        }
    },
    series: [{
        name: 'Humdity',
        data: []
    }, {
        name: 'Temperature',
        data: []
    }],

    responsive: {
        rules: [{
            condition: {
                maxWidth: 500
            },
            chartOptions: {
                legend: {
                    layout: 'horizontal',
                    align: 'center',
                    verticalAlign: 'bottom'
                }
            }
        }]
    }

});


const socket = new WebSocket('<Insert-Your-WSS-Endpoint-With-Prefix-Here>')

socket.addEventListener('open', event => {
  console.log('WebSocket is connected, now check for your new Connection ID in Cloudwatch on AWS')
})


socket.addEventListener('message', event => {

    console.log('Your iot payload is:', event.data);
    drawChart(event.data);
    })



let drawChart = function (data) {


var IoT_Payload = JSON.parse(data);
console.log("our json object", IoT_Payload);

    let { humidity, temperature, timestamps } = IoT_Payload;

    humArr.push(Number(IoT_Payload.humidity));
    tempArr.push(Number(IoT_Payload.temperature));
    upArr.push(Number(IoT_Payload.timestamps));

    myChart.series[0].setData(humArr , true)
    myChart.series[1].setData(tempArr , true)
}

The only change you need to make to the code is on line 61 of the main.js file.

Before:

const socket = new WebSocket('<Insert-Your-WSS-Endpoint-With-Prefix-Here>')

After:

const socket = new WebSocket('wss://4astring7.execute-api.us-east-1.amazonaws.com/production')

You will need to insert the external AWS WebSocket endpoint you got from API Gateway here. This is the external address that starts with wss://. Make sure to include the 'wss://' prefix when pasting your external address into the main.js file.

After changing this line of code in 'main.js', you are now ready to save it locally and then upload the files you just saved into your S3 bucket. To do this simply select the 'Objects' tab in your S3 bucket and drag both files to the base level of your bucket. Both files should be on the same level of the partition hierarchy.

Press the 'upload' button on the bottom right of your screen, and then after both files have been uploaded select the 'close' button. You should now have two objects in your bucket; both web code files('index.html' and 'main.js').

Now is a good time to initiate your static webhost by opening a new web browser tab with your static website URL. The address of your website can be found by going to the “index.html” object in your bucket and opening the 'Object URL.' Clicking this URL will bring up your website.

The Highcharts code works by using AWS WebSockets with AWS Lambda for asynchronous invocations. The connection function in the main.js first gives you the "WebSocket is connected, now check for your new Connection ID in Cloudwatch and the Parameter store on AWS" message when your website is first connected. You can see this message in the browser by typing 'ctrl + shift + i' in most browsers, in Chrome the shortcut is 'ctrl + shift +j'

Now that our website is up, let's send it some IoT data so we can produce our visualization.

✅Step 7 - Populating and visualizing your IoT data using an automated IoT data producer

For this last step we have three ways to populate the visualization from IoT Core to our webhost.

A) Use a device to publish IoT JSON payloads under our topic name.
B) Manually publish JSON data payloads from the MQTT test client in IoT Core as demonstrated earlier in the tutorial.
C) Use a test script to publish IoT data to our topic automatically at a configurable interval and delay between IoT payloads.

For Option A you can simply program your device to publish data to IoT core as I instruct in my course. For Option B you would have to spend some time manually altering and then publishing JSON payloads in the MQTT test client in IoT Core to generate the line chart in the visualization.

For this tutorial I will explain 'Option C.' For this option you need the AWS CLI installed. It’s easy to install with the directions listed here:

https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

This bash IoT data producer script was provided by AWS and can be originally found on https://github.com/aws-samples. I have already altered the test script to send just temperature and humidity data. Simply insert your AWS region and MQTT topic name (iot/whatever) into the test script where indicated. The bash script uses your AWS CLI to deliver the payload to IoT Core (using your SigV4 credentials from the AWS CLI). You can also change the number of payloads published (iterations) and wait time between each payload publish (interval) to produce as much fake IoT data as you like.

#!/bin/bash

mqtttopic='<Insert-Your-IoT-Topic-Here>'
iterations=10
wait=5
region='<Insert-Your-AWS-Test-Region-Here>'
profile='default'

for (( i = 1; i <= $iterations; i++)) {

  #CURRENT_TS=`date +%s`
  #DEVICE="P0"$((1 + $RANDOM % 5))
  #FLOW=$(( 60 + $RANDOM % 40 ))
  #TEMP=$(( 15 + $RANDOM % 20 ))
  #HUMIDITY=$(( 50 + $RANDOM % 40 ))
  #VIBRATION=$(( 100 + $RANDOM % 40 ))
  temperature=$(( 15 + $RANDOM % 20 ))
  humidity=$(( 50 + $RANDOM % 40 ))

  # 3% chance of throwing an anomalous temperature reading
  if [ $(($RANDOM % 100)) -gt 97 ]
  then
    echo "Temperature out of range"
    TEMP=$(($TEMP*6))
  fi

  echo "Publishing message $i/$ITERATIONS to IoT topic $mqtttopic:"
  #echo "current_ts: $CURRENT_TS"
  #echo "deviceid: $DEVICE"
  #echo "flow: $FLOW"
  echo "temperature: $temperature"
  echo "humidity: $humidity"
  #echo "vibration: $VIBRATION"

 #use below for AWS CLI V1
 #aws iot-data publish --topic "$mqtttopic" --payload "{\"temperature\":$temperature,\"humidity\":$humidity}" --profile "$profile" --region "$region"

 #use below for AWS CLI V2
 aws iot-data publish --topic "$mqtttopic" --cli-binary-format raw-in-base64-out --payload "{\"temperature\":$temperature,\"humidity\":$humidity}" --profile "$profile" --region "$region"

  sleep $wait
}

You have to change fields at the top of the page in the bash script to customize it for your MQTT topic name (iot/whatever) and AWS region ('us-east-1 or other) in which you developed your AWS services for this tutorial. The other two fields, 'iterations' and 'wait time', are optional to edit.

Edit these fields for your own info:

mqtttopic=''
iterations (number of payloads to send)
wait time (number of seconds between transmissions)
region=''

Now save the above code, giving it a name like "IoT_tester.sh". You can run the script by simply installing the above bash script locally and then from the command prompt typing the name of the bash script. Bash scripts are neat because they should work on any operating system. Activating the test script in MS Windows looks like this:

😀 🏁

Congratulations! You finished the second tutorial in the series and created an asynchronous Serverless AWS IoT Dashboard using WebSockets. Make sure to stay tuned for parts three and four of this hands-on tutorial series as we get more advanced with Serverless IoT on AWS.

World's Simplest Synchronous Serverless AWS IoT Dashboard

Stephen Borsay — Fri, 07 Jan 2022 20:20:44 +0000

Part 1 of a 4 part series on AWS Serverless IoT

When working with IoT devices, which transmit data to AWS, serverless IoT workflows can save the customer a tremendous amount of money. Instead of setting up an “always on” EC2 instance the client can engage individual AWS services only as needed. This multi-part IoT series will cover a variety of methods, with increasing levels of sophistication and functionality, to visualize IoT data on a static web host using various IoT centric services on AWS. The overall cost of using these AWS serverless services, even assuming you are off the free tier, will be pennies for normal use.

This hands-on workshop series will start off with an easy use case: synchronous polling of IoT data from a S3 bucket being held as a IoT data repository delivered from AWS IoT Core. For reasons which will soon be obvious this isn't an optimal design. However there is an undeniable, inverse correlation between complexity and functionality in this use case, so this is a good place to start. If you are ok with “near real-time” IoT, and some lost IoT data payloads are acceptable, than this simplified technique, explained in this first tutorial in the series, will be of interest. As a special bonus I will provide all the code necessary to complete this lab and visualize your own IoT data on your AWS account. It is sure to impress your friends and family (as long as we keep them in the dark about some initial shortcomings to be remedied later).

An assumption of this first AWS IoT serverless workshop is that you have a device programmed to send IoT data payloads to AWS IoT Core. I’m not going provide device code here or explain how to implement the code on a device. You can reference my classes on AWS IoT on Udemy for coding various devices to communicate with AWS IoT Core utilizing a variety of IDE’s. However, good news, you don’t need to program any devices for this tutorial series as I will show you how to use the AWS MQTT test client as well as an automated Bash script to send fake IoT JSON payloads to the MQTT broker on AWS IoT Core. This is a functional substitute for a real embedded device producing real IoT data. For actual use cases you can always implement code on your device later if you want to add the device component for real IoT data publishing.

This first article in this hands-on workshop series will use synchronous polling in JavaScript to extract data from an S3 bucket in near real time. The next article in the series will switch to use AWS WebSockets with Lambda using API Gateway to transmit “real-time" data to our JavaScript visualization without the need to store data in S3 as a temporary repository. From there we will move on to using WebSockets with Lambda, MQTT, and the AWS IoT JavaScript SDK in the browser for a more professional look and feel while also taking advantage of real-time IoT transmissions. Finally we will conclude the series by using the newest real-time techniques for serverless IoT that utilize GraphQL for real-time data visualizations which should obviate the need, if not the performance, of AWS WebSockets. To get the advantage of asynchronicity we are reliant on using a "server push" model as opposed to the "client pull" model we use in this tutorial. The "server push" model has been traditionally problematic for a serverless environment.

Update February 2022

I've received email about this simple process being inefficient. As I said it isn't a "production ready design." It is intended as just a super simple and easy example.

As always when dealing with S3 writes/puts it is always best to "Batch" data as a rule. Also writing large data blocks is the same cost as small PUTs so keep that in mind as well.

Request pricing For standard storage, the cost for PUT, COPY, POST, or LIST requests ranges from $0.005 per 1000 requests in US regions to $0.007 in other regions.

Cost to write a JSON payload are as follows, and the cost of running this process for 24 hours at one payload per second would be:

((60 x 60 x 24)/1000) x .005 = 43 cents

Budget accordingly

Table of contents:

Step 1 - Creating a public bucket in S3 for your IoT data
Step 2 - Creating an Action and Rule in AWS IoT Core
Step 3 – Testing your Serverless IoT design flow
Step 4 – Converting your S3 bucket into a static webhost
Step 5 – Uploading your HTML and JavaScript code to create a Visualization for your IoT data.
Step 6 – Populating the visualization using an automated IoT data producer

All the code posted in this tutorial can also be found at:

https://github.com/sborsay/Serverless-IoT-on-AWS/tree/master/Level4_design/1_Synchronous_IoT

✅ Step 1 - Creating a public bucket in S3 for your IoT data

Whenever we create a public bucket the first caveat is to confirm the bucket will only store data that we don’t mind sharing with the world. For our example we are just using the S3 bucket to hold IoT JSON data showing temperature, humidity, and timestamps. I think sharing basic environmental data from an unknown location is not too much of a privacy risk. The advantage of using a public bucket for our static webhost, with an open bucket policy and permissive CORS rule, is that it makes the website easily accessible from anywhere in the world without having to use a paid service like AWS CloudFront and Route 53.

Since re:Invent 2021 AWS has changed the process in which to make a S3 bucket public. They have added one extra default permission which must be proactively changed to insure you are not declaring a public bucket by mistake. AWS is especially concerned with people making buckets public unintentionally, the danger being that they will hold sensitive or personal data, and in the past unethical hackers have used search tools to find private data in S3 public buckets to exploit them. Fortunately for our use case, we don’t care about outsiders viewing our environmental data.

Many of you already know how to make a S3 public bucket for a static webhost on AWS. For those that don’t know how to do this in 2022, I will document it below.

Making a Public S3 Bucket

The process of creating a public S3 bucket for website hosting

Go to AWS S3 and then select “Create bucket”

A) Give your bucket a globally unique name, here I call mine a catchy name: mybucket034975

B) Keep your S3 bucket in the same region as the rest of your AWS services for this lab.

C) Switch “Object Ownership” to “ACL’s enabled”, this is new for late 2021! We now must first enable our Access Control Lists to make them public.

D) Unblock your S3 bucket and acknowledge your really want to do this. Scary anti-exculpatory stuff! 😧

F) Finally, select the “Create bucket” button at the bottom of the screen. That's all you have to do for this page, but don’t worry, we are going to have more opportunity to make sure we really, really, and truly want to create a public bucket soon. 👍

G) Now go back into your newly created bucket and click on the “Permissions” tab.

F) Go to Bucket Policy and choose “Edit.” We will paste and save a basic read-only policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicRea2411145d",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectVersion"
            ],
            "Resource": "arn:aws:s3:::<Paste-Your-Bucket-Name-Here>/*"
        }
    ]
}

G) Now we get a chance to visit that ACL we enabled earlier in this process. Click “Edit” then make the changes as shown below:

We are giving “Everyone,” or at least those know or can discover our unique bucket URL, permission to read our bucket info. Click on the 'List' and 'Read' buttons where shown and then acknowledge again that you are extra special certain that you want to do this 😏. Then click “Save changes.”

H) Wow, we are at our last step in creating a public bucket. Now we should set the CORS policy so we don’t get any pesky “mixed use” access-control non-allowed origin issues for cross domain access – I hate those 😠! CORS rules used to be in XML only format and then AWS decided to keep everything consistent and switch the CORS format to JSON. Even though this change caused some legacy conflict issues with existing XML CORS rules it was the right choice as JSON is clearly better than XML despite what the SOAP fans on social media will tell you 👍. Below is a generic CORS JSON document you can use in your own S3 bucket:

[
    {
        "AllowedHeaders": [
            "Authorization"
        ],
        "AllowedMethods": [
            "GET"
        ],
        "AllowedOrigins": [
            "*"
        ],
        "ExposeHeaders": [],
        "MaxAgeSeconds": 6000
    }
]

That’s it for making your cheap and easily accessible public bucket!

✅ Step 2 - Create an Action and Rule in AWS IoT Core

AWS IoT Core is a great service with a built-in server side MQTT broker that has the functionality to dispatch our incoming IoT data payloads to a variety of AWS services. For this first lab we will simply be sending our data to our S3 bucket. To do this we need to create an 'Action' and 'Rule' in IoT Core, then we design our rule to send our IoT data to the S3 public bucket that we just created.

The first step is to create a new rule in IoT Core:

Now give your Rule a name of your choice. Next, we need to edit the Rules Query Statement (RQS) to select what information we will extract or add to our JSON IoT Payload. To make things easier we will use one of the built-in functions AWS provides for the RQS to enrich our IoT data payload:

For our use case we are adding a Unix/Epoch timestamp to our incoming IoT JSON payload. I rename the timestamp as 'timestamps'. The reason for this specific name is that I want the name to be a literal match for how I designate the variable in the JavaScript Code on our upcoming website. The MQTT topic name itself is unimportant for this first tutorial, you can call your MQTT topic whatever name you like, here I call mine ‘outTopic’ (as it is coming ‘out’ from my device). In the tutorials coming up it will be more important how we name and format our topic in the RQS.

Next, we have to add an 'Action' for our 'Rule.' We want to send our IoT message to the S3 bucket we just created so select that as your rule:

Now press the “Configure action” button at the bottom of the screen.

Next we must select the bucket we just created and name the 'Key' in which we will save our IoT data. We also have to create a Role which gives our Action a permission policy to send our IoT data between IoT Core and our S3 bucket. We will let AWS automatically create this Role.

The things to do here:

1. Select the S3 public open bucket you just created in the S3 bucket field.
2. Give your key (blob object) a name.
3. Create a role to give your Action the correct permissions.
4. Press the “Add action” button.

Finally, insure your action is “Enabled” in the next screen by using the breadcrumbs next to the Rule name you just created (usually at the bottom of the list as your last Rule).

✅ Step 3 - Test your Serverless IoT design flow

At this point let's test the serverless IoT design flow we developed to make sure everything is working before we move on to uploading our code to our static webhost on S3. To test our design flow we should send some fake IoT data to S3 from the MQTT Test client in AWS IoT Core. To do this go to the "MQTT test client" tab on the left and select the “Publish” tab. This will allow us to send IoT JSON payloads to our public S3 bucket using the Action/Rule we just created. Let's enter a sample IoT JSON payload of temperature and humidity as shown below. Remember, we don’t need to use a "timestamps” key value in our IoT payload because our RQS adds a UNIX timestamp to our payload automatically. We will publish our JSON IoT payload under to topic name of ‘outTopic’ or whatever you choose to name you topic that matches the RQS.

Things to do here:

1. Select MQTT test client.
2. Select the Publish tab.
3. Type a test payload in proper JSON format like: { “temperature”: 44, “humidity”: 55}
4. Press the “Publish” button.

Now go to the public bucket you just created in S3. Look under the key object that you designated in your Rule/Action in IoT Core. It should be something like an anomalous object named “myKey” in our example. Go ahead and download the blob object named "myKey" and open it in the editor of your choice:

Now if everything was done correctly you should see the JSON payload you just sent from IoT Core. If you sent multiple payloads you will only see the last payload sent as the object is overwritten in S3 with each successive payload. You can't concatenate or edit blob objects in S3. As an aside there is an easy way to create a data lake with multiple objects with the S3 Action we just created but I won't go over that here. For our purposes we are only going to fetch the last JSON payload held within the S3 object store on a given interval (polling).

✅ Step 4 - Convert your S3 bucket into a static Webhost

As I said before AWS makes it so the same S3 bucket can be enabled to both hold IoT data and to host a static website with a static IP address for pennies a month.

We are now ready to convert our public bucket so that it can facilitate hosting a static website. We could have easily have done this in Step 1 and still use the same bucket as a blob object store, as well as a website, but converting it to a static website now makes more procedural sense. The conversion is quite simple.

Go to your S3 public bucket, select the "Properties" tab, then scroll down to the bottom where we can edit "Static website hosting" and select "Edit."

Now enable website hosting and name your index file “index.html”, this will be our landing page for our visualization website. Click “Save changes” at the bottom of the page and you are good to go.

✅ Step 5 - Upload your HTML and JavaScript code to create a visualization for your IoT Data.

We have two files to upload to our public bucket and our newly created webhost. The files are called 'index.html' and 'main.js'.

The index.html is our launch page. Copy the following code and save it locally as "index.html":

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <title>Dashboard</title>
</head>

<body>
    <div class="container">
        <h1>Synchronous Weather Data on Interval</h1>

        <div class="panel panel-info">
            <div class="panel-heading">
                <h3 class="panel-title"><strong>Line Chart</strong></h3>
            </div>
            <div class="panel-body">
                <div id="container1"></div>
            </div>
        </div>
    </div>

    <script src="https://code.jquery.com/jquery-3.1.1.min.js"></script>
    <script src="https://code.highcharts.com/highcharts.js"></script>
    <script src="./main.js"></script>
</body>

</html>

The main.js is our JavaScript page. Copy the following code save it as "main.js":

let humArr = [], tempArr = [], upArr = [];
let myChart = Highcharts.chart('container1', {

    title: {
        text: 'Line chart'
    },

    subtitle: {
        text: 'subtitle'
    },

    yAxis: {
        title: {
            text: 'Value'
        }
    },

    xAxis: {
        categories: upArr
    },

    legend: {
        layout: 'vertical',
        align: 'right',
        verticalAlign: 'middle'
    },

    plotOptions: {
        series: {
            label: {
                connectorAllowed: false
            }
        }
    },
    series: [{
        name: 'Humdity',
        data: []
    }, {
        name: 'Temperature',
        data: []
    }],

    responsive: {
        rules: [{
            condition: {
                maxWidth: 500
            },
            chartOptions: {
                legend: {
                    layout: 'horizontal',
                    align: 'center',
                    verticalAlign: 'bottom'
                }
            }
        }]
    }
});

let getWheatherData = function () {
    $.ajax({
        type: "GET",
        url: "<Insert-Your-IoT-Data-Bucket-With-Key-Here>",  //example: https://mydatabucket.s3.amazonaws.com/myKey"
        dataType: "json",
        async: false,
        success: function (data) {
            console.log('data', data);
            drawChart(data);
        },
        error: function (xhr, status, error) {
            console.error("JSON error: " + status);
        }
    });
}

let drawChart = function (data) {

    let { humidity, temperature, timestamps } = data;

    humArr.push(Number(humidity));
    tempArr.push(Number(temperature));
    upArr.push(Number(timestamps));

    myChart.series[0].setData(humArr , true)
    myChart.series[1].setData(tempArr , true)
}

let intervalTime = 3 * 1000; // 3 second interval polling, change as you like
setInterval(() => {
    getWheatherData();
}, intervalTime);

The only change you need to make to the code is on line 62 of the main.js file. You need to insert the URL of your 'key' which is a "Object URL" listed in your S3 bucket.

You can find your data object address (URL) by copying it from your bucket as demonstrated by the image below. It is the ‘Object URL’ with the 'https://' prefix. This object URL should look something like this:

https://yourbucket.s3.amazonaws.com/myKey

After changing this line of code in 'main.js' to your own data location URL in S3, you are now ready to upload the files you just saved locally into your bucket. To do this simply select the 'Objects' tab in your S3 bucket and drag both files to the base level of your bucket. Both files, and your IoT data object ('myKey'), should be on the same level of the partition hierarchy.

Press the 'upload' button on the bottom right of your screen, and then after both files have been uploaded select the 'close' button. You should now have three objects in you bucket; your IoT data object with your JSON readings (myKey), as well as your two web code files('index.html' and 'main.js').

The Highcharts code works by fetching data from or S3 bucket by a configurable number of seconds. Obviously it can over and under fetch data on the set interval but it will provide a nice visualization given a certain amount of delay and inaccuracy, assuming that is acceptable. We will remedy most of these issues in the coming workshops when we use AWS WebSockets with AWS Lambda for asynchronous invocations.

Don't worry if you see a couple of straight lines for temperature and humidity on your website. The visualization is simply extending the last test IoT data point you manually published from the MQTT test client. You will know the data point is stale as the timestamp is duplicated across the X-axis of the chart.

✅ Step 6 - Populating the visualization using an automated IoT data producer

For this last step we have three ways to populate the visualization from IoT Core to our webhost.

For option A you can simply program your device to publish data to IoT core as I instruct in my course. For option B you would have to spend some time manually altering then publishing JSON payloads in the MQTT test client to generate the line chart in the visualization.

For this tutorial I will explain 'option C.' For this option you need the AWS CLI installed. It’s easy to install with the directions listed here:

https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

This bash IoT data producer script was provided by AWS and can be originally found at https://aws.amazon.com/blogs/iot/integrating-iot-data-with-your-data-lake-with-new-aws-iot-analytics-features/. I have already altered the test script to send just temperature and humidity data. Simply insert your AWS region and MQTT topic name (outTopic) into the test script where indicated. The bash script uses your AWS CLI to deliver the payload to IoT Core (using your SigV4 credentials from the AWS CLI). You can also change the number of payloads published (iterations) and wait time between each publish (interval) to produce as much fake IoT data as you like.

#!/bin/bash

mqtttopic='<Insert-Your-IoT-Topic-Here>'
iterations=10
wait=5
region='<Insert-Your-AWS-Test-Region-Here>'
profile='default'

for (( i = 1; i <= $iterations; i++)) {

  #CURRENT_TS=`date +%s`
  #DEVICE="P0"$((1 + $RANDOM % 5))
  #FLOW=$(( 60 + $RANDOM % 40 ))
  #TEMP=$(( 15 + $RANDOM % 20 ))
  #HUMIDITY=$(( 50 + $RANDOM % 40 ))
  #VIBRATION=$(( 100 + $RANDOM % 40 ))
  temperature=$(( 15 + $RANDOM % 20 ))
  humidity=$(( 50 + $RANDOM % 40 ))

  # 3% chance of throwing an anomalous temperature reading
  if [ $(($RANDOM % 100)) -gt 97 ]
  then
    echo "Temperature out of range"
    TEMP=$(($TEMP*6))
  fi

  echo "Publishing message $i/$ITERATIONS to IoT topic $mqtttopic:"
  #echo "current_ts: $CURRENT_TS"
  #echo "deviceid: $DEVICE"
  #echo "flow: $FLOW"
  echo "temperature: $temperature"
  echo "humidity: $humidity"
  #echo "vibration: $VIBRATION"

 #use below for AWS CLI V1
 #aws iot-data publish --topic "$mqtttopic" --payload "{\"temperature\":$temperature,\"humidity\":$humidity}" --profile "$profile" --region "$region"

 #use below for AWS CLI V2
 aws iot-data publish --topic "$mqtttopic" --cli-binary-format raw-in-base64-out --payload "{\"temperature\":$temperature,\"humidity\":$humidity}" --profile "$profile" --region "$region"

  sleep $wait
}

You have to change fields at the top of the page in the bash script to customize it for your MQTT topic name (outTopic) and AWS region ('us-east-1 or other) in which you developed your AWS services for this tutorial. The other two fields, 'iterations' and 'wait time', are optional to edit.

mqtttopic=''
iterations (number of payloads to send)
wait time (number of seconds between transmissions)
region=''

Now save the above code, giving it a name like "iot_tester.sh". You can run the script by simply installing the bash script locally on your computer and then from the command prompt typing the name of the bash script. Bash scripts should work on any operating system. Activating the test script in MS Windows looks like this:

You can now return to your websites index page and see you visualization getting populated by new data points on the delay of your setInterval() function in 'main.js.'

A few troubleshooting tips for most common issues.

Did you keep your S3 bucket and other AWS services all in the same region?
Does your web browsers cache refresh automatically. On my computer Chrome doesn't inherently refresh upon each new data point, thus I get stale data from S3 resulting in a flat chart. My other five browsers refresh by default for new data. Try the index page on other browsers if you are not getting data point updates for your visualization in your current browser.

😀 🏁

Congratulations! You finished the first tutorial in the series and created the World's Simplest Synchronous Serverless IoT Dashboard. I bet all your friends will be impressed. Make sure to stay tuned for parts two, three, and four of this hands-on tutorial series as we get more advanced with Serverless IoT on AWS.

AWS Serverless design for IoT

Stephen Borsay — Fri, 24 Jul 2020 20:16:39 +0000

This IoT walk-through lab will show you how to send IoT data from your ESP8266 or ESP32 device, through AWS API Gateway, to Lambda, to a data lake in S3, and finally design a static web page for IoT data visualization.

You may be asking, "why would you want to deploy a HTTP API when AWS has a well functioning MQTT broker on AWS IoT Core?" Well, there are a few good reasons that we may want to send our IoT data through AWS API Gateway directly rather than through AWS IoT Core. As an example, I had a student who was using a SIM7000A cellular modem for his ESP32. The hardware abstraction layer on his device was poorly integrated so MQTT(s) wasn't enabled, but HTTP worked well on his device. For this reason a AWS serverless design flow, utilizing the HTTP protocol instead of MQTT, can make sense. Some other possible reasons for using HTTP rather than MQTT are:

A) Your embedded device may not be capable of MQTT(s).

B) You may want to utilize REST instead of MQTT(s), and don't mind losing the key advantage of MQTT: lightweight duplex communication.

C) You may simply want to take advantage of the built-in features of API Gateway such as caching, throttling, velocity templates, payload modeling, and data transformations.

After having said all this, 90% of my course curriculum on Udemy still focuses on AWS IoT Core. However, it is important to determine how to handle these exceptions. In an effort to explore these interesting IoT scenarios I have designed this tutorial and walk-through IoT lab to better help you understand this serverless IoT implementation on AWS. It is important to note that the ESP32 has better built in security than the ESP8266, so the Arduino sketches at the end of the tutorial will reflect these differences.

It is also worth noting that charges for the AWS services used in this tutorial are free, or minuscule, as a serverless design without much compute utilized. AWS S3, Lambda, and API Gateway are all extremely inexpensive for prototyping and testing for non-commercial loads. It's unlikely the following lab will cost you more than a few cents even if you are no longer on the "AWS free tier."

Prerequisites for the tutorial:

A) An AWS free tier or normal AWS account
B) Ability to navigate between AWS services
C) An ESP8266 or ESP32 development board
D) The free Arduino IDE with the device libraries and board manager for your ESP 8266 or ESP32 device

How it works - Serverless IoT

Deploy the Serverless IoT infrastructure

You will create a S3 bucket as the final repository of your IoT Data.

You will create a Lambda function to send your IoT data from API Gateway to S3.
You will configure API Gateway to handle incoming data from our Arduino sketch.
You will create an API Key to secure your deployed URL created in API Gateway.
You will copy the provided Arduino sketch for your ESP8266 or ESP32 and provide your own API Gateway URL.
You will change the permissions on your IoT data bucket and web page bucket from private to public.
You will copy the provided 'index.html' file to visualize your IoT data on a static web host held in a second S3 bucket.

Create a S3 bucket to hold your IoT Data

Create a new S3 bucket in the region of your choice. Choose a globally unique name for your bucket and make sure to keep the region consistent between AWS services.

✅ Step-by-step Instructions for S3

1. Navigate to the AWS S3 console
2. Create a new S3 Bucket in the same region you decide to use consistently throughout this lab. Name your bucket something globally unique (this AWS requirement is so every bucket has its own static URL)
3. You don't need to set a ACL, Bucket policy or CORS at this time, so just select "Create".

4. Finally create and save a folder/partition within your newly created S3 bucket. Name the folder whatever you like.

We are now ready to move on to creating a lambda function to enhance our IoT data and dispatch it to our newly created S3 bucket.

Create your Lambda function in Node.js

A Lambda function programmed in Node.js will be used to format, enrich, and dispatch our incoming JSON payload, sent through API Gateway, to our S3 bucket to hold our IoT sensor data readings

✅ Step-by-step Instructions for Lambda

1. Navigate to the Lambda console and create a new Lambda function ("Author from scratch") in the AWS Region of your S3 bucket
2. Choose the latest runtime of Node.js
3. Chose a new basic execution role
4. Press button to create your lambda function

5. Paste the Node.js code listed below into your lambda function console. Make sure to add your own bucket name and folder name that you created in the previous section where indicated in the lambda code. Uncomment the (event) line of code but keep the (event.queryStringParameters) line of the code commented out for now. We will want to see the entire test payload "event" (object) at this point in the lab. Later, when we utilize our device, we will limit the incoming IoT payload to just the query string parameters.

After pasting in the code listed below, save your lambda function.


        var AWS = require('aws-sdk');
        var s3 = new AWS.S3();

        exports.handler = (event, context, callback) => {  
            var bucketName = "<Your-Bucket-Name>/<Your-folder-Name>";
            var keyName = JSON.stringify(Date.now());
            var content = JSON.stringify(event);                        //uncomment this statement for testing in lambda
          //var content = JSON.stringify(event.queryStringParameters);  //uncommnet this statement after integration with API Gateway
                                                                        //keep only one of the above uncommented!


            var params = { Bucket: bucketName, Key: keyName, Body: content};

            s3.putObject(params, function (err, data) {
                if (err)
                    console.log(err);
                else
                    console.log("Successfully saved object to " + bucketName + "/" + keyName 
                                                                + "and data=" + JSON.stringify(content));
            });
        };

Link to the lambda code: https://github.com/sborsay/Serverless-IoT-on-AWS/blob/master/API_Gateway_Direct/My-Arduino-lambda-Proxy.js

This lambda function writes incoming JSON data into our newly created S3 bucket and the folder/data partition within our s3 bucket. Notice that this function 'enhances' our IoT data payload by adding 'Date.now(),' this is a function that returns a epoch/UNIX timestamp. This is useful as an alternative to the 'UUID' package as we can sequentially label our data objects/payloads with no fear of collision (i.e. duplicate names). In addition we don't have to "roll up" a NPM package as this time-stamping function is native to the language.

6. Currently our lambda function does not have permission to access our newly created S3 bucket. Next, let's give our lambda function the necessary permission, added to then lambda role to give it the ability to write data from our lambda function to our S3 bucket. In Lambda click on the "Permissions" tab (It's between the "Configuration" and "Monitoring" tabs) under the function name.

7. Open the Execution Role we initially created in S3 by clicking on the "Role name."

8. Now we will open a new browser window in the IAM console, click the blue "Attach Policies" button so that we can add our new S3 policy to our lambda execution Role. Type in "S3" in the search bar and select the "AmazonS3FullAccess" managed policy. We are not using the standard AWS "least privilege" model, but don't worry too much about that, we are going to add better security later. If you know what you are doing feel free to limit the role to a stand-alone unmanaged "S3Put" role as a best practice. After making your managed policy selection click the "Attach Policy" blue button.

9. After attaching the managed policy you can now close the IAM window, return to lambda, and click the "Configuration" tab in lambda. That should return you to the coding window. It is now time to test our lambda function to ensure it has the ability to send data to our S3 bucket.

10. Make sure you have entered your S3 bucket name and S3 folder name correctly within your lambda node code, and have already saved the file. Note: we aren't using environmental variables for macros. Next click the "Configure test events" drop down in the upper right of your lambda configuration window.

11. Within the test console, name your test whatever you like, here I call my test payload event "t1", you can leave the JSON data as is, or alter it to better help you remember what you are sending to your S3 bucket as a test. Make sure to keep your test payload in proper JSON format or it won't work. Next hit "Create" to save your 't1' test event as a new test template.

12. After creating your test template in JSON format you should be back in lambda. We are now ready to test our Lambda functions ability to send JSON data to S3. Click the test button on the upper right of the screen to send off your test data to your S3 bucket and folder.

If everything was done correctly you should have received a null response in an 'Execution result: succeeded ' when you scroll up to the log. It is a 'null' response because we haven't written any response code.

13. The last step of verifying our lambda function is correct is to ensure that our test data object was indeed written to our S3 data bucket. To check this go back to your S3 bucket and folder and check that the data object holding the JSON test payload from lambda is indeed in your S3 bucket (you may need to refresh your S3 folder to see your new data object). Click on your test data object which will be be listed by the Date.now() function as an epoch timestamp, and download it.

You will likely have to download your data object to view it rather than simply clicking the URL. If you try to click the URL without making your bucket and partition public you will get an "Access denied" message. We will be changing this later by making our buckets public.

14. After you download the data object, open the JSON payload in the editor of your choice. If you are down with the cool kids you will likely be using VS Code which I find to be overkill in many cases, since I am both uncool and lame i'm using Notepad++ here to open and inspect the test payload.

Awesome! I hope you see your JSON test data object dispatched from your lambda function then sent through to S3. If not, then you need to review the previous steps as nothing going forward will work. Assuming you were successful thus far, let's move on to configuring AWS API Gateway to work with our new lambda function.

Create a Rest API to connect your ESP device to Lambda

API Gateway will be used to configure a publicly facing URL that we can access from both our computer and device to send IoT data to our lambda function.

✅ Step-by-step Instructions for API Gateway

1. Navigate to the API Gateway Console in the same region you have been using for the first two sections of this lab.

2. Select "Rest API" (public) as your API Choice and check "Build."

3. Leave all the defaults and name your API, enter an optional description, then click"Create API."

4. On the next screen use the drop down "Actions" menu to create a new "Method". Choose the "Get" method and click the check mark next to it.

5. Choose "Proxy Integration." This will inject our HTTP headers with our 'QuesryStringParametrs' into the 'event' object which we will parse out later.

6. Select the lambda function you created in the previous section.

Click the "Save button"

7. After saving your work, go back to the same "Actions" button drop down menu you used to select the GET method, and click it. Now choose to "Enable CORS."

8. Remove all the headers from the "Access-Control-Allow-Headers" field (since we are using an embedded device our HTTP headings are not standard).

9. Click the "Enable CORS...headers" button and then "yes...replace current values."

10. Next go back to the "Actions" drop down menu and choose to "Deploy API." Choose a "[New Stage]" and name your stage something short. Then click "Deploy."

11. Now that you connected your API to your lambda function and deployed your API, it is now time to test it. Click the "Invoke URL" address at the top of the page.

12. Clicking "Invoke URL" should open up a new browser window stating "{"message": "Internal server error"}".

Don't worry, this is the correct response, as we haven't configured a custom response. Now let's test our work thus far. Enter a query string in our browser window so that we can check that our data is actually getting sent to our S3 bucket. Enter a test query string such as listed below in your browser window.

YOUR-API-ID.YOUR-REGION.amazonaws.com/DEPLOYMENT-NAME?temperature=55&humidity=66

This is just your unsecured deployment URL concatenated with an arbitrary test query string.

13. Now return to your S3 bucket and the folder within your S3 bucket. Refresh your screen and you should have two new data objects with recent epoch timestamps as names. One object was created by simply opening the unsecured deployed URL, and the latest data object will have the temperature and humidity variables added in the queryStringParameters section of the payload. Download the most recent data object and open it in your editor of choice. Verify that the query string parameters contain your variables entered from your browser's URL pane.

Congratulations! We now have a working Lambda connected to a working, publicity facing URL created in API Gateway. Now it is time to add some security.

Create an API Key to secure our deployed URL

You may notice that we have virtually no security other than keeping your initial deployed URL private. While it is also a good idea to limit total requests and bursts requests on our API, it is a better idea to create and enforce an "API Key" that the client must possess in order to initiate a successful request against our lambda function. Fortunately we can do both by creating an "API Key" which we can then provide to the client to ensure that they have a valid access mechanism to use our deployed URL for their requests. API keys are especially appropriate for IoT, most third party IoT visualization sites such as Losant, Ubidots, and ThingsSpeak will issue their registered users an API key for external requests. Now is a great time to make an API Key and associated usage plan so that we can use the key in our Arduino sketch to confirm that our device has the right credentials to invoke our API. At the same time we will configure a "usage plan" to limit request overload and other potential abuses.

✅ Step-by-step Instructions for Creating an API Key

1. Go back to your API Resources Configuration screen and in the"Method Request" section change "API Key Required" from false to true.

2. Now we have to repeat the deployment process. This time create a new stage with another name like "Dep_with_api_key" or whatever name you like. Our old stage will remain open to the public and our new stage will require an API key which we will create next. You can also delete your old deployment if you no longer wish to have an unsecured URL.

Re-deploy your new stage using the "Actions" drop down button. Now test the new URL associated with this API Key required stage. The browser should now return a {"message": "Forbidden"} alert. This is the built-in notice that you are not allowed to use this new URL as is.

3. Now let's create our API Key, to do this navigate back to API Gateway. On the pane to the left select "Usage Plans." Once in the "Usage Plans" tab select "Create."

4. Next we will limit requests per second, bursts, and total monthly requests. You can set your request configuration to something to meet your own needs. Limiting total monthly requests to under 1000 constrains your account to nominal, if any expense. This is helpful if your client, who possesses a valid API Key, exceeds their request limits. After you select your rates for throttling and quota select "Next."

5. Next we will attach the new usage plan to our currently deployed URL. Choose the API we created in the previous step. Now choose the new deployment you just created and deployed with a API key requirement. Click the check mark then click "Next."

6. Next click "Create an API Key and add to Usage Plan" (that's the box on the right, do not click the box on the left). This will bring up a box to name your specific API Key. Name it something, then click "Save", then click "Done".

7. Now we have to retrieve and copy the alphanumeric cipher for the API Key we just created. To see your new key click on the "API Keys" tab on the screen.

8. Click the "API key" in blue, and now click "Show."

9. Now copy the alphanumeric code for your API Key and keep it handy, you will need it next.

As a side note we don't need to redeploy our API at this point because we are just changing things on the server side on AWS with a new usage plan and X-API-Key. If you watched other API Key most instructionals assume you have to redeploy after creating a usage plan and API Key but this is not needed as long as you deployed when you set the API Key requirement to "true" in the "Method Request" window as we did previously.

Now we are ready to test our new deployment which requires an API key. Unfortunately we can't simply test our API Key in a browser as the headers don't format correctly in the browsers address bar. At this point you can move on and see if it works in the next section in the Arduino Sketch, or we can test the API-Key with a free API testing tool like cURL or Postman. Here I will test our new deployment with our API Key in Postman.

10. To test our API in Postman simply select the GET method. Now paste your API Key secured deployment URL into Postman's address bar. You can try this process first without the API Key added and you should receive the same "Forbidden" message. Now add the "X-API-KEY" (letter case doesn't matter), in the headers box (as circled in the picture below) , and resend your GET request. You should now get the "Internal server error" as before, and the data object should appear in your S3 bucket. Make sure you insert your key in the Header section and not the Body section in Postman. Also confirm that this test is successful by checking your S3 folder for the new data object before moving on to the next step.

Congratulations, now your API Gateway URL can connect with your lambda forwarding IoT data to S3 as long as you provide your API Key along with your GET request for added security. In the next section we will add the API Gateway deployment URL (endpoint) along with our working API Key to our Arduino sketch so that we can send HTTP requests directly to API Gateway from our ESP device.

Program our device sketch in the Arduino IDE for our ESP device

I have provided sketches for the ESP8266 and the ESP32, however in this section I will focus on the ESP8266. It's worth noting that the ESP32 has built in HTTPS along with other WiFi security capabilities while the ESP8266 does not. Given this, we will focus on the more complicated sketch employing SHA-1 security on the ESP8266 device, which we can use as a minimum, to meet API Gateway's security requirements. However we will add pretty good security (PGS) by adding our AWS API Key to the Arduino sketch running on the device.

For a more professional deployment I would rotate an API Key on the device by using a MQTT subscription topic from a lambda MQTT publisher with an AWS.IoTData object provided by the AWS-SDK. However this method would be part of a more advanced lab.

✅ Step-by-step Instructions for the device sketch

1. At this point we only want to extract the query string parameters from the overly explicit information coming from API Gateway. AWS inherently adds a lot of potentially useful information to our incoming IoT data payload which we don't need for the purposes of this tutorial. In order to remove this spurious data simply go to your lambda function and comment out:

//var content = JSON.stringify(event);

and uncomment

var content = JSON.stringify(event.queryStringParameters);

Make sure to re-save your lambda functions after coding the simple change above.

2. Our Arduino ESP8266 sketch is based on the script found here: https://github.com/esp8266/Arduino/blob/92373a98370618dea09718010b30d311a97f3f25/libraries/ESP8266WiFi/examples/HTTPSRequest/HTTPSRequest.ino

I have altered the sketch to work with AWS and API Gateway. There are a number of fields to fill out with your own information. If you are using the ESP8266 rather than the ESP32 there is one extra field we have yet to explore, and that's our SHA-1 fingerprint. So let's acquire that alphanumeric cipher now. For this you should be using Chrome as your browser.

3. First, go back to the URL of your recent API Gateway deployment after you set the "API Key Required": true and deployed it. The web page should be the website displaying the "Forbidden" alert (as this page requires the API Key we created in the previous section). We can retrieve the SHA-1 thumbprint from here.

To acquire the fingerprint (Chrome calls it "Thumbprint")for this web page, go to the breadcrumbs icon in the upper right corner of your Chrome browser. Then go to:

More tools-->Developer tools-->Security(tab)-->view certificate(button) -->Details(tab)-->Thumbprint

4. you will see the SHA-1 Thumbprint as something like this:

53f2ZX9XX6zoqGAupqyXX5yNoOdgzm8qew8hC41

put a space between every other character so it now looks like:

53 f2 ZX 9X X6 zo qG Au pq yX X5y No Od gz m8 qe w8 hC 41

Now the thumbprint is ready to be inserted in your sketch, so copy your own SHA-1 thumbprint.

5. Now fill out the following fields in the provided sketch.

You will need to fill in the following fields respectively:

A) WiFi Network Name (make sure your networks at 2.4GHz not 5GHz)

B) WiFi Password

C) Host name (First part of API Gateway URL, do not include "https://")

D) URL (API Gateway deployment name)

E) API Key

F) Formatted fingerprint (found in the Chrome thumbprint SHA-1)

(above sketch is just an example, for a different region and thumbprint)


/*
    HTTP over TLS (HTTPS) example sketch
    This example demonstrates how to use
    WiFiClientSecure class to access HTTPS API.
    We fetch and display the status of
    esp8266/Arduino project continuous integration
    build.
    Limitations:
      only RSA certificates
      no support of Perfect Forward Secrecy (PFS)
      TLSv1.2 is supported since version 2.4.0-rc1
    Created by Ivan Grokhotkov, 2015.
    This example is in public domain.
    * This example modified by Stephen Borsay for AWS Serverless course on Udemy
    * to Connect your device directly to AWS API Gateway
    * modified for sending fake data buffer, connect any sensor as desired
    * 
*/

#include <ESP8266WiFi.h>
#include <WiFiClientSecure.h>

#ifndef STASSID
#define STASSID "<YOUR-WIFI-NETWORK>"
#define STAPSK  "<YOUR-NETWORK-PASSWORD>"
#endif

const char* ssid = STASSID;
const char* password = STAPSK;
const char* host = "<YOUR-API-GATEWAY-ENDPOINT>.execute-api.<YOUR-REGION>.amazonaws.com"; //do not include "https://"
String url = "<YOUR-API-GATEWAY-DEPLOYMENT-NAME>";

const char* API_KEY = "<YOUR-API-GATEWAY_API-KEY-HERE>";

const int httpsPort = 443;
unsigned long uptime;

// Use web browser to view and copy  SHA1 fingerprint of the certificate 
//to acquire the thumbprint for this webpage, go to the breadcrumbs in the upper right corner of your browser.
//Then go to Tools-->developer tools-->security-->view certificate-->details(tab)-->thumbprint
//const char fingerprint[] PROGMEM = "98 f8 5e fc 87 65 43 5f 0f c1 1e fe e9 81 c9 9c c2 43 27 4c"; //example thumbprint with proper formatting

const char fingerprint[] PROGMEM = "<YOUR-SHA-THUMBPRINT>";

 WiFiClientSecure client;

void setup() {
  Serial.begin(115200);
  Serial.println();
  Serial.print("connecting to ");
  Serial.println(ssid);
  WiFi.mode(WIFI_STA);
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());

  // Use WiFiClientSecure class to create TLS connection

  Serial.print("connecting to ");
  Serial.println(host);

  Serial.printf("Using fingerprint '%s'\n", fingerprint);
  client.setFingerprint(fingerprint);

  if (!client.connect(host, httpsPort)) {
    Serial.println("connection failed");
    return;
  }
  //String url = "/dep1";

  Serial.print("requesting URL: ");
  Serial.println(url);

}


void loop() {
  int t =  random(30,110); //fake number range, adjust as you like
  int h =  random(50,100);


  Serial.print("uptime: ");
  uptime = millis()/1000;
  Serial.println(uptime); //prints time since program started


  client.print(String("GET ") + url + "/?uptime=" + (String) uptime
       + "&temperature=" + (String) t + "&humidity=" + (String) h + " HTTP/1.1\r\n" +
               "Host: " + host + "\r\n" +
               "x-api-key: " + API_KEY + "\r\n" +
               "User-Agent: 14 ESP8266\r\n" +
               "Connection: close\r\n\r\n");

  Serial.println("request sent");
  while (client.connected()) {
    String line = client.readStringUntil('\n');
    if (line == "\r") {
      Serial.println("headers received");
      break;
    }
  }
  String line = client.readStringUntil('\n');
  if (line.startsWith("{\"state\":\"success\"")) {
    Serial.println("esp8266/Arduino CI successfull!");
  } else {
    Serial.println("esp8266/Arduino CI has failed");
  }
  Serial.println("reply was:");
  Serial.println("==========");
  Serial.println(line);
  Serial.println("==========");
  Serial.println("closing connection");
  delay(1000);

//unlike MQTT, HTTP/HTTPS has to be reconstructed every time a request is processed
// so reconnect after GET request is completed and key/value URL payload is dispatched
  if (!client.connect(host, httpsPort)) {
    Serial.println("connection failed");
    return;
  }
   delay(1000);
}

Here is a link to the whole sketch for the ESP8266 on Arduino, you can now upload the sketch to your device after filling out the required fields as listed above.

https://github.com/sborsay/Serverless-IoT-on-AWS/blob/master/API_Gateway_Direct/ESP8266-to-API-Key-Secured.ino

The sketch just generates random values for temperature and humidity as well as uptime. You can easily integrate a DHT11/22, BME280, or numerous other sensors to report actual sensor readings. If you have done everything right you should receive readings on your serial monitor which looks similar to the readings below. Again, ignore the "internal server error" message in the terminal due to not developing a request response.

If you are using the ESP32 then the sketch is significantly easier as WiFi is secure without having to use SHA-1. There are a few very good HTTP sketches available on the internet, I decided to modify Rui Santos's open source ESP32 sketch and add in our AWS specific code, and X-API-Key header. Below is the github link to the simplified ESP32 sketch with API Key secured.

https://github.com/sborsay/Serverless-IoT-on-AWS/blob/master/API_Gateway_Direct/ESP32-to-API-Key-Secured.ino

Next let's go back to our S3 bucket and ensure that our IoT data payloads landed successfully in our folder.

Now we see our S3 bucket contains our data objects with the "humidity", "temperature", and "uptime" variables within each data object.

Congratulations! You now have completed the base lab. I have added a stretch lab below if you wish to continue with a visualization of your IoT data.

Visualizing our IoT data with Highcharts on a static web host in S3

✅ Step-by-step Instructions for Visualization of IoT data

Now that your data is in your bucket there are all types of manipulation you can do with the IoT data lake besides just visualizations. You can use AI, machine learning, BI, as well as tie in many other AWS services like SageMaker, Glue, Athena, Redshift, and QuickSight to name a few. You can utilize many of these AWS services with your IoT data while it is still in your S3 bucket. For this lab we will be creating a second public bucket in S3 to host our visualization website. To do this we will make our new S3 bucket completely open and public as we aren't using AWS CloudFront, Route53, or a VPN. We will then extract our IoT data from our public web host in S3 directly from our soon to be public IoT data bucket. It's important to note that it is NOT appropriate for professional deployments to use public buckets. A professional implementation would involve using a Lambda function as a private layer to extract, ingest, and consume data from a private S3 data bucket. See my Udemy course for details on this more professional method.

1. We now need to create a new S3 bucket to host our static web site for IoT data visualization. Go back to S3 and create a new bucket and give it a globally unique name. Remember to keep all your buckets and AWS services in the same region.

2. After creating your bucket (I called mine "webhost76"), set your bucket up as a static web host. To do so go to: properties-->static website hosting and "Use this bucket to host a website." Now name the "index document" as index.html then "save."

3. Now click on the next tab labeled "permissions." Click and deselect "Block all public access," then save and confirm. AWS wants to ensure you know you are allowing your buckets data to be seen publicly, as they have experienced security breaches in the past with hackers grabbing info in other users' public buckets. In our case we aren't holding sensitive data so it's permissible to make our buckets public in an effort to make this tutorial easier.

4. Next go to the "Access Control List" and click on "Public access" Everyone. Under access to the objects and select "List objects." This gives everyone the ability to read our info. Then click "Save." Notice we aren't giving write permissions so we can prevent cross origin injection attacks.

5. Go to the next box which is "Bucket Policy." We will insert a JSON formatted document granting public access to our bucket (see below). I have added some simple security- IP range limiting. By adding this additional IP field we make using our website only available to IP's in our predesignated range. To find your IP simply google "my IP." Insert your bucket name and IP in the designated areas of the Bucket Policy that I have listed below, and then click "Save." As a note IP's can be spoofed but this is a simple way to add some security with minimal extra complication. I have also included a non IP protected bucket policy as well if you want to see your webpage from any remote location.

Later on, when you are done with this section, you can test that your IP limiting was successful by trying to bring up your visualization website on your smartphone. Smartphones use IPv6 instead of IPv4 by default, and thus your website should not be accessible with your smartphone if you used the bucket policy that limits access by IP range.

IP range limited Bucket Policy:



{
    "Version": "2012-10-17",
    "Id": "S3PolicyId1",
    "Statement": [
        {
            "Sid": "IPAllow",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::<YOUR-BUCKER-NAME-HERE>/*",
            "Condition": {
                "IpAddress": {
                    "aws:SourceIp": "<YOUR-IP-HERE>/24"
                }
            }
        }
    ]
}

https://github.com/sborsay/Serverless-IoT-on-AWS/blob/master/PublicBucket/LimitByIPBucketPolicy

Open Bucket Policy :

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicRead",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<YOUR-BUCKET-NAME>/*"
        }
    ]
}

https://github.com/sborsay/Serverless-IoT-on-AWS/blob/master/PublicBucket/PublicBucketReadPolicy

6. The last thing we need to do to configure our public bucket is to add a CORS policy in the next box. This is a XML document setting cross origin resource sharing which will allow us to ingest the IoT data held in our S3 IoT data bucket. You don't need to customize the XML document below. Simple copy and past it into your CORS window and save.

CORS XML:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
    <AllowedHeader>Authorization</AllowedHeader>
</CORSRule>
</CORSConfiguration>

https://github.com/sborsay/Serverless-IoT-on-AWS/blob/master/PublicBucket/PublicReadCORS

7. Now you have to repeat the exact same process with the S3 IoT data bucket that you created previously in the first section of this lab. This is the bucket that is filled with our test JSON data objects. We need to make that bucket public as well so that our website can access the IoT data within the buckets folder. The one difference between configuring this other bucket is that we are not setting our IoT data bucket for "static website hosting," as we are still just using our original bucket as a data repository for our IoT data lake holding our fake sensor readings.

Now it is time to edit our index.html web page to prepare it for upload to our new s3 bucket. The two fields you will need to customize in my index.html to work with your IoT data bucket are:

A) Your base bucket name

B) The folder name that holds your sensor reading in the base bucket

7. We can get both our folder and bucket URL with the same process. We can simply copy our "Object URL" and extract both the needed info within the URL. To do this go to your IoT data bucket and then go to:

overview-->click on your data folder--> click on a data object

Now click the object URL and at the bottom of the page you can now copy the Object URL.

In my IoT data bucket my Object URL is:

https://globallyuniquebucketname76.s3.amazonaws.com/IoTDataFolder/1582578233424

From this Object URL I can extract the base bucket name as : https://globallyuniquebucketname76.s3.amazonaws.com/

The base bucket will have the format:

https://bucketname.s3.amazonaws.com

And my folder name is: IoTDataFolder

*Note: if your bucket is not in your home region you may also have the region listed in your base bucket address which you will need as well.

8. Now insert both URL's in the index.html provided below. Simply replace my URL and folder name with yours. There are two places in the index.html below that you need your base bucket URL, and one location that will need your folder name. As a note the program works by going to the base bucket level URL, and once the program knows where to grab your data objects it can effectively parse them.


<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <title>Document</title>
</head>

<body>


    <script src="https://code.highcharts.com/highcharts.js"></script>

    <div class="container">
        <h1>Dashboard</h1>


        <div class="panel panel-info">
            <div class="panel-heading">
                <h3 class="panel-title"><strong>Line Chart</strong></h3>
            </div>
            <div class="panel-body">
                <div id="container1"></div>
            </div>
        </div>


        <div class="panel panel-info">
            <div class="panel-heading">
                <h3 class="panel-title"><strong>Bar Chart</strong></h3>
            </div>
            <div class="panel-body">
                <div id="container"></div>
            </div>
        </div>

    </div>

    <script>

        var x = new XMLHttpRequest();
        x.open("GET", "https://<YOU-BUCKET-NAME>.s3.amazonaws.com/", true);
        // x.setRequestHeader("Content-Type", "application/xml");
        x.onreadystatechange = function () {
            if (x.readyState == 4 && x.status == 200) {
                let promiseArr = [];
                let data = [];
                var doc = x.responseXML;
                let keys = doc.getElementsByTagName("Key");

                let index = 0;
                createDataSet(index);

                function createDataSet(index) {
                    if (index >= keys.length) {
                        generateGraph();
                        return false;
                    }
                    let element = keys[index];
                    element = element.textContent;


                    let splitName = element.split('/');
                    if (splitName[0] === '<YOUR-FOLDER-NAME>' && splitName[1] !== '') {
                        promiseArr.push(new Promise((resolve, reject) => {
                            var innerReq = new XMLHttpRequest();
                            innerReq.open("GET", "https://<YOU-BUCKET-NAME>.s3.amazonaws.com/" + splitName[0] + "/" + splitName[1], true);
                            // innerReq.setRequestHeader("Content-Type", "application/xml");
                            innerReq.onreadystatechange = function () {
                                if (innerReq.readyState == 4 && innerReq.status == 200) {
                                    let parseData = JSON.parse(innerReq.responseText);
                                    if (parseData.humidity) {
                                        data.push(Object.assign({}, parseData, { timestamp: splitName[1] }));
                                    }
                                    resolve('Done')
                                    index++;
                                    createDataSet(index);
                                } else {
                                    // reject(innerReq)
                                }
                            }
                            innerReq.send(null);
                        }));
                    } else {
                        index++;
                        createDataSet(index);
                    }
                }




                function generateGraph() {
                    Promise.all(promiseArr.map(p => p.catch(e => e)))
                        .then(res => {

                            abcData = data;
                            let barGraphXaxisName = ['Humidity', 'Temperature', 'Uptime'];
                            let humiditySum = 0, temperatureSum = 0, uptimeSum = 0;
                            let lineXaxisData = [], humArr = [], tempArr = [], upArr = [];
                            for (let i = 0; i < abcData.length; i++) {
                                humiditySum += Number(abcData[i].humidity);
                                temperatureSum += Number(abcData[i].temperature);
                                uptimeSum += Number(abcData[i].uptime);

                                humArr.push(Number(abcData[i].humidity));
                                tempArr.push(Number(abcData[i].temperature));
                                upArr.push(Number(abcData[i].uptime));
                                // lineXaxisData.push(new Date(Number(abcData[i].timestamp)).toLocaleString());
                            }

                            var chart = Highcharts.chart('container', {

                                chart: {
                                    type: 'column'
                                },

                                title: {
                                    text: 'Bar Chart'
                                },
                                xAxis: {
                                    categories: barGraphXaxisName
                                },

                                yAxis: {
                                    title: {
                                        text: 'Value'
                                    }
                                },

                                series: [{
                                    data: [humiditySum, temperatureSum, uptimeSum]
                                }],

                                responsive: {
                                    rules: [{
                                        condition: {
                                            maxWidth: 500
                                        },
                                        chartOptions: {
                                            chart: {
                                                className: 'small-chart'
                                            }
                                        }
                                    }]
                                }
                            });

                            Highcharts.chart('container1', {

                                title: {
                                    text: 'Line chart'
                                },

                                yAxis: {
                                    title: {
                                        text: 'Value'
                                    }
                                },

                                xAxis: {
                                    categories: upArr
                                },

                                legend: {
                                    layout: 'vertical',
                                    align: 'right',
                                    verticalAlign: 'middle'
                                },

                                plotOptions: {
                                    series: {
                                        label: {
                                            connectorAllowed: false
                                        }
                                    }
                                },
                                series: [{
                                    name: 'Humdity',
                                    data: humArr
                                }, {
                                    name: 'Temperature',
                                    data: tempArr
                                }],

                                responsive: {
                                    rules: [{
                                        condition: {
                                            maxWidth: 500
                                        },
                                        chartOptions: {
                                            legend: {
                                                layout: 'horizontal',
                                                align: 'center',
                                                verticalAlign: 'bottom'
                                            }
                                        }
                                    }]
                                }

                            });
                        }).catch(err => {
                            console.log('err', err)
                        })
                }

            }
        };
        x.send(null);

    </script>
</body>

</html>

Github link to our index.html for visualizing our IoT Data:

https://github.com/sborsay/Serverless-IoT-on-AWS/blob/master/API_Gateway_Direct/Visualizations/Highcharts/index.html

9. Now that you have customized my index.html file for your own URL and folder name you are ready to upload the file to your new bucket. To accomplish this, simply drag and drop your customized index.html to your newly created web host bucket.

I have made four videos on YouTube that cover this entire tutorial.

The first video in the series that can be found here:

https://youtu.be/NALTj_ZepNE

If any part of this lab is unclear then I would encourage you to watch the videos, or better yet, take one of my courses on Udemy covering AWS IoT extensively! I hope you enjoyed learning about AWS IoT as well as getting some hands on experience with different serverless services within the AWS framework for IoT. Feel free to email me with any questions.