DEV Community: Grid Smarter Cities

Automating the Deployment of CloudWatch Canaries with CloudFormation

James Harrington — Wed, 30 Jun 2021 15:04:12 +0000

CloudWatch Canaries can monitor your sites and apis, while CloudFormation automates the deployment of AWS resources. In this post, I will show you how these two technologies can work together to automate the deployment of these canaries to monitor a range of applications.

Why CloudFormation?

At Grid Smarter Cities we use CloudFormation to automate the deployment of a wide range of stacks, including:

CI/CD processes for our Front End and Back End applications
Management of users and permissions
Deployment of tools like CloudWatch Canaries and Lambda Performance Tuning

We do this because it makes the process of automatically deploying resources efficient and less time consuming. CloudFormation treats infrastructure as code, meaning both resources and their dependencies can be created, validated and deployed from a single template file.

If the stack needs to be updated, all the user has to do is make the necessary changes to the template, reupload it, and let CloudFormation verify and make all the necessary changes to your resources. If a stack needs duplicating, upload the template again and adjust the parameters and redeploy. If the stack is no longer needed, everything can be deleted with a single button click.

Resources Required Separate To the Stack

Before the stack is deployed, we will need to have a prepared script for the canary and two S3 Buckets. The first bucket is used by the Canary to store its artifacts saved, while it is running - this could be screenshots, logs or generated reports. The second bucket will be used to store the source code used by the canary - we are using a Node script for this canary. The code for this can be found here.

Uploading the Files

These files need to be in a particular folder structure - this wasn't clear on the AWS CloudFormation Documentation. However, I eventually found that your canary's code file structure should be /nodejs/node_modules/ .

The following two commands will create this folder structure, and prepare the zip folder:

cp canary-function.js ./nodejs/node_modules/
zip -r sourcecode.zip ./nodejs/*

The new sourcecode.zip file just needs to be uploaded directly into the S3 bucket you are using to store the Canary source code.

Writing The Template

In this template, we have added parameters for:

The name of the Service we're monitoring
The name of the Canary
The name of Source Code Bucket
The name of the object containing the code inside the SourceCode bucket
The email address for alert emails to be sent to if the canary fails

All of these values will be inputted by the user and passed as parameters when the user has uploaded the template.

MetaData

Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - Label:
          default: 'Service'
        Parameters:
          - ServiceName
          - CanaryName
          - AlertEmail
      - Label:
          default: 'Buckets'
        Parameters:
          - CanarySourceCodeBucketName
          - CanarySourceCodeKey
          - CanaryArtifactBucketName
    ParameterLabels:
      ServiceName:
        default: Service Name
      CanaryName:
        default: Canary Name
      AlertEmail:
        default: Email
      CanarySourceCodeBucketName:
        default: Source code bucket name
      CanarySourceCodeKey:
        default: Key for the object 
      CanaryArtifactBucketName:
        default: The name of the S3 bucket where the artefacts will be store

Parameters

Parameters:
  ServiceName:
    Description: Enter a lower case, high level service name without environment details. Used to autofill service names. For example, your-service-name
    Type: String
  CanaryName:
    Description: Enter a lower case name for the canary, as they should be known. For example dave not Dave
    Type: String
  AlertEmail:
    Description: Email address to send staging build alerts to, or example you@example.com
    Type: String
  CanarySourceCodeBucketName:
    Description: S3 Bucket Name where the source code lives
    Type: String
  CanarySourceCodeKey:
    Description: Key of the object which contains the source code
    Type: String
  CanaryArtifactBucketName:
    Description: S3 Bucket Name where the canary saves the screenshots and other 
    Type: String

Resources

The resources we need are:

A CloudWatch Canary
A Role for the CloudWatch Canary
A policy for the CloudWatch Canary's Role
A CloudWatch alert to detect when the Canary fails
An SNS topic which will send the email alerts

CloudWatch Canary

The first resource is of type AWS::Synthetics::Canary. The property specifying the canary's name and relevant S3 buckets have been read from the parameters defined earlier in the template. For now, the runtime version, handler and schedule have been hard coded, but these could easily be extracted as parameters to make the template more flexible.

CloudwatchCanary: 
    Type: AWS::Synthetics::Canary
    Properties: 
        ArtifactS3Location: !Sub s3://${CanaryArtifactBucketName}
        Code: 
            Handler: canary-function.handler
            S3Bucket: !Sub ${CanarySourceCodeBucketName}
            S3Key:  !Sub ${CanarySourceCodeKey}
        Name: dave-the-canary
        RuntimeVersion: syn-nodejs-puppeteer-3.1
        Schedule: 
            Expression: rate(5 minutes)
        StartCanaryAfterCreation: true
        ExecutionRoleArn: !GetAtt CloudwatchCanaryRole.Arn

CloudWatch Canary Policy and CloudWatch Canary Role

The second and third resources are of type AWS::IAM::Policy and AWS::IAM::Policy. We chose to give the CloudWatch Canary a policy which had been modified from CloudWatchSyntheticsFullAccess. This gave our canary permission to:

Read data from the source code buckets
Save and retrieve data from the artefact bucket
Trigger alarms when the canary detects an error.

We had to add a few extra actions and modify the resource selectors to match our resource names. There were also some unnecessary actions which - for best practices and security - were removed.

CloudWatch Canary Policy

CloudwatchCanaryPolicy:
  Type: AWS::IAM::Policy
  Properties:
    PolicyName: !Sub ${ServiceName}-dave-the-canary-policy
    PolicyDocument:
    Version: '2012-10-17'
    Statement:
    - Effect: Allow
        Action:
        - synthetics:*
        Resource: "*"
    - Effect: Allow
        Action:
        - s3:CreateBucket
        - s3:PutEncryptionConfiguration
        Resource:
        - arn:aws:s3:::*
    - Effect: Allow
        Action:
        - iam:ListRoles
        - s3:ListAllMyBuckets
        - s3:GetBucketLocation
        - xray:GetTraceSummaries
        - xray:BatchGetTraces
        - apigateway:GET
        Resource: "*"
    - Effect: Allow
        Action:
        - s3:GetObject
        - s3:ListBucket
        - s3:PutObject
        Resource: arn:aws:s3:::*
    - Effect: Allow
        Action:
        - s3:GetObjectVersion
        Resource: arn:aws:s3:::*
    - Effect: Allow
        Action:
        - iam:PassRole
        Resource:
        - !Sub arn:aws:iam::*:role/service-role/${CloudwatchCanaryRole}
        Condition:
        StringEquals:
            iam:PassedToService:
            - lambda.amazonaws.com
            - synthetics.amazonaws.com
    - Effect: Allow
        Action:
        - iam:GetRole
        Resource:
        - !Sub arn:aws:iam::*:role/service-role/*
    - Effect: Allow
        Action:
        - cloudwatch:GetMetricData
        - cloudwatch:GetMetricStatistics
        Resource: "*"
    - Effect: Allow
        Action:
        - cloudwatch:PutMetricAlarm
        - cloudwatch:PutMetricData
        - cloudwatch:DeleteAlarms
        Resource:
        - '*'
    - Effect: Allow
        Action:
        - cloudwatch:DescribeAlarms
        Resource:
        - arn:aws:cloudwatch:*:*:alarm:*
    - Effect: Allow
        Action:
        - lambda:CreateFunction
        - lambda:AddPermission
        - lambda:PublishVersion
        - lambda:UpdateFunctionConfiguration
        - lambda:GetFunctionConfiguration
        Resource:
        - arn:aws:lambda:*:*:function:cwsyn-*
    - Effect: Allow
        Action:
        - lambda:GetLayerVersion
        - lambda:PublishLayerVersion
        Resource:
        - arn:aws:lambda:*:*:layer:cwsyn-*
        - arn:aws:lambda:*:*:layer:Synthetics:*
    - Effect: Allow
        Action:
        - ec2:DescribeVpcs
        - ec2:DescribeSubnets
        - ec2:DescribeSecurityGroups
        Resource:
        - "*"
    - Effect: Allow
        Action:
        - sns:ListTopics
        Resource:
        - "*"
    - Effect: Allow
        Action:
        - sns:CreateTopic
        - sns:Subscribe
        - sns:ListSubscriptionsByTopic
        Resource:
        - arn:*:sns:*:*:Synthetics-*
    Roles:
    - !Ref CloudwatchCanaryRole

CloudWatch Canary Role

CloudwatchCanaryRole:
    Type: AWS::IAM::Role
    Properties:
        RoleName:  !Sub ${ServiceName}-${CanaryName}-the-canary-role
        AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
            - Effect: Allow
            Principal:
                Service:
                - lambda.amazonaws.com
            Action:
                - sts:AssumeRole

On its own, the deployed canary will monitor resources as canary's source code files. Any issues found by the canary will be visible on the relevant dashboard in CloudWatch. Unfortunately, this will not trigger any email alerts or notifications. Adding this functionality to our stack is straightforward, we need to add a CloudWatchAlarm and an SNS Topic.

Adding the Topic and Alarm

By adding the following sections, alert emails will be sent to the specified email address when submitting the template.

The CloudWatchCanaryAlarm is triggered when the average success count falls below 100% for one evaluation period; this period was hard coded to five minutes. If this failure is detected, the CloudWatchCanaryAlarmTopic is triggered.

CloudwatchCanaryAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
        ActionsEnabled: true
        AlarmDescription: !Sub ${ServiceName}-${CanaryName}-the-canary-alert
        ComparisonOperator:  LessThanThreshold
        EvaluationPeriods: 1
        DatapointsToAlarm: 1
        MetricName: SuccessPercent
        Namespace: CloudWatchSynthetics
        Period: 60
        Statistic: Average
        Threshold: 100
        TreatMissingData: notBreaching
        AlarmActions:
        - !Ref CloudwatchCanaryAlarmTopic
        Dimensions:
            - Name: CanaryName
            Value: !Sub ${CanaryName}-the-canary

The CloudWatchCanaryAlarmTopic sends a message to the AlertEmail address, specified by the CloudFormation parameters inputted by the user when deploying the stack.

CloudwatchCanaryAlarmTopic:
    Type: AWS::SNS::Topic
    Properties:
        DisplayName: !Sub ${CanaryName}-cw-canary-alarms
        Subscription:
            - Endpoint: !Ref AlertEmail
            Protocol: email

Deploying The Stack

All that's left to do is deploy Dave the Canary. To do this, upload the template, fill in the form defining the stack parameters as shown below, click next(we'll not configure any of the default stack options), click next again, confirm that you "acknowledge that AWS CloudFormation might create IAM resources with custom names" then click the Create Stack button. Your resources should then be deployed and running.

Closing Notes

This CloudFormation template was written to help developers at Grid Smarter Cities deploy tools which monitor our products. Deploying a canary from the console is straightforward but using CloudFormation allows us to automate the deployment process, allowing us to set up monitoring tools and alerts for all of our products.

Writing this template may have required a bit of research and some trial and error but, in doing so, our developers and QAs will never have to manually check that our products are live. If the products aren't, the CloudWatch canary will automatically inform us quickly and more reliably than if we were manually checking ourselves.

Full copies of both templates (with and without the alerts) and example canary function can be found in the Repo

Integration testing websockets with Python

Rob Anderson — Mon, 05 Aug 2019 09:21:03 +0000

TL;DR - Testing is important but can be difficult. I've written the Python library pywsitest to help with integration testing websockets and included some examples.

At Grid Smarter Cities we believe that testing is a key component of good software.

Our attitude towards testing is reflected primarily in our development process where we practice test driven development, perform code analysis using Prospector, and require 100% test coverage (which we check using Coverage.py). This process is re-iterated in our automated build pipelines, which run linting, the full suite of unit tests against our code, and integration tests against a staging environment.

A lot of our ideas about testing are roughly based on a testing pyramid (see below), where because unit tests are fast to write and fast to run, they make up the majority of tests we use. Integration tests take longer to run, generally incur some cost running against a live system, and are more complicated to write; so we write less of them, but they're still an important part of ensuring we have a good product.

UI tests and manual testing are both more time consuming and more expensive processes, so they make up a smaller portion of our tests. Because they're not something I have to worry too much about as a back-end developer I'm going to skip over them a bit today, though I appreciate the importance of a full testing suite and process.

However many unit tests we write, and how much we can trust our code in isolation, we can't have full confidence in our products unless we also test interactions between modules on a live system. Automated integration testing of REST APIs is a reasonably straight forward process using tools like Dredd alongside OpenAPI templates, but Grid ran into difficulties testing some of our more complicated interactions with a websocket based API.

pywsitest

One of our products that has been developed recently at Grid required different users interacting with and updating a request object in real time. We chose to use a websocket-based API for all of the asynchronous parts of our application.

When each user type connects to the websocket host, they receive a set of messages about their current request status, other active users, and some geolocation-based data. When one user modifies a request object, certain other users receive the status update, and users are restricted as to which actions they can perfom on a request.

Testing asynchronous interations between the users became a challenge, and is what lead us to creating our Python library pywsitest (python websocket integration testing framework).

The pywsitest library allows a user to connect to a websocket host, assert that a series of messages have been received, and that any messages that need to be sent were sent successfully. Messages can either be sent on connection, or be triggered by receiving any specified message. This ability to listen for responses, and send messages out when they're received allows for writing tests that target the interaction between different users.

Testing scenario

I've written a websocket based chat server that allows a user to connect using a username, send messages to all connected users through the server, and disconnect from the server when they're finished.

The user's name will be included in the url as a query parameter. Since I'm running this server locally for testing, the url I'll use to connect will be wss://localhost?name=Rob.

When a user connects to the server, the server will broadcast a message to all connected users to tell them the user has connected:

{ "message": "Rob has connected" }

When a user sends a message to the server, it will be broadcast to all connected users with the format:

{ "message": "Rob: Hello, world!" }

When a user disconnects, the server will broadcast to all connected users:

{ "message": "Rob has disconnected" }

For my first test I want to ensure a user can connect to the websocket host, and assert that they receive a broadcasted message saying that they've connected.

To do this I set up a WSTest object pointing at the correct url, I then added my name as a query parameter. Because we are expecting a message from the host letting us know we've successfully connected, I added an expected response using the with_response method on the WSTest instance. Passed into the with_response method is an instance of WSResponse with a single attribute matching with the json message I'm expecting to receive from the host.

import asyncio
from pywsitest import WSTest, WSResponse

ws_test = (
    WSTest("wss://localhost")
    .with_parameter("name", "Rob")
    .with_response(
        WSResponse()
        .with_attribute("message", "Rob has connected")
    )
)

asyncio.get_event_loop().run_until_complete(ws_test.run())

assert ws_test.is_complete()

As the run method is asynchronous I'm running it synchronously using the asyncio library in this example with the line:

asyncio.get_event_loop().run_until_complete(ws_test.run())

The next functionality I want to test is the user's abililty to send a message after connecting to the websocket host. This test is set-up in a very similar way to the previous test, but with a message triggered when the connected message is received. The test runner will also expect a response from the host displaying the message that the test runner sent.

import asyncio
from pywsitest import WSTest, WSResponse, WSMessage

ws_test = (
    WSTest("wss://localhost")
    .with_parameter("name", "Rob")
    .with_response(
        WSResponse()
        .with_attribute("message", "Rob has connected")
        .with_trigger(
            WSMessage()
            .with_attribute("message", "Hello, world!")
        )
    )
    .with_response(
        WSResponse()
        .with_attribute("message", "Rob: Hello, world!")
    )
)

asyncio.get_event_loop().run_until_complete(ws_test.run())

assert ws_test.is_complete()

The last functionality I want to demonstrate will be testing two clients running simultaneously. To do this I'll create methods to run each of the tests, and use asyncio.gather() to run both of the methods simultaneously. I'll also add a short delay in one of the methods to help ensure that both users have connected before one of them broadcasts a message.

In this test, user_1 will connect to the websocket host, and wait until it receives a broadcasted message that user_2 has connected before sending a message. user_2 will then reply to that message, at which point user_1's test will finish and user_2 will listen for the broadcasted message notifying everyone that user_1 has disconnected.

import asyncio
from pywsitest import WSTest, WSResponse, WSMessage

async def test_user_1():
    ws_test_1 = (
        WSTest("wss://localhost")
        .with_parameter("name", "user_1")
        .with_response(
            WSResponse()
            .with_attribute("message", "user_1 has connected")
        )
        .with_response(
            WSResponse()
            .with_attribute("message", "user_2 has connected")
            .with_trigger(
                WSMessage()
                .with_attribute("message", "Hello from user_1!")
            )
        )
        .with_response(
            WSResponse()
            .with_attribute("message", "user_2: Hello from user_2!")
        )
    )

    await ws_test_1.run()
    assert ws_test_1.is_complete()

async def test_user_2():
    await asyncio.sleep(2)

    ws_test_2 = (
        WSTest("wss://localhost")
        .with_parameter("name", "user_2")
        .with_response(
            WSResponse()
            .with_attribute("message", "user_2 has connected")
        )
        .with_response(
            WSResponse()
            .with_attribute("message", "user_1: Hello from user_1!")
            .with_trigger(
                WSMessage()
                .with_attribute("message", "Hello from user_2!")
            )
        )
        .with_response(
            WSResponse()
            .with_attribute("message", "user_1 has disconnected")
        )
    )

    await ws_test_2.run()
    assert ws_test_2.is_complete()

async def run_tests():
    await asyncio.gather(test_user_1(), test_user_2())

asyncio.get_event_loop().run_until_complete(run_tests())

Closing notes

pywsitest was written to help enable Grid Smarter Cities to better test our products, and simplifies the process of writing integration tests targeting a websocket-based service.

It's an open source project, so if you want to get involved in future development it'd be greatly appreciated!