DEV Community: orangejellyfish

Setting up CloudWatch alarms with Serverless

James Allardice — Tue, 18 Aug 2020 07:42:43 +0000

Monitoring and alerting are essential aspects to any production software system. An effective monitoring solution aggregates data across a range of metrics and presents it in a readable, human-friendly manner. A useful monitoring dashboard might allow you to quickly answer questions around high-level statistics such as "how many users have signed up for our application today" or "how many Lambda invocations have resulted in errors over the last hour". On the other side of the coin, a good alerting system will allow you to respond to changes in these metrics, perhaps by notifying an on-call engineer that something has gone wrong.

In the fast-paced development environment of the Serverless Framework, our preferred tool of choice for applications deployed to AWS Lambda with API Gateway, it's easy to forget about building a robust monitoring or alerting solution. Fortunately, it is also relatively straightforward to do so, thanks to the vast array of metrics available by default in AWS CloudWatch.

CloudWatch metrics

A good set of metrics forms the basis of any monitoring or alerting system. CloudWatch exposes metrics from a wide range of AWS services, including those commonly used in a Serverless application such as API Gateway, Lambda, Cognito and DynamoDB. You can experiment with various representations of the available metrics from the CloudWatch console. Note that you do not have to do anything to get these metrics flowing into CloudWatch. AWS handles it all for you.

The AWS CloudWatch console graphing an API Gateway metric

Metrics can be visualised in a number of ways, including graphs as shown in the above screenshot, and these visualisations can be added to CloudWatch dashboards for quick at-a-glance monitoring. However, this requires you to be physically watching the screen to spot any possible anomalies. There must be a better way!

CloudWatch alarms

Alarms are the mechanism exposed by CloudWatch to build an automated alerting system. They can be configured to respond to changes in any of the metrics we previously explored by notifying an SNS topic of the change. SNS topics are flexible and allow a range of automatic responses, such as sending an email to a specific address, and custom handlers built on AWS Lambda.

CloudFormation has good support for CloudWatch so it's possible to write your alerting system infrastructure as code in the "custom resources" section of a Serverless Framework project. The following example configures a CloudWatch alarm that will trigger when any number of 5xx errors are detected by a specific API Gateway stage.

Resources:

  ApiGatewayAlarm5xx:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmDescription: 5xx errors detected at API Gateway
      Namespace: AWS/ApiGateway
      MetricName: 5XXError
      Statistic: Sum
      Threshold: 0
      ComparisonOperator: GreaterThanThreshold
      EvaluationPeriods: 1
      Period: 60
      Dimensions:
        - Name: ApiName
          Value:
            Fn::Join:
              - "-"
              -
                - Ref: ApiGatewayStage
                - ${self:service}
        - Name: Stage
          Value:
            Ref: ApiGatewayStage

Some of the important properties here can be explained as follows:

Namespace - the AWS service namespace whose metric you want to alert on. The available namespaces are listed in the documentation.
MetricName - the specific metric you want to alert on. These are usually listed somewhere in the documentation for the service in question. For example, API Gateway lists them here.

The Statistic, Threshold and ComparisonOperator properties define a change in metric state that will trigger the alarm. In this case the alarm will trigger if the 5XXError metric exceeds a total of 0 over the Period (a value in seconds).

The Dimensions property effectively restricts the alarm to a subset of available metrics. In this example the alarm will only trigger for a specific stage of a specific API Gateway. If you have multiple stages or APIs deployed in a single account it will be important to ensure your alarms are specific enough to not trigger false positives.

Adding actions to alarms

With the example configuration above we have a CloudWatch alarm configured and it will successfully transition between states as the value of the underlying metric changes. To make this alarm a useful part of our monitoring and alerting strategy we need to add an action to it.

In a Serverless application it is likely that the action will always be a notification to an SNS topic. Other actions include certain EC2 and Auto Scaling actions which are outside the scope of this article. Like the CloudWatch alarm itself, an SNS topic can be codified in CloudFormation:

Resources:

  TopicCloudwatchAlarm:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: ${self:service}-${self:custom.stage}-topic-cloudwatch-alarm

An SNS topic needs a "subscription" to be useful. SNS topics are capable of automatically sending emails to a given address for every message published to them. We can add a subscription in CloudFormation too. In this example the TopicArn property references the TopicCloudwatchAlarm resource defined above via the Ref function:

Resources:

  TopicCloudwatchAlarmSubscription:
    Type: AWS::SNS::Subscription
    Properties:
      Endpoint: alerts@example.com
      Protocol: email
      TopicArn:
        Ref: TopicCloudwatchAlarm

With these resources deployed to AWS any message published to the new SNS topic will be sent to the email address specified by the SNS subscription. All that remains is for us to connect the CloudWatch alarm to the SNS topic. The AlarmActions property on the CloudWatch alarm resource takes the ARN of the
SNS topic. Add the following to the original example to wire it all up:

  AlarmActions:
    - Ref: TopicCloudwatchAlarm

All that remains is to trigger the alarm and check your inbox!

An email sent in response to a CloudWatch alarm

Next steps

The alarm we've looked at in this article barely scratches the surface of what's possible with CloudWatch. You can create alarms that take many metrics into account at once. You can create alarms to warn you when an AWS resource is costing more money than you would like. You can even configure alarms based on "anomaly detection" where CloudWatch will analyse past metric data to create a model of expected values and alert on deviations from that baseline. As with most AWS services, the CloudWatch documentation is helpful and is definitely recommended reading if you'd like to learn more about these more advanced alarms.

Custom icon fonts with React Native

James Allardice — Wed, 17 Jul 2019 09:26:08 +0000

When working with icons in React Native apps we're spoilt for choice with
a wide range of free and open-source icon sets such as FontAwesome,
Material and Ionicons. To make things even easier, the
wonderful react-native-vector-icons project bundles all of those icon
sets plus more into one single package. But sometimes free and open-source icon
sets just don't cut it and you're left wondering how to achieve something that
has the same developer experience for a custom icon set. Fortunately,
react-native-vector-icons and a bunch of other projects have us covered here
too.

Setting up react-native-vector-icons

If you're using Expo and have not ejected to ExpoKit then there's
nothing to do here. Expo bundles a wrapper around react-native-vector-icons in
the @expo/icons package.

Otherwise, the installation of the react-native-vector-icons package is as you
would expect for a React Native app. It's published to npm so you can add it to
your project with the CLI or equivalent (we tend to use Yarn when
working with React Native because it plays better with Expo):

$ yarn add react-native-vector-icons
$ react-native link react-native-vector-icons

Generating an icon font

With react-native-vector-icons set up in your project you are ready to work on
the icons themselves. In our experience IcoMoon is the most effective
tool here. IcoMoon is a web application that allows you to import SVG files and
produce font files in various formats from collections of those SVGs, as shown
in the following screenshot:

An example of creating an icon set in IcoMoon

Once all of your icons are imported to the IcoMoon app you can select them and
"Generate" the font file (note that in the screenshot below it shows the number
of selected icons to the left of the highlighted "Generate Font" button):

An example of generating an icon font from an icon set in IcoMoon

There are a few options to configure the resulting font but most of the time the
defaults will suffice. When you're happy download the bundle and unzip it to
find a selection of font files, some examples and a selection.json file. It's
that file plus the *.ttf font file that we need. Copy those files to a
sensible directory within your React Native codebase. We usually go for a top-
level assets directory which contains all of the static assets used by the app
including fonts and images.

Using the custom icon font

It's recommended that you pre-load any fonts that your app is going to use and
your new custom icon font is no exception. In your main app entry point you can
use the Font.loadAsync method. If you have used the Expo CLI to initialise
your project then you probably have something that looks like this already:

import React from 'react';
import { registerRootComponent, AppLoading } from 'expo';
import * as Font from 'expo-font';

class App extends React.Component {
  state = {
    isLoadingComplete: false,
  };

  loadResourcesAsync = async () => Promise.all([
    Font.loadAsync({
      'custom-icons': require('../assets/fonts/custom-icons.ttf'),
    }),
  ]);

  handleLoadingError = (error) => {
    // In this case, you might want to report the error to your error
    // reporting service, for example Sentry
    console.warn(error);
  };

  handleFinishLoading = () => {
    this.setState({ isLoadingComplete: true });
  };

  render() {
    const { isLoadingComplete } = this.state;

    if (!isLoadingComplete) {
      return (
        <AppLoading
          startAsync={this.loadResourcesAsync}
          onError={this.handleLoadingError}
          onFinish={this.handleFinishLoading}
        />
      );
    }

    return (
      <App />
    );
  }
}

registerRootComponent(App);

// Export the App component for unit testing purposes. Expo handles rendering
// via the "registerRootComponent" call above and does not require an export.
export default App;

With this configuration your custom icon font file will be loaded at app start-
up rather than at first usage which would otherwise result in flashes of
unstyled (or missing) content.

Next up you need a normal React component to render icons from your new font.
The react-native-vector-icons package provides some utility methods to make this
process simpler. The following few lines are all that are needed. We usually
place this in a src/components/icon/index.js file:

import { createIconSetFromIcoMoon } from 'react-native-vector-icons';
import icoMoonConfig from '../../../assets/fonts/selection.json';

// We use the IcoMoon app (https://icomoon.io) to generate a custom font made up
// of SVG icons. The actual font file is loaded up-front in src/index.js.
export default createIconSetFromIcoMoon(icoMoonConfig, 'custom-icons');

The key points to note here are the import of the selection.json file from the
bundle downloaded from IcoMoon and the name of the font, custom-icons, as
defined in the Font.loadAsync call in the main app entry point.

The createIconSetFromIcoMoon function could be thought of as a factory that
returns a React component. You can now import that component from your other
components to render icons. The following example imagines a simple "button"
component in src/components/button/index.js:

import React from 'react';
import { TouchableOpacity, Text } from 'react-native';
import Icon from '../icons';

const Button = () => (
  <TouchableOpacity>
    <Icon name="settings" />
    <Text>Settings</Text>
  </TouchableOpacity>
);

export default Button;

The new Icon component supports all of the props that the open-source icon
sets bundled with react-native-vector-icons support. This means you can apply
custom styles, such as sizes and colours, from React Native stylesheets.

Serverless and the principle of least privilege

James Allardice — Wed, 02 Jan 2019 13:37:49 +0000

Every program and every privileged user of the system should operate using the least amount of privilege necessary to complete the job.

— Jerome Saltzer, Communications of the ACM

That quote, by American computer scientist Jerome Saltzer, underpins the concept that became known as the "Principle of Least Privilege". The idea is that by ensuring each part of a system has the ability to access the data it needs to do its job and nothing beyond that. It applies to pretty much any software system, be it a web application or a program running on an embedded chip.

The rest of this article is going to focus on the principle of least privilege as it applies to the Serverless framework, and, in particular, Serverless applications deployed to AWS. Before we dig into Serverless it will help if you have an understanding of the Identity and Access Management, or IAM, feature of AWS.

IAM

AWS IAM is a comprehensive identity and access management tool. It provides functionality to manage users and control their access to other AWS services and resources. It provides the concept of roles that can be assumed by other entities to grant access to resources. It even provides the ability for web or mobile applications to directly access AWS resources.

IAM provides the concept of "identities" which are created to "provide authentication for people and processes in your AWS account". Identities can be, amongst others, "users", such as the personal account you might use to log in to the AWS Console, or "roles". For now we're primarily interested in IAM roles.

Imagine an AWS Lambda function that when invoked needs to read from a DynamoDB table, manipulate some data, write the modified document back to the table and then publish a message to an SNS topic to notify other interested parties that it has done its job. By default, that Lambda will not have permission to access any of those resources. To ensure the function has the ability to access the resources it depends on it can assume an IAM role when it executes. The Lambda documentation refers to this as the "execution role", and every Lambda function has one that is specified at the time it is created. Likewise, other AWS resources, such as API Gateway, may need a role that allows them to invoke Lambda functions. The Lambda web console has a helpful diagram that shows both sides of this:

We can see that the API Gateway has permission to invoke the Lambda, and the Lambda execution role gives it permission to access CloudWatch, DynamoDB and SNS.

Serverless

When developing a Serverless application for AWS you have the ability to either configure an IAM role that will be assumed by all the Lambda functions in your service (the default approach), or to define a different role for each function.

If you stick with the default you'll end up with something like this in your serverless.yml file. This example will allow all Lambda functions in the service to perform the "publish" action on a specific SNS topic:

provider:
  name: aws
  iamRoleStatements:
    - Effect: Allow
      Action:
        - sns:publish
      Resource:
        - Ref: MySNSTopic

There's a couple of things to think about here. We've gone a long way towards ensuring our functions "operate using the least amount of privilege necessary to complete the job". They are able to perform only a single type of action against a specific resource. If your function code was to attempt to publish a message to a different SNS topic it would not be able to do so. However, we can do better. As we add functions to our service it's likely that they have different access requirements. By following the default configuration you will soon end up with a wide-ranging IAM role that gives all of your Lambda functions access to everything that any one of them might need. To better adhere to the principle of least privilege we need to define an IAM role per Lambda function.

Unfortunately, achieving this with the Serverless framework is fairly involved. When not using the provider-level role configuration you don't benefit from any of the permissions that you usually get for free from Serverless, meaning you need to manually configure access to things like CloudWatch for logging. If this is the approach you want to take you can find plenty more information in the official documentation. But thanks to the wonderful open-source community there is a much easier route.

The serverless-iam-roles-per-function plugin takes care of all that other stuff you'd usually get out of the box and lets you focus just on the resources that are specific to your individual Lambda function. Activate the plugin in serverless.yml:

plugins:
  - serverless-iam-roles-per-function

And then you can define iamRoleStatements at the function level:

myLambdaFunction:
  handler: src/functions/hello/index.default
  events:
    - http:
        path: hello
        method: get

  iamRoleStatements:
    - Effect: Allow
      Action:
        - sns:publish
      Resource:
        - Ref: MySNSTopic

With this configuration any additional Lambda functions in our service will not be able to use the sns:publish action on the MySNSTopic resource. And this function itself will not be able to use any other SNS actions even on that resource, fulfilling our original goal of giving each Lambda function to least amount of privilege necessary to complete the job.

The orangejellyfish Serverless starter kit includes the aforementioned plugin by default and contains an example Lambda function along with a range of other best-practice configuration defaults and helpers.

Get good with git: bisect

James Allardice — Wed, 21 Nov 2018 13:12:34 +0000

Have you ever been in a situation where you needed to find out which commit in the history of a git repository was responsible for causing some change in behaviour? It's a fairly common debugging technique, especially in larger, older codebases. If you know where to find the code that might have introduced the change you may be able to use commands like git blame or git log to search through history for the offending commit, but often that is not the case, and even when it is, there is often an easier approach. Enter git bisect, a useful git command that performs a binary search of history to identify the commit that introduced a change.

Do you learn better with a more hands-on approach?

If a face-to-face approach to learning works better for you or your team, orangejellyfish run a popular git workshop which can take place remotely or on-site with you, giving you an opportunity to go deeper into some more advanced git concepts.

Binary search

If that description alone is enough to put you off reading further, don't despair and follow this section to get a better understanding of what "binary search" is. If you've already got that concept down, feel free to jump ahead.

"Binary search" is an algorithm designed to find a target value in a sorted list of values. It works by comparing the target value to the middle element of the list. If they do not match, half of the list has been eliminated (because the list was already sorted) and the process repeats by comparing the target value to the new middle element, and so on, until the target value is found.

A visual representation might make it easier to follow. Here's our sorted list, or array, of numbers. Since this array contains numbers we can see that it in a sorted state because each element holds a value greater than the previous. Let's say that our target value is 42.

We start by comparing our target value to the middle element. We can see that the middle element is greater than our target:

Because the middle element is greater than the target we can discard the second half of the array and perform the test again. This time we can see that the middle element is smaller than our target:

Because the middle element is smaller than the target we can discard the first half of the array (bear in mind that we are now working with a new array, having already discarded half of the original) and perform the test one more time. Note that this time, since we have an even number of elements in the array, it is up to the algorithm whether we round up or down to determine the mid-point. In this example we have rounded up for the sake of brevity. This time we can see that the middle element matches the target:

Because the middle element matches the target the binary search is complete and the position of the target in the array is the result. In our case the target is at position 4. But how does any of this apply to git?

`git bisect` basics

The git bisect command allows us to perform a binary search on a range of commits in a repository. Try to think of a range of commits as an array where each element is an individual commit. This array is sorted in chronological (time of commit) order, rather than numerical order as in the example above, but since it is sorted we are able to perform a binary search on it. The target of the search is the commit that introduced a given change.

In order to determine the relevant range of commits we need to know of a start point at which the change we are trying to find had not yet been made and an end point at which it had been. The end point will often be the HEAD of the current branch. The start point can be more difficult to determine but a good strategy might be to check out an old but likely stable tag and go from there.

Let's assume that we know a bug has been introduced to our codebase sometime between the v1.2.0 tag and now (the HEAD of the master branch). Armed with that information we can start the bisect and tell it which range we're working with:

$ git bisect start          # Start the binary search process
$ git bisect bad            # Identify the end of the range
$ git bisect good v1.2.0    # Identify the start of the range

Git now identifies the middle point of the range of commits, checks it out and pauses to give you the opportunity to work out whether or not the change you are trying to locate had been introduced at that point in time. You might do so by running an automated test suite, examining a file, or running your app and manually testing the relevant feature. Git tells you what has happened by printing something like this:

Bisecting: 64 revisions left to test after this (roughly 6 steps)
[bf96dbd8768d94e935e0bb874b319ae9b65a9aaf] The mid-point commit message

Having determined whether or not the mid-point contains the change you are searching for you can tell git bisect and it will continue the process by discarding one half of the range, checking out the new mid-point and pausing again. Let's assume that the change we wanted to find is already present at this point in time, meaning the current mid-point is "bad":

$ git bisect bad    # Mark the current mid-point as bad, discard later commits
Bisecting: 32 revisions left to test after this (roughly 5 steps)
[e68d1eed5051744ad21bf7391b582939b5c996fc] New mid-point commit message

Git has paused again at the new mid-point so we get the opportunity to repeat our test. Let's assume this time that the change has not been introduced at this point in time, meaning this mid-point is "good":

$ git bisect good    # Mark the current mid-point as good, discard earlier commits
Bisecting: 12 revisions left to test after this (roughly 4 steps)
[1ee2cc616963f57900d3bc2b635d9efbd1cd168d] New mid-point commit message

Continue this process, running your test at each step and marking the mid-point as either "good" or "bad" until git is able to definitively determine the commit that introduced the change. At that point it will helpfully tell you:

ebd6cd99667703a856b20c87c2307427006fcd2f is the first bad commit
commit ebd6cd99667703a856b20c87c2307427006fcd2f
Author: First Last <first.last@example.com>
Date:   Thu Aug 23 13:58:20 2018 +0000

    This is the commit that introduced the issue

All that remains now is to get back to the normal state by telling git that you are done:

$ git bisect reset

Automating `git bisect`

What we've covered so far is a powerful addition to your git debugging toolbox in its own right, but we can take it a step further to cut down some of the manual work.

The part of the previous example that would slow us down in reality is the test to determine whether each mid-point commit exhibits the behaviour we are trying to locate or not. If it's possible to script that test then git is more than happy to take that burden off us. What we need is a script that exits with code 0 if the current commit is "good" and with 1 (the Unix catch-all error code) if it's "bad". Most automated test runners will already do this so in many cases you'll be able to use an existing script rather than writing a new one.

Let's assume you're trying to track down a bug in a codebase that uses Jest as a unit test runner. The Jest CLI tool exits with helpful codes so we're able to use it to automate a git bisect:

$ git bisect start          # Start the binary search process
$ git bisect bad            # Identify the end of the range
$ git bisect good v1.2.0    # Identify the start of the range
$ git bisect run jest       # Run unit tests against each mid-point

There are some additional tricks you can use in your test script. For example, exiting the script with exit code 125 will tell git bisect that the commit cannot be tested and should be skipped. In that situation the command will choose another commit for you and continue automatically.

What next?

At this point you should have gained a solid understanding of git bisect and be in a position to use it either manually or automatically to discover problems that were introduced by an unknown commit. If you'd like to take it further a good place to start is the official git documentation.

If a more hands-on approach works better for you or your team, orangejellyfish run a popular git workshop which can take place remotely or on-site with you, giving you an opportunity to go deeper into some more advanced git concepts.

A bleeding edge Serverless Framework boilerplate

James Allardice — Wed, 15 Aug 2018 14:35:53 +0000

At orangejellyfish we are big fans of the Serverless framework and use it in a number of projects. Over time we have realised the need for a boilerplate Serverless app to save time when getting started, and that's what we have built.

orangejellyfish / serverless-starter

Serverless framework starter kit for AWS Lambda

serverless-starter

An opinionated starter kit by orangejellyfish for Serverless framework apps running in AWS. Built to be future-proof Inspired by and adapted from the excellent serverless-babel-starter project by Postlight.

Features

Lambdas run Node 14 by default making your functions faster and giving you the ability to use more recent ECMAScript features including async/await, optional chaining and nullish coalescing.
Lambda code is bundled with Webpack 5 via the serverless-webpack plugin reducing the amount of code deployed to AWS.
Lambda code is compiled with Babel 7 and babel-preset-env meaning you can use even more cutting-edge ECMAScript features if you need to without unnecessarily compiling code that would be supported by Node 14.
Lambda config is located alongside the function code and referenced from the top-level Serverless configuration file, offering greater separation of concerns and keeping the configuration file readable.
IAM roles are configured per-Lambda via the serverless-iam-roles-per-function plugin, meaning…

View on GitHub

The starter kit is an opinionated boilerplate for a Serverless app running in AWS. It's been designed to offer as much future-proofing as possible, a tricky challenge in the fast-moving JavaScript ecosystem! Our starting point was the fantastic "Modern Serverless Starter Kit" by Postlight. While that project has "modern" in the name we felt that we could safely go a bit closer to the cutting edge.

Features

When designing the Serverless starter kit we had a few key requirements in mind
which directly translate into features:

Node 8 support. This was a key one for us. AWS has supported Node 8 as a Lambda runtime environment since April 2018, bringing native support for a whole host of recent ECMAScript features (including async/await), performance improvements and reduced memory consumption thanks to the more recent version of V8.
Webpack 4 support. It was important that we could bundle our Lambda function code with Webpack to reduce the size of the package uploaded to AWS. Webpack 4 is the latest and greatest version, offering major speed improvements and much simpler configuration compared to previous iterations.
Babel 7 support. This one is more a straightforward case of future-proofing. There's currently little technical reason to favour Babel 7 over the arguably more stable version 6 but it's fun to push the boundaries sometimes!
Local development support. If you have to deploy functions to AWS every time you make a change the developer experience is somewhat poor. Fortunately this problem has been solved by the serverless-offline plugin which emulates AWS Lambda and API Gateway locally.
Jest support. Unit testing is important. Jest is, in our opinion, the best tool for the job. The starter kit includes Jest configuration to automatically collect code coverage information.
ESLint configuration. We're firm believers in the need for consistent code and therefore the starter kit ships with a pre-configured ESLint set up, along with Husky and lint-staged to efficiently lint and auto-fix code at commit time.

Usage

The Serverless CLI allows you to scaffold a new Serverless application from a template on GitHub. Run the following command to use our starter kit:

serverless create --template-url https://github.com/orangejellyfish/serverless-starter --path your/local/path

What's next?

We think that the feature set provided by this starter kit give you a really solid start for any Serverless application but it's only the basics. Watch this space for more advanced kits, built upon this one, in the future, adding pre-configured support for things like DynamoDB.

DEV Community: orangejellyfish

Setting up CloudWatch alarms with Serverless

CloudWatch metrics

CloudWatch alarms

Adding actions to alarms

Next steps

Custom icon fonts with React Native

Setting up react-native-vector-icons

Generating an icon font

Using the custom icon font

Serverless and the principle of least privilege

IAM

Serverless

Get good with git: bisect

Binary search

git bisect basics

Automating git bisect

What next?

A bleeding edge Serverless Framework boilerplate

orangejellyfish / serverless-starter

Serverless framework starter kit for AWS Lambda

serverless-starter

Features

Features

Usage

What's next?

`git bisect` basics

Automating `git bisect`