DEV Community: Matt Tyler

Call Your AWS API Gateway With Native IAM

Matt Tyler — Thu, 07 May 2020 11:12:15 +0000

Please reach out to me on Twitter @MatthewTyler13 if you have any questions!

Introduction

In a previous installment, we investigated converting over the HTTP API we had built to a REST API. In doing so, we switched out the persistence layer to DynamoDB, and moved to VTL-based service integrations over lambdas. We also used IAM authorization instead of using JWTs. We used Postman to test our API because it is easy way to set up Authorization headers that are compatible with IAM authorization. In previous installments though, I showed how to set up generate client code from our OpenAPI definition, and then apply the JWT to the header. This allowed us to write tests in javascript that could be used for end-to-end testing of the API. Can we do the same when using IAM Authorization? Of course we can! Let's see how!

All code for this tutorial is available here. It may help to go over the client tool generation section that I wrote previously.

IAM Authorization and AWS Signature V4

IAM Authorization uses a different method to validate that requests are authorized, and it is called AWS Signature V4. It is a special signature that is applied to a request in the Authorization header. The signature contains information about the request itself, and is signed with an access key and secret of the user making the request. This is in contrast to a JWT, which only signs claims that are asserted by the authorization server and does not contain any information about the particular request that is being sent.

The header typically looks something like this

Authorization: AWS4-HMAC-SHA256 Credential=AKIA****************/20200320/ap-southeast-2/execute-api/aws4_request, SignedHeaders=host;x-amz-date, Signature=39467d7f8e91e137a49a2713ceb9538d189fdc1e5f76b6939f8027f2ee8c8170

This consists of several parts.

AWS4-HMAC-SHA256

This indicates the particular signature type and signing algorithm used.
Credential=

This indicates the owning credential. It consists of the principal ID (AKIA****************), the date the request was sent (20200320), the scoped region (ap-southeast-2), the service being called (execute-api), and the request type (aws4_request).
Signed Headers

This indicates the headers that were in the canonical string, which is used to calculate the signature. Not all headers need to be included, so you need to specify if any optional headers were included - otherwise the signature calculation will fail on the AWS end.
The signature that was created by signing the hashed canonical string

That brings us to the next point - the canonical string. The canonical string is an encapsulation of the request, into a single string, which is then hashed. That has is signed by the secret access key. When you request is sent, AWS will attempt to reconstruct this string, sign it, and then compare signatures. If the signatures match, AWS determines that request is valid, and can then apply futher checks (e.g. IAM Policy).

A canonical request looks like the following;

POST
/prod/

content-length:9
content-type:text/plain
host:3r47x2ktzh.execute-api.ap-southeast-2.amazonaws.com
x-amz-date:20200318T063056Z

content-length;content-type;host;x-amz-date
b526aef1a341cfe6e5c377ed4c222888eeb81f913a107110a867e009c1758f24

It contains a few things

The method of the HTTP Request.
The path being accessed, relative to host.
A query string, if present (it isn't here).
The canonical headers that are to be signed with the request.
A list of the headers that are in the signed request
A hex-encoded SHA2 Hash of the content in the request body

Additional information about constructing a canonical request is available here.

A hash is then calculated on the canonical request, and a signature is calculated by signing this with the secret access key. This is then included as the signature in the Authorization header.

Client Implementation

That's all well and good, but how can we practically use that in a client? In an earlier installment we pre-generated a client that uses a node library, Axios, to send the request. Adding a header that doesn't depend on the content, like a JWT, is fairly easy. How can we do it in this scenario, without having to write signature calculation code every time we want to send a request?

The answer is pretty easy - most good HTTP client libraries will provide some way to intercept requests before they are sent off, and responses before they are received. Axios providers 'intercepters' which can transform the request before it is sent to the server. Michael Hart has written a library to perform the hard work of constructing the signature, so all we have to do is create an interceptor to do the work.

What follows is an excerpt.

import axios from "axios";
import { createHash } from "crypto";
import { URL } from "url"
import { Config } from "aws-sdk"

const aws4 = require('aws4');

function hash(string: string) {
    return createHash('sha256').update(string, 'utf8').digest('hex')
}

    const instance = axios.create()

    // the interceptor
    instance.interceptors.request.use(async (config) => { 
        // load AWS credentials
        const { credentials: {
            accessKeyId, secretAccessKey
        }} = new Config();

        const url = new URL(config.url);
        const data = config.data ? config.data : "";

        const headers: { [key: string]: string }[] = [
            { 'X-Amz-Content-Sha256': hash(data) },
        ];

        if (!new Set(['OPTIONS', 'GET']).has(config.method.toUpperCase())) {
            headers.push({ 'Content-Type': config.headers['Content-Type'] })
        }

        const req = aws4.sign({
            service: 'execute-api',
            region: 'ap-southeast-2',
            method: config.method.toUpperCase(),
            path: `${url.pathname}${url.search}`,
            headers: Object.assign({}, ...headers),
            body: data,
            host: url.host
        }, { accessKeyId, secretAccessKey });

        config.headers = req.headers;
        return config
    })

    const api = new DefaultApi({}, process.env["ENDPOINT"], instance);

Assuming your API Gateway endpoint is loaded, this can now be used to sign requests that require IAM authorization. Assuming that the credentials that have been used have access to the invoke the relevant API Gateway endpoint.

A Comparison with JWT Authorizers

It makes sense to talk about the difference between this method and JWT's, given that JWT support is available in HTTP APIs for API Gateway and IAM authorization isn't (it's limited to REST APIs). I don't think this means that AWS is abandoning IAM authorization for API Gateway - JWT's are extremely popular and every customer was implementing their own JWT Authorizer with Lambda (sometimes incorrectly). I think IAM authorization has several advantages over JWTs.

It provides a different signature per request, thereby providing a way to ensure the request isn't tampered with.
The secret is not exposed in the request, thereby limiting the opportunities to expose the secret, either via a man-in-the-middle attack or similar vector.
Because the request is tied to an IAM entity, all the powers of IAM are available to determine whether the caller is allowed to perform a specific action.

The downside is that this method is limited to AWS. If you were to port the API to another provider you would need to implement another authorization method. You also need to aquire the credentials in the first place. Embedding credentials in an application is usually not a good idea - so most applications will use cognito federation, to enable clients to exchange a JWT from an Identity Provider for AWS Access Tokens. So even if you do decide to use IAM Authorization for your API, for a public API you will likely still end up with a JWT somewhere. I personally believe that it is worth it, given how powerful and flexible IAM policies can be (that being said, new HTTP APIs do not have support for IAM authorization). The addition of session tags and scope propogation to Cognito would also offer flexible ways to control access to protected resources, but we might be waiting awhile.

Conclusion

The previous installment showed how to create a REST API with IAM Authorization, but didn't show how IAM authorization worked. We've fixed that now by introducing the AWS Signature V4 signing process. We showed how the signature in this process is created, and how it is used in the Authorization header of a HTTP request in order to authorize requests to protected resources on AWS. We showed how to implement the process in generated client code, by showing how to write a request interceptor in Axios using the aws4 node library. Finally, we compared the AWS Sig V4 method with JWTs.

Serverless is More Contact Mechanical Rock to Get Started!

How To Split Functions From Your SAM API Definition

Matt Tyler — Thu, 30 Apr 2020 04:29:25 +0000

Please reach out to me on Twitter @MatthewTyler13 if you have any questions!

A few people have asked whether it’s possible to split lambda functions from SAM templates when creating a lambda-backed API Gateway. The answer to that question is a little bit complicated.

Are you defining lambda functions using the 'AWS::Serverless::Function' type, and you are intending to use the ‘event’ property to hook these functions up? The answer in this case is unfortunately 'no'. The macro transformation, which is called via the “Transform: AWS::Serverless-2016-10-31” directive at the top, does not work this way. It relies on being able to resolve the presence of both the API resource and the function resource from within the same template. It needs to do this to be able to modify the API resource with additional details about the lambda functions. Other function events operate in the same manner.

If either of these resources is missing it cannot do anything. CloudFormation cannot descend into the execution context of nested templates in order to make the necessary modifications. Nested templates simply do not work that way. In spite of how much easier SAM makes it to do Infrastructure-as-Code, in reality it is still limited by the underlying CloudFormation service; CDK has similar limitations.

However, this does not mean that defining lambda functions outside of the API resource context is completely impossible. So how do you do it?

The trick is to use OpenAPI documents to define the API. In doing so, it is possible to define the API in the parent stack, and reference lambda functions from correctly configured nested stacks.

Let’s run through a quick example (You can find the complete code example here).

First, define the lambda function in it's own template.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  HelloWorld:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/
      Handler: app.lambdaHandler
      Runtime: nodejs12.x

Outputs:
  HelloWorldFunctionArn:
    Description: API Gateway endpoint URL for Prod stage for Hello World function
    Value: !GetAtt HelloWorld.Arn

You'll need to ensure you output any/all lambda function ARNs. You will need to pass the function ARNs to resources defined in the parent template. Let's look at that now.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  LambdaTemplate:
    Type: AWS::Serverless::Application
    Properties:
      Location: ./template-function.yaml

  Api:
    Type: AWS::Serverless::HttpApi
    Properties:
      CorsConfiguration:
        AllowCredentials: true
        AllowHeaders: "*"
        AllowMethods:
          - GET
          - POST
          - DELETE
          - HEAD
        AllowOrigins:
          - https://*
      DefinitionBody:
        'Fn::Transform':
          Name: AWS::Include
          Parameters:
            Location: api.yaml

  HelloWorldLambdaPermission:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !GetAtt LambdaTemplate.Outputs.HelloWorldFunctionArn
      Principal: apigateway.amazonaws.com
      SourceArn: !Sub "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${Api}/*/GET/"

Outputs:
  Endpoint:
    Description: API Gateway endpoint URL for Prod stage for Hello World function
    Value:
      Fn::Sub: https://${Api}.execute-api.${AWS::Region}.amazonaws.com/

Note that we are using 'AWS::Serverless::Application' resource to reference the nested template. When using the 'sam package' command, sam will upload the template to an S3 bucket and rewrite the reference appropriately. When deploying the packaged template, the referenced template will be instantiated as a nested stack. As the nested template in this example is using a CloudFormation macro, you will need to ensure that you enable 'CAPABILITY_AUTO_EXPAND' when deploying the template. Note that we 'AWS::Include' the api.yaml file; this function will insert the API definition into the template, and allow us to resolve any references that are in it.

Now let's inspect the 'Api' resource, of which the most important aspect is the 'DefinitionBody' property. This references our OpenAPI document which in turn references our lambda function. I've extracted the most important part.

paths:
  /:
    get:
      operationId: HelloWorld
      x-amazon-apigateway-integration:
        $ref: '#/components/x-amazon-apigateway-integrations/helloWorld'
      responses:
        "200":
          description: "Hello World!"
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Message"
        "400":
          description: "Bad Request Exception"
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/BadRequestException"
        "500":
          description: "Internal Server Error"
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/InternalServerErrorException"
components:
  x-amazon-apigateway-integrations:
    helloWorld:
      type: aws_proxy
      httpMethod: POST
      uri:
        Fn::Sub: "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${LambdaTemplate.Outputs.HelloWorldFunctionArn}/invocations"
      passthroughBehavior: when_no_templates
      payloadFormatVersion: 2.0

Here we define the helloWorld operation, which references a lambda integration that is defined in the components section. When the template is instantiated, it constructs the 'uri' to reference the lambda function ARN that was output from the nested template. When the API resource is created, it is then able to 'wire up' to the lambda function.

There is one more thing that needs to be done; Permissions must be granted on the lambda function to allow it to be invoked by API Gateway. This can be done with following snippet.

  HelloWorldLambdaPermission:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunction
      FunctionName: !GetAtt LambdaTemplate.Outputs.HelloWorldFunctionArn
      Principal: apigateway.amazonaws.com
      SourceArn: !Sub "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${Api}/*/GET/"

I imagine most of the folks trying to do this are probably trying to define all the lambdas for a particular resource in a particular template. E.g. if I was writing an application that enabled people to post comments to different message boards (like a forum), I may want to locate all the lambda functions for messages and message boards in separate templates.

Now it’s obviously a lot of additional work if you are not using OpenAPI currently to build your SAM-powered API’s. That said, if you aren’t using OpenAPI I would suggest reconsidering your position. APIs are fundamentally designed to be consumed by multiple clients; if you only ever intend your API to be consumed by one application you may not need an API. Publishing an OpenAPI specification for your API gives you and your clients a complete reference that can be used to generate various helpful assets; from documentation to complete SDK’s for various languages.

The negative in all this is that you cannot use the events property in the serverless function definition to define the API, which can be pretty convenient. On the other hand that doesn’t mean you have lost all the usefulness of the SAM template format. You can still use other useful elements of the definition such as easy function aliasing, canary deployments and the SAM template policy mechanism.

Seeking a source of serverless support? Contact Mechanical Rock to Get Help!

Serverless With 100% Less Lambda

Matt Tyler — Wed, 22 Apr 2020 14:58:57 +0000

Please reach out to me on Twitter @MatthewTyler13 if you have any questions!

Introduction

It often comes as a surprise to many developers that you don't actually need lambda when building certain kinds of API's with API Gateway. Many simple CRUD applications don't need it at all and can get away with service integrations to DynamoDB. In doing so, you no longer need to pay for the cost of a lambda execution, or incur additional latency from needing to invoke a lambda function. If all your lambda function does is store data in DynamoDB, you probably don't need that lambda function. The serverless express guestbook application that I've been using as an example is a good case study in this. In a previous installment, we implemented X-Ray tracing and noticed that storing and retrieving comments from S3 is quite slow. Let's refactor that to use DynamoDB, and remove the lambda functions in the process!

All code for this tutorial is available here. Aside from the tools required from previous installments (The AWS SAM CLI), it will help to have Postman installed to exercise the API later.

Database Design

Let's begin by designing the DynamoDB table. Theses are the following access patterns I want to cover;

I want users to be able to post comments.

This will logically require a field to hold author and message data.
I want users to be able to delete their own comments.

This means I will need a way to uniquely identify a particular comment, via an ID field.
I want to be able to list comments by user, most recent comments first.

This will require some sort of time field.
I want to able to list all comments, most recent comments first.

This adds nothing new field-wise (or does it?), but it may influence our indexing.

I've settled on the following fields/indexes

pk: This is the partition key - and I will store author data in this.
sk: This is the sort key - and I will store the comment ID in this field. Together, these two fields uniquely identify every comment in the database, and allow me to CRUD a particular comment
m: This field will contain the comment message.
d: This will store the time that a comment was made, in epoch (unix) time
pk_d: A local secondary index (LSI) that uses 'd' to sort the entries. This allows me to query a users comments in order by the time they were made
t: A static value that represents the type of entry. This will contain the string 'comment'
t_d: A global secondary index (GSI) to sort all comments by the date they were made. This is required to be able to query all comments and return them in the order they were made. Table scans do not return items in global order (only by partition order), so we require an additional partition key that all comments can belong to, and an associated sort key.

To create the table in CloudFormation, you can use the following definition.

  Database:
    Type: AWS::DynamoDB::Table
    Properties:
      AttributeDefinitions:
        - AttributeName: pk
          AttributeType: S
        - AttributeName: sk
          AttributeType: S
        - AttributeName: d
          AttributeType: N
        - AttributeName: t
          AttributeType: S
      BillingMode: PAY_PER_REQUEST
      GlobalSecondaryIndexes:
        - IndexName: t-dt
          KeySchema:
            - AttributeName: t
              KeyType: HASH
            - AttributeName: d
              KeyType: RANGE
          Projection:
            ProjectionType: ALL
      KeySchema:
        - AttributeName: pk
          KeyType: HASH
        - AttributeName: sk
          KeyType: RANGE
      LocalSecondaryIndexes:
        - IndexName: pk-d
          KeySchema:
            - AttributeName: pk
              KeyType: HASH
            - AttributeName: d
              KeyType: RANGE
          Projection:
            ProjectionType: ALL

The Rest of the Template

Previously we used a lambda function and an HTTP API - we remove both of these and replace it with the following REST API.

  GuestBookApi:
    Type: AWS::Serverless::Api
    Properties:
      DefinitionBody:
        'Fn::Transform':
          Name: AWS::Include
          Parameters:
            Location: api.yaml
      StageName: prod
      TracingEnabled: true
      OpenApiVersion: '3.0.0'
      Cors:
        AllowOrigin: "'*'"
        AllowHeaders: "'authorization, content-type'"
      MethodSettings:
        - ResourcePath: '/*'
          HttpMethod: '*'
          DataTraceEnabled: true
          LoggingLevel: INFO
          MetricsEnabled: true
          ThrottlingRateLimit: 5
          ThrottlingBurstLimit: 10

This is pretty similar to the HTTP API definition from before but adds a couple of things:

An explicit stage name of 'prod'
Enables X-Ray Tracing (not supported yet in HTTP API - but it is on the roadmap)
Adds some settings around logging and throttling that are not supported in HTTP API (but once again, are on the roadmap)

Finally, we will need two roles to manage read and writing to the database. These will be referenced in our OpenAPI definition, and will be used by our API Gateway service integrations to perform actions against our DynamoDB table. As we are splitting up our methods and endpoints, we can narrow permissions needed by a specific resource/method to a specific set. This ensures each action has the minimum permissions needed to perform the job. This is a massive advantage over using a monolithic lambda function that controls routing - as paths do not have access to more permissions than they require to perform their intended function.

 PostCommentsRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: apigateway.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: root
          PolicyDocument:
            Statement:
              - Effect: Allow
                Action: dynamodb:PutItem
                Resource: !Sub "${Database.Arn}"

  ReadCommentsRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: apigateway.amazonaws.com
            Action: sts:AssumeRole
      Policies:
        - PolicyName: root
          PolicyDocument:
            Statement:
              - Effect: Allow
                Action: 
                  - dynamodb:GetItem
                  - dynamodb:Query
                Resource: 
                  - !Sub "${Database.Arn}"
                  - !Sub "${Database.Arn}/index/*"

The OpenAPI Definition

Our OpenAPI template requires several adjustments. Most of these I based off of the (Real World Serverless)[https://github.com/awslabs/realworld-serverless-application] application. At the time this was the only public example I could find of an application that used the OpenAPI version 3 template format successfully.

We start with the following definitions to enable request validation and define CORS headers for error responses. This is a bit more difficult to configure correctly in an API Gateway REST API than is in HTTP APIs; so if you hate CORS, you'll probably love HTTP APIs.

openapi: 3.0.1
info:
  title: simple-node-api
  description: A simple API for a guestbook application
  version: 2019-10-13

x-amazon-apigateway-request-validators:
  all:
    validateRequestBody: true
    validateRequestParameters: true
x-amazon-apigateway-request-validator: all

x-amazon-apigateway-gateway-responses:
  # Provide more detailed error message for bad request body errors. See doc: https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-swagger-extensions-gateway-responses.html
  BAD_REQUEST_BODY:
    responseTemplates:
      application/json: '{"errorCode": "BadRequestBody", "message": "$context.error.validationErrorString"}'
    responseParameters:
      gatewayresponse.header.Access-Control-Allow-Origin: "'*'"
  DEFAULT_4XX:
    responseParameters:
      gatewayresponse.header.Access-Control-Allow-Origin: "'*'"
  DEFAULT_5XX:
    responseParameters:
      gatewayresponse.header.Access-Control-Allow-Origin: "'*'"

Then we need to add some definitions to our 'Create Message' endpoint. In it's entirety, it looks like this.

  /:
    post:
      operationId: CreateMessage
      requestBody:
        content:
          text/plain:
            schema:
              type: string
        required: true
      responses:
        "201":
          description: "Successfully created a message."
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Message"
        "400":
          description: "Bad Request Exception"
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/BadRequestException"
        "500":
          description: "Internal Server Error"
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/InternalServerErrorException"
      x-amazon-apigateway-integration:
        type: aws
        uri:
          Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:dynamodb:action/PutItem
        httpMethod: POST
        credentials:
          Fn::Sub: "${PostCommentsRole.Arn}"
        requestParameters:
          "integration.request.header.X-Amzn-Trace-Id": "context.xrayTraceId"
        requestTemplates:
          "application/json":
            Fn::Sub: |
              { 
                "TableName": "${Database}",
                "Item": {
                  "pk": {
                      "S": "$context.identity.caller"
                  },
                  "sk": {
                      "S": "$context.requestId"
                  },
                  "d": {
                      "N": "$context.requestTimeEpoch"
                  },
                  "m": {
                      "S": "$input.body"
                  },
                  "t": {
                      "S": "comment"
                  }
                }
              }
          "text/plain": 
            Fn::Sub: |
              { 
                "TableName": "${Database}",
                "Item": {
                  "pk": {
                      "S": "$context.identity.caller"
                  },
                  "sk": {
                      "S": "$context.requestId"
                  },
                  "d": {
                      "N": "$context.requestTimeEpoch"
                  },
                  "m": {
                      "S": "$input.body"
                  },
                  "t": {
                      "S": "comment"
                  }
                }
              }
        responses:
          "2\\d{2}":
            statusCode: 201
            responseTemplates:
              "application/json": |
                #set($inputRoot = $input.path('$'))
                {
                    "id": "$context.requestId",
                    "author": "$context.identity.caller",
                }
        passthroughBehavior: never
      x-amazon-apigateway-auth:
        type: AWS_IAM
      security:
        - sigv4: []

The start of the definition should be familiar territory, but it begins to diverge with the 'x-amazon-apigateway-integration' property. This property is an API Gateway extension to the specification that defines the service integration for this endpoint.

      x-amazon-apigateway-integration:
        uri:
          Fn::Sub: arn:${AWS::Partition}:apigateway:${AWS::Region}:dynamodb:action/Query
        httpMethod: POST
        credentials:
          Fn::Sub: "${ReadCommentsRole.Arn}"
        type: aws
        passthroughBehavior: never

The start of the definition includes a few things;

This defines the service integration we are going to use. We can see from this example, that we have chosen to use a dynamoDB Query action.

httpMethod

Regardless of whether we are reading or writing, most service integrations use a 'POST' http method. This refers to invoking the particular service integration action - not the method of the particular endpoint.

credential

Here we have subbed in the ARN of the read comments role that we create in the CloudFormation Template.

type

This refers to the particular integration type that we are using - a standard 'aws' integration in this example.

passthroughBehaviour

This determines whether non-matching content types are passed through to the integration. I usually default this to 'never'. If a request comes through with a non-matching content-type header, API Gateway will auto-respond with 415 Unsupported Media Type.

Lastly, we define the security of the endpoint. This is done via the 'x-amazon-apigateway-auth' property on each method, in conjunction with security schemes within the components defintion.

Now we need to define a request template.

        requestParameters:
          "integration.request.header.X-Amzn-Trace-Id": "context.xrayTraceId"
        requestTemplates:
          "application/json":
              Fn::Sub: |
                #set($token = $input.params("token"))
                {
                    "TableName": "simple-node-api-Database-5IHXRFDA8AAX"
                    ,"IndexName": "t-dt"
                    ,"KeyConditionExpression": "t = :v1"
                    ,"ExpressionAttributeValues": {
                        ":v1": {
                            "S": "comment"
                        }
                    }
                    ,"ScanIndexForward": false
                    #if($!token != "")
                    #set($startKeyString = $util.base64Decode($token))
                    #set($startKey = $startKeyString.replaceAll("\\""", """"))
                    ,"ExclusiveStartKey": $startKey
                    #end
                    #if($!{input.params("maxItems")} != "")
                    ,"Limit": $input.params('maxItems')
                    #else
                    ,"Limit": 10
                    #end
                }

Firstly I've ensured that the X-Ray Trace ID header is propagated into the request via the request parameters. This will allow me to see DynamoDB in the request trace. The next step is to define a VTL mapping template. The templates are defined on a per-Content-Type basis. I've decided to only accept 'application/json', so only one template is present.

The template defines the payload that is sent to the DynamoDB query endpoint, which follows the specification detailed here. Several rows start with a '#' symbol - I've used these to inject additional properties where needed. For example, if the 'maxItems' query parameter was specified, I'll include it in the query, otherwise default to the value 10. I additionally check for a base64 encoded token, and inject it as the ExclusiveStartKey if it is present. This allows the user to paginate through the results provided by the endpoint.

Further information is available on special VTL parameters here.

I've also implemented an endpoint to create comments - which is far simpler. Peruse it at your own leisure. I've left additional endpoints as an exercise for the reader.

components:
  securitySchemes:
    sigv4:
      type: apiKey
      name: Authorization
      in: header
      "x-amazon-apigateway-authtype": awsSigv4

This defines a security scheme, namely that authorization information will be the header under the Authorization Key, and that will confirm to AWS Signature V4. This is the correct Authorization scheme when using native IAM controls to invoke API Gateway.

Each endpoint will have the following additional property. This enables AWS_IAM authentication on the endpoint and indicates that AWS Signature V4 is in use.

      x-amazon-apigateway-auth:
        type: AWS_IAM
      security:
        - sigv4: []

Deploy and Exercise the API

Deploying the API, as always, can be done through a simple sam build && sam deploy command initiated at the root of the repository.

Once the API is deployed, let's use Postman to send some messages. The first thing you'll need to is select 'POST' and then set the neccesary authorization header. To do this, you'll need to select AWS authorization and fill out the appropriate credentials. Assuming that you have been using AWS SAM CLI successfully up till now, you can grab keys from your ~/.aws/config file. This assumes that you have permissions to invoke the API. The auth section should look like the below.

You can post a message by entering some text in the 'body' section. Just make sure you set the content type correctly by setting it to 'raw', and then selecting 'text' from the dropdown menu.

Once sent, you should get a result like the following image.

Performing a GET is similar - you will still need to set the authorization header, but you won't need to enter anything in the body.

Likewise - you should get a result similar to the following.

We can also check things out in X-Ray. You'll notice the trace-map now looks like the following.

We can review the latency of each request - either the API Gateway as a whole or just the request to DynamoDB.

We can list all those traces...

And review a trace for a specific request.

For those who followed our previous installment, you'll remember that to get all of the comments originally took around 1.6 seconds when each comment was stored in S3. This is significantly faster at 60ms per request. That's 26 times faster, which is a pretty big improvement. The moral of the story is to perhaps not use S3 in such a scenario - use DynamoDB.

Other Considerations and Commentary

HTTP API's do not yet have all the features of REST API's. This largely seems to be due to the result of customer feedback; a simple lambda proxy that provides JWT authorization covers a fairly large number of use-cases. Still, it's worth some additional discussion.

HTTP API vs REST API

At the moment HTTP API's do not support direct service integrations but they are probably on the roadmap. AWS has stated that HTTP API's will eventually hit feature parity with REST API's. The performance improvement was derived mainly from switching out S3 for DynamoDB - in a future installment, I will do a more Apples-to-Apples comparison of REST API vs HTTP API. AWS has claimed that HTTP API's are 60% faster than REST APIs, so I expect that HTTP API with Lambda will have comparable performance to REST APIs with Service Integrations - at least for this application anyway.

Authentication - IAM vs JWT

The serverless express applications used JWTs authorization because it is all that HTTP API's support. REST API's have a more robust selection. In this case I chose to use IAM Authorization. I personally prefer native IAM controls, because it lets me piggy-back on to a more robust RBAC mechanism that I do not need to write myself. In practice this can make things complicated, because in practice it can require using cognito identity pools to vend out AWS credentials via a token exchange. As mentioned in prior installments, Casbin (or some other policy engine) can be used if you want to stick to just using JWTs.

OpenAPI Document Pollution

It bothers some developers that they must include AWS extensions in the OpenAPI document. I can understand wanting to keep the document 'pure' from vendor pollution. To do this, it is possible to define the vendor extensions in a separate file, and then merge the two files afterwards as part of your build process. Alternatively, AWS have their own IDL, called Smithy. Smithy can be used to generate an OpenAPI definition file with and without API Gateway extensions. Some users may find this useful if they want to publish their OpenAPI document free from vendor properties that may expose implementation details.

Pagination

There are limited ways to implement pagination when using VTL extensions. In my example, I used base 64 encoding in a vain attempt to hide implementation details, but anyone can simply decode the token. They could then rely on implementation detail that may change in future, which may break their application. The real-world serverless application example instead uses a KMS key to encrypt the pagination data, so that this cannot occur. There is no way to do this in VTL though, so you must use more flexible compute, like lambda, to do so.

Testing

Testing is much harder with VTL - as it requires deploying and exercising the API directly. This is more akin to an End-to-End test, but you may be able to get away with a unit test when using lambda. That said - you should be performing end-to-end testing on your API anyway so I don't normally consider this a deal-breaker, personally.

Instrumentation

A cool side effect of going the VTL path is we didn't really need to write any custom logging or instrumentation code - it's provided entirely out of the box via X-Ray integration, and in-built logging. It's a little more work to do this via lambda functions.

Conclusion

We decided to take a step back and implement our API using REST API. We used service integrations to remove our lambda functions from the equation. We built a CloudFormation template to deploy our API with DynamoDB. We updated our OpenAPI definition with API Gateway extensions, which allowed us to use the DynamoDB service integration. We implemented authorization via native IAM controls. We then sent a few requests off using postman, and review the performance of the application using X-Ray. Finally, we finished with a discussion of the differences between this approach and that of the HTTP API-Lambda method.

Do more with (server)less! Contact Mechanical Rock to Get Started!

The Hitchhiker's Guide to S3 Access Controls

Matt Tyler — Wed, 15 Apr 2020 11:42:16 +0000

Introduction

S3 was the first service to become generally available (GA) in AWS, debuting in 2006. It would be fair to say that since then it has become an essential building block of the internet. It is the foundation for both services internal to AWS and externally by service providers. No service lasts for 14 years (as of now, it's 2020) and doesn't accumulate some level of baggage, and there are probably a few developers out there who would liken dealing with the myriad of S3 access mechanisms as being akin to dealing with a moody teenager. I deal with S3 on an almost daily basis, and have a boatload of relevant AWS certifications - and I still regularly get tripped up when dealing with S3. With that in mind, I've decided to take the time write out how S3 policies work, with extra attention to my method of determining the access level.

TLDR

If you don't need to grant access to a bucket outside identities within your own account then you probably don't need anything beyond IAM Policies
ACLs, Bucket Policies, and Access Points are useful when you need to grant access to identities outside your own account (though they do work for internal identities as well)
Bucket Policies and Access Points can be far more granular than ACL's, so they are typically more useful. Avoid ACLs wherever possible.
Access Points are useful if you want to create named aliases for buckets for specific bucket consumers. This is helpful if your bucket policies become too large to fit within the default limits. This is often the case for analytics workloads, e.g. publishing large data-sets for many consumers (universities, private sector etc.)
Buckets and Objects have a concept of 'owners'; this is part of the ACL system, and critical to understand even if you hardly ever touch ACLs.
The owner of either entity is always the identity that created it.
The bucket owner can never be changed.
Only the bucket owner can view the bucket in the console.
Object owners can only be changed by overwriting the object.
By default, only the owner of an entity can make administrative changes to the entity. This can cause confusion if another entity has put an object into a bucket owned by someone else, without setting the 'bucket-owner-full-control' ACL when the object was created.
Checks on the owner sometimes occur in strange places - e.g. a CloudFront Origin Identity can only serve an object from a bucket if that object is owned by the account that owns the bucket.

Terminology

Lets get some terminology out of the way first.

Access Control List - to be honest, all types of the access mechanisms used by S3 are access control lists but most of the time when we are talking S3, we specifically mean the S3 ACL mechanism. This is the original access mechanism in S3. I tried to avoid it most of the time - the other access mechanisms are far more flexible and easy to work with (IMHO).

Policy

A policy is the 'rule' for determining access - it is the sum of the statements in the policy (eg. Resource, Action, Effect, Principal and Conditions)

Resource

In S3, this generally refers to things; buckets, and the objects that are inside those buckets. Wildcards work in resource policies for specifying multiple of something. e.g.

  {
      "Resource": "arn:aws:::s3:mybucket/*"
  }

The above will match all items in the bucket. However, if you are giving bucket level permissions this won't work as the statement does not match "arn:aws:::s3:mybucket". Because of this, it's common to see the following in a resource policy.

  {
      "Resource": [
          "arn:aws:::s3:mybucket",
          "arn:aws:::s3:mybucket/*"
      ]
  }

Note: A bucket policy can only have statements in the resource statement that refer to itself. It doesn't make sense to have a bucket policy on a bucket refer to the contents of another bucket. This might cause you to wonder why they include the resource statement at all - why not remove it like principal statements in IAM policies? The big difference is the ability to refer to specific objects within the bucket, so I believe they stuck with specifying ARNs for consistencies sake.

Action

Actions are the things we are allowed to do with resources. This will usually be things like PuObject, GetObject, ListBuckets - etc. Some actions work on buckets, whilst others work on the objects in those buckets.

It's usually reasonably obvious what is going on, except for the two things - "ListAllBuckets" and "ListBucket". The former lists all the buckets in the account, the latter lists all of the objects in a particular bucket. Yes, it's easy to mix up.

Effect

The effect is the result of various conditions matching the rest of the policy. We can do two things - we can either ALLOW things to happen or DENY them.

Principal

The principal is the identity of the 'thing' performing the action. For example, this lets us to block Mark from reading an object but allows Alice to read it. Users, roles, services, accounts, all of these things are 'principals'. Here are some examples.

Give access to the AWS Config service.

  {

      "Principal": {
          "Service": "config.amazonaws.com"
      }
  }

Give access to all accounts within AWS.

  {
      "Principal": {
          "AWS": "*"
      }
  }

Give access to two accounts - the following two examples are equivalent.
Important! Note the usage of 'root' doesn't mean root user, or root account. It just means the account is free to delegate permission as it sees fit.

  {
      "Principal": {
         "AWS": [ "0123456789", "123456790" ]
      }
  }

  {
      "Principal": {
         "AWS": [ "arn:aws:iam::0123456789:root", "arn:aws:iam::123456790:root" ]
      }
  }

Give access to everyone in the world

  {
      "Principal": "*"
  }

Give access to a specific role within an account

  {
      "Principal": {
          "AWS": "arn:aws:iam::1234567890:role/my-cool-role"
      }
  }

Note that the only time wildcards work in a policy is when they are specified as the only thing in a value - "*" will work, but "arn:aws:iam::1234567890:role/*" will not.

Condition

Condition statements allow us to further restrict access based on various circumstantial attributes. Maybe we block Mark all the time, except for when Mark is accessing the bucket from a particular office.

There are a lot of conditions - some of which can be found here in the cloudonaut reference, and more specific condition keys can be found in the official reference which is available here.

The following is an bucket policy (you can tell because there is a principal property in the policy) that blocks access to objects within 'examplebucket' that will only allow access to objects within a specific IP range (192.0.2.0/24), unless the address is 192.0.2.188.

Note that this does not specifically deny accessing the bucket from 192.0.2.188 - so it could be still granted access by some other policy.

    {
        "Version": "2012-10-17",
        "Id": "S3PolicyId1",
        "Statement": [
            {
                "Sid": "statement1",
                "Effect": "Allow",
                "Principal": "*",
                "Action":["s3:GetObject"]  ,
                "Resource": "arn:aws:s3:::examplebucket/*",
                "Condition" : {
                    "IpAddress" : {
                        "aws:SourceIp": "192.0.2.0/24" 
                    },
                    "NotIpAddress" : {
                        "aws:SourceIp": "192.0.2.188/32" 
                    } 
                } 
            } 
        ]
    }

Canonical ID

Canonical ID's pop up in a few places but you are more likely to see them in ACLs than anywhere else. They usually identify what accounts may have access to certain buckets. More information can be found here.

Resource Policy & IAM Policies

Resource policies are access policies that apply to 'resources' as opposed to 'identities' (like IAM Policies). Resource policies are most notable in that they include a 'principal' statement to narrow down the identity accessing the resource. Identity policies do not have a principal statement because they attached to identities - the principal is therefore implied.

Note: As dicussed before, principal statements can not include wildcard ('*') statements.

Owner

All resources in S3 have a concept of an 'owner'. The owner of the bucket is always the account that created it, and so far as I know can't be changed. However, objects in that bucket may be owned by other accounts. The bucket owner pays the bills, and can therefore deny access to objects and delete objects that are owned by others in the bucket. The bucket owner only has full access to objects that they own, unless the object owner grants them full access.

The object owner can only be changed by overwriting the object - there is no way to change ownership of an existing object.

This admittedly doesn't come up too often, but it is good to be aware of it.

With that out of the way, let's get a bit more practical by looking at most of the access mechanisms in S3.

What is Public Access?

A public bucket is one that grants access to its objects without requiring any proof of identity. Because of how confusing S3 policies can be, AWS now has methods to deny public access to buckets, regardless of whether a policy will technically allow it. If you are trying to either allow or deny public access to a bucket, you should check those settings first - of which we will detail now.

Public Access Restrictions

Public access restrictions are easiest way to ensure that a bucket can't ever be made public. These settings can be set at both an account level (active across all buckets within the account) or at an individual bucket level.

Block public access to buckets and objects granted through new access control lists (ACLs)
Block public access to buckets and objects granted through any access control lists (ACLs)
Block public access to buckets and objects granted through new public bucket policies
Block public access to buckets and objects granted through any public bucket policies
Block all public access (union of the above four)

Basically, the two mechanisms for allowing access to an entity that isn't in your account are bucket policies and ACLs. You have two settings that force them to be private ('any' policies'), and another two settings that will allow any buckets that are already public to remain public - but no others!

More information can be found in the official documentation.

S3 ACLs

ACLs are the oldest mechanism and as such are very 'spray-and-pray'. Access is more or less limited to read and/or write - no other calls to change ACL's or enabled object-locks or other advanced features. Access is granted via 'Canonical ID's' which is just another identifier for an AWS account (think of it as a stand-in for an account ARN). ACL's can be used at both the bucket and object level.

Note that ACL's at the bucket level do not give access to objects within the bucket - just bucket level operations. E.g. A read ACL on a bucket won't let you read objects inside the bucket - it'll let you list things inside the bucket.

If you are only ever using the bucket within your account, and have no need to make the bucket public, you will likely never touch one. But if you need to grant access to one or more principals outside the account that owns the bucket, ACL's are one mechanism to do so (though there are better ways to do it).

There are also 'canned' ACLs which are ACL's which a special name that grants access for certain kinds of scenarios e.g. PublicRead, PublicReadWrite. These are not user-configurable, and are the only ACL's that can applied via CloudFormation. For me, that's enough reason alone to prefer bucket policies over ACLs which are capable of doing the same thing but more.

Resource Policies

Resource policies are similar into ACL's in that they are away to grant access to resources that are outside the account boundary. Where they differ is that they are in a similar format to IAM polices so they have a lot more flexibility. 99.99999999 percent-of-the-time, if you are dealing with cross-account bucket access, what you want to use is a bucket policy. Additionally, if you need to give bucket access to an AWS service, bucket policies are how you will do this.

One important thing to mention - they can provide access to resources within your account boundary, provided that the principals within your account do not have an policy attached to them explicitly denying them access. The absence of an IAM policy, and the presence of an 'Allow' bucket policy will grant the privileges encoded within the bucket policy.

IAM Policies

As said before, these are more less the same as bucket policies, but are scoped against the IAM identities that they are attached to. As such, These are exactly like the bucket policies, but will not have a 'principal' statement in the policy. These are only used to grant and revoke privileges for identities that are within the account that owns the bucket. If all you want to do is manage access between bucket and IAM principals with your account, IAM policies are the only policies you need.

IAM Policies are sometimes referred to as 'User policies' in AWS documentation for S3.

Access Points

Access points are the newest way to grant access to S3 Buckets. They have come about as a way to manage access at a larger scale; there is a limit to how large the bucket policy can be, and only one bucket policy document can be attached to a bucket. Using bucket access points creates an 'alias' for a bucket, and all the access through that alias is managed by a policy document designed specifically for that alias.

If you find that confusing, think of like this; you may have a bank account (bucket) and multiple debit cards (access points) for different family members. You might have different daily spending limits on those cards, but they all fundamentally draw from the same source.

Access points are still relatively new, but I imagine they may become more popular than bucket policies due to the ability to have multiple of them, and to have bucket aliases that are scoped to a particular account (e.g. no issues with global naming collisions like normal bucket names).

Evaluating Multiple Policies

It's a little confusing to understand how all of these policies apply

Are block-public-access restrictions enabled?

If you're accessing the bucket publicly, and these are enabled, stop now. You can't access it. If only a couple of them are, you'll need to scrub out the associated policy (assuming it is an 'Allow' policy) and evaluated the remaining policies.

Note: If you are blocking all public ACLs, this doesn't mean that the bucket is not public - You could have a '*' in the principal of a bucket policy which may be granting public access.

Check if anything has an explicit deny.

If even /one/ thing, no matter what it is, matches a deny condition for the principal, they will be denied access to the bucket.

Check if anything has an explicit allow.

As long as there are no matching 'deny' statements, and at least one allow statement, the principal will be allowed to access the bucket.

No explicit allow anywhere?

The principal will be denied access to the bucket.

A bucket that is created via the console (without changing any default settings), will tend to only allow access via IAM to the owning account.

Conclusions

This was a whirlwind tour of S3 access controls. We had a look at some common terminology that is both specific and common to all S3 access mechanisms. We also learnt a little about public access, and we can prevent public access by using the public access restriction controls. We had a look at each mechanism which includes ACLs, Bucket Policies, IAM Policies and Access Points. We learnt about the problems that each access mechanisms solves, and how to recognise the appropriate scenarios to use them in. Finally, we learn how S3 evaluates whether to allow access in the prescence of multiple policies.

Need help avoiding the S3 Bucket Negligence Award? Contact Mechanical Rock to Get Started!

Please reach out to me on Twitter @MatthewTyler13 if you have any questions!

Serverless Express: Measuring With Metrics

Matt Tyler — Wed, 08 Apr 2020 12:50:22 +0000

Please reach out to me on Twitter @MatthewTyler13 if you have any questions!

Introduction

The last installment started to cover off on observability of our applications. Observability helps us to answer questions about how well our applications are running in production. We can than use this information to improve our applications over time. Logging, Tracing and Metrics are commonly referred to as the 'three pillars of observability'. We previously had a look at logging and tracing. Logging allows us to keep persist statements about things that are occuring in our application for various purposes. Tracing is somewhat similar to logging, except it works across system boundaries to 'trace' how a request travels through various parts of a distributed systems architecture. I have not covered the third and final pillar, metrics, so let's dive into it!

Metrics, Dashboards, Alerts - What Are They Good For?

Metrics differ a fair bit from the other two pillars in that they are not statements - they are measurements. Metrics usually have a defined unit-of-measurement, whether that be duration, or a count of events etc. This allows them to be aggregated and vizualized for the purposes of understanding approximations of system performance at different points in time.

Time of an event
A measurement
Dimensions

In practice metrics are used for a few things;

To compare system performance at different points in time

e.g. metrics can be used to see how a system performs under various different conditions.

To measure and audit a system to ensure it is meeting various obligations

e.g. It may have been agreed that a system must be available 99.95% of the time, or that 90% of requests should complete in under 50ms, lest the user of the system be compensated.

To alert when various thresholds are met

e.g. If the system breaches a threshold of errors in a defined period, we should alert somebody so they can investigate the issue.

Metrics need not be restricted to software engineering concerns like uptime and requests-per-second. Metrics that are related to the domain your business operates in (e.g. number of orders) can be very valuable.

In most cases, metrics only help to answer fairly generic questions of performance over time e.g. the 'what'. To understand 'how', and 'why', tracing and logging are generally more helpful. This doesn't stop metrics from getting a disproportionate amount of attention. Metrics are extremely seductive because they are so easy to understand at a glance (green = good, red = bad). A time-poor engineering manager can easily make the transition to a poor engineering manager by placing too much emphasis on metrics. It's important to realize metrics are like sugar; best taken in moderation as part of a balanced diet.

The worst outcome with metrics is something known as 'Dashboard Hell'. A team will create a new dashboard and alert for every incident that occurs which wastes significant amounts of engineering effort. Either a team will resolve the underlying causes of the incident (rendering the alerts useless), or the team will eventually be unable to cope with the number of dashboards and incoming alerts, leading to alarm fatigue. Building metrics and alerts at the expense of addressing underlying engineering issues will always result in a dysfunctional team and terrible services.

Unfortunately, most vendors of observability products place too much emphasis on marketing pretty graphs and pictures, and too little on providing engineers with the tools needed to effectively understand their services in production.

Fight me.

Metrics, Alerts, Dashboard on AWS

This pillar is primarily serviced as part of the CloudWatch suite on AWS. CloudWatch includes the ability to use metrics that are natively generated by AWS services and custom metrics that are created via the CloudWatch API. CloudWatch additionally allows users to create alarms when certain thresholds on these metrics are met. Whilst alerts are not created out of the box, many services automatically record sets of metrics.

API Gateway is a good example of a service that has good default metrics collection out of the box - under specific circumstances. API Gateway collects metrics on each route and method e.g. /mypath for method POST. This does require that you configure separate paths in API Gateway; building a lambda-lith, as we have done with express, does not capture this level of detail because we delegating the handling of routing to the lambda function. Generally, the more configuration that is captured at the platform layer, the more information is available to AWS, and hence AWS can provide more out-of-the-box.

The following is an example of metrics and dashboards that are provided out-of-the-box for the API Gateway service. They must be enabled by setting 'DetailedMetricsEnabled' in CloudFormation, or setting it via an API.

I am of the opinion that you should avoid custom instrumentation as much possible and allow the platform to take care of it. For AWS that will generally mean configuring your application through platform-level configuration (e.g. CloudFormation) as much as possible. The less code you need to write the better. Other platforms offer similar capability at the code-level but this is usually limited to virtual-machine based languages like Java and C#, where tools are capable of injecting hooks into the language runtime based on intimate knowledge of particular frameworks. This is a bit harder to do in languages like Go and Rust (and even Python and Javascript), and usually restricts developers to a handful of frameworks. I personally find the AWS approach far more flexible.

Custom metrics and the embedded metric format

Whilst you can get some great metrics out-of-the-box, the reality is that you may have a need for something else. This is particularly true of business-level metrics. After all, there is no way for AWS to know what business value your application is providing so I wouldn't expect any future releases to include them. There are two ways to create custom metrics;

Use the CloudWatch metrics API, or,
Use the CloudWatch Embedded Logs Format.

The former used to be the only way to create custom metrics. This involved needing to make a call to the API at some point in your code. A big disadvantage of this is that you need to account for failure of the API call, and it's impact upon your system. For example, a failing API call to create a custom metric should not result in the failure of a customer ordering a book. Just because you can't record the metric isn't a reason to fail a customer order.

Your other alternative is to use the CloudWatch Embedded Logs format. This is the best option if you are already shipping logs to CloudWatch, either via native integration (e.g. because you are using Lambda) or the CloudWatch Logs agent. By structuring your logs in a specific JSON format, CloudWatch will parse your logs for metrics that are embedded within your log messages; creating and recording metrics automatically. This does mean that you need to move to a JSON-based structured logging format, but if you are not currently structuring your logs this is good step to take.

AWS has released a few libraries to make using the embedded format a little easier. The library for node.js (which includes TypeScript definitions) is available here.

The structure of embedded format is fairly similar to the API calls you would have made using the PutMetrics call, so I'll stick to explaining just the embedded format. Each log message is limited to 256kb. Each log message must be in JSON format and include the following node at the root of the document.

{
    "_aws": {
        "TimeStamp": 1559748430481, // A timestamp in ms time (unix epoch time)
        "CloudWatchMetrics": [] // Metadata about the metrics
    }
}

'CloudWatchMetrics' is an array of metric definitions objects. This object includes;

'Namespace'

The 'grouping' for the following metrics. This will generally be the type or name of the application.

Dimensions

Dimensions are typically what you would 'slice-and-dice' on. Dimensions is an array of DimensionSets. This is potentially the most confusing part to wrap your head around, so let's try an example.

Say you were measuring response times for requests that resulted in various status codes. Status codes would be an appropriate dimension. But you may also want to group successful codes (200) together. Maybe it is also worth grouping codes as 'Success' (for 200's) and 'Failure' for everything else?. We have a dimension set that looks like '[Status, Status_Code]'. Every dimension in the dimension set forms a unique combination. If we had two items in the set for Status (Success, Failure), and eight for status code (200, 201, 404, 403, 401, 501, 502, 503), this would result in sixteen custom metrics created. As you are billed by the custom metric, this can be important to keep in mind.

You usually would not use a high cardinality attribute (e.g. something with lots of unique values) as a metric dimension. This would include things like user identifiers or request identifiers. Each unique dimension in DimensionSet results in a corresponding unique combination with all the other dimensions in the set, and an associated custom metric. This has the potential to get quite expensive, so be conservative with the number of dimensions within a DimensionSet that you define.

Metrics

Metrics define the name and units of particular metric that you are recording. This is metadata about the values you are recording elsewhere in the structure. E.g. you may declare that there will be an entry with name 'Time' with units 'Milliseconds'. Units are strictly optional and must be of a particular pre-defined datum.

The CloudWatch metrics array is essentially calling out definitions of various attributes that are at the root of your JSON logging message. The following is the example given in the documentation.

{
  "_aws": {
    "Timestamp": 1574109732004,
    "CloudWatchMetrics": [
      {
        "Namespace": "lambda-function-metrics",
        "Dimensions": [["functionVersion"]],
        "Metrics": [
          {
            "Name": "time",
            "Unit": "Milliseconds"
          }
        ]
      }
    ]
  },
  "functionVersion": "$LATEST",
  "time": 100,
  "requestId": "989ffbf8-9ace-4817-a57c-e4dd734019ee"
}

In this example, we can see 'functionVersion' has been called out as dimension in the array, with a corresponding metric of 'time' of unit milliseconds. The requestId will be more or less ignored, as it is a simple logging attribute. The 'PutMetrics' API call follows more or less the same structure, it just wouldn't include any simple logging attributes.

Kicking The Tyres

Code is available here.

We will adjust our logging by installing the node logging library that AWS provides that conforms to the embedded metrics specification.

npm install --save aws-embedded-metrics

Like before, we will import some functions from the library

const { createMetricsLogger, Unit } = require("aws-embedded-metrics");

and we'll configure the library by adjusting some middleware.

app.use((req, res, next) => {
    req['segment'] = xray.getSegment();
    const logger = createMetricsLogger();
    logger.setNamespace("simple-node-api");
    logger.setProperty("RequestId", req.headers["x-request-id"])
    req['logger'] = logger;
    next();
});

Here I have created a new namespace ("simple-node-api"), and added a property to record the request identifier.

OK, great. Now we need to record a metric. For the purposes of demonstration, I'll record the time taken for the authorization middleware to make pass/fail decision. I'm obviously already getting that from X-Ray, but this is purely for demonstration purposes. The middleware now looks like this...

app.use((req, res, next) => {
    const { headers, segment, method, logger, path: obj } = req;
    xray.captureAsyncFunc('Auth Middleware', subsegment => {
        const token = headers['authorization'].replace("Bearer ", "");
        const decoded = jwt.decode(token, { json: true });
        const { sub } = decoded;
        const groups = decoded['cognito:groups'] || [];
        const act = methodToAction[method];

        req.logger.setProperty("subject", sub);
        req.logger.setProperty("object", obj);
        req.logger.setProperty("groups", groups);
        req.logger.putDimensions({ "action": act});

        const currentTime = new Date().getTime();

        rbac.addRolesToUser(sub, groups).then(() => {
            rbac.enforce(sub, obj, act)
                .then(pass => {
                    subsegment.close();
                    if (pass) {
                        req.logger.putDimensions({ "Authorization": "success" })
                        req.logger.putMetric("evaluationTime", new Date().getTime() - currentTime, Unit.Milliseconds)
                        // the call to 'flush' will log out the message
                        req.logger.flush().then(() => next())
                    } else {
                        req.logger.putDimensions({ "Authorization": "failure" });
                        // the call to 'flush' will log out the message
                        req.logger.flush().then(() => res.status(403).json({message: "Forbidden"}))
                    }
                })
        }).catch(() => subsegment.close());
    }, segment);
});

The first thing to occur is setting various properties to record the subject, object and group. I'll make a dimension out of 'action' which is only set to read or write, and therefore is not a high-cardinality attribute. I take the current time, and when evaluation has finished, I can record the time it finished. I then record the difference in time as a metric. The metric will have a dimension to indicate whether it succeeded or failed.

The output in the CloudWatch logs will look like the following...

{
    "LogGroup": "simple-node-api-ExpressBackend-V53ZHQ8TGB1Y",
    "ServiceName": "simple-node-api-ExpressBackend-V53ZHQ8TGB1Y",
    "ServiceType": "AWS::Lambda::Function",
    "action": "read",
    "Authorization": "success",
    "RequestId": "KgRJujF0SwMEPLQ=",
    "subject": "0348f283-442b-4e5c-a9a8-da6d3f284ea9",
    "object": "/",
    "groups": [
        "writer",
        "deleter",
        "reader"
    ],
    "executionEnvironment": "AWS_Lambda_nodejs12.x",
    "memorySize": "128",
    "functionVersion": "$LATEST",
    "logStreamId": "2020/04/05/[$LATEST]8514dba7bc7d4a8bbb48505f02ad6380",
    "traceId": "Root=1-5e899571-26ba38ebe8846762aedb813e;Parent=dc867b62be8a635d;Sampled=1",
    "_aws": {
        "Timestamp": 1586074994255,
        "CloudWatchMetrics": [
            {
                "Dimensions": [
                    [
                        "LogGroup",
                        "ServiceName",
                        "ServiceType",
                        "action"
                    ],
                    [
                        "LogGroup",
                        "ServiceName",
                        "ServiceType",
                        "Authorization"
                    ]
                ],
                "Metrics": [
                    {
                        "Name": "evaluationTime",
                        "Unit": "Milliseconds"
                    }
                ],
                "Namespace": "simple-node-api"
            }
        ]
    },
    "evaluationTime": 241
}

Once the code is deployed through sam build && sam deploy --guided, we can send some requests off which should make the metrics appear in the console. A script like the following can help with that.

#!/bin/zsh

# Your API endpoint address is available from the output of your deployment
ENDPOINT=https://otax9va024.execute-api.ap-southeast-2.amazonaws.com

# Aquire a token through your cognito endpoint
TOKEN=''

# this should return nothing e.g. {"Items":[]}
curl $ENDPOINT

# now send some data
for i in {1..10}; do
    curl -XPOST -H "Content-Type: text/plain" -H "Authorization: Bearer $TOKEN" -d "Message: $i" $ENDPOINT
done

curl -H "Authorization: $TOKEN" $ENDPOINT

In the console we can find the metrics that we previously defined.

We can graph metrics with various kinds of aggregations including averages...

or percentiles...

We can also define alarms...

when certain conditions are reached...

and preview them against recent metric collection.

Simples!

Conclusion

In this article we took a look at the last pillar of observability: metrics. We discussed various ways metrics are used out in the real world, and how they compare in usage to logging and tracing. We then look at ways we can use metrics in AWS, either through in-built metrics provided by the platform, or by defining custom metrics. We defined our custom metric in the serverless express app using the CloudWatch embedded logs format through an AWS provided logging library. We then viewed this metric in the console, and saw how to set up alarms on it.

'Everything-on-Fire' becoming 'Business-as-Usual'? Contact Mechanical Rock to Get Help!

Why You Working So Hard?

Matt Tyler — Sun, 05 Apr 2020 11:51:28 +0000

I was about 22 and just starting out with ZZ Top.

I was in the dressing room and BB King said to me, ‘Can I play your guitar?’

'Sure man.’

He strummed it a few times and handed it back to me. He looked at me rather quizzically and said,

‘Why you working so hard?’

‘What do you mean?’

'Those strings. You got real heavy, heavy strings.’

‘Well, isn’t that how to get the heavy, heavy sound?’

He said, ‘No! Don’t be working so hard!’

Billy Gibbons

You wouldn't think the world of music would offer much in the way of advice to the world of programming, but I find myself coming back to this story fairly frequently. It's a good example of how perception begins to take over and lead you astray.

In Gibbon's case, it was the accepted knowledge that to sound good, you needed to use heavy, thick guitar strings. An encounter with BB King, who by that time was already a living legend, lead Billy to try using lighter strings and work out things for himself.

In this story, Gibbon was making things difficult for himself based on his own perceptions and those of people around him. I think we very prone to doing this in technology, perhaps more so. We see this manifest in various different ways;

We over-engineer things in case we need something in future.
We gate-keep, saying that 'real' developers run their own infrastructure, or only use the terminal, or use .
We try to justify the difficulty we have in implementing the solutions we want to build.
We fight with people who are trying to help us do things a better way.
We disregard the solutions of others because we feel we know better, EVEN if we know the simple solution is better.
We sit in our comfort zone and refuse to learn new things, dismissing them out-of-hand rather than trying them out and forming our own opinion.

I've come to serverless primarily out of a means of dissatisfaction with the rising complexity that is elsewhere in software development. I want small, solutions that I can understand within their specific context, that I can compose together to solve problems. I don't want to worry about a whole host of things I used to worry about - because I just don't have time to do that anymore!

I asked myself, 'Why you working so hard?', and chose serverless.

Whenever I'm knee-deep in, and starting to get tangled up in my own code, I ask myself, 'Why are working so hard?', and try to find a simpler solution.

Why are you working so hard?

Re:Invent Yourself and Your Company With Serverless

Matt Tyler — Fri, 27 Mar 2020 13:47:40 +0000

Please reach out to me on Twitter @MatthewTyler13 if you have any questions!

Introduction

I became a father for the first time recently. Turns out your hands are pretty much full for a lot of the time, rocking an upset infant back-and-forth in an attempt to get them to settle. Watching all the re:invent content I wasn't able to attend when I was at the conference seemed like a good way to fill in the time. I thought I'd put together an expansive list of my favourite videos from the last couple of re:invents. These sessions run a bit of a gamut; some are what I consider base-line knowledge, some are advanced use-cases, some are just interesting. Generally, they focus around the serverless ecosystem and topics that I think developers needs to know to become expert serverless practictioners.

I've broken these up into certain 'focus areas'. 'API & Mobile' is sort of self explanatory, but I've also included videos that are related to specific frameworks. 'Serverless Transformation' includes videos that are primarily targeted at decision makers, and convincing them to give this serverless thing a go. 'Serverless Backend Architecture' focuses more on stitching different services together to create robust services. 'Data Management' focuses on the services used to persist data in your applications. 'CI/CD' focuses on the tooling and techniques used to releases serverless applications to your end-users. Finally, I've included a bunch of videos aimed at upping your security game.

All the videos include a 'code' that has a number in it. These videos are in the 200, 300, and 400 ranges. 200 videos tend to be for beginners, 300 for intermediate, and 400 for advanced use cases.

I've compiled this list into a YouTube playlist, for easy viewing.

API & Mobile

(SVS402) Building APIs from front to back

This is a great presentation by Eric Johnson, in which he builds out a simple application using "serverless-first" features in API Gateway. It's a great example of how you can build an API without writing lambda functions.

(SVS212) I didn't know Amazon API Gateway did that

Once again from Eric Johnson, this goes over a bunch of features that API Gateway has that don't always see a lot of use from people new to serverless. It's a thorough overview of the product - so if you have only ever used API Gateway by proxy (e.g. through Serverless Framework) it's worth a watch.

(NET310) Building serverless micro-frontends at the edge

This goes over CloudFront and Lambda@Edge, focusing on delivering content closest to users. For the longest time, this has been the only "edge-compute" functionality in AWS, but last year we saw an increasing number of releases around bring AWS closer to users. I'm referring to Outposts, Wavelength, and Local Zones which were all released at re:invent 2019. It's a good time to become more familiar with edge concepts, as it is probably going to become increasingly important in mobile and IoT applications.

(MOB402) Data-driven mobile and web apps

This is the sessions that contains information on Amplify Datastore. Amplify Datastore is Amazon's approach to handling offline sync for mobile applications, built into the Amplify Framework. It's a very flexible approach that should be ultimately should see some interesting things being built with it. Offline sync is particularly important when using mobile applications in remote areas, or places with unreliable network connections.

(MOB306) Amplifying fullstack serverless apps with AWS AppSync

This is a good overview of what is capable with AppSync - a managed graphQL service. It's great for federating from multiple data sources, and this can help to accelerate development of web and mobile applications.

(MOB308) Production-grade full-stack apps with AWS Amplify

Amplify is a framework for building web and mobile applications that is tightly integrated with the AWS ecosystem. In the future I imagine most application development will be frontend development connected to specialized managed services. Amplify is the harbinger of that time to come, so come take a look at what the future is going to look like.

(SVS341) An in-depth tour of AWS SAM

AWS SAM is my favourite framework for serverless application development, and this is a decent introduction to it.

(DOP402) Deep dive into AWS Cloud Development Kit

Of course, when things start to get too complicated with SAM, I find the AWS CDK is the cure for it.

Serverless Transformation

(ARC373) Amazon.com: Reducing time to market & TCO using serverless

If you need to convince business leaders on the value of serverless, this is the presentation for you.

(SVS320) The serverless journey of shop.LEGO.com

Sheen Brisals talks about the experience of LEGO's move to serverless. It's cool to learn how larger enterprises have been able to use serverless to deliver better experiences. He was also on Jeremy Daly's Serverless Chats podcast not too long ago.

(SVS242) Selecting your first serverless pilot

The first step in any organization moving to serverless is quite simple: Build Something! But what? This presentation gives some ideas on selecting an appropriate project to build serverless-ly, so you can maximise your learning and chance of success.

(GPSBUS210) Accelerate business innovation with serverless applications

If you need to convince business leaders on the value of serverless, this is the other presentation for you.

(SVS226) From prototype to org-wide serverless adoption: Key considerations

You've had your first successful production pilot, now what do you do? How can you get your organization to adopt serverless on a wider scale? A great overview of the technical considerations and cultural changes that teams will need to adjust to in order succeed with serverless.

Serverless Backend Architecture

(SVS401) Optimizing your serverless applications

Chris Munns takes you on a journey, explaining many of the best practices out there for building serverless applications. This is required viewing - it'll bring you up to speed with the current best practices for serverless circa 2019.

(SVS308) Moving to event-driven architectures

A great presentation from Tim Bray. Event-driven architecture is the only way to deliver scalable applications, and happens to combine nicely with other approaches like Behaviour Drive Development and Domain Driven Design.

(API320) Building event-driven applications with Amazon EventBridge

Amazon EventBridge is up there as one of my favourite services, as it is a powerful tool for building decoupled backend services. This allows me to add new services and improve existing ones in an incremental manner, helping me to deliver software sooner.

(SVS406) Asynchronous-processing best practices with AWS Lambda

Once you've moved to event-driven architectures, you are in a position to take advantage of asynchronous processing. Done well, this enables you to build a considerable amount of resiliency into your applications.

(API304) Scalable serverless event-driven applications using Amazon SQS and Lambda

Queues form the foundation of asynchronous processing, so understanding how to use queues well is critical.

(ARC307) Serverless architectural patterns and best practice

Heitor Lessa goes over the common patterns that many of us see frequently implemented in serverless applications. A useful presentation to watch as it may have a solution to a problem you are currently experiencing.

(ARC349) Beyond five 9's: Lessons from our highest available data planes | (STG331) Beyond 11 9's: Lessons from Amazon S3 culture of durability

Two cool videos on how Amazon builds ultra-reliable services. Less useful for building a service on a technical level, but excellent advice on how manage development and operational teams to deliver critical services.

(SVS405) A serverless journey: AWS Lambda under the hood

Want to learn how AWS Lambda actually works? Watch this video.

Data Management

(CMY304) Data modeling with Amazon DynamoDB

Most of us are more familiar with relational databases like MySQL or Postgres, so using a NoSQL solution like DynamoDB can be a bit of an adjustment. Here Alex DeBrie gives a gentle introduction to working with NoSQL databases.

(DAT403) Amazon DynamoDB deep dive: Advanced design patterns

Rick Houlihan is a database deity. You'll be studying this presentation for years, and it's worth the price of a re:invent ticket alone. If you want to know how to something in DynamoDB, chances are it is in this presentation. If you want to build high performance applications with virtually infinite scalability, throw your RDBMS and hook your brain up to this.

(DAT336) Aurora Serverless: Scalable, cost-effective application deployment

Of course, if your not ready for DynamoDB or it doesn't fit your use-case, you have a good serverless option in Aurora - a serverless RDBMS.

(STG302) Best practices for Amazon S3

Amazon S3 is such a critical service to AWS and the rest of the internet. It might just be plain old storage, but it's ubiqituity in various serverless applictions make it an important service to learn.

(ANT307) Deep dive into Amazon Athena

Serverless analytics is a growth area. Athena is such a cool tool - dump some data into S3 and analyze it. It couldn't be simpler. The whole service is basically dark margic.

CI/CD

(DOP302) Best Practices for authoring AWS CloudFormation

If you are doing any work with AWS, you have probably written a CloudFormation template or two. A lot of people love to complain about CloudFormation, but I still think it is a pretty great tool once you get used to it. One of the challenges to templates is understanding all the features available in authoring templates. This presentation does a good job of showing them off.

(SVS336) CI/CD for serverless applications

If you want to make a rock-solid service, you need rock-solid CI/CD. Learn how to do it for serverless applications.

(DEV319) Continuous integration best practices

The first step to CI/CD nirvana is having good CI. Learn how to get it right the first time so you can create tight, effective feedback loops that will help enable to write better software, faster.

(DEV317) Advanced continuous delivery best practices

Continuous Delivery is a subject area that doesn't get nearly as much attention as CI. Why? Because it's hard.

(DOP404) Amazon's approach to high availability deployment

Now you have learned the best practices, come see how Amazon actually does it. There is a lot of useful information in this presentation that goes along way to explaining why AWS CodePipeline is the way it is.

Security

(SEC209) Getting Started with AWS Identity

Given you are going to need to secure access to various services, your going to need to write some IAM policies. This is a good overview of the IAM service.

(SEC316) Access Control Confidence: Right access to the right things

A more in depth guide to IAM with various examples. It also covers 'Attribute-Based Access Control' which is going to become a more important method of authorization in the AWS ecosystem, as it allows for AWS to source attributes from external identity providers and use those to determine the right level of access. Contains traces of Pickles.

(SVS310) Securing enterprise-grade serverless apps

A good review of different patterns for securing access to serverless applications. If you want to know what your various options are for securing your serverless API's, this a good one to review.

(DOP310) Amazon's approach to security during development

Want to learn how AWS handles security? Hint: it's everyones responsibility.

Starting out with Serverless? Contact Mechanical Rock to Get Started!

8 Tips For Better CloudFormation Templates!

Matt Tyler — Thu, 19 Mar 2020 12:22:33 +0000

Please reach out to me on Twitter @MatthewTyler13 if you have any questions!

Introduction

I've been writing CloudFormation for about half a decade now. A lot of people love to complain about CloudFormation, but to be honest, I still feel it's the best option out there for handling cloud infrastructure. Most of the tools in the AWS ecosystem compile down to it, so having a really good understanding of CloudFormation can still benefit you even if you interface to it via some other means. Along the way, I've picked up some various tips and tricks that make dealing with CloudFormation easier. In most cases, I find the big problem with CloudFormation is writing templates in such a way that other people can understand them. That includes you, the author, when you inevitably need to make some adjustments in six months time.

1. Use !Sub

CloudFormation has a bunch of intrinsic functions, but none is as useful as Fn::Sub when it comes to cleaning up a template. Sub was introduced in 2016, and since then has pretty much replaced all the Fn::Join tomfoolery that was present in a lot of templates. Consider constructing an ARN in IAM trust principle statement. Using Fn::Join might look something like this;

AWSTemplateFormatVersion: 2010-09-09
Resources:
  RootRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              AWS:
              - !Join
                - ""
                - - "arn:"
                  - !Ref AWS::Partition
                  - ":iam::"
                  - !Ref AWS::AccountId
                  - ":root"
            Action:
              - sts:AssumeRole
# ...

Using !Sub is a lot cleaner.

AWSTemplateFormatVersion: 2010-09-09
Resources:
  RootRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              AWS:
              - !Sub "arn:${AWS::Partition}:iam::${AWS::AccountId}:root"
            Action:
              - sts:AssumeRole
# ...

You can also take advantage of !Sub when build out step functions definitions. This would be a complete nightmare without Sub. Consider this template;

 Publisher:
    Type: AWS::StepFunctions::StateMachine
    Properties:
      DefinitionString: !Sub |
        {
          "StartAt": "FindProduct",
          "States": {
            "FindProduct": {
              "Type": "Task",
              "Resource": "${FindProduct.Arn}",
              "Next": "Fork",
              "Retry": [{
                "ErrorEquals": [ "States.ALL"]
              }]
            },
            "Fork": {
              "Type": "Choice",
              "Default": "AmbiguousProductName",
              "Choices": [
                {
                  "Variable": "$.products.count",
                  "NumericEquals": 0,
                  "Next": "CreateProduct"
                },
                {
                  "Variable": "$.products.count",
                  "NumericEquals": 1,
                  "Next": "CreateVersion"
                }
              ],
              "OutputPath": "$.candidate"
            },
            "AmbiguousProductName": {
                "Type": "Fail",
                "Cause": "Product name belongs to multiple products",
                "Error": "ErrorAmbiguousProductName"
            },
            "CreateProduct": {
              "Type": "Task",
              "Resource": "${CreateProduct.Arn}",
              "End": true,
              "Retry": [{
                "ErrorEquals": [ "States.Timeout" ]
              }]
            },
            "CreateVersion": {
              "Type": "Task",
              "Resource": "${CreateVersion.Arn}",
              "End": true,
              "Retry": [{
                "ErrorEquals": [ "States.Timeout" ]
              }]
            }
          }
        }
      RoleArn: !GetAtt StatesExecutionRole.Arn

In the above template, I've used the !Sub function to substitute in the correct ARNs for each function that has been defined elsewhere in the template. As an aside, multi-line string support in YAML is god-send for resource parameters that need to take JSON strings, as is the case with some properties within Step Functions and EventBridge.

2. Using Lists of ARNs

This doesn't mean that other pseudo functions like !Split and !Join have been supplanted by !Sub. There are still some cool tricks you can do. Imagine you wanted to pass a list of Account Identifiers, that needed to be expanded into a resource policy. e.g.

Parameters:
  AccountIds:
    Type: CommaDelimitedList

Ideally we want to turn '01234567890,1111111111,2222222222' into:

 - "arn:aws:iam::01234567890:root"
 - "arn:aws:iam::11111111111:root"
 - "arn:aws:iam::22222222222:root"

We can do this with one weird trick. Imagine we needed to give a set of accounts access to a KMS key.

  KmsKey:
    Type: AWS::KMS::Key
    Properties:
      KeyPolicy:
        Id: KeyPolicy
        Statement:
          - Sid: Root Permissions
            Effect: Allow
            Principal:
              AWS: !Sub 'arn:aws:iam::${AWS::AccountId}:root'
            Action: kms:*
            Resource: '*'
          - Sid: Target Accounts
            Effect: Allow
            Principal:
              AWS: !Split
                - ','
                - !Sub
                  - 'arn:aws:iam::${inner}:root'
                  - inner: !Join
                    - ':root,arn:aws:iam::'
                    - !Ref AccountIds
            Action: 
              - kms:Decrypt
              - kms:DescribeKey
            Resource: '*'

The magic is happening here:

  Principal:
    AWS: !Split
      - ','
      - !Sub
        - 'arn:aws:iam::${inner}:root'
        - inner: !Join
          - ':root,arn:aws:iam::'
          - !Ref AccountIds

To understand it, we start from the inner most function.

The first join statement ('inner') will create the following string,

'01234567890:root,awn:aws:iam::1111111111:root,arn:aws:iam::2222222222'

This string is now substituted into the first parameter of the !Sub function.

'arn:aws:iam::01234567890:root,awn:aws:iam::1111111111:root,arn:aws:iam::2222222222:root'

Which will then be split along the commas into the following.

['arn:aws:iam::01234567890:root','awn:aws:iam::1111111111:root','arn:aws:iam::2222222222:root']

Pretty cool!

3. Use SSM Parameters

There are myriad of parameter types in CloudFormation. The SSM types are my favorites, in particular AWS::SSM::Parameter::Value<String>. You might use it like this:

Parameters:
  TargetAccountId:
    Type: AWS::SSM::Parameter::Value<String>
    Default: '/<some-prefix/default'

I'll often create reusable templates which can be composed together. One template can create an SSM parameter that can be resolved in another template. If the parameter doesn't exist, CloudFormation will throw an error. This can act as a prompt to ensure the other template is instantiated first.

You could also StackSet deploy a set of 'base stacks' that include a bunch of SSM parameters to configure accounts in a default manner. Then other templates can use those parameters to build upon them. For example, your organization might give teams a set of accounts for non-production and production workloads. They could deploy SSM parameters that define those accounts - and any pipeline stacks can use SSM parameters to resolve to the correct targets.

I'll also often use SSM parameters to replace a lack of environment variables in Lambda@Edge environments. To do this, I'll instantiate parameters that use the distribution ID of the CloudFront distribution. I can then pull information from SSM parameters that are prefixed using the distribution ID.

4. Prefer default template parameters instead of hard-coding magic strings

If you need store a reference to something that is likely only ever going to be one value, it can be tempting to hard-code that value within the body of a template. I find that this can be confusing for those that may come to template and wonder what the source of the value is. A good example of this can be found within CloudFront - you may need to use the value 'Z2FDTNDATAQYW2', which is the default Hosted Zone ID for CloudFront distributions, and is required when setting up Route 53 Alias for CloudFront. Now, you could embed this in your template, but if your unfamiliar with this value, it'll likely confuse the next person who sees it in the template. There's also the unlikely chance it may change, but it's worth guarding against.

So I'll often do this;

  CloudFrontHostedZoneId:
    Type: string
    Default: Z2FDTNDATAQYW2
    AllowedValues:
      - Z2FDTNDATAQYW2
    Description: >-
      The Default Hosted Zone for CloudFront Distributions

This way it has;

A relevant default value,
With only one allowed value (so no-one changes it unknowingly), and
A description about what it is for.

You could also add a link to the documentation that explains the value in the description.

5. Use Parameter Metadata

Parameters, where possible, should be documented and grouped together in order to make it easier for users to fill in parameters. This really only comes into play when using the console to launch a template - so if you are fully automated the benefits are more in documentation. But if you are building templates for things like tutorials, or providing simple infrastructure for someone to stand up via something like AWS Service Catalog, it can help certain demographics to provision bits of infrastructure. Here is a sample that demonstrates this;

Parameters:
  Name:
    Type: String
    Description: >-
      The name of the application. 
      This should include a stage qualifier if this is a nonproduction instance.
      Stage qualifiers should only include 3-5 characters
    AllowedPattern: ^[a-z]{3,}(-[a-z]{3,5})?$
    MinLength: 3
  Domain:
    Type: String
    Default: app
    AllowedValues:
      - example.com
      - otherdomain.com
    Description: The domain for the application.
  SigningKeyParameter:
    Type: 'AWS::SSM::Parameter::Name<String>'
    Default: /system/signing-key/default
    Description: >-
      The name of SSM parameter from which to retrieve the cookie signing key
    AllowedValues:
      - /system/signing-key/default
  SigningKeyParameterKeyId:
    Type: string
    Description: >-
      The ID of the KMS key used to encrypt signing key in SSM Parameter Store.
  ClientId:
    Type: String
    Description: The Application Client ID that has been assigned to the application
  CloudFrontHostedZoneId:
    Type: string
    Default: Z2FDTNDATAQYW2
    AllowedValues:
      - Z2FDTNDATAQYW2
    Description: >-
      The Default Hosted Zone for CloudFront Distributions
Metadata:
  'AWS::CloudFormation::Interface':
    ParameterGroups:
      - Label: 
          default: Authentication
        Parameters: 
          - ClientId
          - SigningKeyParameter
      - Label: 
          default: Application Domain
        Parameters:
          - Name
          - Domain
      - Label:
          default: System
        Parameters:
          - CloudFrontHostedZoneId
    ParameterLabels:
      SigningKeyParameter:
        default: Signing Key Parameter
      ClientId:
        default: Client ID
      CloudFrontHostedZoneId:
        default: CloudFront Hosted Zone ID

A set of parameters is defined, and then we use the 'Metadata' section to group-like parameters, and order them. In this case, we create a few groups; an authentication group, an application domain group, and system parameters groups. This gives users hints as to the purpose of each parameter.

6. Nested Stacks

Nested stacks are a good way to split templates up into components. This has a few benefits;

Templates start to form building blocks, which enables reuse.
Large templates are split up into chunks that can make it easier to understand what is going on.
It can help with template authorship - splitting templates into chunks forces you to do a lot of thinking up front. This can help prevent you get lost in a million lines of YAML.
It can help you get around the 200 resource limit.

7. Try SAM! - Nested Applications but be aware of limitations

The serverless application model is superset of CloudFormation that is designed to build and deploy serverless applications. It does this via some additional resources and a CloudFormation macro. Using it can make it easier to build and deploy serverless applications; the behavior of deploying a new template SAM is pretty close to what developers expect. Defining resources using SAM-specific resources is typically less verbose than the equivalent template defined in plain CloudFormation.

There are some limitations;

You can't StackSet deploy serverless applications (although you can if you include them in a nested template, called from the main template - go figure)
Some resources in a SAM template must be included together - this includes any resource that references other 'serverless' resources that are defined in the template. Serverless::Functions that reference a RestApiId from a Serverless::Api are a good example of this.
You need to be aware that the serverless transform is a macro - as such, if you don't want to or can't create and approve a changeset, you will need to use the CAPABILITY_AUTO_EXPAND to deploy a template without reviewing the changeset. This is particular relevant to those who are deploying templates via AWS Service Catalog.

8. Don’t name things! - use tags if you must

There are loads of resources in CloudFormation that cannot be updated or replaced when names are defined for them; S3 Buckets being noticeable examples. When possible, prefer auto-generated names. If some semantic naming must be given to a particular resource, I would generally give them an appropriate tag instead. The auto-generated names can be provided via CloudFormation outputs. If you are finding it difficult to work with auto-generated names that need to be provided to other stacks, consider creating SSM parameters under an application specific prefix, and storing the auto-generated name inside it. The next template can then reference the name using an SSM parameter input.

I often find the naming of various resources can be become very contentious in large enterprises who have decided on specific naming conventions for different resources. This is largely because systems in the past have not had robust mechanisms for dealing with attribute metadata. In these instances,it is worth putting in the effort to push back and insist on using a lightweight labeling mechanism like tags. I have seen enterprises insist on duplicating information in names (stuffing information about the region inside a resource name), and in doing so exhaust the character limit of some resource names. This results in using various abbreviations inside a resource name, to the point at which the names start to like look randomly generated names anyway. Fight hard to avoid this silliness if you can, early.

Conclusion

I hope you enjoyed my tips for writing CloudFormation. Come back in a few weeks time, in which I'll enumerate a few more tips for writing better templates!

Infrastructure-as-Code causing you pain? Contact us to get started!

How to Add Observability to Your Serverless HTTP API on AWS

Matt Tyler — Wed, 11 Mar 2020 12:57:50 +0000

Please reach out to me on Twitter @MatthewTyler13 if you have any questions!

Introduction

If you have been following along, you will have a built an extremely simple API, auto-generated client code, and provided ways to authenticate users and authorize their requests. In this installment we will begin to take a look at how to determine the operational health of our API. In particular, we begin to look at our observability options for APIs hosted in AWS. The zeitgeist defines three pillars of observability - logging, tracing, and metrics. For the sake of brevity we will focus on logging and tracing. We will leave the third pillar, metrics, for a future post.

The goal here is to give you enough insight into the health and behaviour of your API. Without this information it is difficult/impossible to diagnose the source of errors in your application. The performance of your application is critical and you may find that varying types of load cause your application to behave differently. Logging and tracing can help you to triage and diagnose errors and bottlenecks in your code which results in a better experience for consumers of your application.

Logging

If you have ever written 'console.log' you will be familiar with logging. Printing out to the console is often one of the first things we learn, as part of the canonical 'hello world' example. Beyond that, most of us will write out to the console the moment we have an issue in our programs. That said, there are many things we can do to write better logs.

The first, and main thing we can do it to improve our logs is to introduce something popularly called 'structured logging'. This primarily means settling on a standard format in which to log. This is not just limited to the 'shape' of the output, which include lines in JSON or some other format, but typically includes what various attributes should be included in the output.

A list of outputs that may be included for a typical structured logging line for a HTTP API, may include:

The date-time of the log entry
The log 'level', which may include,
- Error,
- Info,
- Debug, etc.
Header attributes, which may include,
- Trace ID
- User-Agent
The route of the API
The method of the API
The actual log message

Be careful not to include secret material or personally identifiable information (PII) into your logs. There exist various ways to scramble this information to prevent running afoul of this, whilst still retaining enough information to be useful.

The more information and attributes you able to log, the more likely the information will be useful in some capacity. Events with a lot attributes (resulting in lot 'unique' entries) are often referred to 'high cardinality events'.

Tracing

Tracing is relatively new when compared to the now ancient practice of logging, and has come about as a means to understand actions rippling through distributed systems. In a monolith, the system has the complete context of the entire system at any one point in time. In a microservices architecture this is no longer true as the entire state of the system may be spread throughout many different services. Simple logging will no longer help us in understanding an event or action as it propogates through the system.

Tracing offers a deceptively simple solution to this problem; begin by adding what is called a 'correlation identifier' or 'trace-id' to every request/response, and propogate this through the system. If one of your services makes a call to another service, it should continue to pass this identifier through to the other service, and so on and so forth. Each service should log this information correlation ID out in addition to everything else it was already logging. If the logs for all services are then centralized, it is possible to use the correlation ID to construct a complete map of how a request propogated through the system.

Tracing is usually very request-response orientated and includes the time taken for each request-response cycle. This makes tracing very powerful in identifying bottlenecks and performance degradation in a system.

There exist many different standards/implementations for tracing, which have included OpenTracing, Jaegar, AWS X-Ray etc.

When to Log vs When to Trace?

It can be very difficult to distinguish when you should be logging versus when you should be tracing. There is an incredible amount of overlap between the two, so I'm going to limit myself to discussing what to do within the context of a serverless application running on AWS. There will be a fair amount of 'opinion' going on here so of course, take everything with a grain of salt and be prepared to use your own judgement.

We'll start with tracing, because there is one clear advantage that is built in with X-Ray that you do not get with your own homegrown logging/tracing solutions, and this service integration. X-Ray is integrated with many different AWS services, and this gives you information in your traces that you will simply not be able to get any other way. An example of this is dynamodb service integration, in which the traces produced by the dynamodb service will include a lot of useful information about query performance. If you are building with serverless best practices which would include usage of as many managed services as possible, it would be senseless not to take advantage of this.

In addition, various AWS services will happily propogate your X-Ray trace IDs. This will enable you to create maps of request as they propogate through your services. Unfortunately, not every AWS service will propogate trace ID's (event bridge being a notable example), and creating traces across account boundaries is a laborious process. If you have separated your services into different accounts, you may find 'logging' your traces or using a third party tracing service is necessary.

Which brings me to logging. I generally prefer to keep logging pretty simple. I will generally:

Log the request as soon it is received, to give me enough context as to what 'started' the particular transaction.
Log any information that would indicate branching logic, if it occurs. e.g. if, else, case statements.
Log any unhandled errors that might occur, e.g. those I allow to flow up to the handler. If something is caught and recovered from - I consider that branching logic.

Most Lambda function runtimes will generally log any errors that are thrown from the handler. However, this does use the standard logging format provided by the specific runtime. If you have your own standard output format, you will probably want to catch any errors and log them. You will still probably want to rethrow those errors (thereby logging the output error twice), to ensure the lambda platform recognises that an error has occurred. This is particularly important if you are invoking lambda functions asynchronously and require the retry semantics that it offers.

Some of this will depend on how you structure your lambda code. I try to branch as little as feasibly possible, deferring to step functions for logic control/flow if it is required. As a result of this, it's not uncommon to see only two logging statements in my code. I usually do not bother to log a statement that indicates success of a particular function, unless I need this information for constructing metrics. A lot of services have reasonable support for constructing this without needing to me explicitly console.log (like API Gateway), so this is not a step I would usually need to take for those scenarios.

However, if your organization specifies that they want you to log certain things in certain ways, I'd usually go with that even if it seems redundant. At the end of the day if your organization has invested in specific tooling you may be following the path of least resistance in doing so. Consider your own context when making decisions.

Understanding the Defaults

Before we get into custom logging and tracing, let's investigate what is provided out-of-the-box and enable that.

Logs are provided in two locations in CloudWatch.

API Gateway logs are provided in the log group API-Gateway-Execution-Logs_{rest-api-id}/{stage-name}.

If you have deployed the API in this guide, your stage name will likely be '$default'. This doesn't seem to be configurable via CloudFormation yet for HTTP APIs - but you can configure it via the console.

Logs for your lambda function are provided at '/aws/lambda/'.
If you are making API calls to AWS services - you will find a fair amount of things logged in CloudTrail
There's addition logging for Cognito as well if you happen to need it. Which you may, if you concerned about specific logs around authentication.

Some tracing features are available out of the box; We previously enabled lambda X-Ray tracing through the use of the following snippet in the template.

Globals:
  Function:
    Timeout: 3
    Tracing: Active

Specifying this in the 'Globals' section of the template ensure that tracing is enabled for all lambda functions. The trace header identifier will be populated into an environment variable called '_X_AMZN_TRACE_ID', that you can access within the lambda function. This trace ID will need to be used if to instrument downstream calls to other services, AWS or otherwise.

Add More Logging

If we were to view the logs for our existing API there would not be much additional information in the logs, particularly for successful invocations. Lets see what we can do to improve that.

We'll start by installing a structured logging library. For this, we'll use pino. Pino will output our logs in JSON format. Assuming you are in the directory of app.js file, executing the following command to install and save pino as a runtime dependency.

npm install --save pino

Next, we will need to configure Pino.

const newLogger = require('pino');

app.use((req, res, next) => {
    // We attach it to the request object to be used later
    // This creates a new logger per request.
    req['logger'] = newLogger(); 
    next();
});

We can use the logger in other places in the codebase, e.g. In our authorization middleware.

app.use((req, res, next) => {
    const { headers, method, logger, path: obj } = req;
    const token = headers['authorization'];
    const decoded = jwt.decode(token, { json: true });
    const { sub } = decoded;
    const groups = decoded['cognito:groups'] || [];
    const act = methodToAction[method];
    // By overwriting the logger, we can propogate the sub, obj, act
    // variables after the middleware has exited
    req.logger = req.logger.child({ sub, obj, act, groups })
    rbac.addRolesToUser(sub, groups).then(() => {
        rbac.enforce(sub, obj, act)
            .then(pass => {
                req.logger.info("Evaluating Access");
                if (pass) {
                    req.logger.info("Access Allowed");
                    next()
                } else {
                    req.logger.info("Access Denied");
                    res.status(403).json({ message: 'Forbidden' });
                }
            })
    })
});

This results in logs that look like the following:

Most centralized logging systems will have no issues ingesting, parsing and searching over JSON structured logs.

Keen eyes will notice that the 'sub' field is not particular instructive as to the identity of the caller. Sometimes that can be a handy feature - it's removed a way to personally identify anyone from the log entry alone. Within certain industries, countries, etc, it can be particularly important to keep sensitive or personally identifiable information (PII) out of your logs.

How to Trace?

We start by installing the xray SDK for node. I needed to install the experimental branch which has support for async/await syntax. This will probably be merged soon.

npm install --save aws-xray-sdk@^2.5.0-experimental.1

XRay can be configured in a myriad of different ways, but this is the way I found simplest.

const xray = require('aws-xray-sdk');

const app = new express();

app.use((req, res, next) => {
    req['segment'] = xray.getSegment();
    req['logger'] = newLogger();
    next();
});

This will cause X-Ray to open a new segment per request, and attach it the request object.

Let's try to trace our authorization middleware, by using 'captureAsyncFunc'.

app.use((req, res, next) => {
    const { segment } = req;
    captureAsyncFunc('Auth Middleware', subsegment => {
        const token = req.headers['authorization'];
        const decoded = jwt.decode(token, { json: true });
        const { sub } = decoded;
        const groups = decoded['cognito:groups'] || [];
        const { path: obj } = req;
        const act = methodToAction[req.method];
        req.logger = logger.child({ sub, obj, act, groups })
        rbac.addRolesToUser(sub, groups).then(() => {
            rbac.enforce(sub, obj, act)
                .then(pass => {
                    if (pass) {
                        next()
                    } else {
                        res.status(403).json({ message: 'Forbidden' });
                    }
                })
        }).finally(() => subsegment.close())
    }, segment);
});

This is a fair amount to swallow, but basically...

We get the segment that was opened earlier and attached to the request object.
We open a subsegment to track our authorization middleware. The subsegment is named 'Auth Middleware', and the logic is passed in as second function, and the third argument is the segment to open this subsegmment on.
When are 'done' with the logic, we close the subsegment, which will occur as soon as the rbac functions have finished.

Instrumenting AWS calls is a bit more straight-forward...

function newS3Client() {
    return xray.captureAWSClient(
        new s3({ 
            params: { Bucket: env.get('BUCKET').required().asString() },
        })
    );
}

This will ensure our S3 calls are traced. Note that you do need to ensure that they wrapped in their own subsegment when called. E.g. we need open a new subsegment in our particular route handlers. For example, getMessages will now look like the following;

app.get('/', ({ segment, query }, res) => {
    xray.captureAsyncFunc('Get Messages', subsegment => {
        const client = newS3Client();
        const maxItems = query.maxItems || 20;
        const token = query.token;
        getMessages(client, parseInt(maxItems), token).then(response => {
            res.status(200).json(response);
        }).finally(() => subsegment.close());
    }, segment);
});

This is the bare minimum we need to do start adding useful traces. You can add a lot of additional metadata to traces, but for brevity I've left this out. It's worth looking into.

You can check GitHub for the completed code examples. Let's view the results. I've fired off a few random requests, and we'll inspect the traces to see what we can learn.

The first is a trace map of our service.

This is a the birds-eye view of our service. From this, we can the average time per invocation and average number of calls per minute, for a particular time scale. We can also see the average calls we made to downstream services, represented by the S3 and S3 Bucket nodes at the right side of the map.

You can get a similar map per trace as well, as shown below.

Let's take a look at those calls below the trace map. This is a trace for one of our 'get' calls.

We can learn several things from this trace:

Time spent in the auth middlware pales in comparison to our remote SDK calls (as you would expect).
Most of our time is spent retrieving comments from the bucket e.g. the list call is pretty quick in comparison.
We do our best to fire off all the requests in parallel, but it's not perfect due to node's single thread nature.
It takes a fair amount of time to retrieve even just one comment from the bucket.

Going back to the trace map, we can click on a particular node and view the distribution of response times.

Odds are pretty good a request for a particular item from the bucket is going to take more then half a second. DynamoDB would probably be a better choice - and because we have traced the calls you could make a particular compelling case to change the storage backend.

Conclusion

We took a whirlwind tour of observability, focusing on two of the pillars of Observability: Logging and Tracing. We learned a little about how to configure the default logging and tracing that is inherent in the AWS platform. We then looked enhancing our applications with our own custom instrumentation. We first enabled structured logging in our application, and then showed how to configure tracing in our application using X-Ray. Finally, we had a look at some of the traces in our application and drew some conclusions about how could improve performance.

Starting out with Serverless? Contact Mechanical Rock to Get Started!

Role-Chaining in AWS Made Simple

Matt Tyler — Wed, 04 Mar 2020 13:24:18 +0000

Please reach out to me on Twitter @MatthewTyler13 if you have any questions!

Introduction

At some point in your career using AWS, you'll find it necessary to learn a little about how assuming roles in other accounts works. If your working on a personal account, chances are you used to logging in with an IAM user that you have created for yourself. When you join a company using AWS, it's more likely that they have a multi-account set-up using AWS Organizations with AWS SSO - in which case you will log into a specific account using a role via federated access. Likewise, you are probably used to needing to create roles for various services (like Lambda), and provide a service trust so that the service can use a role.

I've done a lot of control-plane work in my time, and this has required understanding a fair amount about how assuming roles works. A more complicated trick I've had to pull off is building automation that required role chaining - assuming a role into an account and from there, assuming a role into another account. You can think of this as using an account similar to how one would use a jump-box (or bastion host for non-Australians). Most of the time this has been to meet a security policy, delegating permissions management to an account managed by some central authority. This allows that party responsibility for access control, and the ability to closely monitor what is happening.

Assuming a role via the Javascript SDK is relatively simple, but is has become easier in recent times through the addition of a new credential provider in late 2018, known as 'ChainableTemporaryCredentials'. Prior to this, I used my own custom library which allowed me to do perform role chaining. However, my library did not not refresh credentials when they expired; this was less important for me because I tended to only use the library within lambda functions, and not long running compute. 'ChainableTemporaryCredentials' does handle credential refreshing, so it is a better solution than what I came up with.

Before we get into the specifics though, let's discuss a little bit about how role-assumption works in the simple two-account model.

Cross Account Role Assumption

Setting up cross account role assumption can be a little confusing if you have never done it, but it will become second nature the more you do it. It works like this:

Create a role in the target account, that will ultimately be assumed by another account. Give it the necessary permissions to do what will be required of it.
Adjust the 'AssumeRolePolicy' or 'trust', of the target role.

The role will need to have a trust policy like the following;

    {
      "Version": "2012-10-17",
      "Principal": {
        "AWS": [
          "arn:aws:iam::1234567890:root"
        ]
      },
      "Action": "sts:AssumeRole"
    }

By calling out the account root, you are effectively delegating responsibility to the other account to manage who is allowed to access this role. Note however, that you cannot use wildcards in the trust policy, so you either trust the whole account or something more specific

Within the source account, create a role that is capable of assuming the role in the target account

It will require IAM permissions that look like the following;

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": "sts:AssumeRole",
          "Resource": "arn:aws:iam::098765431:role/role-to-assume",
        }
      ]
    }

Imagine we were to assume a role in another account to access S3 from within that context. The following code will assume role using the javascript SDK for this scenario, and provide those credentials to the S3 account. Using plain STS client calls, it looks like the following;

import { S3, STS, Credentials } from "aws-sdk";

const { 
  Credentials: { 
    AccessKeyId: accessKeyId, 
    SecretAccessKey: secretAccessKey,
    SessionToken: sessionToken 
  } 
} = await new STS().assumeRole({
  RoleArn: "arn:aws:iam::0987654321:role/role-to-assume"
}).promise();

const client = new S3({
  credentials: new Credentials({ accessKeyId, secretAccessKey, SessionToken })
});

There's obviously a lot of boilerplate here, primarily because of the changing case of the input and output parameters between the response of the STS call and the credentials object. Removing this boilerplate was my reason for writing my own helper library in the first place. Now that ChainableTemporaryCredentials is around, we get rid of some the ceremony. Check this out;

import { S3, ChainableTemporaryCredentials } from "aws-sdk";

const credentials = new ChainableTemporaryCredentials({
  params: {
    // Any parameters used by STS AssumeRole can be used here eg; RoleSessionName etc
    RoleArn: "arn:aws:iam::0987654321:role/role-to-assume"
  }
});

const client = new S3({ credentials });

Role Chaining

Extending this to a third role that is assumable from a 'middle' role isn't an awful lot different from the example with two roles. We simply add another role, and place a trust on the role in the middle.

Using ChainableTemporaryCredentials we can perform the double-assumption by adding an additional parameter. 'masterCredentials' can be used to specify how the credentials to the top level call should be acquired.

import { S3, ChainableTemporaryCredentials } from "aws-sdk";

const credentials = new ChainableTemporaryCredentials({
  params: {
    RoleArn: "arn:aws:iam::0101010101:role/next-role-to-assume"
  },
  masterCredentials: new AWS.ChainableTemporaryCredentials({
    params: { 
      RoleArn: "arn:aws:iam::0987654321:role/role-to-assume"
    }
  })
});

const client = new S3({ credentials });

Simples! You can probably imagine how ugly it gets when directly using STS calls, hence why I wrote my own library to handle it - But this is a lot better!

Conclusion

We had a brief look at how cross-account role assumption works, and how to set it up in the simple two account case. We showed how to do this STS calls and how the ChainableTemporaryCredentials provider in Javascript SDK makes this easier. Then we added a third role, and showed how to perform role chaining via the credential provider. We gained an appreciation for how this makes the entire process simpler!

IAM got you feeling chained up? Contact Mechanical Rock to Get Started!

How to Add Role-Based-Access-Control to Your Serverless HTTP API on AWS

Matt Tyler — Wed, 26 Feb 2020 11:17:53 +0000

Introduction

We added a JWT Authorizer to our API in the last installment. A user that wanted to submit comments would therefore need to authenticate with an Identity Provider (IdP) first. At the end of that piece we also discussed some of the limitations inherent within our implementation, and touched briefly on claims/scopes. Claims/Scopes are a part of the OAuth2 specification that define the properties of the token we passed to our API. It's time to have a bigger discussion about them, and how they relate to various forms of access control, like role-based-access-control (RBAC) and attribute-based-access-control (ABAC).

Code for this tutorial can be found here.

matt-tyler / simple-node-api-rbac

Claims and Scope - Practically

A better way to describe these is to consider a practical example. Scopes were originally conceived as a way for the user to offer consent to a third-party. The canonical example everyone uses is LinkedIn, but that's a little worn out, so let's use a bank as an example.

Imagine a company (completely independent of the bank) launches a new web service. This service aims to analyze your spending history of your savings account and produce detailed reports and suggestions to help you save money. To do this they require you to give over your username and password for your banking account, as this will need login in to your account to scrape the information.

This is bad because they have access to credentials which are not limited to the job that they are intending to perform, and also because there is no way for the user to consent to the specific activities they want to perform.

OAuth2 solves these both these problems. With OAuth2, registering with the service would result in a redirect to the bank's authorization page. The bank would list the permissions that the service is requesting (e.g.; read statements), allowing the user to explicitly consent to the delegation of permissions. If they accept, credentials would be issued that would allow the service to request information about the users bank statements.

OAuth2 works well in this case. However, the restrictions of permissions leads people to incorrectly assume that all that is required for access control is the scopes and claims which is not strictly true. A typically token issued by a bank (like the one in the example) might look like this;

{
  "iss": "http://auth.bank.com",
  "sub": "my-user@bank.com",
  "aud": "76616b84-ad91-4718-8672-fc7d4c0975ae",
  "scopes": [
    "mybank.com/statements.read"
  ],
  "exp": "...",
  "nbf" "...",
}

Note the 'mybank.com/read.statements' scope, which we could assume to mean 'the token allows the user to read statements'. But whose statements are they allowed to read? Their own? everyone? Someone else? The OAuth specification does not detail this! Does this mean we need to explicitly create scopes for every scenario? How large would that make the token? And does that mean that the token issuing server now needs knowledge of every single permission and user in the system? Is this practical?

Sometime it is, and sometime it isn't. I think it is a fair assertion that some kind of additional form of policy evaluation logic is needed in most cases.

Policy Evaluation

There are quite a few different policy evaluation mechanisms out there, though they often follow a fairly basic pattern. Most use some kind of declarative language that works on subjects, actions, and objects/resources, and indicates whether a user is allowed to do something.

Subject

The subject is the actor that is attempting to do something; in most cases, this is a user or some system identity.

In AWS, this is usually the identity of the caller for a typical IAM permission, or the identity in the principal statement for a resource based policy.

Action

This is the 'something' that the subject is attempting to do. This could be reading or writing, or some other kind of method.

This is (not surprisingly) the action property in a typical IAM policy.

Object

The object is what is being acted upon; e.g. we are creating a 'message', we reading 'statements'. In terms of a typical HTTP API this is the resource.

In AWS this refers to the resource section in a policy statement.

Effect

Indicates whether a matching policy results in 'Allow' or 'Deny'. Different systems result in difference precedence e.g. Does a 'Deny' result in overriding a matching 'Allow' case? Are all permission default-deny or default-allow?

This is obviously the 'Effect' clause in an IAM policy statement and AWS has chosen to implement default-deny with deny override.

There are obviously extensions to this, and AWS has implemented many of them via the condition statements, but this is the basic language that is required to begin implementing some form of access control policy that goes beyond what is available in OAuth2 scopes.

But How?

Ben Kehoe

@ben11kehoe

Give us the right capabilities to stay in IAM. Let me turn attributes in Cognito User Pools into session tags through Cognito Identity. Let me tag API Gateway resources and methods, so I can leverage that in IAM policies. Let me compare path & query params against principal tags.

17:29 PM - 26 Jan 2020

2 10

AWS has been adding a lot of features to use OAuth directly with API Gateway, skipping Cognito Identity Pools and AWS IAM. I think this is regressive. A lot of useful functionality is coming out of it, but we should hope to get that IAM-side instead.

In a perfect world this would all be handled by some native mechanism that is present in the cloud provider, as alluded to by Ben Kehoe's statement. There exists various mechanisms in AWS to do parts of the process but they do not currently all align to solve the whole problem. Fundamentally, some mechanism is required to enable us to practically use the IAM policy evaluation engine upon the principals, attributes and resources that WE define, and not just the ones available natively in the platform.

Cognito does a good job of handling user registration and various token related tasks, but it does not currently propagate the information necessary to perform these kinds of policy decisions. This is a future that is probably coming, as illustrated by new ABAC mechanisms introduced via tags, and exemplified by propagating session tags in AWS SSO.

We could see a world where a user would log in via Cognito and receive access to an IAM role via a pair of credentials. These credentials would be bound to session-tags that were created by the platform, that would include information about the users precise identity, which could then be used to scale-back their permissions e.g. prevent them from reading certain rows from DynamoDB via the leadingkey condition, or restrict reading of S3 files to specific prefix. Likewise, requested scopes or group membership within user pools (or other third party directories) could propagate other information to session tags to enable further flexibility within access policies.

This would keep the policy definition and evaluation mechanism inside the platform/infrastructure level, and outside of the application domain.

Unfortunately this isn't supported yet via Cognito and API Gateway. HTTP API is even more restrictive, only allowing the use of a JWT, so we are even further away from native IAM controls. So until the time comes that the feature set of HTTP API authorizers increases, and until a robust session tag mechanism appears in Cognito, we will need to take a code-wise, cloud-foolish approach and implement our own mechanism for defining and evaluating access policies.

To make matters worse, HTTP API Gateway JWT authorizers must have an Aud claim on the token, which Cognito access tokens do not include. Scopes are also not included on Cognito ID tokens. As far as I can tell, this means that you can't use the scope check feature on JWT authorizers if you are using Cognito. You can get around this using Cognito user pool groups, which is what I will demonstrate going forward.

Policy Evaluation Engines

There are a few policy evaluation engines available, but I'm only familiar with two of them.

Open Policy Agent

Open Policy Agent is a project that is currently under incubation status with the Cloud Native Computing Foundation. It is written in Go.

Casbin

Casbin is a open source project that has been around for a few years. It was originally written in Go, but now supports multiple different languages and policy storage backends.

I've used Casbin in production services written in Javascript and Go, so due to familiarity I will use Casbin for our examples. It's possible to do some very funky things in Casbin using either ABAC or RBAC-style policy controls (or a mix of both), but I'll stick to a fairly simple/common RBAC model.

Using Casbin and Cognito, we will enhance our existing guestbook application;

We will create cognito 'groups' that will indicate whether a user can
- read comments, (reader)
- write comments, (writer)
- delete comments (deleter)
We will write a policy that determines
- What groups map to what roles in the policy engine
- What the roles in the policy engine are allowed to do

I will include some examples demonstrating the results of the policy evaluation.

Implementing RBAC with Casbin

Let's start by defining our policy and model. The model determines how the actors in the policy interact, and the policy is the list of valid statements. It is much easier to understand with an example, so let's start by looking at the casbin policy.

[request_definition]
r = sub, obj, act

[policy_definition]
p = sub, obj, act

[role_definition]
g = _, _

[policy_effect]
e = some(where (p.eft == allow))

[matchers]
m = g(r.sub, p.sub) && keyMatch2(r.obj, p.obj) && r.act == p.act

This takes a fair amount of explaining. I'll go over each block one-by-one.

request_definition

The 'request_definition' describes that there are going to be three actors in any request; the subject, the object and the action.

policy_definition

The 'policy_definition' describes how we can construct policies. Any inbound request will be later 'matched' against the policy to determine the policy effect.

role_definition

This is the most confusing aspect of the model, but essentially says that there is one role definition 'g', and that roles can contain other roles. This can be used to establish role-inheritance and heirarchy e.g. writer contains the permission to write, plus all the permissions that were granted to the reader role.

policy_effect

The 'policy_effect' determines whether we allow or deny a matching request. This statement is saying that we have 'default deny', but one matching statement will result in 'allow' - so if we had a statement later that had a 'deny' action, it would overridden by the 'allow'. (I don't actually like this but I figure we'll keep things simple).

matchers

The section defines how the matching logic works, and is specific to casbin. It states that

the subject in the request must belong to a group/role, and,
the object in the request match via a glob,
and the actions defined in the request,

Must match those specified in the policy document.

The documentation explains how to build all sorts of different models for different situations. Understanding the model documents is difficult and I personally find that the policy documents are far easier to grok.

p, role:reader, /messages, read
p, role:writer, /messages, write
p, role:deleter, /messages, delete

g, role:deleter, role:writer
g, role:writer, role:reader

At the top we have defined the roles along with their related unique permissions. The section at the bottom is used to define the heirarchy. Here we stated that the deleter role includes the permissions granted by the writer, which in turn is granted the permissions assigned to the reader.

The next step is to wire this all up in Express. As a first step, I tried to locate all the policy related logic in a single file.

const casbin = require('casbin');

const enforcerPromise = casbin.newEnforcer(
    // I have inlined the model and policy as a string literal.
    // I have not repeated it here because it is already above.
    casbin.newModel(model),
    new casbin.StringAdapter(policy));

async function enforce(sub, obj, act) {
    const e = await enforcerPromise;
    return await e.enforce(sub, obj, act);
}

async function addRolesToUser(sub, roles) {
    const e = await enforcerPromise;
    await Promise.all(roles.map(role => e.addRoleForUser(sub, `role:${role}`)));
}

module.exports.enforce = enforce;
module.exports.addRolesToUser = addRolesToUser;

We initialize a casbin enforcer, and then export two functions. The first of these functions is for policy evaluation against the request. The second is to load the users groups/roles into casbin, so that policy evaluation can function correctly.

The next step is too hook into the express system via middleware.

// ...
const rbac = require('./rbac');
const jwt = require('jsonwebtoken')

// ...

const methodToAction = {
    GET: 'read',
    PUT: 'write',
    POST: 'write',
    DELETE: 'delete'
}

app.use((req, res, next) => {
    const token = req.headers['authorization'];
    const decoded = jwt.decode(token, { json: true });
    const { sub } = decoded;
    const groups = decoded['cognito:groups'] || [];
    const { path: obj } = req;
    const act = methodToAction[req.method];
    console.log({ sub, obj, act});
    console.log(sub, groups);
    rbac.addRolesToUser(sub, groups).then(() => {
        rbac.enforce(sub, obj, act)
            .then(pass => {
                if (pass) {
                    next()
                } else {
                    res.status(403).json({ message: 'Forbidden' });
                }
            })
    })
    .catch(err => {
        console.log(err);
        throw err;
    });
});

Now every time a request is sent, the following happens;

The token is copied from the header.
The token is decoded, and subject and groups claim from the header is extracted.
The user and their groups are registered with Casbin.
The object is extracted from the path, and the action determined from the method.
The subject, object, and action of the request are evaluated against the policy.
Either it evaluates successfully against the policy and the request continues, or a 400 client error is returned.

Cognito requires a little bit of additional configuration. The template is available in the repository, but let's call out some new additions.

  User:
    Type: AWS::Cognito::UserPoolUser
    Properties:
      UserPoolId: !Ref UserPool
      Username: !Ref Email
      DesiredDeliveryMediums:
        - EMAIL
      UserAttributes:
        - Name: email
          Value: !Ref Email

  CommentReaderGroup:
    Type: AWS::Cognito::UserPoolGroup
    Properties: 
      Description: Comment Reader
      GroupName: reader
      Precedence: 0
      UserPoolId: !Ref UserPool

  CommentDeleterGroup:
    Type: AWS::Cognito::UserPoolGroup
    Properties: 
      Description: Comment Deleter
      GroupName: deleter
      Precedence: 0
      UserPoolId: !Ref UserPool

  AttachUserToWriterGroup:
    Type: AWS::Cognito::UserPoolUserToGroupAttachment
    Properties: 
      GroupName: !Ref CommentWriterGroup
      Username: !Ref User
      UserPoolId: !Ref UserPool

  AttachUserToReaderGroup:
    Type: AWS::Cognito::UserPoolUserToGroupAttachment
    Properties: 
      GroupName: !Ref CommentReaderGroup
      Username: !Ref User
      UserPoolId: !Ref UserPool

  AttachUserToDeleterGroup:
    Type: AWS::Cognito::UserPoolUserToGroupAttachment
    Properties: 
      GroupName: !Ref CommentDeleterGroup
      Username: !Ref User
      UserPoolId: !Ref UserPool

Most of this involves the addition of some groups that match the roles referenced in the policy; reader, writer and deleter. I've added the generated user to all of these groups. As I've said previously, make sure to use an email address you own when instantiating the cognito template, as it will send a password to your email address.

To get everything going, download the repository, and deploy the cognito-template.yaml file. Use the outputs from this stack as inputs to the SAM template that defines the API, by invoking sam build && sam deploy --guided. The outputs of the SAM template contains a login URL that can be used to access the login page. From this, you can login and acquire the ID token from the callback URL.

Fill in the ENDPOINT variable using the address of your API, and use the id_token from the login callback URL for the TOKEN variable.

ENDPOINT=''

TOKEN=''

curl -H "Authorization: $TOKEN" $ENDPOINT

curl -XPOST -H "Content-Type: text/plain" -H "Authorization: $TOKEN" -d "Message: My Message" $ENDPOINT

You'll find that both calls will succeed, as we have given the user identified by the token permissions to read, write and delete.

Now we'll remove our user from the groups. To do this go to Cognito in the AWS Console. Select 'User Pools' and click on the one that we created. From here, select users, and click on the only user. The groups will be displayed at the top. Click the 'x's to remove all the groups from the user.

Try to run the above script again. It still succeeded, why?

Well, we are still sending a verified token that contains all the users groups, and we did not regenerate this token after we removed the groups. It will eventually expire, but until then it will still confer the privileges associated with the user. You could instead query the users groups from Cognito directly on every request, but this will add additional latency. Like most things, it's a trade-off. Try logging in again and issuing the requests with a new token. You'll find that the request is rejected as expected.

Try adding different combinations of groups, hit the API, and see what happens! Modify the policy and redeploy! Experiment a bit!

Summary

We had a brief discussion around the limitations of scopes, and raised a scenario to explain what is not covered by the specification. We then briefly introduced ABAC and RBAC styles of access policy, and introduced the possibility of better implementation within AWS Cognito in future. We then considered policy authorization, and discussed some popular access policy evaluation libraries. Of these libraries, we chose to use Casbin to demonstrate how to build a policy model. We use Casbin to add a middleware to our guestbook express application, which evaluated whether a user had access to specific resources based on their membership of Cognito groups.

Feeling RBAC'ed into a corner? Mechanical Rock can help!

Async Iterators: A Practical Example feat. AWS SDK for Node.js

Matt Tyler — Wed, 19 Feb 2020 13:23:18 +0000

Introduction

A little while ago I was having some issues with a particular piece of code that was making a fairly large number of external calls to an AWS service. The sheer volume of calls was causing the service to throttle my client. This was largely due to the fact the client would make a few calls, which would be fired off all at once, then resolved with Promise.all. From those results, it would list more data, then make more calls, then list more, ad-nauseum. Each listing call would page through the full set of data before making the next 'nested' call.

The big problem here is that each next set of calls is multipled by the previous set of calls. The solution to this is to remove the promise.all mechanism and page through explicitly, using for-loops. However, if you have ever used the Javascript AWS SDK, this can look messy.

private async listProvisionedProducts() {
    const provisionedProducts: ServiceCatalog.ProvisionedProductAttributes = []
    let response: ServiceCatalog.SearchProvisionedProductsOutput = {}
    do {
        response = await this.serviceCatalog.searchProvisionedProducts({ 
          PageToken: response.NextPageToken 
        }).promise();
        provisionedProducts.push(...response.ProvisionedProducts)
    } while (response.NextPageToken);
    return provisionedProducts;
}

This extract lists all of the provisioned products in an account. As the API is paged we need to fetch each set of results in turn. If we want to act on each set of results in this manner, we would need to either return the entire result set first (as is done here), or perform some sort of transformation within the loop. The former is what led to this problem, and the second leads to the mess that I was trying to avoid.

It would look a lot cleaner if you could define an iterator over a collection of promises. It turns out you can, and it's the result of a (relatively) new feature called async iterators.

What is an Async Iterator?

Async iterators enable the use of the for await...of syntax in javascript. This enables you to loop over something which returns an iterable of promises. For more information, you can see the following documentation over at Mozilla.

Async iterators are natively supported in Node.js 10 and up. If you are using 8 or 9, you can run node with the --harmony_async_iteration flag to enable support. If you are using typescript, ensure your configuration is enabled for compatibility with ES2018 and then everything should be fine.

Most of the time I prefer to write a bit more functional, making use heavy use of map, reduce, et. al, rather than using for loops. There are two big reasons related to making calls to external services where I find using for-loops can have a significant advantage, particular when making remote calls. I'll cover this soon, but let's first see an example.

A Practical Example.

We'll make a very simple script that can be invoked via the command line to demonstrate the mechanism. When run, it'll wait for input for you to press a key before it fetches any output. It will exit once it has finished.

const S3 = require('aws-sdk/clients/s3');
const pressAnyKey = require('press-any-key');

// Here we hide the pagination details
async function* ListObjects(s3, params) {
  let isTruncated = false;
  let token;
  do {
    const response = await s3.listObjectsV2({ 
        ...params, ContinuationToken: token
    }).promise();

    // One could also yield each item separately
    yield response.Contents;

    ({ IsTruncated: isTruncated, NextContinuationToken: token  } = response);
  } while (isTruncated)
}

async function main() {
  const Bucket = process.argv.slice(2)[0];
  const s3 = new S3({ params: { Bucket }});

  // Usage of the for-await syntax hides the pagination details
  for await (const contents of ListObjects(s3, { MaxKeys: 2})) {
    const objects = contents.map(({ Key }) => Key).join(', ')
    console.log(objects);
    await pressAnyKey('Press any key to fetch next result...');
  }
}

main().then(() => console.log('Finished'))

Of particular note is async function* ListObject declaration. The asterix that is appended to the 'function' statement indicates that we defining this as a 'generator', with the 'async' qualifier denoting it is an 'async generator'. In doing this, yielding from this function will result in returning a promise, with the return type of the function being an asynchronous iterable - thereby fulfilling the async iterator protocol.

There are other ways to define async iterables, but I find the generator method is usually the easiest to understand, without needing to dive into all the details. Though, if you do want to know the details you could do worse than to read this article.

You can run the example via cloning this repository and executing npm run script -- <BUCKET-NAME> from within the base directory. Just make sure you have your AWS profile set up correctly!

So why is this better?

It's perhaps not entirely clear why this is a better way to do things, but I think it is generally superior for two reasons.

The code is easier to follow

Separating the paging mechanism from the logic makes the code easier to understand. If I were to come back to the code later, or I was new to the continuation-token mechanism of the AWS APIs, I would still be able to understand what was going on. I would not be confused by the continuation-token loop - all I need to understand is that I'm 'listing objects' and performing something on each object in turn. Whilst the paging mechanism is important from the service point of view e.g. I'm not unintentionally pulling more data unless I actually have to - it's probably not relevant to understanding the top-level logic of the code. We have hidden that implementation detail away.

It can reduce 'thundering herds' of API calls

This isn't obvious from the above example. Utilizing async iterators can help reduce the number of API calls that are being made, and this helps to reduce the chance of being throttled. I'll often make a bunch of calls which return promises, and resolve them simultaneously with a Promise.all() call. Most of the time this is ok, but this can have consequences when making external API calls, which in turn resolve and then make other external calls. If I first wanted to list all my buckets, and then returned 50 buckets, listed all the objects, and then performed calls against all those objects... this can result in a huge number of calls in a short space of time, and it's highly likely I will start to encounter throttling errors. The number of calls potentially being made in parallel from the same source also makes it difficult to implement a good backoff strategy. Whilst it is efficient to make a external calls in parallel, a balance needs to be maintained to prevent flooding the remote service.

Further Thoughts

Some of the SDKs offer, IMHO, better ways to page through sets of data.

The Boto3 Python API provides paginators in various service clients making the need to create an async iterable (as in Javascript) unnecessary e.g.

import boto3

# Create a client
client = boto3.client('s3', region_name='us-west-2')

# Create a reusable Paginator
paginator = client.get_paginator('list_objects')

# Create a PageIterator from the Paginator
page_iterator = paginator.paginate(Bucket='my-bucket')

for page in page_iterator:
    print(page['Contents'])

The Go v2 and Rust clients do something similar. The following is can example of searching through AWS Service Catalog, using the paging mechanism of the AWS Go Client.

    request := catalog.SearchProductsAsAdminInput{
        Filters: map[string][]string{
            "FullTextSearch": {productName},
        },
    }

    req := p.client.SearchProductsAsAdminRequest(&request)
    pager := req.Paginate()
    for pager.Next() {
        page := pager.CurrentPage()
        productIDs = append(productIDs,
            findProductsWithName(page.ProductViewDetails, productName)...)
    }

As an aside, the Go client is particular interesting, because the API design feels quite different from the Python and Javascript clients. In the Go client, you construct requests which are then actioned via a 'send' call on the request. Python and Javascript instead directly dispatch the call by providing parameters to the method. Interestingly enough, version 3 of the Javascript SDK is moving towards a similar interface.

At any rate, I hope they also make paging a bit nicer, because pagination is not handled in a standard manner across the Javascript SDK. Ian Mckay put together this interesting survey of various pagination rules in AWS.

Conclusion

We looked at async iterators in Javascript, and how to implement them to perform paginated requests in the Javascript AWS SDK. We can see that this allows up to write cleaner code that can avoid throttling errors. Finally, we had a quick look at how pagination is implemented in other language SDKs, and how this might propagate over to version 3 of the Javascript SDK.

Struggling with serverless? We can help! Contact us to get started!