DEV Community: Joshua Kahn

Exploring API Gateway Lambda Authorizers

Joshua Kahn — Tue, 08 Sep 2020 16:24:00 +0000

More “learning in public”, capturing knowledge that will be useful for myself and others.

Over the past few weeks, I have spent quite a bit more time discussing Amazon API Gateway than I have in years. API Gateway currently offers three types of APIs: (1) REST, (2) HTTP, and (3) WebSocket. The REST and HTTP flavors have quite a bit of overlap, with the HTTP option being the new generation with a growing set of capabilities (while driving towards parity with REST).

In working with APIs, authorization is often a key concern. Unless you are building a truly public API, some level of authorization will be needed to control or limit access to the API. API Gateway offers several options that allow developers to separate authorization from business logic. While these differ dependent on the type of API, options include AWS IAM, Amazon Cognito, OpenID Connect, and AWS Lambda authorization.

Lambda authorizers are particularly interesting given the flexiblity they can provide. A Lambda Authorizer is a a Lambda function to which API Gateway will defer authorization decisions. The function receives one of two types of inputs and responds with output that includes a policy statement. As with other API Gateway features, separating authorization to its own function allows developers to focus on writing business logic.

While Lambda authorizers can be useful, consider the requirements for your service before implementing universally. Other authorizer types may be more appropriate. For example, consider AWS IAM authorization for service-to-service calls or Cognito User Pool authoizers when using Cognito. Lambda Authorizers will introduce latency when called as the authorization logic will need to be executed before returning a response. API Gateway attempts to mitigate this latency by optionally caching the response for up to one hour. When cached, API Gateway will not call the authorizer function on subsequent requests.

A Lambda authorizer can take one of two forms: (1) token-based and (2) request parameter-based. The type of authorizer dictates the event payload received by the Lambda function when invoked by API Gateway. The token-based authorizer (TOKEN) receives the caller’s identity encoded as a bearer token (e.g. JWT or OAuth). A request parameter-based authorizer (REQUEST) receives the complete request, including headers, query string parameters, and API Gateway context information. Which you select is determined by the information required to make an authorization decision.

The response from a Lambda authorizer is composed primarily of a policy statement, not unlike an IAM policy. The policy describes the API resources the caller has access to. Resources are described using an ARN, meaning the authorization can have very fine-grained control over access to various APIs, stages, and/or resources. Wildcards are also available, but be careful so as not to provide inadvertent access. In addition to a policy, the authorizer response can also contain two fields, context and usageIdentifierKey. The context map can be used to pass additional information to the backend service. For example, context could contain details about the caller retrieved by the authorizer from a database lookup. If the API has a usage plan, the usageIdentifierKey is an API key associated with that plan.

Looking to go deeper? See Alex DeBrie’s excellent article The Complete Guide to Custom Authorizers with AWS Lambda and API Gateway.

Building a sample project

To better understand Lambda authorizers, doing as I often do, I built a sample project. The premise of the project is an affiliate API for a book seller.

On signing up, affiliates are provides with a means to authenticate with the book seller’s identity provider (IdP). The authentication mechanism is out of scope and does not matter. When an affiliate successfully authenticates, he receives a JWT token encoded with identifying data, including a unique affiliate identifier (orgId). The affiliate uses the JWT token to access the API by passing it in the Authorization header as a bearer token.

Complete sample code for this project can be found on GitHub. Details on deploying the project yourself are included.

Let’s take a deeper look at the Lamdba authorizer included in the project (custom-authorizer/app.rb). The authorizer is a typical Lambda function, wherein the handler method is called on invocation. For our use case, the function performs the following:

Decodes the JWT token received in the incoming event payload.
Checks if the token represents an administrator or an affiliate.
If an affiliate, look up the affiliate in a DynamoDB table using the orgId included in the token.
Builds an authorization policy applicable to the caller. If an affiliate, various affiliate attributes are included in the context map.
Returns the policy.

API Gateway then uses the returned policy to determine if the caller can access the called API method. The authorizer result is cached for five minutes.

The API methods available in the project are simplistic, primarily returning static data. The add-affiliate function performs the task of creating a new affiliate in the DynamoDB table, creating an API key, and associating that key with a usage plan. A real-world onboarding process is likely more complex, but this was sufficient for a sample.

To enable sharing of common resources and libraries, this project makes use of Lambda Layers. For a deeper dive on Layers, specifically for Ruby, see my earlier blog post.

Taking the API for a spin

We can use Postman, cURL, or a similar tool to exercise the sample project.

First, we will act as an administrator to configure our first affiliate. We will use the POST /admin/add endpoint to add a new affiliate, passing a request body such as:

{
  "name": "New Affiliate",
  "plan": "GOLD"
}

If we attempt the request now, it will fail (unauthorized) as we have not provided a JWT token in the Authorization header. Let’s add a bearer token in the Authorization header that specifies the user is an administrator. Again, we’re not concerned with the creation of this token here, only that it is issued by a valid source.

You can create your own token at http://jwt.io or use the one provided below (the payload should contain a field called admin which is set to a value of true):

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkkgYW0gQURNSU4iLCJpYXQiOjE1MTYyMzkwMjIsImFkbWluIjp0cnVlfQ.CiYHMxM1x1y7YGCn1NPpXP8JI006kPSxBVqD58Tgv3o

Attempting the request again with the header results in the new affiliate being created (the orgId value will vary):

{
  "message": "Affiliate New Affiliate created! An API Key has been provisioned.",
  "name": "New Affiliate",
  "plan": "GOLD",
  "orgId": "81c95682-104a-4567-9ff5-e0af794d392e"
}

We now have an affiliate account that can be used to access other API methods. Grab the orgId and create a new JWT token, removing the admin field and adding orgId with the value found in the previous step. Set the secret key to my-secret-key (shhhh, very secret) as shown in the image below.

Return to Postman. We’ll test an affiliate endpoint at GET /products. Be sure to also update the Authorization header with the new token (remove the old body payload).

The result will include an array of books, for example:

[
  {
      "id": 12,
      "title": "A book",
      "author": "By someone",
      "price": 8.99
  },
  ...
]

If you remove the token from the header, you will receive an Unauthorized response.

A light load test

Before closing this post, I wanted to perform a light load test on the API to confirm usage plans were enforced when a Lambda authorizer provided the API Key. To do so, I used Artillery with a simple scenario that simulates a small number of requests per second over one minute. This is not a true load test as much as a means to verify usage plan behavior. The usage plan limits are set artificially low.

The Artillery configuration is simple: call the two affiliate endpoints at a rate of 15 requests per second for one minute. Because the usage plan limits the client to five requests per second, I expected that roughly two-thirds of the requests would yield a 429 response (To Many Requests). The Artillery configuration is as follows (note the bearer token in the Authorization header, I precomputed it using http://jwt.io for the affiliate OrgId created earlier):

config:
  target: "https://abcdefg.execute-api.us-east-1.amazonaws.com/dev"
  phases:
    - duration: 60
      arrivalRate: 15
  defaults:
    headers:
      Authorization: "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ik5ldyBBZmZpbGlhdGUiLCJpYXQiOjE1MTYyMzkwMjIsIm9yZ0lkIjoiODFjOTU2ODItMTA0YS00NTY3LTlmZjUtZTBhZjc5NGQzOTJlIn0.-9tXCDNaAXpOGSlek7ENpMjuFXq1yzWfXmBJUgCtQ3Q"
scenarios:
  - flow:
    - get:
        url: "/products"
    - get:
        url: "/products/1"

The result showed that all requests succeeded, meaning that Usage Plan throttling did not work. After some digging, I came to realize that the source of the API key was inccorrect. By default, API Key Source is set to HEADER, but here we need to change to AUTHORIZER. While available in the API Gateway Console, SAM does not have a property to change this setting (though CloudFormation does). Each deployment of the serverless application also resets the Key Source, meaning we either need to reset via the AWS CLI/Console after each deployment or use an OpenAPI specification.

For now, I’ve elected to manually configure the Key Source in the AWS Console:

With Key Source setting fixed, we can run the “load test” again. Looking at just the first ten seconds, we see that roughly fifty requests are allowed, which aligns with the GOLD usage plan rate of five requests per second. Over the course of the next two ten-second periods, the client exhausts the 100 requests allowed by the usage plan per day (set purposefully low for this demonstration) and all subsequent requests are blocked.

Started phase 0, duration: 60s @ 11:19:47(-0500) 2020-09-08
Report @ 11:19:57(-0500) 2020-09-08
Elapsed time: 10 seconds
  Scenarios launched: 149
  Scenarios completed: 147
  Requests completed: 295
  Mean response/sec: 30
  Response time (msec):
    min: 43.4
    max: 2862.7
    median: 92.6
    p95: 1849.3
    p99: 2197.4
  Codes:
    200: 56
    429: 239

Conclusion

Building a more complex Lambda authorizer was useful in building my own understanding of how they can be used in a broader serverless application. For a deeper dive, I recommend Alex DeBrie’s excellent article The Complete Guide to Custom Authorizers with AWS Lambda and API Gateway.

Complete sample code for this project can be found on GitHub.

Photo by Scott Webb on Unsplash

This post first appeared at https://blog.iamjkahn.com/2020/09/exploring-api-gateway-custom-authorizers.html

Starting an AWS Step Functions State Machine from Java

Joshua Kahn — Mon, 27 Jan 2020 18:41:00 +0000

Just over two years ago, I wrote a quick post to document how to invoke an AWS Lambda function from Java. I recently was asked how to start execution of an AWS Step Function state machine and also found a working example challenging to dig up … so I put together a quick example:

import com.amazonaws.regions.Regions;
import com.amazonaws.services.stepfunctions.AWSStepFunctions;
import com.amazonaws.services.stepfunctions.AWSStepFunctionsClientBuilder;
import com.amazonaws.services.stepfunctions.model.StartExecutionRequest;
import com.amazonaws.services.stepfunctions.model.StartExecutionResult;
import org.json.JSONObject;

// ...

// (1) Define the AWS Region in which the function is to be invoked.
Regions region = Regions.fromName(System.getenv("AWS_REGION"));

// (2) Instantiate AWSStepFunctionsClientBuilder to build the client
AWSStepFunctionsClientBuilder builder = AWSStepFunctionsClientBuilder.standard()
                                          .withRegion(region);

// (3) Build the client, which will ultimately do the work
AWSStepFunctions client = builder.build();

// (4) Construct the JSON input to the state machine
JSONObject sfnInput = new JSONObject();
sfnInput.put("key1", "hello");
sfnInput.put("key2", "world");

// (5) Create a request to start execution with needed parameters
StartExecutionRequest request = new StartExecutionRequest()
                                      .withStateMachineArn("MY_STATE_MACHINE_ARN")
                                      .withInput(sfnInput.toString());

// (6) Start the state machine and capture response
StartExecutionResult result = client.startExecution(request);

// (7) Handle the result, this includes the execution ID

The example above makes use of the DefaultAWSCredentialsProviderChain to pull credentials from environment variables, instance profile, etc. You can use BasicAWSCredentials if you need to hardcode keys, but this is a much less desirable option.

Securely Storing API Secrets for AWS AppSync HTTP Data Sources

Joshua Kahn — Wed, 08 Jan 2020 01:29:00 +0000

At AWS re:Invent 2019, I presented a Chalk Talk on alternative data sources for AWS AppSync. During one repeat of the session, an attendee asked about storing API Keys for web services. The attendee specifically referenced Algolia, though numerous services use API Keys to identify the number and frequency of calls from a particular user.

To build an AWS AppSync HTTP Resolver to “GraphQL-ify” a service that uses API Keys, it would be easy to hardcode the key in the request resolver. Resolvers are not exposed to the end user of the API, but they are committed to source control, which could potentially leak API Keys. An application may also use different API Keys in different environments, for example to allow for higher throughput in production.

This post discusses an approach to securely storing and using API Keys by leveraging Pipeline Resolvers. API Keys are stored in AWS Secrets Manager and injected into the request to the downstream API.

To demonstrate, I extended the GraphQL Weather API built by Adrian Hall. Adrian built an API using OpenWeatherMap, which uses API keys, as an HTTP data source. Here, I’ve used the same GraphQL schema and resolvers for interacting with OpenWeatherMap.

Building the Pipeline Resolver

The Pipeline Resolver for my weather API is composed of two stages: (1) retrieve secret value from Secrets Manager and (2) query OpenWeatherMap. We can define the resolver using AWS CloudFormation as follows:

WeatherByCityQueryResolver:
  Type: AWS::AppSync::Resolver
  Properties:
    ApiId: !GetAtt WeatherApi.ApiId
    TypeName: Query
    FieldName: weatherByCity
    Kind: PIPELINE
    PipelineConfig:
      Functions:
        - !GetAtt GetSecretValue.FunctionId # (1) get secret value
        - !GetAtt GetWeatherByCity.FunctionId # (2) query OpenWeatherApi
    RequestMappingTemplate: |
      $util.qr($ctx.stash.put("SecretId", "/sample/openweathermap/apikey"))
      {}
    ResponseMappingTemplate: |
      $util.toJson($ctx.result)

We’ll take a deeper look at the two functions that compose the resolver shortly, but take note of the request mapping template. Here, we use the AppSync stash (a map that lives through a single resolver execution) to store the unique name of the secret in Secrets Manager. The secret name is hardcoded in the request mapping template, but by elevating that name to the top of the pipeline, our GetSecretValue function is more flexible. We could pass the name of any arbitrary secret (presuming AppSync has permission to access it) and retrieve the secret value with the one function.

For this example, the OpenWeatherMap API Key is stored in Secrets Manager with the name /sample/openweathermap/apikey. To implement this solution on your own, you will need to sign-up for the OpenWeatherMap service.

Getting the Secret Value

In an earlier blog post, I described how to invoke various AWS services directly from AWS AppSync, including Secrets Manager. The GetSecretValue function is almost identical to the resolver described in that post, though here the caller passes the name of the secret in the stash.

In CloudFormation, the function is configured as follows:

GetSecretValue:
  Type: AWS::AppSync::FunctionConfiguration
  Properties:
    ApiId: !GetAtt WeatherApi.ApiId
    Name: GetSecretValue
    Description: >
      Retrieves the value of the specified secrets from
      AWS Secrets Manager.
    DataSourceName: !GetAtt SecretsManagerDataSource.Name
    FunctionVersion: "2018-05-29"
    RequestMappingTemplate: |
      {
        "version": "2018-05-29",
        "method": "POST",
        "resourcePath": "/",
        "params": {
          "headers": {
            "content-type": "application/x-amz-json-1.1",
            "x-amz-target": "secretsmanager.GetSecretValue"
          },
          "body": {
            "SecretId": "$ctx.stash.SecretId"
          }
        }
      }
    ResponseMappingTemplate: |
      #set( $result = $util.parseJson($ctx.result.body) )
      $util.toJson($result.SecretString)

As noted in my previous post as well, it is extremely important to configure the HTTP Data Source to sign requests that invoke AWS Services. We also need to provide an AppSync service role with permissions to get the secret (secretsmanager:GetSecretValue). Again, in CloudFormation:

SecretsManagerDataSource:
  Type: AWS::AppSync::DataSource
  Properties:
    ApiId: !GetAtt WeatherApi.ApiId
    Name: SecretsManager
    Description: AWS Secrets Manager
    Type: HTTP
    ServiceRoleArn: !GetAtt AppSyncServiceRole.Arn
    HttpConfig:
      Endpoint: !Sub "https://secretsmanager.${AWS::Region}.amazonaws.com/"
      AuthorizationConfig:
        AuthorizationType: AWS_IAM
        AwsIamConfig:
          SigningRegion: !Sub "${AWS::Region}"
          SigningServiceName: secretsmanager

Querying OpenWeatherMap

Armed with the secret API Key, we can call the OpenWeatherAPI. Both the Data Source and response mapping template are identical to Adrian’s article; however, we need to slightly modify the request mapping to use the API key retrieved in the previous function. The request mapping template is as follows (note the use of $ctx.prev.result):

{
  "version": "2018-05-29",
  "method": "GET",
  "resourcePath": "/data/2.5/weather",
  "params":{
    "query": {
      "q": "$context.args.city",
      "appid": "$ctx.prev.result"
    },
    "headers": {
      "Content-Type": "application/json"
    }
  }
}

Closing

By using a Pipeline Resolver that directly invokes Secrets Manager, we have alleviated the need to hardcode a secret and secured it using IAM permissions. We also did so without writing a line of code … well, just a few Velocity Templates, which are very easy to maintain. Our new API allows us to query for the current weather in Nashville (or many other cities):

query Weather {
  weatherByCity(city: "Nashville") {
    timestamp
    temperature
    location
  }
}

And the result:

{
  "data": {
    "weatherByCity": {
      "timestamp": "2020-01-07T22:55:43Z",
      "temperature": 48.18,
      "location": "Nashville, US"
    }
  }
}

While I’m not sure if the re:Invent attendee who inspired this post will find it, I hope you have found it useful. Similar approaches can also be used to build more complex data retrievals as well, without the need to write code.

Photo by Dave on Unsplash

Invoking even more AWS services directly from AWS AppSync

Joshua Kahn — Mon, 09 Dec 2019 21:58:00 +0000

A few months ago, I wrote a post on the AWS Mobile Blog that demonstrated how to invoke AWS service directly from AWS AppSync. In that case, I specifically focused on AWS Step Functions, but it is possible to integrate AppSync with many other AWS services. The goal of this post is to document the integration of AppSync with services beyond Step Functions.

AWS AppSync uses GraphQL resolvers to define the interaction between AppSync and each data source. These are Apache Velocity Templates that will be unique per GraphQL operation. By moving integration of AWS services to resolvers, we can minimize the maintenance of integration code. Velocity Templates generally require little upkeep over time. This approach can be even more maintainable than relying on a thin layer of Lambda code for integration, which would still require dependency management and updates.

Adding an AWS data source

The first step in integrating AppSync with an AWS service is to create an HTTP data source for the service. For an AWS service, the data source endpoint is set to the service API endpoint for the given AWS region and we configure SigV4 signing.

Currently, to configure signing of HTTP data sources, we can use either the AWS CLI or CloudFormation. My preference is to utilize CloudFormation for these needs, though my earlier blog post demonstrates the CLI approach as well. For example, we can add a new data source to invoke Amazon SQS using CloudFormation as follows

SQSHttpDataSource:
  Type: AWS::AppSync::DataSource
  Properties:
    ApiId: !GetAtt MyApi.ApiId
    Name: SQSDataSource
    Description: SQS Data Source
    Type: HTTP
    # IAM role defined elsewhere in AWS CloudFormation template
    # Note: this role needs a policy with 'sqs:SendMessage' permission
    ServiceRoleArn: !GetAtt AppSyncServiceRole.Arn
    HttpConfig:
      Endpoint: !Sub https://sqs.${AWS::Region}.amazonaws.com/
      AuthorizationConfig:
        AuthorizationType: AWS_IAM
        AwsIamConfig:
          SigningRegion: !Ref AWS::Region
          SigningServiceName: sqs

AppSync will require an AWS IAM role that allows the service to perform the desired action. For example, SQS would require sqs:SendMessage permission and Step Functions states:StartExecution.

After creating the data source, we can implement the resolvers that allow interaction with the AWS service. Let’s step through a few example integrations.

Amazon SQS

For each service reviewed in this post, we will start with a sample GraphQL schema that defines the interactions and data associated with the resolver.

We will begin with Amazon Simple Queue Service (SQS), one of the earliest AWS services and one of my favorites for building reliable, decoupled workloads. Here, we send a messsage via a queue, though other operations are available as well. For example, we could list all messages on a particular queue.

input SendMessageInput {
  email: String!
  message: String!
}

type Message {
  id: ID!
  email: String!
  message: String!
}

type Mutation {
  sendMessage(input: SendMessageInput!): Message
}

AppSync can be used to deliver a message to an SQS Queue using the following resolver. While SQS also supports a POST API as well, I have implemented here using a GET. Note that the endpoint resource path includes the AWS Account ID and name of the queue. Infrastructure as code approaches, such as CloudFormation, would make wiring this information easy.

sendMessage - Request Mapping

{
  "version": "2018-05-29",
  "method": "GET",
  "resourcePath": "/<ACCOUNT_ID>/<QUEUE_NAME>",
  "params": {
    "query": {
      "Action": "SendMessage",
      "Version": "2012-11-05",
      "MessageBody": "$util.urlEncode($util.toJson($ctx.args.input))"
    }
  }
}

The response from SQS is encoded in XML and can be easily transformed to a JSON payload using the $util functions provided by AppSync.

sendMessage - Response Mapping

#set( $result = $utils.xml.toMap($ctx.result.body) )
{
  "id": "$result.SendMessageResponse.SendMessageResult.MessageId",
  "email": "$ctx.args.input.email",
  "message": "$ctx.args.input.message",
}

AWS Secrets Manager

AWS Secrets Manager allows customers to protect sensitive information such as API keys. In a future blog post, I will share how I incorporated Secrets Manager in an AWS AppSync Pipeline Resolver to first retrieve a secret API key before querying a third-party web service for data, all via AppSync resolvers. For now, we can simply retrieve and return a secret from Secrets Manager.

type Query {
  getSecret(SecretName: String!): String
}

getSecret - Request Mapping

{
  "version": "2018-05-29",
  "method": "POST",
  "resourcePath": "/",
  "params": {
    "headers": {
      "content-type": "application/x-amz-json-1.1",
      "x-amz-target": "secretsmanager.GetSecretValue"
    },
    "body": {
      "SecretId": "$ctx.args.SecretName"
    }
  }
}

The response from SQS is encoded in XML and can be easily transformed to a JSON payload using the $util functions provided by AppSync.

getSecret - Response Mapping

#set( $result = $util.parseJson($ctx.result.body) )
$util.toJson($result.SecretString)

AWS Step Functions

Although I discussed Step Functions in the earlier mentioned blog, I will include starting a Step Functions state machine here for completeness and to document handling more complex input.

type Mutation {
  submitOrder(input: OrderInput!): Order
}

submitOrder - Request Mapping

$util.qr($ctx.stash.put("orderId", $util.autoId()))
{
  "version": "2018-05-29",
  "method": "POST",
  "resourcePath": "/",
  "params": {
    "headers": {
      "content-type": "application/x-amz-json-1.0",
      "x-amz-target":"AWSStepFunctions.StartExecution"
    },
    "body": {
      "stateMachineArn": "<STATE_MACHINE_ARN>",
      "input": "$util.escapeJavaScript($util.toJson($input))"
    }
  }
}

sumitOrder - Response Mapping

{
  "id": "${ctx.stash.orderId}"
}

Please let me know if there are other integrations you would be interested in seeing.

Originally published at https://blog.iamjkahn.com
Photo by Patrick Perkins on Unsplash

Building AWS AppSync Pipeline Resolvers with AWS CloudFormation

Joshua Kahn — Mon, 11 Mar 2019 00:00:00 +0000

Among the many updates from AWS AppSync over the past few months, the addition of Pipeline Resolvers was particularly exciting. Pipeline Resolvers allow us to build a resolver from a series of reusable functions, each of which can query a separate data source. This approach can be useful in performing an authorization check or resolving a batch request across data sources.

For example, I recently built a Pipeline Resolver composed of two functions:

Retrieves a list of recent items, each identified by a unique ID, from Amazon ElastiCache via a AWS Lambda data source.
Retrieves each of the items from an Amazon DynamoDB data source using BatchGetItem. Using a Pipeline Resolver allowed for separation of functionality as well as composition via AppSync.

Instead of reviewing how I built the resolver, in this post, I will share how I *built it using *AWS CloudFormation. If you are interested in an overview of Pipeline Resolvers or an example implementation, see the Resources section below.

Building a Pipeline Resolver with CloudFormation

When building a Pipeline Resolver in CloudFormation, we specify a resolver as well as the functions that compose that resolver. The resolver and its composite functions will contain request and response mapping templates, but only the functions will have a data source. To demonstrate, we will implement theListPosts resolver found in Nader Dabit’s Intro to AWS AppSync Pipeline Functions:

First, we’ll implement the resolver. We’ll assume the AppSync API and other applicable resources have already been created:

ListPostsQueryResolver:  
 Type: AWS::AppSync::Resolver  
 DependsOn: MySchema  
 Properties:
   ApiId: !GetAtt MyApi.ApiId  
   TypeName: Query  
   FieldName: listPosts  
   # configure this resolver as a pipeline  
   Kind: PIPELINE  
   PipelineConfig:  
     Functions:  
       - !GetAtt IsUserCallerFunction.FunctionId  
       - !GetAtt GetPostsFunction.FunctionId  
   # see Nader's article for details on these templates  
   RequestMappingTemplate: |  
     $util.qr($ctx.stash.put("callerId", $ctx.identity.sub))  
     $util.qr($ctx.stash.put("userId", $ctx.args.userId))  
     {}  
   ResponseMappingTemplate: |  
     $util.toJson($ctx.result)

Next, we will implement the first of two pipeline functions:

IsUserCallerFunction:  
 Type: AWS::AppSync::FunctionConfiguration  
 Properties:  
   ApiId: !GetAtt MyApi.ApiId  
   Name: is_user_caller  
   Description: Checks to see if the user is also the caller  
   DataSourceName: !GetAtt UsersDynamoDBDataSource.Name  
   FunctionVersion: "2018-05-29"  
   RequestMappingTemplate: |  
     #if($ctx.stash.callerId == $ctx.stash.userId)  
       #return($ctx.prev.result)  
     #else  
     {  
       "operation": "GetItem",  
       "key": {  
         "id": $util.dynamodb.toDynamoDBJson($ctx.stash.callerId),  
       }  
     }  
     #end  
   ResponseMappingTemplate: |  
     #if($ctx.result.level != "admin")  
       $util.error("User is not authorized to make this query")  
     #else  
     $util.toJson($ctx.result)  
     #end

Finally, we’ll add the second function:

GetPostsFunction:  
 Type: AWS::AppSync::FunctionConfiguration  
 Properties:  
   ApiId: !GetAtt MyApi.ApiId  
   Name: get_posts  
   Description: Scan table and retrieve items created by user  
   DataSourceName: !GetAtt PostsDynamoDBDataSource.Name  
   FunctionVersion: "2018-05-29"  
   RequestMappingTemplate: |  
     {  
       "operation" : "Scan",  
       "filter" : {  
         "expression" : "#userId = :userId",  
         "expressionNames" : {  
           "#userId" : "userId"  
         },  
         "expressionValues" : {  
           ":userId" : $util.dynamodb.toDynamoDBJson($ctx.stash.userId)  
         }  
       },  
       "nextToken": $util.toJson($util.defaultIfNullOrBlank($ctx.args.nextToken, null))  
     }  
   ResponseMappingTemplate: |  
     #if($ctx.error)  
       $util.error($ctx.error.message, $ctx.error.type)  
     #end  
     {  
       "items": $util.toJson($ctx.result.items),  
       "nextToken": $util.toJson($util.defaultIfNullOrBlank($context.result.nextToken, null))  
     }

After creating the CloudFormation stack, we can test the resolver via the AppSync Console. As usual, CloudFormation brings the value of reuse, portability, configurability.

Thanks for reading and please feel free to leave a comment with any questions.

Resources

Understanding AWS Batch: A Brief Introduction and Sample Project

Joshua Kahn — Mon, 14 May 2018 00:00:00 +0000

Of late, I have focused most of my development work as being “serverless-first” — it I can build it with AWS Lambda, I do. That said, there are some types of compute workloads that simply do not fit the serverless paradigm. In most cases, these tend to be long-running jobs or big data analysis that require multiple hours of ongoing work.

Solutions built on AWS Batch allow developers to build efficient, long-running compute jobs by focusing on the business logic required, while AWS manages the scheduling and provisioning of the work. I recently undertook an investigation of Batch for a customer and built a sample project that I would like to share in this post.

Brief Introduction to AWS Batch

Batch computing run jobs asynchronously and automatically across multiple compute instances. While running a single job may be trivial, running many at scale, particularly with multiple dependencies, can be more challenging. This is where using a fully managed service such as AWS Batch offers significant benefit.

AWS Batch organizes its work into four components:

Jobs — the unit of work submitted to AWS Batch, whether it be implemented as a shell script, executable, or Docker container image.
Job Definition — describes how your work is be executed, including the CPU and memory requirements and IAM role that provides access to other AWS services.
Job Queues — listing of work to be completed by your Jobs. You can leverage multiple queues with different priority levels.
Compute Environment — the compute resources that run your Jobs. Environments can be configured to be managed by AWS or on your own as well as the number of and type(s) of instances on which Jobs will run. You can also allow AWS to select the right instance type. Lastly, a scheduler owned by AWS evaluated when and where to run Jobs that have been submitted to the Job Queues.

The sample project described in the following section defines each of the above (except for the scheduler) for our simple batch job. Our focus is solely on the development of business logic (to complete the batch work) and configuration of how that work is to be completed…AWS manages the rest.

Sample Project: Image Processing

AWS Batch can support a variety of resource intensive tasks, ranging from analyzing complex financial data to screening for drug interactions. That said, our sample is rather simple … processing and tagging an image via Amazon Rekogniton (yes, this could be a Lambda function, but we’re focused on Batch today).

As images are put in an S3 bucket, a Lambda function is invoked, which will submit a new AWS Batch job. The job is implemented as a Docker container image, which is stored in Amazon Elastic Container Registry (ECR). Our job is a simple Ruby application that takes a set of command line arguments as input to its work. I’ve used Terraform to manage the infrastructure of this project (see template.tf)

A few comments / learnings from my exploration into AWS Batch:

Three different IAM Roles were required: (1) Batch service role; (2) EC2 instance role for underlying instances; and (3) ECS task role (to interact with DynamoDB and Rekognition). The task role will be most familiar as it defines the AWS services the job can interact with; however, the instance role is also important (use the AmazonEC2ContainerServiceforEC2Role managed role).
AWS Batch offers a troubleshooting guide to help with common issues. For example, I ran into “Jobs Stuck in RUNNABLE Status” due not using the aforementioned instance role.
As part of the Job Definition, you can define both parameters and environment variables. I used parameters as values that I might override when creating the job and environment variables as values that should not be changed (e.g. AWS Region) but need to be passed to the job.
The command defined in the Job Definition is similar to the Docker RUN command. Here, I chose to interact with my job via the command line, passing parameter values via the Job Definition command.

For further details on implementation, please see my aws-batch-image-processor repository on GitHub.

Calling Amazon API Gateway Authenticated Methods with axios and aws4

Joshua Kahn — Wed, 03 May 2017 00:00:00 +0000

Amazon API Gateway provides a convenient, easy-to-use service that allows developers to publish, monitor, and maintain APIs. It also provides a separation of concerns between your custom business logic and common needs such as caching, throttling, and authorization.

For a recent project, I needed to secure my APIs such that only authorized users could call them (e.g. administrator endpoints). API Gateway supports a number of approaches to controlling access to your services. I also needed to provide authentication for a pool of users and opted to leverage AWS’s powerful IAM capability to control access via Amazon Cognito. Cognito provides both user management as well as federated identity to provide secure access to AWS resources, including calling an API Gateway method.

Enough background, on to the code…

On the frontend, I used the popular axios HTTP library in addition to aws4, a library to sign requests using AWS Signature v4. While the configuration of API Gateway is beyond the scope of this post, know that we need to sign and provide an Authentication header in order for the call to be allowed by secured APIs. This is what aws4 helps to enable. Signing the requests allows the frontend to assume an AWS Role authorized to call the API.

Note: the following code snippets assume the user has already authenticated via Cognito and retrieved temporary credentials (including an access key, secret key, and session token).

First, the following code demonstrates a GET to an API secured with AWS_IAM authorization:

400: Invalid request

Next, let’s consider how the above changes for a PUT request. Note the addition to the request body as well as a content-type header.

400: Invalid request

I hope you found the above useful as you work with these great frontend packages and Amazon API Gateway.