DEV Community: Harprit Singh

A Serverless App with AWS SAM to Rediscover Liked Tweets

Harprit Singh — Sat, 23 Jan 2021 15:05:16 +0000

Introduction

Following the tradition of solving our little Twitter problems with serverless, this post describes an app that would help you rediscover tweets that you have liked in the past. Of course, with the goal of learning how to do serverless along the way. We will be building this simple yet interesting app using AWS Lambda and related services. For configuring and gluing together all the components we will be making use of AWS SAM.

On Twitter, you may be following a bunch of smart folks and often tap the like icon for interesting thoughts they share. However, once you have liked a tweet, it's quite hard to rediscover that after a while. Twitter's native option for that is a chronological display of all liked Tweets with an endless scroll. That, let's just say, isn't that cool.

The serverless app we are going to build will show you back a liked tweet of yours with some serendipity added to the mix. Demo page is here. Though, once Twitter API rate limit is reached, it may not work :-( Below is a snapshot of how it should look though. Also, as we shall see, you may follow along the steps and deploy the same for yourself from the source here on Github.

Prerequisites

Before starting out building this app, here's a list of things you need to have. You may find further details in the readme as well.

AWS SAM CLI Installed

Check the introduction from these docs.

Twitter API Keys and Tokens

Get yours using a Twitter Developer account. We shall be storing them in AWS SSM Parameter Store.

Java 8 with Maven

Lambda function is written in Java. You may skip this if you prefer to build a similar Lambda function in a different programming language.

Components

Here's a view of different components of the app and how they interact:

The flow gets triggered when a user, via the browser, invokes the API Gateway endpoint URL passing along their Twitter username.
API Gateway, in turn in an event driven fashion, invokes the Lambda function with a request object containing the username.
Lambda function on invocation fetches Twitter Authentication keys from AWS SSM Parameter Store. Followed by using these keys to invoke Twitter API to fetch a batch of tweets liked by the user. Finally, one random tweet out of this list is returned as response and eventually gets shown back to the user.

In the upcoming sections of this post, let's look at each of these components one by one.

Parameter Store

For invoking the Twitter APIs, we need to first obtain authentication keys using a Twitter Developer account. Once we have the keys, to be able to securely store and access them from Lambda function, we will make use of AWS SSM Parameter Store. To keep things simple and fetch all authentication related details in a single call to SSM, we shall be storing them as a comma separated StringList param as follows:

Next, we need to define this parameter in the SAM template. This, as we shall see later in the SAM config, will be referred to as an environment variable by the Lambda function. This practice helps in achieving customization of stacks during deployment using SAM CLI. We may pass along different parameter names at the time of deployment, that correspond to respective values for different environments.

For example, let's say that, here we have defined a parameter resource TwitterAuthParam with default value as TWITTER_AUTH for all non prod environments. When we need to run this app on prod environment, we may override the same during deployment with value TWITTER_AUTH_PROD. Accordingly, the Lambda function with no change in code will be able to access the respective SSM param depending on the environment that it's running in.

Parameters:
  TwitterAuthParam:
    Type: String
    Default: TWITTER_AUTH

Lambda

AWS Lambda function, defined for this app, contains the code that for a given username invokes Twitter APIs to get a list of liked tweets for the user. This is followed by simple randomization to choose one liked tweet and return it back as a response. Lambda function is written in Java and makes use of the awesome Twitter4J library. Of course, the same can be achieved in any language using a similar Twitter library.

Lambda function code follows the best practice of performing initializations in the constructor code like fetching authentication keys from Parameter Store and creating Twitter connection instance. Keeping these initializations outside the handler function means that they get carried out only once during initial Cold Start for a given instance. This helps in making faster all subsequent handler function invocations on the same object.

Here's the part of the SAM Template defining config for the Lambda function:

RandomLikesFunction:
  Type: AWS::Serverless::Function
  Properties:
    CodeUri: RandomLikesFunction
    Handler: com.hs.randomlikes.App::handleRequest
    Runtime: java8
    MemorySize: 512
    Timeout: 15
    Environment:
      Variables:
        TWITTER_AUTH_PARAM: !Ref TwitterAuthParam
    Policies:
      - SSMParameterReadPolicy:
          ParameterName: !Ref TwitterAuthParam
    Events:
      GetResource:
        Type: HttpApi
        Properties:
          Method: get
          Path: /u/{name}
          ApiId: !Ref RandomLikesHttpApi

API Gateway

Frontend for the app uses vanilla JS which invokes the API Gateway URL with user's Twitter username as a parameter. It's using HTTP flavor of API which is faster and simpler to use compared to REST (pun intended :-)

Further, as we are using HTTP proxy integration, API Gateway converts the HTTP request into a JSON and passes this to the Lambda function. In Java world, this translates into an event of type APIGatewayProxyRequestEvent as an input to the Lambda function. On the way back Lambda function returns APIGatewayProxyResponseEvent as output, which eventually gets converted by API Gateway into an HTTP response eventually returned back to the caller.

API Gateway, other than providing an interface to the Lambda function, also helps secure the app. This is required as the API endpoint is not authenticated and we need to define throttling limits to avoid any abuse. Or, in other words, to avoid ending up with a big AWS bill due to numerous Lambda invocations.

Finally, we need to enable CORS config to ensure that the browser allows our domain to invoke API Gateway endpoint and is able to access the response.

  RandomLikesHttpApi:
    Type: AWS::Serverless::HttpApi
    Properties:
      DefaultRouteSettings:
        ThrottlingBurstLimit: 5
        ThrottlingRateLimit: 20
      CorsConfiguration:
        AllowMethods:
          - GET
        AllowOrigins:
          - "https://random-likes.harprit.dev"

SAM Template

Finally, let's have a look at the complete AWS SAM configuration template for the app. Here's a visual of the same that describes different sections of the template and shows how SAM glues together all the components discussed above.

Check here for a better resolution version of this image.

Conclusion

You may may find further details like how to build and deploy the app from the source code and README. As mentioned above, purpose behind this app was to build something fun and a bit useful while learning to build serverless apps with AWS SAM along the way. I hope it met that goal somewhat.

A visual introduction to AWS Lambda permissions

Harprit Singh — Thu, 01 Oct 2020 04:53:45 +0000

Introduction

Big value in using AWS Lambda comes from its interplay with other AWS services. To achieve that, however, it's important to understand the role AWS IAM plays in both allowing and securing these interactions.

When working with Lambda using AWS console or frameworks like Serverless or SAM, most of the connectivity details are sorted out for us behind the scenes. But when dealing with scenarios like cross account access and applying principle of least privilege, there's a need to have a better understanding of how Lambda's permissions work. In this post, we will go about looking at a general model for this followed by applying the same to a couple of scenarios.

Permissions Model

At the heart of AWS IAM are the policy documents defining "who can do what" in the AWS world. Clearly, when trying to make sense of Lambda's interactions, it's best to look at things from relevant policies point of view. In this context, for a given Lambda function, following two questions come up which in turn get answered by two different kinds of policies:

1. Which service is allowed to invoke a Lambda function?

This is decided by Resource Based Policy of a Lambda function. Such policies are called so as they are applied directly to a specific resource in an inline manner. In case of a Lambda function, the "Resource" element in the policy refers to the function's ARN. While the "Principal" config refers to other AWS resources to whom function invocation access is granted. Previously, Lambda's Resource Based Policies used to be referred to as Function Policies.

2. Once invoked, which service is a Lambda function in turn allowed to invoke?

This is controlled by policies attached to Execution Role of a Lambda function. Execution Role is the role that a Lambda function assumes during its execution. A "Principal" isn't defined in such policies as entities like Lambda function that assume the role themselves act as the principal. The "Resource" part of this policy refers to AWS resources to which the Lambda function is granted access to. We must also define a trust policy on the Execution Role so as to explicitly grant the function permission to assume the role as a principal.

An important difference to understand is that while policies attached to the Execution Role allow a set of permissions to Lambda function as a Principal. Resource Based Policies take care of the other side of things by stating what other Principals can do to a Lambda function as a resource. Put together, both the policies let us control complete access to and from a Lambda function.

Invocation Modes

Given the above general model, things change a bit based on the manner in which Lambda function gets invoked. Of which, there are following two modes:

1. Push Mode

In this case, an external service pushes an event into Lambda and triggers the function. This event could originate due to some action in an another service or a user initiated web request and so on. In this scenario, from permissions perspective, the service that's the source of the event requires rights to invoke the Lambda function. As we saw above, for a Lambda function, the place to define such permissions is the Resource Based Policy.

2. Pull Mode

There are some services like SQS and Kinesis that do not push events directly to invoke Lambda. For these event sources, Lambda as a service polls them for events and invokes the Lambda function i.e. the function doesn't gets invoked by an external service. From Lambda's permissions point of view, this implies that Resource Based Policies are not relevant here. Secondly, Lambda's Execution Role permissions need to be broadened so as to be able to poll and read messages from services like SQS and Kinesis.

Next, we shall look at how things really look when applying Lambda's permissions model with examples on both types of invocation modes.

Push Mode Permissions

For this scenario, let's consider a simple design where a Lambda function subscribes to a SNS topic. When a message is published to the SNS topic, the subscribed Lambda function gets invoked by SNS with the message as payload. Upon invocation, the Lambda functions reads the message, extract some data of interest and inserts it into DynamoDB. While the logs go to CloudWatch. This flow is as visually shown below along with a summarized view of how policies applied to Lambda should look like to support the interactions.

In this example, we are assuming that both SNS and Lambda belong to the same account. We will look at the cross account scenario in the next section.

Looking into more detailed look at the permissions, here's the Resource Based Policy allowing invocation of Lambda function by the subscribed SNS Topic.

{
  "Version": "2012-10-17",
  "Id": "default",
  "Statement": [
    {
      "Sid": "sns-to-lambda-same-account",
      "Effect": "Allow",
      "Principal": {
        "Service": "sns.amazonaws.com"
      },
      "Action": "lambda:InvokeFunction",
      "Resource": "arn:aws:lambda:us-east-1:ACCOUNT-A:function:MyLambda",
      "Condition": {
        "ArnLike": {
          "AWS:SourceArn": "arn:aws:sns:us-east-1:ACCOUNT-A:MyTopic"
        }
      }
    }
  ]
}

While the Execution Role Policy as shown below grants access to the Lambda function to insert entries into DynamoDB table and write logs in CloudWatch.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "dynamodb:PutItem",
            "Resource": [
                "arn:aws:dynamodb:us-east-1:ACCOUNT-A:table/MyTable",
                "arn:aws:dynamodb:us-east-1:ACCOUNT-A:table/MyTable/index/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        }
    ]
}

Cross Account Scenario

Next, let's see how things change in case the Lambda function and the subscribed SNS topic sit in different accounts. Say, Lambda function is defined in ACCOUNT-A while the subscribed SNS topic belongs to ACCOUNT-B.

To allow this cross account invocation simply update SNS topic's ARN, belonging to ACCOUNT-B, in the Lambda function's Resource Based Policy.

{
  "Version": "2012-10-17",
  "Id": "default",
  "Statement": [
    {
      "Sid": "sns-to-lambda-cross-account",
      "Effect": "Allow",
      "Principal": {
        "Service": "sns.amazonaws.com"
      },
      "Action": "lambda:InvokeFunction",
      "Resource": "arn:aws:lambda:us-east-1:ACCOUNT-A:function:MyLambda",
      "Condition": {
        "ArnLike": {
          "AWS:SourceArn": "arn:aws:sns:us-east-1:ACCOUNT-B:MyTopic"
        }
      }
    }
  ]
}

While we are focusing here on Lambda, it's important to note that additional cross account permissions are required on SNS side of things as well. SNS Access Policy is counterpart of Lambda's Resource Based Policy on the SNS side. As the SNS Access Policy gets applied directly at the specific SNS topic level and let's us control who all can access the same.

For the above scenario, SNS Access Policy needs to be updated with following statement allowing access to ACCOUNT-A to be able to subscribe to the SNS topic.

{
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::ACCOUNT-A:root"
  },
  "Action": [
    "SNS:Subscribe",
    "SNS:ListSubscriptionsByTopic",
    "SNS:Receive"
  ],
  "Resource": "arn:aws:sns:us-east-1:ACCOUNT-B:MyTopic"
}

Pull Mode Permissions

For an example of pull mode scenario, let's consider a design where Lambda polls to read messages from a SNS Queue and invokes Lambda function. As in the push mode example, the function on invocation processes the messages and writes to DynamoDB table and CloudWatch. This is as shown below along with a summarized view of permissions that need to be defined in Lambda's execution role. As discussed above, in pull mode invocation there's no need for Resource Based Policies. This is because Lambda function is invoked by Lambda service itself and not any external entity.

Above visual shows a concise view of Lambda's Execution Role permissions required for this scenario. Check the following code snippet for more details on the same. To enable Lambda to continuously poll the SQS Queue for new messages, "GetQueueAttributes" and "ReceiveMessage" are required. "DeleteMessage" is required to remove the messages that get successfully processed by Lambda function. Remaining permissions are as discussed in above example for allowing updates to DynamoDB and CloudWatch Logs.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sqs:GetQueueAttributes",
                "sqs:ReceiveMessage",
                "sqs:DeleteMessage"
            ],
            "Resource": "arn:aws:sqs:us-east-1:ACCOUNT-A:MyQueue"
        },
        {
            "Effect": "Allow",
            "Action": "dynamodb:PutItem",
            "Resource": [
                "arn:aws:dynamodb:us-east-1:ACCOUNT-A:table/MyTable",
                "arn:aws:dynamodb:us-east-1:ACCOUNT-A:table/MyTable/index/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        }
    ]
}

Cross Account Scenario

For the cross account scenario, let's assume that the Lambda function is defined in ACCOUNT-A while the polled SQS Queue resides in ACCOUNT-B. In this case, Lambda's execution role needs to point to the ARN of the SQS Queue in ACCOUNT-B. This allows Lambda to pull messages from the queue in ACCOUNT-B. As rest of the services like DynamoDB are in the same account, no change is required for them.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sqs:DeleteMessage",
                "sqs:GetQueueAttributes",
                "sqs:ReceiveMessage"
            ],
            "Resource": "arn:aws:sqs:us-east-1:ACCOUNT-B:MyQueue"
        },
        {
            "Effect": "Allow",
            "Action": "dynamodb:PutItem",
            "Resource": [
                "arn:aws:dynamodb:us-east-1:ACCOUNT-A:table/MyTable",
                "arn:aws:dynamodb:us-east-1:ACCOUNT-A:table/MyTable/index/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        }
    ]
}

As detailed in the push mode scenario, permissions in Lambda function's Resource Based Policy allow cross account invocation of the function. In case of SQS, similar resource level access control is achieved using SQS Access Policies. That's where we can define what all identities like users and roles can access a specific SQS queue and the actions they are allowed to perform.

In this scenario, a statement needs to be added in the Access Policy of SQS queue as follows. Specifically, to allow access to Lambda function's role in ACCOUNT-A so that it's able to poll and read messages from the SQS queue in ACCOUNT-B.

{
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::ACCOUNT-A:role/service-role/LambdaCrossAccountSQSRole"
  },
  "Action": [
    "SQS:ChangeMessageVisibility",
    "SQS:DeleteMessage",
    "SQS:ReceiveMessage",
    "SQS:GetQueueAttributes"
  ],
  "Resource": "arn:aws:sqs:us-east-1:ACCOUNT-B:MyQueue"
}

To summarize, as shown in the visual below, cross account access needs to be opened by both the parties. With updates required in Execution Role for the Lambda function and in the Access Policy for the SQS queue.

Conclusion

AWS IAM is a vast yet profoundly important subject. It's vital for everyone who builds on AWS to take time to understand IAM well. As they say, security is job zero. Though this writeup only scratches the surface, I hope it helped you in some ways to have a basic model for working with AWS IAM and Lambda permissions.

References: