Terraforming AWS: a serverless website backend, part 3

This is part three of my article series on using Terraform to build a serverless backend in AWS. Check out part one to get started.

Mailgun

Remember that Terraform supports a wide variety of cloud providers, and you can mix them together to produce the design you want.

The code we're deploying uses Mailgun.

We've already got the Mailgun API key configured as a Terraform variable - let's add an SMTP password to terraform.tfvars:

mailgun_smtp_password = "hunter2"

And then we can setup everything up in terraform.tf.

variable "mailgun_smtp_password" {}

provider "mailgun" {
  api_key = "${var.environment_configs["mail_api_key"]}"
}

resource "mailgun_domain" "serverless" {
  name          = "${var.environment_configs["mailgun_domain_name"]}"
  spam_action   = "disabled"
  smtp_password = "${var.mailgun_smtp_password}"
}

This is a good moment to touch on arguments and attributes - all the values you configured, like spam_action = "disabled", are arguments - just like calling constructors or functions in other languages.
Attributes are values on the resource that exist once it is created - for example an assigned IP address or URL, just like an instance property in other languages. We used it earlier when creating the aws_kms_alias for example:

   target_key_id = "${aws_kms_key.LambdaBackend_config.key_id}"

The mailgun_domain resource provides two attributes for the DNS records you need to create in order for Mailgun to send and receive email for your domain. There are two ways you can get this setup.

Manually setup Mailgun DNS records

The first way is simply to output the DNS records from Terraform and apply them manually at your DNS provider.

Back when we setup the SSM Parameter Store entries for our Lambda function, we used variables - specifically input variables that were passed from our terraform.tfvars file and onwards into ssm_parameter_map.tf. Terraform also supports output variables - either from modules like ssm_parameter_map or from our 'root' terraform.tf module.

Add these output variables to output the DNS records from the deployed mailgun_domain:

output "send" {
  value = "${mailgun_domain.serverless.sending_records}"
}

output "receive" {
  value = "${mailgun_domain.serverless.receiving_records}"
}

Then just terraform apply and the required DNS records will be shown at the end of the run. (If that didn't work - what might you have forgotten to do?)

Setup Mailgun DNS records using Terraform

Alternatively, at this point you probably know enough Terraform to try setting up the DNS records automatically! Terraform supports a variety of DNS providers, including AWS (you might consider using this module), DNSMadeEasy, DNSimple, Cloudflare, Dyn and generic RFC 2136 access.

If you aren't using anything more than your domain registrar's default settings, AWS has worked well for me personally, but does incur a small charge.

Make use of the sending_records and receiving_records attributes mentioned above to setup your provider, and make sure you've run terraform apply before proceeding. If you need to work with the list of records or adjust the format of any strings, check out the interpolation documentation for details.

Test Lambda

You can test your Lambda function by heading to the Lambda console, and using the test event:

{ "body-json": "email=user%40domain.com"}

Alternatively, you can do this on the command line:

aws lambda invoke --function-name LambdaBackend --payload "{ \"body-json\": \"email=user%40domain.com\"}" out.txt

Either way, you should receive an email at that address, and the result should look like this:

{
  "errorMessage": "Email.MovedPermanently : Redirecting.",
  "errorType": "http://your/redirect/URL",
  "stackTrace": [
    "redirect (/var/task/index.js:50:19)",
    "/var/task/index.js:111:20",
    "transporter.send (/var/task/node_modules/nodemailer/lib/mailer/index.js:187:21)",
    "/var/task/node_modules/nodemailer-mailgun-transport/src/mailgun-transport.js:149:9",
    "Request.finalCb [as callback] (/var/task/node_modules/nodemailer-mailgun-transport/node_modules/mailgun-js/lib/request.js:99:12)",
    "IncomingMessage.<anonymous> (/var/task/node_modules/nodemailer-mailgun-transport/node_modules/mailgun-js/lib/request.js:313:17)",
    "emitNone (events.js:91:20)",
    "IncomingMessage.emit (events.js:185:7)",
    "endReadableNT (_stream_readable.js:974:12)",
    "_combinedTickCallback (internal/process/next_tick.js:80:11)",
    "process._tickDomainCallback (internal/process/next_tick.js:128:9)"
  ]
}

The Lambda function returns an error for reasons you will see later.

If your function isn't working, take a look at the Cloudwatch logs for your function.

API gateway

Now that the function is working within AWS, we need to hook it up to the API gateway to expose it to the outside world.

This section sets up logging from the API Gateway:

resource "aws_api_gateway_account" "gateway" {
  cloudwatch_role_arn = "${aws_iam_role.cloudwatchlog.arn}"
}

resource "aws_iam_role" "cloudwatchlog" {
  name = "cloudwatchlog"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "apigateway.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

resource "aws_iam_policy_attachment" "cloudwatchlog" {
  name       = "cloudwatchlog"
  roles      = ["${aws_iam_role.cloudwatchlog.name}"]
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs"
}

This section sets up the actual API, resources and integrations. There are also Terraform modules available for this if you want to go down that route.

resource "aws_api_gateway_rest_api" "service" {
  name = "BackendService"
}

resource "aws_api_gateway_resource" "api" {
  rest_api_id = "${aws_api_gateway_rest_api.service.id}"
  parent_id   = "${aws_api_gateway_rest_api.service.root_resource_id}"
  path_part   = "api"
}

resource "aws_api_gateway_resource" "email" {
  rest_api_id = "${aws_api_gateway_rest_api.service.id}"
  parent_id   = "${aws_api_gateway_resource.api.id}"
  path_part   = "email"
}

resource "aws_api_gateway_method" "post" {
  rest_api_id   = "${aws_api_gateway_rest_api.service.id}"
  resource_id   = "${aws_api_gateway_resource.email.id}"
  http_method   = "POST"
  authorization = "NONE"
}

resource "aws_api_gateway_integration" "integration" {
  rest_api_id             = "${aws_api_gateway_rest_api.service.id}"
  resource_id             = "${aws_api_gateway_resource.email.id}"
  http_method             = "${aws_api_gateway_method.post.http_method}"
  integration_http_method = "POST"
  type                    = "AWS"
  uri                     = "arn:aws:apigateway:${var.region}:lambda:path/2015-03-31/functions/${aws_lambda_function.LambdaBackend_lambda.arn}:$${stageVariables.alias}/invocations"
  passthrough_behavior    = "WHEN_NO_TEMPLATES"

  request_templates {
    "application/x-www-form-urlencoded" = <<EOF
##  See http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html
##  This template will pass through all parameters including path, querystring, header, stage variables, and context through to the integration endpoint via the body/payload
#set($allParams = $input.params())
{
"body-json" : $input.json('$'),
"params" : {
#foreach($type in $allParams.keySet())
    #set($params = $allParams.get($type))
"$type" : {
    #foreach($paramName in $params.keySet())
    "$paramName" : "$util.escapeJavaScript($params.get($paramName))"
        #if($foreach.hasNext),#end
    #end
}
    #if($foreach.hasNext),#end
#end
},
"stage-variables" : {
#foreach($key in $stageVariables.keySet())
"$key" : "$util.escapeJavaScript($stageVariables.get($key))"
    #if($foreach.hasNext),#end
#end
},
"context" : {
    "account-id" : "$context.identity.accountId",
    "api-id" : "$context.apiId",
    "api-key" : "$context.identity.apiKey",
    "authorizer-principal-id" : "$context.authorizer.principalId",
    "caller" : "$context.identity.caller",
    "cognito-authentication-provider" : "$context.identity.cognitoAuthenticationProvider",
    "cognito-authentication-type" : "$context.identity.cognitoAuthenticationType",
    "cognito-identity-id" : "$context.identity.cognitoIdentityId",
    "cognito-identity-pool-id" : "$context.identity.cognitoIdentityPoolId",
    "http-method" : "$context.httpMethod",
    "stage" : "$context.stage",
    "source-ip" : "$context.identity.sourceIp",
    "user" : "$context.identity.user",
    "user-agent" : "$context.identity.userAgent",
    "user-arn" : "$context.identity.userArn",
    "request-id" : "$context.requestId",
    "resource-id" : "$context.resourceId",
    "resource-path" : "$context.resourcePath"
    }
}

EOF
  }
}

resource "aws_api_gateway_method_response" "301" {
  rest_api_id = "${aws_api_gateway_rest_api.service.id}"
  resource_id = "${aws_api_gateway_resource.email.id}"
  http_method = "${aws_api_gateway_method.post.http_method}"
  status_code = "301"
  depends_on  = ["aws_api_gateway_integration.integration"]

  response_parameters = {
    "method.response.header.Location" = true
  }
}

resource "aws_api_gateway_integration_response" "default" {
  rest_api_id       = "${aws_api_gateway_rest_api.service.id}"
  resource_id       = "${aws_api_gateway_resource.email.id}"
  http_method       = "${aws_api_gateway_method.post.http_method}"
  status_code       = "${aws_api_gateway_method_response.301.status_code}"
  selection_pattern = "^Email.MovedPermanently.*"

  response_parameters = {
    "method.response.header.Location" = "integration.response.body.errorType"
  }

  depends_on = ["aws_api_gateway_integration.integration"]
}

resource "aws_api_gateway_deployment" "deploy" {
  depends_on = ["aws_api_gateway_method.post", "aws_api_gateway_integration.integration", "aws_api_gateway_integration_response.default"]

  rest_api_id = "${aws_api_gateway_rest_api.service.id}"
  stage_name  = "${terraform.env}"

  variables = {
    "alias" = "${terraform.env}"
  }
}

resource "aws_api_gateway_method_settings" "settings" {
  rest_api_id = "${aws_api_gateway_rest_api.service.id}"
  stage_name  = "${aws_api_gateway_deployment.deploy.stage_name}"
  method_path = "${aws_api_gateway_resource.api.path_part}/${aws_api_gateway_resource.email.path_part}/${aws_api_gateway_method.post.http_method}"

  settings {
    metrics_enabled = true
    logging_level   = "INFO"
  }
}

resource "aws_lambda_alias" "alias" {
  name             = "${terraform.env}"
  function_name    = "${aws_lambda_function.LambdaBackend_lambda.arn}"
  function_version = "$LATEST"
}

resource "aws_lambda_permission" "invoke" {
  statement_id  = "${terraform.env}Invoke"
  action        = "lambda:InvokeFunction"
  function_name = "${aws_lambda_function.LambdaBackend_lambda.arn}"
  principal     = "apigateway.amazonaws.com"
  source_arn    = "arn:aws:execute-api:${var.region}:${data.aws_caller_identity.current.account_id}:${aws_api_gateway_rest_api.service.id}/*/${aws_api_gateway_method.post.http_method}/${aws_api_gateway_resource.api.path_part}/${aws_api_gateway_resource.email.path_part}"
  qualifier     = "${terraform.env}"
  depends_on    = ["aws_lambda_alias.alias"]
}

Back when we setup the SSM Parameter Store entries for our Lambda function, we used variables - specifically input variables that were passed from our terraform.tfvars file and onwards into ssm_parameter_map.tf. Terraform also supports output variables - either from modules like ssm_parameter_map or from our 'root' terraform.tf module.

Add this output variable to output the path from the deployed API Gateway:

output "invoke_url" {
  value = "${aws_api_gateway_deployment.deploy.invoke_url}/${aws_api_gateway_resource.api.path_part}/${aws_api_gateway_resource.email.path_part}"
}

When you terraform apply you will get the value of the variable printed at the end of the output:

Outputs:

invoke_url = https://api_id_here.execute-api.eu-west-1.amazonaws.com/default/api/email

Test API

Now your Lambda function is exposed as an API.

1. Start up Postman, and enter the value from invoke_url as your URL.
2. Change the method to POST
3. On the Body tab, set the type to x-www-form-urlencoded
4. Add the key email and the target email address as the value.
5. Hit Send!

Postman should follow the redirect to your configured URL, and you should receive the email in your inbox. Congratulations!

Mayhem and destruction

If at any point you want to tear down the configuration of these services at AWS and Mailgun, you can simply run terraform destroy and confirm when asked. Your terraform.tf file will be unaffected.

DynamoDB provisioned throughput does incur a running cost for example, so you may wish to consider doing this when you have completed the article series.

Next steps

Rob's original article goes on to add much more functionality, and make use of additional AWS services. However, you probably don't need to know any more about Terraform to do so, so I am going to pause the article series here to gather some feedback. If enough people find this useful I will finish the series.

Please let me know in the comments if you have any questions or comments, and especially if you found this useful!