How I Scaled My AWS Side Project from Local Lambdas to Full Infrastructure with Terraform and AWS Amplify

EventSpark is the event management platform we’ve done a few months ago, which was only done locally with NextJS and AWS services like Lambda and DynamoDB. Every developer knows, that moving from a locally built code to production is one of the first steps when looking at the bigger picture of any project.

Armed with some new knowledge of tools and AWS services, I’d love to show you how you can take your full-stack applications to production by using Terraform - an Infrastructure as Code tool we’ve used in my latest blog post.

We will add a new service to our arsenal of knowledge - AWS Amplify.

AWS Amplify is an AWS service used for easier deployment of full-stack applications straight to the Internet. In this project, since our backend is done with AWS Lambda, we will use Amplify for deploying our NextJS frontend.

For deployment, we are using Terraform, so we can deploy our whole application from one folder inside our codebase?

You can visit the GitHub repository of the project by clicking on this link.

Sounds cool? Let’s get to it!

1. The Evolution: From Manual to Automated Infrastructure

In my experience, it is very difficult to manually go through the AWS Management console to create resources and code directly inside the Lambda IDE (been there, done that in my first student 24h hackathon!) — this is a great start if you’ve never worked with advanced systems and any of the Cloud platforms, but as soon as you step into the real production systems, this approach comes unsustainable very quickly.

That’s why Infrastructure as Code solutions, like CloudFormation, AWS CDK and Terraform (we’ve covered all of these tools in previous blogs!) are a true savior — once you define your resources and the code used, you can be sure that you can deploy same environments on different AWS accounts, e.g. dev, testing or prod accounts.

Terraform is a great IaC tool for automating deployment, codifying architectural decisions and ensuring consistency in any environment. It also enables us to use CI/CD solutions for easy and scheduled deployments, if a project requires it.

2. Modular Infrastructure Design

Inside the code repository, this is the folder structure for the Terraform IaC code:

Rather than putting every resource inside one file, I’ve created one root module and 3 child modules:

• authentication - containing all necessary Lambda logic and resources for fully scalable authentication service
• events - containing all resources for the Event micro-service, enabling fast fetching of events data
• client - Terraform code defining necessary environment variables, like authentication and events micro-service’s URLs with GitHub configuration to fetch the code straight from the main branch

One of the great features of terraform was the depends_on feature, I’ve defined the deployment order with it:

1. Authentication
2. Events
3. Client

Because the NextJS application needs to wait for the events and authentication micro-services to deploy, so the environment variables are populated, and the events micro-service needs to wait for the authentication micro-service to deploy, so our Lambdas are protected by Cognito Authorizer inside AWS API Gateway. This eliminates manual coordination and makes deployment so much easier.

3. Solving the Lambda Dependencies Challenge

When I’ve developed EventSpark initially, defining Lambda dependencies was very simple — put the requirements.txt file inside the Lambda folder and Serverless Application Model CLI tool will take care of everything!

First time I’ve deployed a Lambda with Terraform, I hoped that it’s the same as for SAM CLI, but I was wrong — CloudWatch logs were full of ImportModuleError: No module named 'insert_any_library_here' errors.

Since automatic dependency download isn’t available in Terraform, I needed to come up with a new solution, and I’ve come up with 2 ideas, both with their pros and cons:

1. Download the dependencies and bundle them up with the Lambda code Easy to do, however it increases package size and deployment time
2. Create a Lambda layer to share dependencies across multiple functions A little bit more difficult than the first option, but this choice enables us the efficiency and reusability

I’ve chosen the second option — I’ve defined the process of creating Lambda layers with necessary dependencies and attaching them to the Lambdas.

An example of creating a Lambda layer and then attaching them to the Lambda in Terraform looks like this:

# Define the process in as bash commands to build the Lambda layer
resource "null_resource" "powertools_layer_build" {
  provisioner "local-exec" {
    command = <<-EOT
      mkdir -p "${path.module}/builds/python"
      python3 -m pip install aws-lambda-powertools -t "${path.module}/builds/python"
      zip -r powertools_layer.zip python
    EOT
  }
}

# Define the necessary information to build the Lambda layer itself
resource "aws_lambda_layer_version" "powertools" {
  filename            = "${path.module}/builds/powertools_layer.zip"
  layer_name          = "${var.user_pool_name}-powertools"
  compatible_runtimes = [var.lambda_runtime]

  depends_on = [null_resource.powertools_layer_build]
}

# Attach the Lambda layer to the Authentication Lambdas
resource "aws_lambda_function" "auth" {
  for_each         = local.lambda_config
  filename         = data.archive_file.lambda[each.key].output_path
  function_name    = "EventSparkAuth-${each.key}"
  role             = aws_iam_role.lambda_exec.arn
  handler          = "lambda_handler.lambda_handler"
  source_code_hash = data.archive_file.lambda[each.key].output_base64sha256
  runtime          = var.lambda_runtime
  timeout          = 30
  layers           = [aws_lambda_layer_version.powertools.arn] # Attached here!
  environment {
    variables = local.lambda_env
  }
  tags = var.tags
}

The code leverages Terraform’s null_resource with local-exec provisioner to build the layer at deployment time. It runs the bash code we’ve defined, puts it into a layer and then that same layer can be used in our Lambdas.

4. AWS Amplify: Beyond Simple Static Hosting

One of the most popular ways of hosting websites on AWS is the S3 + CloudFront combination - this ensures global delivery so the load times of static files are lightning fast, SSL configuration and at the end of the day, it’s very simple to do even if you don’t have AWS, or any Cloud experience at all.

However, since our frontend is a NextJS application with Server Side Rendering, it makes it difficult to deploy to S3 as some parts of the application, like NextAuth library used for session management, require some sort of server resource, providing additional support to the application.

Here is where AWS Amplify comes along. This service offers a more elegant solution, as it does the building of the code for you, implements environment variables and keeps them safe and you can deploy from different branches in your GitHub repository — this makes it a great service for deploying any full-stack application in multiple environments we’ve mentioned before, like staging and production.

As I’ve mentioned before, the client module gets deployed last, as the NextJS application requires URLs to our Authentication and Event micro-services. To be more precise, our frontend requires the following 2 environment variables:

• NEXT_PUBLIC_EVENTSPARK_AUTH_API
• NEXT_PUBLIC_EVENTSPARK_EVENT_API

These variables are provided automatically by our other child Terraform modules, which represent our micro-services.

Finally, another great feature of Amplify is automatic deployment when it notices that there is a new commit on our main or master branch. In case of failure, it rolls back to the previous working version. Isn’t this amazing?

5. The Deployment Experience using Terraform

Deploying with Terraform is easy. The following 3 commands do it all:

terraform init # Downloads and initializes all necessary dependencies
terraform plan # Shows inside the terminal what resources will be created
terraform apply # Deploy the resources to AWS

This is very similar to Serverless Application Model CLI or AWS CDK, the only difference, excluding the syntax, is that for each of the child modules you’d need to write a separate YAML / CDK file. Here, you can leverage Terraform to deploy multiple micro-services from one command.

After the deployment is complete, go to your AWS management console, go to AWS Amplify service and you’ll need to manually start a build job, so your frontend gets deployed.

Before starting this process, please make sure that you’ve filled the terraform.tfvars file inside the terraform/ folder. I’ve prefilled most of the variables for you, but you’ll need to add the following 3 variables on your own: github_repository github_token and github_repository_branch , where the first variable is the URL to your repository, second variable you can get by creating a classic token inside your GitHub settings (link to tutorial here - make sure you have “*admin:repo_hook”* and “repo” checked) and the third variable is your default branch.

The steps are as following:

1. you will see the following popup to migrate to AWS GitHub app - click on “Start Migration”

1. click on your branch deployment which looks like this

1. click on “Run Job”

1. wait for the job to complete (my build took around 2-3 minutes) and after the deployment is complete, you can click on the link under the “Domain” label to see your application!

6. Lessons Learned

Since this repository is a monorepo — a Git repository containing multiple parts of the project, in this case both backend and frontend code — inside the Terraform configuration for the Amplify deployment, I needed to set the AMPLIFY_MONOREPO_APP_ROOT environment variable and after getting all of the necessary URLs from other child modules, Terraform and Amplify took care of everything else. In this case, the client folder contains all the frontend code, therefore the AMPLIFY_MONOREPO_APP_ROOT environment variable had the value client.

Another feature which I loved about Terraform is the terraform plan command. It’s always a good idea to see and confirm all of the resources you are creating, or updating if your resources are already deployed. Just by doing this step and double-checking your work could save you money at the end of the month.

Let’s go over some useful Terraform patterns which you could reuse in your projects, especially if you are a beginner in Terraform.

7. Terraform Patterns Worth Reusing

Dynamic Resource Creation

I found it very useful to use the for_each method inside the Terraform code when defining Lambda functions. I defined the names of the Lambdas, their source folder, environment variables by defining a map with the keyword locals .

locals {
  lambda_config = {
    get_events = { source_dir = "GetEventsLambda", env = {...} }
    create_event = { source_dir = "CreateEventLambda", env = {...} }
  }
}

resource "aws_lambda_function" "functions" {
  for_each = local.lambda_config
  # ...
}

Module Output Chaining

Module output chaining represents another crucial pattern that enables us to use a modular architecture. For example, authentication module exposes outputs like user_pool_id and auth_api_endpoint , which are used for creating Cognito Authorizer for our Events module and for calling Authentication APIs respectively.

In addition, remember how I’ve mentioned how we can define the order of deployment by using depends_on — you can see it in this code example, it’s very simple to do it. Here, we’ve said to Terraform that the events module is dependent on the authentication module, meaning that it needs to deploy our Authentication module to get the user_pool_id , so we can protect our Events Lambdas.

# Root main.tf
module "events" {
  user_pool_arn = module.authentication.user_pool_id
  depends_on = [module.authentication]
}

Creating Lambda layers

As mentioned before, I’ve chosen Lambda layers to provide necessary dependencies to the Lambdas in this project. Here is the code snippet:

# Define the build process of the necessary Python dependencies
resource "null_resource" "layer_build" {
  provisioner "local-exec" {
    command = <<-EOT
      mkdir -p "${path.module}/builds/python"
      python3 -m pip install <insert_python_dependencies_here> -t "${path.module}/builds/python"
      zip -r layer.zip python
    EOT
  }
}

# Attach the built dependencies and bunch them up into a layer
resource "aws_lambda_layer_version" "layer_name" {
  filename            = "${path.module}/builds/layer.zip"
  layer_name          = "${var.user_pool_name}-layer"
  compatible_runtimes = [var.lambda_runtime]

  depends_on = [null_resource.powertools_layer_build]
}

8. Conclusion & Future Steps

Scaling a practice project like EventSpark with a few Lambdas into a production-ready system requires more than just writing code.

It’s about building repeatable, modular infrastructure that can be deployed in any environment and to remove any deployment headaches you might have if you took any other way. By combining Terraform for backend automation and AWS Amplify for frontend deployment, you get the best of both worlds: infrastructure consistency and effortless user-facing delivery.

I believe that modular Infrastructure as Code is necessary for enabling painless deployments in any environment and to ensure that you have a scalable project going forward.

As for future steps I would add a CI/CD pipeline, like we’ve done in the previous project — on any commit to the master or main branch, update the code in production. In addition, I would add multi-environment support like testing, staging and production, but we’ll do that in upcoming blog posts.

Thank you so much for reading! Until next time!