DEV Community: Tanseer

AWS Elemental MediaConvert: The Video Service Every Developer Should Know

Tanseer — Mon, 03 Aug 2026 05:30:00 +0000

About this series

Most AWS learning stops after EC2, S3, IAM, and Lambda. But AWS has over two hundred services, and many of the most useful ones rarely appear in tutorials.

AWS Hidden Gems covers those underrated services you shouldn't ignore. Each article picks one, then explains why it exists, what it does, where it fits, and how to set it up from the console. Know the four basics above and you can follow along. Everything else gets explained as it comes up.

Today's service: MediaConvert

Almost every app has video in it, and every video hides one boring step: converting the raw file into formats that phones, laptops, TVs, and slow networks can all play. That is what MediaConvert does.

Why does this service exist?

Transcoding means rewriting a video into a different format, quality, or resolution. You need it because one file cannot serve every viewer. A weak mobile connection needs a small version. A fast home connection wants the crisp one.

Before AWS, teams ran FFmpeg (a free tool that converts almost any video) on their own server fleets. Those servers were expensive, sat idle between uploads, and needed constant patching. Getting broadcast quality output took specialist knowledge most teams did not have.

MediaConvert removes all of that. You hand it a file, it does the work on infrastructure you never manage, and you pay only for the minutes you process.

What is MediaConvert?

MediaConvert is a fully managed, file based video transcoding service. Fully managed means AWS runs the servers for you. File based means it works on complete video files that already exist, usually in S3, not on live streams.

What it does:

Changes the codec, the compression method, such as H.264 or the newer H.265
Changes resolution, so one 4K master becomes 1080p, 720p, and 480p
Adjusts bitrate, the data used per second and the main quality versus size lever
Adds captions, extra audio tracks, logo overlays, and thumbnails
Packages video for streaming with HLS and DASH, the formats behind adaptive bitrate, where the player switches quality as the network changes What it does not do is store or deliver your videos. Storage is the job of S3. Delivery is the job of CloudFront, a network that keeps copies of your files close to users.

A real world problem

An online course startup lets instructors upload lectures from their phones. Files arrive in every format and size. A student on wifi watches fine. A student on mobile data sees the giant 4K file stall every few seconds. Some older devices cannot play certain formats at all.

The startup cannot control how instructors record or how students watch. It can only fix the middle: convert every upload into a consistent set of streaming friendly versions. That is what MediaConvert is for.

Real world use cases

Streaming platforms turn studio master files into every quality level for phones, browsers, and TVs
Education platforms standardize instructor uploads so playback never depends on someone's recording setup
Healthcare processes recorded procedures and telehealth sessions into required formats for review and archiving
Finance and enterprise transcode earnings calls and training, captioned for accessibility and stored for the long term
Ecommerce converts product and review clips into fast loading versions that do not slow the page
Gaming and social apps normalize huge volumes of user uploaded clips into formats the app can play The pattern is always the same. Video comes in messy, and it leaves clean and ready to play.

Where it fits in AWS

A common setup: a video lands in S3, the upload triggers a Lambda function, Lambda starts a MediaConvert job, MediaConvert reads the file and writes converted versions back to S3, and CloudFront delivers them worldwide. MediaConvert reports progress through EventBridge, the AWS service for reacting to events, and it accesses S3 using an IAM role, a set of permissions a service borrows to act for you.

flowchart LR
    A[User uploads video] --> B[S3 input bucket]
    B -->|Upload event| C[Lambda function]
    C -->|Start job| D[MediaConvert]
    B -->|Reads source file| D
    D -->|Writes outputs| E[S3 output bucket]
    E --> F[CloudFront]
    F --> G[Viewers on any device]
    D -->|Job status| H[EventBridge]

MediaConvert is the processing engine in the middle. Everything around it is handled by services built for that job.

How the workflow runs

The unit of work is a job: one input file, the outputs you want, and the IAM role to use. A job is submitted, waits in a queue, gets processed on managed infrastructure, and writes its outputs to S3 when done. Status updates flow through EventBridge the whole time, and failures come back with an error message.

Two features save repetition. Presets store output settings, so you set 1080p H.264 once and reuse it. Job templates store a whole job's settings, so your standard outputs become a one line request.

flowchart TD
    A[Job submitted] --> B[Job waits in queue]
    B --> C[MediaConvert processes it]
    C --> D{Success?}
    D -->|Yes| E[Outputs written to S3]
    D -->|No| F[Job marked with error]

Setting it up in the AWS Console

Before you start, upload a short sample video to an S3 bucket. Note the bucket name and the folder where you want the output. Keep the file small so your first job finishes quickly.

Sign in to the AWS Console, type MediaConvert into the search bar at the top, and open it. Look at the region shown in the top right corner and make sure it matches the region of your S3 bucket. MediaConvert reads from and writes to S3, so keeping both in the same region avoids extra data transfer charges and keeps the job faster.
Click Create job. The page splits into two halves. The left panel is where you add the parts of your job, meaning the input and the output groups. The right panel is where you edit the settings for whatever part you have selected.
In the Input section on the left, paste the full S3 path to your sample video, for example s3://your-bucket-name/sample.mp4. This is the file MediaConvert will read. Check the path carefully, because a wrong path or a missing file extension is the most common reason a job fails right away.
In the left panel, find Output groups and click Add. MediaConvert asks what kind of group you want. Choose File group for this first test, which produces one converted file rather than a segmented streaming package, so it is the simplest way to confirm the flow works. Open the group and set its Destination to the S3 folder for your output, for example s3://your-bucket-name/output/.
Inside the output group there is an output entry. Select it and set three things on the right: the video codec to H.264, which every device supports, the resolution to 1280 by 720, and the audio settings left at their defaults. These few values are enough to produce a clean, playable file.
Scroll to the Job settings section and find the service access or IAM role option. MediaConvert needs a role that lets it read your input bucket and write to your output bucket. For a first job, pick the option that lets the console create this role for you. If you prefer to build it yourself, the role needs a trust policy allowing the MediaConvert service to assume it, plus s3:GetObject on the input bucket and s3:PutObject on the output bucket.
Leave the default on demand queue selected. You do not need reserved capacity to start.
Click Create at the bottom. The job appears with a status of Progressing. For a small file it changes to Complete within a minute or two. Open your output folder in S3, download the converted file, and play it to confirm the transcode worked. Common mistakes: a permissions error almost always means the IAM role cannot reach one of your buckets, so check that it covers both the input and the output. A job that fails right away usually means the input path is wrong, so confirm it exactly, including the file extension.

Starting a job with code

Clicking through the console is fine for learning, but in a real system you want jobs to start on their own. The usual pattern is a Lambda function that runs whenever a new video lands in S3 and tells MediaConvert to start a job.

One detail trips up newcomers. MediaConvert gives each account its own endpoint, which is its own address, so you fetch that endpoint once and build the client with it. To keep the code short, first save your settings from steps 4 and 5 as a job template in the console. The template holds the codec, resolution, and output destination, so the function only has to pass the input file.

import boto3

# MediaConvert has a per account endpoint, so fetch it once when the function loads.
default_client = boto3.client("mediaconvert")
endpoint = default_client.describe_endpoints()["Endpoints"][0]["Url"]
mediaconvert = boto3.client("mediaconvert", endpoint_url=endpoint)

def lambda_handler(event, context):
    # The S3 event tells us which file was just uploaded.
    record = event["Records"][0]["s3"]
    source = f"s3://{record['bucket']['name']}/{record['object']['key']}"

    response = mediaconvert.create_job(
        Role="arn:aws:iam::123456789012:role/MediaConvertRole",
        JobTemplate="my-video-template",
        Settings={
            "Inputs": [{"FileInput": source}]
        },
    )

    print("Started MediaConvert job:", response["Job"]["Id"])
    return {"jobId": response["Job"]["Id"]}

To make this run on its own, add an S3 trigger on your input bucket that invokes the function on new uploads, and give the function's role permission to call MediaConvert and to pass the MediaConvert role.

Pricing

Item	Detail
Billing unit	Per minute of output video, prorated per second
Billed on	Output, not input
Output math	One 10 minute video into 3 versions = 30 output minutes
Basic tier HD	About $0.015 per minute (US, on demand)
Basic tier SD	Lower than HD
Basic tier 4K	Higher than HD
Professional tier	Higher rate, for advanced broadcast features
Cost example	100 videos x 10 min x 2 HD versions = 2,000 min, about $30 per month
Free tier	Monthly allowance of basic tier output minutes for 12 months
Reserved pricing	Lower rate for large, predictable monthly volume

The AWS media services family

AWS Media Services
├── MediaConvert   file based transcoding for existing videos
├── MediaLive      real time encoding for live broadcasts
├── MediaPackage   packages and protects streams for delivery
├── MediaConnect   reliable transport of live video
├── MediaTailor    inserts personalized ads into streams
└── Amazon IVS     managed low latency interactive live streaming

The key split is file versus live. MediaConvert handles finished files. MediaLive and IVS handle video happening right now. If your video already exists, MediaConvert is the tool.

Wrapping up

MediaConvert takes video in any shape and produces clean, playable versions for every device and network, with no servers to run. Next time you build anything with video, you know the managed service that does the heavy lifting and how to wire it up.

Series progress

You are on stop one of AWS Hidden Gems.

AWS Elemental MediaConvert (you are here)
Amazon IVS
Amazon Rekognition
Amazon Personalize
AWS AppSync Next up is Amazon IVS, which picks up where MediaConvert stops, handling live and interactive video.

Let's connect

Questions, corrections, or want to talk through where this fits in your own project? Reach me at khantanseer43@gmail.com.

Terraform State Locking Without DynamoDB: S3 Native Locking Explained

Tanseer — Tue, 21 Jul 2026 04:53:34 +0000

A beginner friendly guide to S3 native state locking and why it makes your Terraform setup so much cleaner

Setting up remote state in Terraform used to come with a small annoyance. You could not just use an S3 bucket on its own. You also had to create a DynamoDB table sitting right next to it, purely to handle locking. Two AWS services for something that felt like one job.

That has changed. Starting with Terraform 1.10, S3 can handle state locking by itself. No DynamoDB table, no extra resource to provision, no extra permissions to hand out. In this post I will walk through what state locking is, why it needed DynamoDB in the first place, what changed under the hood, and the exact steps to switch to the simpler setup.

This is written for people who are still new to AWS and Terraform, so I will explain each term the first time it shows up.

A quick refresher on the Terraform state file

When you run Terraform, it needs to remember what it built. It keeps this record in a file called the state file, usually named terraform.tfstate. Think of it as a map that connects the resources written in your code to the real resources living in your AWS account.

If you work alone on your laptop, that file can just sit on your machine. But the moment a second person joins, everyone needs to share the same map. If two people have two different copies, they will fight over what exists and what does not.

The common fix is to store the state file in a shared location that the whole team points to. This shared location is called a backend. An S3 bucket is one of the most popular backends because it is cheap, durable, and simple. A bucket is just a storage container in S3 where you keep files, which AWS calls objects.

Why state locking matters

Here is the problem that locking solves.

Say two engineers on your team both run terraform apply at almost the same moment. Both runs read the current state file. Both make their changes. Both try to write the updated state back. One write lands on top of the other, and now your state file is scrambled. In the worst case Terraform gets confused about what already exists and starts creating duplicates or destroying things it should not touch.

State locking stops this. Before Terraform makes any change, it grabs a lock. While that lock is held, any other run that tries to start is told to wait. Once the first run finishes, it releases the lock and the next person can go. Only one operation writes at a time, so the state file stays clean.

The question is where that lock actually lives.

The old way: a DynamoDB table just for locks

For a long time, the S3 backend could store your state, but it could not lock it. S3 on its own had no simple way to say "only write this if nobody else is holding it." So Terraform borrowed a second service to keep track of locks: DynamoDB, which is a fast key value database from AWS.

The setup looked like this. You created a DynamoDB table with a single partition key called LockID. A partition key, also called a hash key, is just the main field DynamoDB uses to find a row. When Terraform started an operation, it wrote a lock item into that table. When it finished, it deleted the item. If a second run showed up and saw a lock item already there, it knew the state was busy.

The Terraform code for that table usually looked like this.

resource "aws_dynamodb_table" "terraform_locks" {
  name         = "terraform-locks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

And the backend block that tied it all together looked like this.

terraform {
  backend "s3" {
    bucket         = "my-terraform-state-bucket"
    key            = "global/s3/terraform.tfstate"
    region         = "ap-south-1"
    encrypt        = true
    dynamodb_table = "terraform-locks"
  }
}

Here key is the path to the state file inside the bucket, and encrypt = true tells S3 to encrypt the file at rest.

None of this is hard on its own. But notice how much you are carrying. You provision a whole extra service. You manage its name and region and make sure they match the backend config. You grant IAM permissions for it, where IAM is the AWS system that decides who is allowed to do what. And you have one more thing that can silently break, like a table created in the wrong region or a name typo that only shows up when a lock fails.

What actually changed

Two things had to line up for S3 to do locking on its own.

First, in August 2024, AWS added conditional writes to S3. This is the important piece. A conditional write lets you upload an object only if it does not already exist. Technically this works through an HTTP header called If-None-Match. In plain terms, you can now tell S3 "create this file, but only if it is not already there, otherwise refuse." That refuse behavior is exactly what a lock needs. If the lock file is already there, your write bounces, which means someone else has the lock.

Second, Terraform 1.10 (released in November 2024) added a new backend option called use_lockfile. When you turn it on, Terraform uses that S3 conditional write trick to manage locks directly in the same bucket as your state. DynamoDB is no longer part of the picture.

Terraform 1.11 went a step further and marked the old dynamodb_table argument as deprecated, which is the maintainers' way of saying the DynamoDB path is on its way out and you should move over.

One quick clarification so you do not get tripped up. S3 native state locking has nothing to do with a feature called S3 Object Lock. Object Lock is a compliance feature that stops files from being deleted or changed for a set period. That is not what we are using here. Terraform state locking uses conditional writes, and it works on a plain, normal S3 bucket with no special mode turned on.

How it works under the hood

When you enable use_lockfile and run an operation, Terraform creates a small lock file in the same bucket, right next to your state file. It reuses your state path and adds a .tflock extension.

So if your state file is at:

global/s3/terraform.tfstate

the lock file will appear at:

global/s3/terraform.tfstate.tflock

While an operation runs, that .tflock object exists. When the operation ends, Terraform deletes it. If someone else runs Terraform during that window, S3 rejects their conditional write because the lock file is already there, and Terraform shows them an error instead of letting two runs collide.

The error looks something like this.

Error: Error acquiring the state lock

Error message: operation error S3: PutObject, https response error
StatusCode: 412, RequestID: ..., HostID: ..., api error PreconditionFailed:
At least one of the pre-conditions you specified did not hold

Lock Info:
  ID:        3f6a1c9e-...-b2d4
  Path:      my-terraform-state-bucket/global/s3/terraform.tfstate.tflock
  Operation: OperationTypeApply
  Who:       sahil@laptop
  Created:   2026-07-21 09:14:22.5 +0000 UTC

Notice the StatusCode: 412 and PreconditionFailed. That 412 is the HTTP status S3 returns when a conditional write is refused because the object already exists. In other words, the lock is doing its job. Everything you need to identify who holds the lock is right there in the output.

Steps to set it up

Here is the full walkthrough for a fresh setup.

1. Check your Terraform version

use_lockfile needs Terraform 1.10.0 or newer. Check what you have.

terraform version

If you see anything older than 1.10, upgrade before continuing.

2. Create the S3 bucket for your state

If you already have a state bucket you can skip ahead. If not, create one with the AWS CLI, which is the command line tool for talking to AWS. Bucket names have to be globally unique across all of AWS, so pick something specific to you.

aws s3api create-bucket \
  --bucket my-terraform-state-bucket \
  --region ap-south-1 \
  --create-bucket-configuration LocationConstraint=ap-south-1

A small gotcha. Every region except us-east-1 needs that --create-bucket-configuration LocationConstraint line. If you are using us-east-1, drop that last line, because that region rejects it.

3. Turn on versioning

Versioning tells S3 to keep old copies of a file each time it changes. For a state file this is a safety net. If something ever corrupts your state, you can roll back to an earlier version.

aws s3api put-bucket-versioning \
  --bucket my-terraform-state-bucket \
  --versioning-configuration Status=Enabled

4. Write the backend block

Now the good part. In your Terraform project, add the backend block with use_lockfile = true. Notice there is no dynamodb_table line anywhere.

terraform {
  backend "s3" {
    bucket       = "my-terraform-state-bucket"
    key          = "global/s3/terraform.tfstate"
    region       = "ap-south-1"
    encrypt      = true
    use_lockfile = true
  }
}

That single use_lockfile = true line is the whole feature.

5. Set the IAM permissions

Because DynamoDB is gone, your permissions get shorter too. The role or user running Terraform now only needs S3 actions, and no DynamoDB actions at all. A minimal policy looks like this.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::my-terraform-state-bucket"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::my-terraform-state-bucket/global/s3/terraform.tfstate*"
    }
  ]
}

The trailing * on the resource is doing quiet but important work. It matches both the state file and the .tflock lock file, since the lock file shares the same path with .tflock tacked on. One rule covers both.

6. Initialize Terraform

Run init to wire everything up. Terraform reads your backend block and gets the bucket ready.

terraform init

7. Test that the lock works

Run a normal plan and apply.

terraform plan
terraform apply

While the apply is running, open the S3 console or run aws s3 ls on your bucket path. You will see a terraform.tfstate.tflock object appear during the run and vanish once it finishes. If you want to see the lock in action, open a second terminal and start another terraform apply while the first one is still going. The second one will hit that 412 lock error from earlier and refuse to run until the first is done. That is the whole point working exactly as designed.

Moving over if you already use DynamoDB

If you have an existing setup with a DynamoDB lock table, switching is easy and you do not have to do it all in one nervous jump.

The cautious approach is to keep both options set for a short transition period. You can list use_lockfile = true and dynamodb_table at the same time, and Terraform will acquire locks in both places. That way nothing breaks while you and your team catch up.

terraform {
  backend "s3" {
    bucket         = "my-terraform-state-bucket"
    key            = "global/s3/terraform.tfstate"
    region         = "ap-south-1"
    encrypt        = true
    dynamodb_table = "terraform-locks"
    use_lockfile   = true
  }
}

After changing the backend, run init again with the reconfigure flag so Terraform picks up the new settings.

terraform init -reconfigure

Once you are confident everyone is on Terraform 1.10 or newer and the S3 locking is working, remove the dynamodb_table line, run terraform init -reconfigure one more time, and then delete the DynamoDB table you no longer need.

When a lock gets stuck

Every so often a Terraform run crashes halfway, or someone closes their laptop mid apply. When that happens the .tflock file can be left behind even though no operation is actually running. The next person then gets the lock error for a lock nobody holds.

To clear it, grab the lock ID from the error output (the ID field) and run force unlock.

terraform force-unlock 3f6a1c9e-...-b2d4

Use this carefully. Only force unlock when you are sure no real Terraform operation is running anywhere, otherwise you are back to the exact race condition that locking was meant to prevent. If in doubt, ask your team before you unlock.

Why this genuinely reduces complexity

Pull back and look at what you removed. You went from two services to one. There is no DynamoDB table to create, name, tag, watch, or clean up. Your lock now lives in the same bucket as your state, so there is one place to look when you want to understand what is going on.

The bootstrap problem shrinks too. Every Terraform team hits the same chicken and egg puzzle at the start. To use remote state you first need the backend infrastructure to exist, but that infrastructure is itself something you want to manage. With the old approach that meant standing up a bucket and a table before anything else. Now it is just a bucket.

Your permissions shrink. No DynamoDB actions to grant means a shorter IAM policy and fewer things a reviewer has to reason about. Your Terraform code shrinks, because the whole aws_dynamodb_table resource can be deleted. And you cut out a category of quiet failures, like a lock table sitting in the wrong region or a name that does not match the backend config.

Same safety, fewer moving parts. That is a rare and welcome trade.

Wrapping up

State locking has always been one of those things you set up once, forget about, and only remember when it saves you from corrupting shared state. The mechanics used to lean on DynamoDB because S3 could not do the job alone. Now that S3 supports conditional writes, that reason is gone, and Terraform 1.10 gives you a one line switch to take advantage of it.

If you are starting a new project, reach for use_lockfile from day one. If you already have a DynamoDB table doing this work, plan a calm migration, keep both settings on for a bit, then retire the table. Your future self, staring at a much simpler backend block, will thank you.

Let us connect

If you have questions, ran into a snag while trying this, or just want to talk Terraform and AWS, reach out at khantanseer43@gmail.com. I am always happy to help someone get unstuck.

Build and Deploy a 3 Tier Serverless App on AWS with Terraform Modules

Tanseer — Tue, 14 Jul 2026 04:59:11 +0000

Introduction

Most tutorials show you how to create a Lambda function by clicking around the AWS console. That works once. Then you need a staging environment, and you click everything again. Then production. Then someone asks "wait, did staging have the 30 second timeout or was that only dev?" and nobody knows, because the infrastructure lives in memory and screenshots.

This post walks through a project where the entire 3 tier stack comes from a single terraform apply. Tier one is the frontend on Amplify, tier two is the API on Lambda and API Gateway, tier three is a MySQL database on RDS. Run the apply in the dev folder, you get a dev environment. Run it in prod, you get prod. Same code, zero clicking.

The app itself is a small todo application. That is on purpose. The app is just a passenger. The real work is the Terraform: reusable modules, three isolated environments, remote state stored in S3, and module outputs wired into module inputs so everything builds in the correct order.

The full source is here: github.com/TanseerS/serverless-web-application

Quick note on the word serverless before we start. It does not mean there are no servers. It means AWS manages the servers and you only bring code. You pay when the code runs, not while it sits idle.

What we are building

The flow is simple to trace. A user opens a React app hosted on AWS Amplify (a service that builds and hosts frontends straight from a GitHub repo). The app calls a REST API on API Gateway (the AWS service that gives your code an HTTPS URL). Every route on that API points to a single Lambda function (code that runs on demand without a server you manage). Inside the Lambda, an Express app handles the routing and talks to a MySQL database on RDS (managed relational databases on AWS).

Deployments are split in two. Push to GitHub and Amplify rebuilds the frontend on its own, since CI/CD comes free with Amplify. Change the backend code or any infrastructure, and terraform apply ships it. Terraform even re-zips the backend folder automatically whenever the source changes.

The repository layout

serverless-web-application/
├── frontend/            # React 18 + Vite app
├── backend/             # Express + Sequelize API, packaged for Lambda
├── assets/              # diagrams and screenshots
└── infrastructure/
    ├── bootstrap/       # run once, creates the S3 state bucket
    ├── modules/
    │   ├── database/    # RDS MySQL, security group, subnet group
    │   ├── backend/     # Lambda, API Gateway, IAM, code packaging
    │   └── frontend/    # Amplify app, branch, IAM
    └── environments/
        ├── dev/
        ├── staging/
        └── prod/

Two ideas make this layout work.

First, modules are reusable building blocks. The database module does not know or care whether it is building dev or prod. It takes a project and an environment variable and names everything ${var.project}-${var.environment}-..., so the same code stamps out todo-app-dev-rds, todo-app-staging-rds and todo-app-prod-rds.

Second, each environment folder is tiny. It looks up the default VPC with data sources, calls the three modules, and passes in values from its own terraform.tfvars file. Promoting a change from dev to prod means changing directory and applying again. You never edit module code to deploy an environment.

Step 1: solve the chicken and egg problem

Terraform keeps a state file, which is its record of everything it has created. By default that file sits on your laptop, which is fine until your laptop is not around or a teammate runs an apply and now there are two versions of the truth. The fix is remote state: store the file in an S3 bucket instead.

But here is the catch. The S3 bucket has to exist before Terraform can store state in it, and we want Terraform to create the bucket. Who creates the creator?

The answer is a small run once setup called the bootstrap pattern. The infrastructure/bootstrap/ folder is a tiny Terraform project with plain local state. Its only job is to create the state bucket. You run it once per AWS account and forget about it.

data "aws_caller_identity" "current" {}

resource "aws_s3_bucket" "this" {
  bucket = "${var.project}-statefile-${data.aws_caller_identity.current.account_id}"
}

Notice the bucket name. S3 bucket names are globally unique across every AWS account on the planet, so todo-app-statefile alone would probably be taken. The aws_caller_identity data source reads your AWS account ID at plan time and appends it, so the name is unique without hardcoding anything.

The bootstrap also turns on three protections for the bucket: versioning (every old copy of your state is kept, which is what saves you when an apply goes wrong), full public access blocking, and AES 256 encryption at rest.

Step 2: remote state with S3 native locking

With the bucket in place, every environment points at it through a backend.tf file. Same bucket, different key per environment:

terraform {
  backend "s3" {
    bucket       = "todo-app-statefile-<account-id>"
    key          = "dev/terraform.tfstate"
    region       = "ap-south-1"
    encrypt      = true
    use_lockfile = true
  }
}

Because dev, staging and prod each write to their own key, an apply in dev can never touch prod. They are fully isolated.

The use_lockfile = true line deserves attention. State locking stops two people from running apply at the same time and corrupting the state. For years the standard answer was a DynamoDB table just for the lock. Terraform 1.10 added native S3 locking: Terraform drops a lock object into the bucket itself while an apply runs. One less resource to create, pay for and explain.

Here is the bucket after all three environments have been initialized, one prefix per state file:

Step 3: the database module

The database module creates three things. A DB subnet group that tells RDS which subnets it can live in. A security group (a virtual firewall) allowing inbound MySQL on port 3306. And the RDS instance itself:

resource "aws_db_instance" "this" {
  identifier        = "${var.project}-${var.environment}-rds"
  db_name           = "TodoAppDb"
  engine            = "mysql"
  engine_version    = "8.0"
  allocated_storage = 20
  storage_type      = "gp3"
  instance_class    = "db.t3.micro"

  username = var.username
  password = var.password

  db_subnet_group_name   = aws_db_subnet_group.this.name
  vpc_security_group_ids = [aws_security_group.this.id]

  publicly_accessible = true
  skip_final_snapshot = true
}

MySQL 8.0 on a db.t3.micro with 20 GB of gp3 storage, which keeps it inside the free tier.

One honest admission. The security group currently allows 3306 from 0.0.0.0/0, meaning anywhere on the internet, and the instance is publicly accessible. That is a demo shortcut so the Lambda can reach the database without living inside the VPC. Production would put RDS in private subnets, place the Lambda inside the VPC, and scope the security group to the Lambda alone. The repo says this out loud in its trade offs section rather than pretending it is fine.

The module ends by outputting db_endpoint, db_name and db_port. Remember those. They are about to become someone else's inputs.

Step 4: the backend module, where most of the Terraform lives

This module packages the code, creates the Lambda, and builds the entire REST API. A few pieces are worth slowing down for.

Packaging code without a build script

data "archive_file" "this" {
  type        = "zip"
  source_dir  = "${path.module}/../../../backend"
  output_path = "${path.module}/../../build/function.zip"
}

Terraform zips the backend/ folder at plan time. The Lambda resource then sets source_code_hash = data.archive_file.this.output_base64sha256, which means the function only gets updated when the code actually changed. Edit a Terraform file and apply, the Lambda is untouched. Edit routes/tasks.js and apply, Terraform notices the hash changed and ships the new zip. No separate deploy pipeline for the backend at all.

The Lambda function

resource "aws_lambda_function" "this" {
  function_name    = "${var.project}-${var.environment}-lambda-function"
  handler          = "server.handler"
  runtime          = "nodejs22.x"
  timeout          = 30
  memory_size      = 512

  environment {
    variables = {
      DB_HOST     = var.db_endpoint
      DB_NAME     = var.db_name
      DB_PORT     = var.db_port
      DB_USER     = var.username
      DB_PASSWORD = var.password
    }
  }
}

Node.js 22, 512 MB of memory, 30 second timeout. The database connection details arrive as environment variables, and look where they come from: var.db_endpoint is the output of the database module. The environment folder wires them together like this:

module "backend" {
  source      = "../../modules/backend"
  db_endpoint = module.database.db_endpoint
  db_name     = module.database.db_name
  db_port     = module.database.db_port
  ...
}

This is the trick that makes the whole project build in the right order. Terraform sees that the backend needs an output from the database, so it builds the database first. Nobody wrote depends_on for that. The dependency graph falls out of the references themselves.

Eight API routes from one map

The API has four paths (/api, /api/health, /api/tasks, /api/tasks/{id}) and eight method and path combinations once you count the OPTIONS methods needed for CORS preflight. Writing eight method blocks and eight integration blocks by hand would be sixteen nearly identical resources. Instead, the routes live in one locals map:

locals {
  routes = {
    health_get    = { resource_id = aws_api_gateway_resource.health.id,     http_method = "GET" }
    tasks_get     = { resource_id = aws_api_gateway_resource.tasks.id,      http_method = "GET" }
    tasks_post    = { resource_id = aws_api_gateway_resource.tasks.id,      http_method = "POST" }
    task_put      = { resource_id = aws_api_gateway_resource.task_by_id.id, http_method = "PUT" }
    task_delete   = { resource_id = aws_api_gateway_resource.task_by_id.id, http_method = "DELETE" }
    # ...plus three OPTIONS entries
  }
}

resource "aws_api_gateway_method" "this" {
  for_each      = local.routes
  rest_api_id   = aws_api_gateway_rest_api.this.id
  resource_id   = each.value.resource_id
  http_method   = each.value.http_method
  authorization = "NONE"
}

The for_each loops over the map and stamps out one resource per entry. The integrations do the same, every one an AWS_PROXY integration pointing at the Lambda, which means API Gateway forwards the raw request and lets Express figure out the routing inside the function. Adding a new route later means adding one line to the map.

The silent API Gateway gotcha

Here is a bug that costs beginners hours. API Gateway keeps a deployed snapshot of your API separate from its configuration. You can change methods and integrations in Terraform, apply successfully, and API Gateway will keep serving the old snapshot without a single complaint. Everything looks green. Your change is just not live.

The fix is to force a redeploy whenever the routing changes:

resource "aws_api_gateway_deployment" "this" {
  rest_api_id = aws_api_gateway_rest_api.this.id

  triggers = {
    redeploy = sha1(jsonencode([
      aws_api_gateway_method.this,
      aws_api_gateway_integration.this,
    ]))
  }

  lifecycle {
    create_before_destroy = true
  }
}

The trigger is a hash of the routing config. Config changes, hash changes, Terraform creates a fresh deployment. The create_before_destroy lifecycle rule makes sure the new deployment exists before the old one goes away, so the stage never has a gap where the API is down.

Step 5: the frontend module

The Amplify app connects directly to the GitHub repo using a personal access token, with the build instructions written inline:

resource "aws_amplify_app" "this" {
  name         = "${var.project}-${var.environment}-amplify-app"
  repository   = var.repository
  access_token = var.github_access_token

  environment_variables = {
    VITE_API_URL = var.api_uri
  }
}

That VITE_API_URL line is my favorite detail in the project. The value comes from module.backend.api_invoke_url, the output of the backend module. So Terraform creates the API, learns its URL, and hands it to Amplify as a build time environment variable. The React app bakes it in during npm run build. The frontend learns its backend address from Terraform and nothing is ever hardcoded.

The module also adds a rewrite rule so deep links like /tasks/5 fall back to index.html instead of returning 404, which every single page app on Amplify needs, and a branch resource with auto build turned on. The branch stage is set with a conditional: var.environment == "prod" ? "PRODUCTION" : "DEVELOPMENT".

Step 6: the backend code that makes Express work in Lambda

A normal Express app ends with app.listen(3000). There is no port to listen on inside Lambda, so the app is wrapped instead:

const serverlessExpress = require('@vendia/serverless-express');
const app = express();
// routes...
exports.handler = serverlessExpress({ app });

The @vendia/serverless-express package translates the event API Gateway sends into the request object Express expects, and translates the response back. Your Express code does not change at all.

One more pattern worth stealing. The database table has to exist before the first query, but you do not want to create it on every request. The code caches the promise:

let dbReady = null;
app.use('/api/tasks', async (req, res, next) => {
  try {
    dbReady = dbReady || sequelize.sync();
    await dbReady;
    next();
  } catch (err) {
    dbReady = null; // don't cache a failed attempt
    res.status(500).json({ error: err.message });
  }
}, tasksRouter);

The first request in each Lambda container runs sequelize.sync() which creates the tasks table if needed. Every later request awaits the already resolved promise, which costs nothing. And if the sync fails, the cache resets so the next request tries again instead of being stuck with a failed promise forever. There is no manual schema step anywhere in the deployment.

Deploying it yourself

You need an AWS account with credentials configured, Terraform 1.15 or newer, and a GitHub personal access token that can read your fork and create webhooks.

First, bootstrap the state bucket, once per account:

cd infrastructure/bootstrap
terraform init
terraform apply

Second, edit backend.tf in each environment folder and set the bucket name to what the bootstrap printed. Backend blocks cannot take variables, so this one value is a literal.

Third, deploy an environment:

cd infrastructure/environments/dev
cp terraform.tfvars.example terraform.tfvars   # fill in DB password, repo URL, token, branch
terraform init
terraform plan
terraform apply

RDS takes a few minutes to come up. The result is 28 resources from one command:

Fourth, trigger the first frontend build. Amplify only builds on new commits, so push something to the configured branch after the apply. An empty commit does the job.

Finally, check it is alive:

curl https://<api-id>.execute-api.ap-south-1.amazonaws.com/dev/api/health
# {"status":"ok","database":"connected"}

What actually gets created

Everything below came from the same modules, applied once per environment folder. Three Lambda functions on Node.js 22:

Three regional REST APIs, each with a stage named after its environment:

A MySQL instance per environment:

And the Amplify apps, connected to GitHub and building on every push:

Notice what did not happen. Nobody opened the RDS console to click Create database three times. The staging environment is not "dev, but with some settings someone changed by hand last month." All three are provably identical because they came from identical code.

Honest trade offs

A few shortcuts were taken deliberately, and it is worth knowing what production would change. The open security group and public RDS would become private subnets with the Lambda inside the VPC. Database credentials would move from Lambda environment variables into Secrets Manager. The single AZ db.t3.micro with skip_final_snapshot would become Multi AZ with deletion protection. And the hardcoded bucket name in backend.tf would be handled with partial backend config through terraform init -backend-config.

The roadmap includes a GitHub Actions pipeline that runs terraform fmt, validate and plan on pull requests and applies on merge, plus tflint and checkov for static checks.

Conclusion

The todo app is forgettable. The patterns are not. The bootstrap trick for remote state, S3 native locking without a DynamoDB table, modules whose outputs feed the next module's inputs so the build order takes care of itself, for_each over a map instead of sixteen copy pasted resources, and a hash trigger that stops API Gateway from silently serving stale config. Every one of these shows up in real production Terraform, and this repo is a small enough place to learn them without drowning.

Clone it, deploy dev, then run terraform destroy when you are done exploring. RDS is the only always on cost here, and tearing it down is one command, the same way bringing it up was.

Repo: github.com/TanseerS/serverless-web-application

Contact

Questions, feedback, or stuck on a step? Reach me at khantanseer43@gmail.com. Happy to help.

Go Live in Minutes: Building Your First Live Stream with Amazon IVS

Tanseer — Tue, 07 Jul 2026 05:08:24 +0000

The same tech that powers Twitch, minus the servers, minus the headache

A few weeks ago I wanted to test a live streaming idea. My first thought was the usual one. I will need a media server, something like Wowza or a self hosted RTMP setup on EC2, then a CDN in front of it, then hours of tuning latency. Before I even opened the console I was already tired.

Then I found Amazon IVS, and honestly, the whole thing took less time than making tea.

If you are a developer who has never touched live streaming on AWS before, this post is for you. By the end of it you will know exactly what IVS is, why it is different from rolling your own setup, and how to go live in four steps.

What exactly is Amazon IVS

Amazon Interactive Video Service, or IVS for short, is a fully managed live streaming service from AWS. Fully managed means AWS runs everything behind the scenes. You do not create servers, you do not install streaming software on the cloud, and you do not worry about scaling.

Here is the part that made me sit up. IVS runs on the exact same infrastructure that powers Twitch. Twitch is one of the biggest live streaming platforms in the world, handling millions of concurrent viewers every single day. When you create an IVS channel, your tiny test stream rides on that same battle tested backbone.

So instead of you building a smaller, buggier version of Twitch infrastructure, AWS just hands you Twitch infrastructure as an API.

The killer feature: ultra low latency

If you have ever streamed with a typical setup, you know the pain. You wave at the camera, and viewers see the wave 15 to 30 seconds later. That delay is called latency, the time it takes for video to travel from your camera to the viewer's screen.

IVS brings that down to under 3 seconds.

Why does that matter? Because under 3 seconds, a stream stops feeling like a broadcast and starts feeling like a conversation. Viewers can react in chat and you can respond almost instantly.

IVS also lets you sync interactive elements with the video. Think live chat, polls, quizzes, or even shopping buttons that appear exactly when the host says "buy this now." This is done through something called timed metadata, small pieces of data you inject into the stream that arrive on the viewer's screen in sync with the video frame. For a marketplace or live commerce product, this is gold.

How is it serverless

The word serverless gets thrown around a lot, so let me be specific about what it means here.

There are no servers for you to manage. No EC2 instances, no patching, no capacity planning. You create a channel and AWS handles the rest.

It scales automatically. Whether 5 people show up to your stream or 5 million, you do nothing. The same channel handles both. There is no "upgrade your instance" moment at 2 AM when your stream suddenly goes viral.

You pay only for what you use. IVS pricing is based on two things: hours of video you send into the service (input) and hours of video delivered to viewers (output). No stream running means no bill running. This is the same pay as you go model that makes Lambda and other serverless tools so friendly for side projects.

Going live in 4 steps

This is the part that genuinely surprised me. Here is the entire setup.

Step 1: Create a channel

Log in to the AWS Console and search for Amazon IVS. Click Create channel, give it a name, and keep the default configuration for now. That is it. AWS provisions the channel in seconds.

Step 2: Copy your Ingest Server and Stream Key

Open your new channel and you will see two important values. The Ingest Server is the address your video gets sent to, something like rtmps://xxxxxx.global-contribute.live-video.net. The Stream Key is a secret token that proves the stream belongs to you. Treat the stream key like a password. Anyone who has it can broadcast on your channel.

Step 3: Paste them into OBS Studio

OBS Studio is free, open source broadcasting software and the de facto standard for streaming. Download it, open Settings, go to Stream, and choose Custom as the service. Paste the Ingest Server into the Server field and the Stream Key into the Stream Key field. Save.

Step 4: Click Start Streaming

Back on the main OBS screen, add a source (your webcam or screen), then hit Start Streaming. Switch to the AWS Console, open your channel, and you will see yourself live in the built in preview player within seconds.

Four steps. No infrastructure. You are now running on Twitch grade streaming systems.

Where to go from here

The console preview is just the beginning. IVS gives you a playback URL for every channel, and AWS provides a player SDK for web, iOS, and Android. That means you can drop your live stream into your own React app or mobile app with a few lines of code. Add the IVS chat feature and timed metadata, and you have a real interactive streaming product without touching a single server.

Conclusion

Live streaming used to be a project measured in weeks. With Amazon IVS it is a project measured in minutes. You get the infrastructure behind Twitch, latency under 3 seconds, automatic scaling from a handful of viewers to millions, and a bill that only grows when your stream does. If you have been putting off a live video feature because the plumbing felt scary, this is your sign to create a channel and press Start Streaming.

Let's connect

If you tried this and got stuck somewhere, or if you built something cool with IVS, I would love to hear about it. Reach out at khantanseer43@gmail.com.

Terraform guide to IAM Policies: The Two Cases You Need to Know

Tanseer — Wed, 01 Jul 2026 11:21:21 +0000

When to attach a ready made policy and when to write your own

When you first start writing IAM setup in Terraform, the examples online can feel like they contradict each other. One tutorial writes a long block called aws_iam_policy_document. Another just pastes a single line with a long string inside it. Both of them work, so which one is correct?

The answer is that both are correct. They simply solve two different problems. Once you can tell which situation you are in, IAM in Terraform stops feeling random and starts feeling obvious.

Before we get into the two cases, here are the words you will keep seeing:

IAM stands for Identity and Access Management. It is the AWS service that decides who is allowed to do what. An IAM role is an identity that an AWS service (like a Lambda function) takes on so it can act with certain permissions. An IAM policy is the document that lists which actions are allowed or denied. An ARN, short for Amazon Resource Name, is the unique address of a resource in AWS.

With that out of the way, let us look at the two cases.

Case 1: AWS Managed Policy (attach directly)

A managed policy is a policy that AWS already wrote and maintains for you. If AWS already provides the exact permissions you need, you do not create your own aws_iam_policy_document. You just point your role at the policy AWS already made.

For example, giving a Lambda function permission to write logs is a very common need, and AWS has a ready made policy for it:

resource "aws_iam_role_policy_attachment" "logs" {
  role       = aws_iam_role.this.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

Another common one is letting a Lambda function run inside a VPC (a Virtual Private Cloud, which is your own private network in AWS):

resource "aws_iam_role_policy_attachment" "vpc" {
  role       = aws_iam_role.this.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole"
}

Notice what is happening here. You are not writing any permissions yourself. You are taking an existing AWS managed policy, referring to it by its ARN, and attaching it to your role. That is the whole job.

Case 2: Your own custom permissions

Sometimes AWS does not have a managed policy that matches exactly what you need. Maybe you want a Lambda function to read and write files in one specific S3 bucket and nothing else. In that case you build the policy yourself, and it takes three small steps.

Step 1: Create the policy document

First you describe the permissions. This aws_iam_policy_document is a data source, which means it does not create anything in AWS on its own. It is a way to write your permission rules in clean Terraform instead of raw JSON.

data "aws_iam_policy_document" "s3_access" {
  statement {
    effect = "Allow"

    actions = [
      "s3:GetObject",
      "s3:PutObject"
    ]

    resources = [
      aws_s3_bucket.uploads.arn,
      "${aws_s3_bucket.uploads.arn}/*"
    ]
  }
}

Reading this in plain English: allow the actions of getting an object and putting an object, and only on the uploads bucket and everything inside it.

Step 2: Create the IAM policy

Now you turn that document into a real IAM policy that exists in AWS. The policy field pulls in the JSON produced by the document from Step 1.

resource "aws_iam_policy" "s3_access" {
  name   = "${var.project}-${var.environment}-s3-access"
  policy = data.aws_iam_policy_document.s3_access.json
}

Step 3: Attach it

Finally you attach the policy you just created to your role. This looks almost the same as Case 1, but the ARN now comes from your own policy instead of an AWS managed one.

resource "aws_iam_role_policy_attachment" "s3_access" {
  role       = aws_iam_role.this.name
  policy_arn = aws_iam_policy.s3_access.arn
}

The mental model

The easiest way to remember the difference is this. In Case 1 the policy already exists, so you skip straight to attaching. In Case 2 you have to build the policy first, then attach it.

The custom flow always moves in the same direction:

Policy Document
      │
      ▼
IAM Policy
      │
      ▼
Attach to Role

You write the document, the document becomes a policy, and the policy gets attached to the role. If you ever feel lost, just ask yourself one question: does AWS already have a policy for this? If yes, attach it directly. If no, build your own with the three steps above.

Conclusion

IAM in Terraform is not as confusing as it first looks. There are really only two situations. When AWS already provides the exact permissions, you attach a managed policy by its ARN in a single resource. When you need something specific that AWS does not offer, you create a policy document, turn it into an IAM policy, and attach that.

Keep that fork in mind and you will always know which approach the moment fits, instead of copying an example and hoping it is the right one.

Let us connect

If you have questions or want to share how you handle IAM in your own projects, reach out at khantanseer43@gmail.com. I am always happy to talk Terraform and AWS.

Lambda MicroVMs Are Kind of Insane: Tiny Virtual Machines That Boot in Milliseconds and Sleep When You Are Not Looking

Tanseer — Tue, 30 Jun 2026 04:53:28 +0000

A beginner friendly guide to the Firecracker powered microVMs behind AWS Lambda, the new suspend and resume superpower, and the big question everyone keeps asking: is this still serverless?

Introduction

Imagine a virtual computer so small and so fast that it starts up in less time than it takes you to blink. Now imagine thousands of them running on a single physical server, each one completely walled off from its neighbours, and each one quietly going to sleep the moment it has nothing to do, so that you stop paying for it.

That is not a science fiction pitch. That is roughly what is happening inside AWS Lambda today, and the newer microVM capabilities push the idea even further.

In this post we will start from zero. We will explain what a microVM even is, meet the tiny engine that makes the whole thing possible, walk through what happens when a single request comes in, and then look at the features that make people say this is a little bit crazy. By the end you will be able to answer the question that is dividing the cloud community: after all of this, is Lambda even serverless anymore?

Let us get into it.

So What Exactly Is a Lambda MicroVM?

A microVM is a tiny, lightweight virtual machine. It behaves like its own small computer with a strong security wall around it, but it throws away almost all the heavy baggage of a normal machine, so it starts in milliseconds and uses very little memory. Picture a tiny, fast, single purpose box that appears, does one job, and disappears.

Here is the part that surprises most people. Lambda has always run your code inside these microVMs. Every time your function runs, AWS quietly spins up a tiny isolated machine just for you. The recent excitement is about giving those microVMs new powers, such as the ability to pause when idle and to be reached directly over the internet. So when people say "Lambda MicroVMs," they are really talking about this next chapter of an idea that has been humming away under the hood for years.

Meet Firecracker, the Tiny Engine Doing All the Work

Powering all of this is Firecracker, a technology that AWS built and then released to the public as open source. AWS created it specifically to run microVMs safely at massive scale, and it is the reason a tiny machine can boot in around 125 milliseconds while still keeping the strong security wall of a real virtual machine. This same AWS made engine quietly runs underneath both Lambda and AWS Fargate.

How It Works: The Journey of a Single Request

Let us follow one request from start to finish so the architecture clicks.

You upload your code. This can be a simple zip file or a container image. You also choose how much memory the function gets, which in turn decides how much CPU power it receives.
Something triggers it. A user hits an endpoint, a file lands in storage, a message arrives on a queue, and Lambda decides your function needs to run.
A microVM appears. AWS spins up a Firecracker microVM with your chosen runtime and your code inside. This little box is called the execution environment. If a fresh one has to be created from scratch, that small delay is known as a cold start.
Your code runs. The environment initialises, your handler runs, and a response goes back.
The environment is kept around. Instead of throwing the microVM away immediately, Lambda often keeps it ready for a little while. If another request arrives soon, it reuses the same warm environment and skips the startup cost entirely. This is called a warm start, and it is why the second request to a function usually feels much faster than the first. So the architecture is really a fleet of these tiny machines being created, reused, and recycled constantly, all managed for you so you never touch a server.

Each One Gets Its Own Front Door: Dedicated HTTP Endpoints

An endpoint is just a web address that something can send a request to. HTTP is the everyday language of the web that browsers and apps use to talk to servers.

For a long time, if you wanted your Lambda function to be reachable over the web, you usually had to put another service in front of it to handle the incoming traffic. The newer approach gives a function its own dedicated HTTPS endpoint, a clean and direct web address that points straight at it. No extra plumbing in between.

The address looks something like this:

https://your-unique-id.lambda-url.your-region.on.aws

You can keep it open or lock it down so only authorised callers get through. For simple use cases, such as a webhook that some other service calls, or a small backend for a single page app, this is wonderfully direct. Your request lands on a private little microVM that exists just to answer you.

The Real Magic: Suspend and Resume

This is the feature that earns the word "crazy."

To understand it, you need one more idea: a snapshot. A snapshot is a complete saved picture of a running machine at a single moment in time, including everything sitting in its memory. If you take a snapshot and then restore it later, the machine wakes up exactly where it left off, as if no time had passed at all.

Firecracker can do this with microVMs. It can freeze a running microVM, save its full state, and bring it back to life almost instantly.

Now think about what that unlocks. Traditionally you had two bad choices. Either you kept a server running all the time so it was ready to respond instantly, and you paid for every idle second. Or you shut it down to save money and then suffered a slow start whenever traffic came back.

Suspend and resume gives you a third option. The microVM can stay paused while nothing is happening, and the headline benefit is simple: you are not paying for a fully running machine while it sits idle. When a request finally arrives, it wakes up quickly and carries on. You get the responsiveness of an always on server with a cost profile much closer to pay only for what you use.

That is the holy grail people have chased for years. Be instantly ready, but only pay when you are actually doing work.

Lambda MicroVM vs EC2 vs Classic Lambda

It helps to see where this sits between the two options most people already know.

EC2 is a rented virtual server that you control and manage yourself. Classic Lambda is short lived, event driven functions with a hard time limit. The microVM approach lands in the middle, borrowing the best of each.

Feature	EC2	Classic Lambda	Lambda MicroVM
What you manage	The whole server	Just your code	Just your code
How long it can run	As long as you like	Up to 15 minutes per run	Designed for longer and more stateful work than the 15 minute limit
Startup speed	Slow, minutes	Fast, with occasional cold starts	Fast, with quick resume from a paused state
Cost when idle	You keep paying while it runs	Nothing, it is not running	The big win is not paying for a fully running machine while idle
Scaling	You set it up	Automatic, scales to zero	Automatic, scales to zero
Best feeling	Total control	Pure simplicity	Control and simplicity together

The short version is this. EC2 gives you full control but you babysit it and pay for idle time. Classic Lambda is beautifully simple but tightly constrained. The microVM model tries to give you longer running, more flexible compute that still scales down to nothing when no one is using it.

What Is This Actually Good For?

A new tool is only exciting if it solves real problems. Here is where this approach shines.

It is great for long running jobs that do not fit neatly into a short window, such as heavier data processing or video work that would bump into the classic 15 minute ceiling.

It suits stateful sessions, meaning work where the machine needs to remember something between requests rather than starting fresh every single time.

It is a natural fit for AI and machine learning inference, where a large model can be expensive to load. You load it once, let the machine pause when traffic is quiet, and resume instantly when the next request comes, instead of paying to keep a big GPU style box running all day.

It works beautifully for interactive sandboxes and code execution, the kind of thing where each user needs their own safe, isolated little environment that spins up fast and tears down cleanly.

And it is useful for bursty traffic, where requests come in unpredictable spikes. You get instant readiness during the spike and near zero cost during the long quiet stretches in between.

When would you not bother? If your workload is genuinely tiny and event driven, classic Lambda is still the simplest possible answer. And if you need deep control over the operating system, custom networking, or hardware that must run nonstop, a traditional server like EC2 is still the right call.

What About the Price?

Cost is usually the deciding factor, so let us compare the three billing styles in plain terms. To follow along, you need one bit of vocabulary: a GB second is just one gigabyte of memory used for one second. It is the unit Lambda uses to measure how much work you actually consumed.

EC2 charges you for the time the server is switched on, whether or not anyone is using it. A box running quietly overnight with zero visitors still appears on your bill. You pay for capacity, not for usage.

Classic Lambda charges you in two parts: a tiny amount for each request, plus an amount for compute measured in those GB seconds. If nobody calls your function, it costs you nothing. You pay for usage, not for capacity.

The microVM model aims for the best of both. You get a more capable, longer lived machine, but the key promise is that you are not charged for a fully running VM while it is paused and idle. You pay mainly for the active time when it is actually doing something.

Important: cloud pricing changes often, and the exact rates for these options vary by region and configuration. Rather than trust any single number you read in a blog, including this one, always check the current AWS pricing pages before you make a decision.

So Is It Still Serverless?

Here is the fun part, and it is genuinely a matter of debate.

The word serverless never meant there were no servers. It meant you, the developer, never had to think about them. The classic definition came with a few expectations: your code is event driven and short lived, it scales automatically all the way down to zero, you never manage infrastructure, and you only pay for what you use.

Now look at what these microVMs offer. Long running. Able to hold state. Reachable through their own dedicated web address. Sitting paused and ready rather than vanishing completely. To some people that starts to look an awful lot like a small, managed server wearing a serverless costume.

The case for "yes, it is still serverless" is strong. You still never patch an operating system. It still scales to zero. You still pay for value rather than for idle capacity. By the spirit of the original promise, nothing has been broken.

The case for "no, the purity is gone" is also fair. The original idea of a tiny, stateless, event triggered function that lives for a moment and dies has clearly been stretched into something bigger and more persistent.

The honest answer is probably this. Serverless is becoming a spectrum rather than a strict box. On one end you have pure functions that flash in and out of existence. On the other you have long lived managed machines. These microVMs live somewhere in the comfortable middle, and they are quietly redefining what the word can mean. Maybe the better question is not "is this serverless," but "does this let me ship without babysitting infrastructure." If the answer is yes, the label matters a lot less than the freedom.

Wrapping Up

Lambda has always been powered by tiny, fast, isolated microVMs running on Firecracker. What is changing is how clever those microVMs are getting. They can now boot in a blink, answer the web directly, and most remarkably of all, doze off when idle so you are not paying for a machine that is doing nothing.

Whether you call it serverless, managed compute, or something brand new, one thing is clear. The line between a server and a function is getting blurrier, and that is a great problem for developers to have. The more the cloud handles for us, the more time we get to spend building the things we actually care about.

If you are new to AWS, the best next step is to try spinning up a simple function, give it a dedicated endpoint, and call it from your browser. Seeing your own tiny microVM answer back is the moment the whole idea finally clicks.

Let Us Connect

If you found this helpful, have a question, or spot something you want to discuss, I would love to hear from you. Reach out at khantanseer43@gmail.com.

Who Wins the Variable Fight in Terraform?

Tanseer — Tue, 30 Jun 2026 04:39:15 +0000

A beginner friendly guide to understanding which value Terraform actually picks when the same variable is set in many places

Introduction

When you start writing Terraform, you quickly learn that you can set the same variable in many different ways. You can give it a default value, you can put it in a file, you can pass it on the command line, and you can even set it through your computer's environment.

This is great for flexibility, but it also raises a confusing question. If the same variable gets a value from five different places, which value does Terraform actually use?

This is called variable precedence. Precedence simply means the order of priority. It is the set of rules Terraform follows to decide who wins when there is a conflict.

In this post we will walk through every way you can set a variable, then look at the exact order Terraform uses to pick a winner. We will use small, repeatable examples so you can try each one yourself.

Quick note on the words we will use. A variable is just a named value you can reuse, like a name tag on a box. A value is what is inside that box. Precedence is the priority order Terraform uses when two boxes have the same name tag but different contents.

First, the ways you can set a variable

Before we talk about who wins, let us list all the players in the game. There are six common ways to give a variable a value in Terraform.

The first is a default value. You write this directly inside the variable block in your code.

variable "region" {
  type    = string
  default = "us-east-1"
}

The second is an environment variable. An environment variable is a value stored in your computer's shell, outside of Terraform. Terraform reads any environment variable that starts with TF_VAR_ followed by the variable name.

export TF_VAR_region="us-west-2"

The third is a file called terraform.tfvars. A tfvars file is a plain file where you list variable names and the values you want to give them. Terraform loads this file automatically without you doing anything.

region = "ap-south-1"

The fourth is a file called terraform.tfvars.json. This is the same idea as above, but written in JSON format instead. JSON is just another way of writing structured data using curly braces and quotes.

{
  "region": "eu-west-1"
}

The fifth is any file ending in .auto.tfvars or .auto.tfvars.json. The word auto here means automatic. Terraform loads these files for you automatically too, just like the standard tfvars file. A common example name is prod.auto.tfvars.

region = "ca-central-1"

The sixth is a command line flag. A flag is an extra instruction you type when running a command. You can pass a value directly with -var, or point to a specific file with -var-file.

terraform apply -var="region=sa-east-1"

The order that decides the winner

Here is the part everyone wants to know. Terraform checks these sources in a fixed order, from lowest priority to highest priority. A source that comes later in the list overrides anything set earlier.

Here is the order from weakest to strongest.

The default value inside the variable block. This is the fallback that is used only when nothing else provides a value.
Environment variables that start with TF_VAR_.
The terraform.tfvars file.
The terraform.tfvars.json file.
Any files ending in .auto.tfvars or .auto.tfvars.json, loaded in alphabetical order of their file names.
The -var and -var-file flags on the command line, applied in the exact order you type them. The simplest way to remember this is that the command line always wins, and the default value always loses. Everything else sits in the middle.

A full example you can run

Let us prove the rules with a tiny project. Create a folder and add the following files.

First the variable declaration in a file called variables.tf.

variable "region" {
  type    = string
  default = "us-east-1"
}

Next a file called main.tf that just prints the value back to us using an output. An output is simply a way for Terraform to show you a value after it runs.

output "selected_region" {
  value = var.region
}

Now run this command to see the result.

terraform apply

At this point only the default exists, so Terraform will show us-east-1. The default wins because nobody else is competing.

Now add a file called terraform.tfvars with this line.

region = "ap-south-1"

Run apply again. This time Terraform shows ap-south-1. The tfvars file beat the default because it has higher priority.

Now add a file called prod.auto.tfvars with this line.

region = "ca-central-1"

Run apply once more. Now the answer becomes ca-central-1. The auto tfvars file sits higher than the standard tfvars file, so it takes over.

Finally, run apply with a command line flag.

terraform apply -var="region=sa-east-1"

The result is now sa-east-1. The command line beats every file, so it has the final say.

What happens with two auto tfvars files

A common question is what happens when you have more than one auto tfvars file and both set the same variable. The answer is alphabetical order.

Imagine you have two files named a.auto.tfvars and b.auto.tfvars, and both set the region. Terraform loads a.auto.tfvars first and b.auto.tfvars second. Because the later one overrides the earlier one, the value inside b.auto.tfvars wins.

So the file whose name comes later in the alphabet has the higher priority. This is easy to forget, so name your files carefully if order matters to you.

Command line order also matters

There is one more detail worth knowing. When you pass several flags on the command line, Terraform applies them left to right, and the rightmost one wins.

terraform apply -var="region=us-east-1" -var="region=us-west-2"

In this case the final value is us-west-2, because it appears last. The same rule applies when you mix -var and -var-file. Whatever comes last on the line takes priority.

A quick warning about missing values

If a variable has no default and you never give it a value through any source, Terraform will stop and ask you to type the value in. This is a safety feature so you never accidentally run with a blank value.

If you want a variable to be optional, give it a sensible default. If you want it to be required on purpose, leave the default out so Terraform forces someone to provide it.

Conclusion

Variable precedence in Terraform looks confusing at first, but it follows one calm and predictable rule. Lower priority sources set the value, and higher priority sources override it.

The order, from weakest to strongest, is the default value, then environment variables, then the standard tfvars file, then the JSON tfvars file, then the auto tfvars files in alphabetical order, and finally the command line flags which always win.

Once this clicks, you gain real control over your setup. You can keep safe defaults in your code, store normal settings in tfvars files, and override anything on the fly from the command line when you need to. That is the whole point of precedence, giving you layers you can trust.

Try the small example above and watch the value change with each step. Seeing it happen is the fastest way to make this stick.

Get in touch

If you have questions or want me to cover another Terraform topic, reach out at khantanseer43@gmail.com. I am always happy to help fellow builders.

Who Wins the Variable Fight in Terraform?

Tanseer — Wed, 24 Jun 2026 11:02:27 +0000

A beginner friendly guide to understanding which value Terraform actually picks when the same variable is set in many places

Introduction

This is great for flexibility, but it also raises a confusing question. If the same variable gets a value from five different places, which value does Terraform actually use?

This is called variable precedence. Precedence simply means the order of priority. It is the set of rules Terraform follows to decide who wins when there is a conflict.

Quick note on the words we will use. A variable is just a named value you can reuse, like a name tag on a box. A value is what is inside that box. Precedence is the priority order Terraform uses when two boxes have the same name tag but different contents.

First, the ways you can set a variable

Before we talk about who wins, let us list all the players in the game. There are six common ways to give a variable a value in Terraform.

The first is a default value. You write this directly inside the variable block in your code.

variable "region" {
  type    = string
  default = "us-east-1"
}

export TF_VAR_region="us-west-2"

region = "ap-south-1"

{
  "region": "eu-west-1"
}

region = "ca-central-1"

The sixth is a command line flag. A flag is an extra instruction you type when running a command. You can pass a value directly with -var, or point to a specific file with -var-file.

terraform apply -var="region=sa-east-1"

The order that decides the winner

Here is the order from weakest to strongest.

The default value inside the variable block. This is the fallback that is used only when nothing else provides a value.
Environment variables that start with TF_VAR_.
The terraform.tfvars file.
The terraform.tfvars.json file.
Any files ending in .auto.tfvars or .auto.tfvars.json, loaded in alphabetical order of their file names.
The -var and -var-file flags on the command line, applied in the exact order you type them. The simplest way to remember this is that the command line always wins, and the default value always loses. Everything else sits in the middle.

A full example you can run

Let us prove the rules with a tiny project. Create a folder and add the following files.

First the variable declaration in a file called variables.tf.

variable "region" {
  type    = string
  default = "us-east-1"
}

Next a file called main.tf that just prints the value back to us using an output. An output is simply a way for Terraform to show you a value after it runs.

output "selected_region" {
  value = var.region
}

Now run this command to see the result.

terraform apply

At this point only the default exists, so Terraform will show us-east-1. The default wins because nobody else is competing.

Now add a file called terraform.tfvars with this line.

region = "ap-south-1"

Run apply again. This time Terraform shows ap-south-1. The tfvars file beat the default because it has higher priority.

Now add a file called prod.auto.tfvars with this line.

region = "ca-central-1"

Run apply once more. Now the answer becomes ca-central-1. The auto tfvars file sits higher than the standard tfvars file, so it takes over.

Finally, run apply with a command line flag.

terraform apply -var="region=sa-east-1"

The result is now sa-east-1. The command line beats every file, so it has the final say.

What happens with two auto tfvars files

A common question is what happens when you have more than one auto tfvars file and both set the same variable. The answer is alphabetical order.

So the file whose name comes later in the alphabet has the higher priority. This is easy to forget, so name your files carefully if order matters to you.

Command line order also matters

There is one more detail worth knowing. When you pass several flags on the command line, Terraform applies them left to right, and the rightmost one wins.

terraform apply -var="region=us-east-1" -var="region=us-west-2"

In this case the final value is us-west-2, because it appears last. The same rule applies when you mix -var and -var-file. Whatever comes last on the line takes priority.

A quick warning about missing values

If you want a variable to be optional, give it a sensible default. If you want it to be required on purpose, leave the default out so Terraform forces someone to provide it.

Conclusion

Variable precedence in Terraform looks confusing at first, but it follows one calm and predictable rule. Lower priority sources set the value, and higher priority sources override it.

Try the small example above and watch the value change with each step. Seeing it happen is the fastest way to make this stick.

Get in touch

If you have questions or want me to cover another Terraform topic, reach out at khantanseer43@gmail.com. I am always happy to help fellow builders.

The Bucket That Must Exist Before Everything Else

Tanseer — Tue, 23 Jun 2026 10:31:31 +0000

How I created a dedicated S3 bucket to hold my Terraform remote backend, and every small lesson I learned along the way

Introduction

When you start working with Terraform, one of the first real puzzles you hit is a classic chicken and egg situation.

Terraform needs a safe place to store its state file. The state file is a record of everything Terraform has created for you, like a memory of your infrastructure. The most common safe place to keep this file is an S3 bucket on AWS. S3 stands for Simple Storage Service, and a bucket is just a container where you store files in the cloud.

Here is the puzzle. Terraform wants to store its state in an S3 bucket, but that bucket does not exist yet. So how do you create the very bucket that Terraform itself needs in order to remember what it created?

The answer is a small, separate Terraform setup called a bootstrap. You run it once, it creates the bucket, and then you never touch it again. In this post I will walk you through exactly how I built mine for my project, and I will explain every piece of jargon the first time it shows up. This is written for developers who are new to AWS, so nothing here assumes prior cloud knowledge.

What a Bootstrap Actually Is

A bootstrap is a tiny Terraform project whose only job is to create the foundation that the main project depends on. In our case the foundation is a single S3 bucket that will store the remote backend.

A backend is simply where Terraform keeps its state. A local backend keeps the state on your own computer. A remote backend keeps the state in the cloud, which is much safer and lets a whole team share it. We want a remote backend, and to have a remote backend we first need the bucket to hold it.

The golden rule of a bootstrap is this. Run it once, before anything else, then leave it alone forever.

The Folder Structure

I kept the bootstrap deliberately small. Here is the entire layout.

bootstrap/
├── main.tf        # everything in one file
├── variables.tf
└── README.md

That is it. Three files. A lot of beginners expect more files because larger Terraform projects have many. But a bootstrap is special, and the missing files are missing on purpose. Let me explain why, because the reasoning here taught me a lot.

Why there is no backend.tf

A file called backend.tf is where you would normally tell Terraform to use a remote backend. We do not have one here, and that is intentional. The bucket does not exist yet, so we cannot point Terraform at it. Instead the bootstrap uses local state, meaning the state file sits on our machine for now. This is the one time local state is the correct choice.

Why there is no outputs.tf

An output in Terraform is a value you ask Terraform to print or pass along after it runs, like the name of something it just created. You might think we need to output the bucket name so the main project can read it. We do not. The bucket name is already known from our variables, and the backend.tf file in the main project is static. Static means it is fixed text that cannot read or react to outputs. So an output here would serve no purpose.

Why there is no providers.tf

A provider is the plugin that lets Terraform talk to a specific cloud, in this case AWS. In big projects people split the provider setup into its own providers.tf file for tidiness. Our bootstrap is so small that everything fits comfortably in main.tf, so there is no need to split it.

The Order of Blocks Inside main.tf

Inside main.tf the order of things matters for readability. I followed this order from top to bottom.

terraform block → provider block → data source → resources

The terraform block holds settings about Terraform itself, such as which version to use. The provider block configures AWS. A data source reads information that already exists rather than creating anything new. Resources are the things Terraform actually creates for you.

The Four Resources, and Nothing More

The bootstrap creates exactly four resources. Each one has a clear reason to exist.

The first is aws_s3_bucket. This is the bucket itself. I named it this in the code, which is a common Terraform convention when there is only one of something.

The second is aws_s3_bucket_versioning. Versioning means S3 keeps old copies of a file whenever it changes. This matters a lot for a state file. If the state ever gets corrupted, versioning lets you roll back to a healthy earlier copy. It is your safety net.

The third is aws_s3_bucket_public_access_block. This shuts the bucket off from the public internet. It has four separate arguments, and I set all four to true. The reason I did this explicitly, rather than trusting AWS to do it for me, leads to one of the biggest lessons in this whole project, which I will come back to shortly.

The fourth is aws_s3_bucket_server_side_encryption_configuration. Encryption scrambles the stored data so that only authorized access can read it. I chose AES256, which is a strong and standard encryption method. Again I set this explicitly rather than relying on any default.

Here is the complete main.tf so you can see how all four resources fit together with the terraform block, the provider, and the data source.

# Add AWS Provider
terraform {
  required_version = "~> 1.15.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

# Configure the AWS Provider
provider "aws" {
  region = var.region
}

# AWS account id
data "aws_caller_identity" "current" {}

# Create s3 bucket
resource "aws_s3_bucket" "this" {
  bucket = "${var.project}-statefile-${data.aws_caller_identity.current.account_id}"
}

# Versioning bucket
resource "aws_s3_bucket_versioning" "this" {
  bucket = aws_s3_bucket.this.id
  versioning_configuration {
    status = "Enabled"
  }
}

# Block public access
resource "aws_s3_bucket_public_access_block" "this" {
  bucket = aws_s3_bucket.this.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

# Encryption
resource "aws_s3_bucket_server_side_encryption_configuration" "this" {
  bucket = aws_s3_bucket.this.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

Notice the small detail inside each resource. The line bucket = aws_s3_bucket.this.id is how Terraform links resources together. It tells the versioning, public access, and encryption resources to attach themselves to the exact bucket we just created, rather than to some other bucket. This linking is how Terraform understands the order to build things in.

Reference image placeholder: a screenshot of the four resources listed in main.tf.

The Variables File

A variable is a named value you can reuse and change in one place. Here is the full variables.tf.

variable "region" {
  type        = string
  description = "AWS region where the statefile must exists"
  default     = "ap-south-1"
}

variable "project" {
  type        = string
  description = "Name of the project"
  default     = "todo-app"
}

The first variable is region. A region is the geographic location of your AWS data center. Mine is a string with a default of ap-south-1, which is the Mumbai region. A string simply means text.

The second variable is project. This is the name of the project, with a default of todo-app. A nice improvement you can add later is validation, which is a rule Terraform checks before running. A validation rule could force the name to use lowercase letters and hyphens only, which keeps bucket names clean and valid. I left it simple here, but it is worth knowing the option exists.

Notice what is missing. There is no account_id variable. Your account ID is the unique number that identifies your AWS account. Instead of typing it in by hand as a variable, where you could easily make a mistake, I read it automatically using a data source called data "aws_caller_identity". This always returns the correct account, so it can never be wrong.

How the Bucket Gets Its Name

S3 bucket names must be unique across all of AWS, not just within your account. So I built the name out of pieces that are guaranteed to be unique together.

${var.project}-statefile-${data.aws_caller_identity.current.account_id}

This combines the project name, the word statefile, and the account ID. Because the account ID is unique to me, the resulting name will not clash with anyone else.

One choice worth pointing out. I did not put the environment, such as dev or prod, into the name. One bucket serves all environments. This keeps things simple, and the different environments are separated inside the bucket by their file paths instead.

Version Constraints, and Why the Tiny Symbols Matter

In the terraform block I pinned the Terraform version like this.

required_version = "~> 1.15.0"

The little ~> symbol is called the pessimistic constraint operator. It controls how much Terraform is allowed to upgrade itself automatically. The exact form you write changes the meaning in a way that is easy to miss.

Writing ~> 1.15.0 allows only patch releases, meaning small bug fix updates like 1.15.1 or 1.15.2, but not 1.16.

Writing ~> 1.15 would also allow minor versions, meaning larger feature updates like 1.16 or 1.17.

These are very different. Patch only is cautious and safe. Minor allowed is more relaxed. The point is to pick one consciously rather than copying a symbol without understanding it.

A Small Rule About String Interpolation

Interpolation is the way Terraform inserts a variable value into a piece of text, using the ${...} syntax. There is a simple rule I learned to keep code clean.

Use the wrapped form "${var.region}" only when you are combining the variable with other text, like inside the bucket name above.

Use the plain form var.region when you are referring to a single variable on its own, with no surrounding text.

Wrapping a lone variable in ${...} for no reason just adds clutter, so I stopped doing it.

Problems I Hit, and What They Taught Me

This is the part I find most valuable, because every one of these came from getting something wrong first.

I tried to use a data source as the default value for a variable. This does not work. A default must be a static literal, meaning a fixed value typed directly in, not something that has to be looked up while Terraform runs.

I learned the account_id must be treated as a string, not a number. Account IDs can begin with a zero, and numbers drop leading zeros, which would silently break the value. Storing it as text keeps it intact.

I confirmed that backend.tf is static and is read during the init phase, which is the very first setup step when you run Terraform. Because it runs so early, it cannot use variables or outputs at all. That is exactly why the bootstrap uses local state instead.

I learned that providers.tf is also static and read at the same early init phase, for the same reason.

I learned never to rely on AWS defaults for security. AWS settings can change over time, and assuming a default protects you is risky. I now set encryption and public access blocking explicitly every single time, so the protection is written down and guaranteed.

And one more lesson that applies to everything, including advice from me. Always verify information. At one point I had a wrong belief about which Terraform version existed, and checking it directly set me straight. Trust, but confirm.

Git Rules

Git is the tool that tracks changes to your code. When using it, some files should be saved and shared, while others must never leave your machine because they contain secrets or local state. Here is how I split them.

# commit
*.tf
.terraform.lock.hcl
*.tfvars.example

# ignore
.terraform/
*.tfstate
*.tfstate.backup
*.tfvars

The files under commit are safe to share. The lock file pins the exact provider versions so everyone uses the same ones. The example file shows the shape of variable values without revealing real ones.

The files under ignore must stay private. The state files describe your live infrastructure and can hold sensitive details. The .tfvars files often contain real secret values. The .terraform folder is just local cache that does not belong in version control.

The Run Order

Finally, here is the exact order to run everything. The bootstrap goes first, the main project follows.

cd bootstrap → terraform init → terraform apply
cd environments/dev → terraform init → terraform apply

The command terraform init prepares the working folder and downloads what Terraform needs. The command terraform apply actually builds the resources. You run them in the bootstrap folder once to create the bucket, then move into your real environment and run them again to build the rest of your infrastructure, now safely backed by the remote state bucket you just made.

Reference image placeholder: a terminal screenshot showing terraform apply finishing successfully in the bootstrap folder.

Conclusion

The bootstrap is small, but it solves a real problem that confuses almost everyone at the start. You cannot store Terraform state in a bucket that does not exist, so you build that bucket once with a tiny standalone setup that uses local state, and then you never look back.

Along the way I learned that the missing files in a bootstrap are missing on purpose, that security should always be explicit rather than assumed, that tiny version symbols carry real meaning, and that even confident advice should be checked. None of these are hard once you see the reasoning, and together they make your foundation solid.

If you are new to AWS and Terraform, I hope walking through my journey makes your own first bootstrap far less mysterious than mine was.

Contact

If you have questions or want to share your own setup, feel free to reach out at khantanseer43@gmail.com.

Stop Hardcoding Passwords: A Beginner's Guide to AWS Secrets Manager

Tanseer — Tue, 23 Jun 2026 05:26:44 +0000

How to keep your database passwords and API keys safe, and why rotating them doesn't break your app the way you think it will

If you are new to AWS, there is a good chance you have done something like this at least once. You needed your app to connect to a database, so you took the database password and pasted it straight into your code, or into a Lambda environment variable, and moved on. It worked, so why worry?

The problem is that a password sitting in plain view is a password waiting to leak. In this tutorial we will walk through AWS Secrets Manager, a service built to store exactly these kinds of sensitive values safely. We will cover what it is, why environment variables are a weak place to keep secrets, what a KMS key actually does, and the one topic that confuses almost every beginner: automatic rotation. By the end you will understand why rotation does not break your application, even though it changes your password while the app is running.

Let us start from the beginning.

Jargon check before we go further: a secret just means any sensitive string your app needs but should never be public. Think database passwords, API keys, and access tokens. An API key is a long secret string that proves your app is allowed to use some service. A token is similar, usually a temporary one.

What Is AWS Secrets Manager?

AWS Secrets Manager is an encrypted vault for your secrets. Instead of writing your database password directly into your code, you store it inside Secrets Manager, and your app asks for it at runtime (meaning while the app is actually running, not while you are writing it).

A simple way to picture it is a hotel room safe. You do not tape your passport to the wall. You lock it in the safe, and only someone with the right key can open it. The hotel also keeps a log of every time the safe is opened. Secrets Manager works the same way.

It gives you three main benefits.

The first is encryption at rest. "At rest" means while the data is sitting in storage, as opposed to while it is travelling over the network. Your secret is scrambled so that even someone who somehow gets to the raw storage cannot read it.

The second is IAM controlled access. IAM stands for Identity and Access Management, which is the AWS system that decides who is allowed to do what. With Secrets Manager you can say "only my application is allowed to read this database password" and nothing else can touch it.

The third is automatic rotation, which means AWS can change your password on a schedule for you. This is the feature we will spend the most time on, because it is the most useful and the most misunderstood.

Why Not Just Use a Lambda Environment Variable?

An environment variable is a setting you attach to your app from the outside, so the value lives next to the app rather than inside your code. In AWS Lambda (the service that runs your backend code without you managing a server) you can set these in the function configuration. It works, and plenty of people do it. So what is wrong with it for secrets?

There are four real problems.

First, the value sits in plain text inside your function configuration. Anyone who can open the AWS console and view your Lambda can simply read the password. There is no lock on it.

Second, the value tends to leak into places you did not expect. If you manage your infrastructure with a tool like Terraform, the password ends up written into your Terraform state file, which is the record Terraform keeps of everything it created. It can also show up in deployment logs. So one password quietly becomes several copies scattered around.

Third, changing the value means redeploying or updating the function. There is no clean way to swap it on the fly.

Fourth, there is no audit trail. You cannot answer the question "who read this secret and when," because nothing was recorded.

Environment variables are perfectly fine for values that are not sensitive, like which region your app runs in or a feature toggle. They are just a weak place to keep real secrets.

What Is a KMS Key?

When we said Secrets Manager encrypts your data, something has to do the actual encrypting. That something is a KMS key.

KMS stands for Key Management Service. A KMS key is the cryptographic key used to scramble (encrypt) and unscramble (decrypt) your secret. Encryption is just the process of turning readable text into unreadable text so that only someone with the key can read it again.

Here is the clever part. You never see or hold the raw key yourself. It lives inside tamper resistant hardware, meaning special equipment built so the key cannot be copied out or stolen. You simply ask KMS "please decrypt this for me," and KMS checks your IAM permission before doing it.

Think of KMS as the master key to a building, locked inside a vault that you personally cannot open. You do not get to carry the master key around. Instead you hand your locked box to the security desk and say "open this please," and they check your badge first. Because the key never leaves the vault, nobody can walk off with it.

When Secrets Manager stores your password, it quietly uses a KMS key to encrypt it. AWS gives you a default key for free, so as a beginner you usually do not need to create your own.

Automatic Rotation: The Part Everyone Misunderstands

Rotation means automatically changing a password to a new one on a schedule. Here is the worry almost every beginner has, and it is a reasonable one:

"If Secrets Manager changes my password, but my database still has the old one, won't my app stop being able to connect?"

The answer is no, and understanding why is the heart of this whole topic.

Rotation does not change the password in only one place. It changes it in both the database and the secret, as a single coordinated operation. Here is the exact sequence the rotation process runs.

It generates a brand new random password.
It connects to the database using the current password (which still works at this point) and runs the command ALTER USER, which is the SQL instruction that sets a new password on the database account. Now the database expects the new password.
It stores that new password back into Secrets Manager as the current value.
It connects once more using the new value to confirm everything works. Because both sides are updated together, they are never out of sync in a way that locks you out.

But there is a second condition you must meet for this to stay smooth. Your app has to fetch the secret fresh from Secrets Manager at runtime, rather than holding its own permanent copy. If your code reads the current value when it needs it, then after rotation it simply reads the new value next time and carries on.

There is one trap here worth naming. To avoid asking Secrets Manager for the password on every single request (which costs a little money and adds delay), developers often cache it, meaning they keep a copy in memory for reuse. That is good practice, but if you cache the password forever and never refresh it, then after rotation your app is holding a stale password and will fail to connect. The fix is a cache with a TTL, which stands for Time To Live, a setting that tells the cache to throw the value away and fetch a fresh one after a few minutes.

So the rule is simple. Rotation does not break your app, as long as your app reads the secret fresh and does not cling to an old cached copy.

Versioning: Why the Old Password Sticks Around for a Moment

A secret in Secrets Manager does not hold just one value. It holds versions, and each version carries a label that tells you what it is for. Three labels matter.

AWSCURRENT is the value you get by default when you ask for the secret. This is the live, in use password.

AWSPREVIOUS is the value from just before the last change. Secrets Manager keeps it valid for a short window on purpose. Picture rotation happening at the exact moment a request is already in flight, meaning a request that started a split second before the change. That request may have grabbed the old value already. Keeping AWSPREVIOUS alive briefly lets that in flight request finish successfully instead of erroring out. It smooths over the handover.

AWSPENDING is the new value during rotation, after it has been created but before it has been promoted to current. It is the password in waiting.

You rarely need to think about these directly, because asking for the secret gives you AWSCURRENT automatically. But knowing the labels exist helps you understand why rotation is so smooth and why the brief overlap does not cause failures.

Cross Region Replication: A Feature for Later

By default, a secret lives in a single AWS region. A region is a geographic location where AWS runs its data centers, such as Mumbai (named ap-south-1) or Singapore.

Cross region replication keeps an automatically synced copy of your secret in other regions you choose. If you store a secret in Mumbai and replicate it to Singapore, then any change in Mumbai is copied across to the Singapore version on its own.

You would want this in two situations. One is disaster recovery, so that if an entire region has an outage, an app running in your backup region can still read the secret locally. The other is when your app runs in several regions at once and you want each one to read the secret nearby instead of reaching across the world every time.

As a beginner running a single region project, you almost certainly do not need this yet. File it away as a "when I grow into multiple regions" feature.

Conclusion

If you take away one idea from this tutorial, let it be this. Secrets stay in sync because rotation changes both sides at once, the database and the stored secret, in a single coordinated operation. And your application stays connected because it reads the current value fresh from Secrets Manager whenever it needs it, rather than holding a hardcoded copy that goes stale.

Everything else builds on that. Environment variables are weak for secrets because they sit in plain text and leak into your state files and logs. A KMS key does the encrypting behind the scenes without ever letting you touch the raw key. Versioning with AWSCURRENT and AWSPREVIOUS keeps a brief safety net so in flight requests survive a rotation. And cross region replication is there for the day you outgrow a single region.

So the next time you are tempted to paste a password into an environment variable, reach for Secrets Manager instead. Your future self, and anyone who inherits your code, will thank you.

Get in Touch

Found this helpful, or stuck on something? I would love to hear from you. Reach out at khantanseer43@gmail.com and I will do my best to help.

Terraform outputs.tf Explained: What It Is, When to Use It, and When to Skip It

Tanseer — Tue, 23 Jun 2026 04:53:47 +0000

Who This Is For

If you are learning Terraform and have come across outputs.tf in project structures or tutorials but are not fully clear on what it actually does or why it exists, this blog is for you.

We will start from the basics and build up to how outputs work across modules, what they look like in practice, and the rules around where they can and cannot be used. Every term will be explained along the way.

What Is an Output in Terraform?

When Terraform creates infrastructure, it stores everything it knows about that infrastructure in a file called the state file. The state file is Terraform's database. It tracks every resource, every attribute, every ID — everything.

But the state file is not something you read directly. It is large, it is internal, and most of the values in it are not things you care about day to day.

Outputs are a way to surface specific values from that state file — the ones you actually need. You decide what gets exposed. Everything else stays internal.

A good way to think about it:

State file = the entire database.

Outputs = a view on top of that database that exposes only what you chose to make visible.

Outputs do not add new information. They do not change what Terraform tracks. They simply choose what to surface from what already exists.

The Three Contexts Where Outputs Are Used

Outputs are not just for one purpose. They serve three distinct purposes depending on where you use them. Understanding the difference between these three will make the entire concept click.

Context 1: Passing Values Between Modules in the Same Project

This is the most common use case, and the one you will use in almost every real Terraform project.

Imagine your project has two modules: a database module and a backend module. The database module creates an RDS instance and stores credentials in AWS Secrets Manager. The backend module is a Lambda function that needs to connect to that database. To do that, it needs the ARN (unique identifier) of the secret.

But here is the problem. Module internals are private. A resource defined inside modules/database/main.tf is not visible anywhere outside that module. It is intentionally hidden. You cannot just reference it from another module directly.

This is where outputs.tf comes in. The database module uses outputs.tf to explicitly expose the values it wants to share. The backend module then receives those values as input variables.

The wiring between them happens in your environment file, for example environments/dev/main.tf. That file is the connection layer. It takes what one module exposes and passes it as input to another.

The flow looks like this:

modules/database/outputs.tf
        |
        v
environments/dev/main.tf   (the wiring layer)
        |
        v
modules/backend/variables.tf

Here is what that looks like in code:

# modules/database/outputs.tf
output "secret_arn" {
  value = aws_secretsmanager_secret.db_credentials.arn
}

output "secret_name" {
  value = aws_secretsmanager_secret.db_credentials.name
}

# environments/dev/main.tf
module "database" {
  source = "../../modules/database"
}

module "backend" {
  source     = "../../modules/backend"
  secret_arn  = module.database.secret_arn
  secret_name = module.database.secret_name
}

# modules/backend/variables.tf
variable "secret_arn" {}
variable "secret_name" {}

The database module is a black box. variables.tf is what it accepts as input. main.tf is its internal implementation. outputs.tf is what it hands back out. Nothing inside the module leaks out unless you explicitly put it in outputs.tf.

Context 2: Displaying Values in the Terminal After Apply

Outputs defined in the root module, meaning the environment folder like environments/dev, are printed to the terminal after terraform apply completes.

This is purely for you as the person running the deployment. It is a convenience feature so you do not have to go hunting through the AWS console for values you commonly need.

Good candidates for terminal outputs are things like:

The API Gateway URL your backend is reachable at.

The Amplify URL where your frontend is deployed.

The RDS endpoint for your database.

# environments/dev/outputs.tf
output "api_gateway_url" {
  value = module.backend.api_url
}

output "amplify_url" {
  value = module.frontend.app_url
}

After apply, these print cleanly to your terminal:

Outputs:

api_gateway_url = "https://abc123.execute-api.ap-south-1.amazonaws.com/dev"
amplify_url     = "https://main.abc123.amplifyapp.com"

No digging through the console. The values are right there.

Context 3: Sharing Values Across Completely Separate Terraform Projects

Sometimes infrastructure is split across multiple completely separate Terraform projects, each with their own state file. A networking team might manage VPCs in one project. An application team might manage Lambda functions in another. The application team needs the VPC ID from the networking project.

This is handled using something called terraform_remote_state. It is a data source (a way to read information from outside the current project) that reads another project's state file and exposes its outputs.

data "terraform_remote_state" "networking" {
  backend = "s3"
  config = {
    bucket = "company-tfstate"
    key    = "networking/terraform.tfstate"
    region = "ap-south-1"
  }
}

# Now you can use:
data.terraform_remote_state.networking.outputs.vpc_id

One important detail here: terraform_remote_state can only access values that were explicitly defined as outputs in the other project. It cannot reach into the raw state and pull arbitrary values. If the networking team did not put the VPC ID in their outputs.tf, you cannot get it from here.

This reinforces the same rule: outputs are the only gateway for values to leave a Terraform boundary, whether that boundary is a module or an entire separate project.

This pattern does not apply if your entire infrastructure lives in one root, which is the case for smaller projects. But it is a real and widely used pattern in team environments.

Where Outputs Cannot Be Used

This is important. Outputs work during the plan and apply phase, when Terraform is evaluating your configuration and building infrastructure. But not every file in your project is evaluated at that phase.

Two files are evaluated earlier, during the init phase, before any HCL evaluation happens:

backend.tf — This is where you configure where Terraform stores its state file, for example an S3 bucket. Terraform reads this file during terraform init, before it knows anything about your resources or variables. Everything here must be a hardcoded string. You cannot reference an output, a variable, or a local.

providers.tf — This is where you configure your cloud provider, for example the AWS region and profile. This is also read during init. Same restriction. Hardcode everything here.

Trying to use a variable or output in either of these files will cause Terraform to throw an error, and the reason is always the same: those values are not available yet at init time.

Here is a simple way to remember the two phases:

Init phase reads backend.tf and providers.tf statically. Downloads providers. Sets up the backend. No HCL logic allowed.

Plan and Apply phase evaluates everything else. Outputs, variables, locals, data sources, resource references — all of this works here.

Why You Cannot Just Skip outputs.tf and Read the State Directly

A common question when learning this is: the state file already has all the values, why do I need outputs at all?

There are two reasons.

Within a module, resources are private by design. Terraform intentionally encapsulates module internals so that modules are independent and reusable. You cannot reach into modules/database/main.tf from environments/dev/main.tf and reference a resource directly. The module must explicitly hand that value out through outputs.tf. This is the same reason functions in a programming language return values instead of letting callers read internal variables directly.

Across separate roots, terraform_remote_state only exposes outputs. There is no mechanism to read arbitrary resource attributes from another project's state file. Outputs are the only thing that crosses that boundary.

When outputs.tf Is Optional

Not every module needs an outputs.tf. The rule is straightforward.

If nothing downstream needs a value from your module, there is nothing to expose. The outputs file is optional.

A common example is a frontend module in a project. If the frontend module deploys an Amplify app and no other module needs the Amplify URL as an input, then outputs.tf in the frontend module is not needed. The URL might still be shown in the terminal via the root module's outputs, but the frontend module itself does not need to expose anything.

The question to ask before adding any output is:

Who is the consumer of this value, and how will they read it?

If the answer is nobody, the output does not belong. Adding outputs that nothing consumes is just noise in your codebase.

A Quick Reference

Here is everything in one place:

Outputs between modules: use outputs.tf in the source module, wire it in the environment's main.tf, receive it in the destination module's variables.tf.

Outputs in the terminal: define them in the root module's outputs.tf. They print after terraform apply.

Outputs across separate roots: use terraform_remote_state to read another project's outputs from its remote state file.

Outputs do not work in backend.tf or providers.tf because those are read during init before HCL evaluation.

You cannot skip outputs.tf because module internals are private and remote state only exposes outputs.

outputs.tf is optional when nothing downstream needs the module's values.

Conclusion

Outputs are one of those concepts in Terraform that seem small but are actually the backbone of how information flows through your infrastructure code. Once you understand that modules are black boxes and outputs are the only thing that leaves them, the entire system starts to make sense.

Before adding any output, always ask who the consumer is. That one question will keep your outputs intentional, your modules clean, and your project easy to follow.

Need Help?

If you are learning Terraform and have questions about project structure, modules, state management, or anything else, feel free to reach out.

Email me at khantanseer43@gmail.com

How to Add a Custom Domain to AWS Cognito Google Login (And the Errors Nobody Warns You About)

Tanseer — Tue, 02 Jun 2026 04:39:24 +0000

Who This Is For

If you are using AWS Cognito to handle user authentication in your app and you have added Google as a social login option, you may have noticed something uncomfortable. When a user clicks "Sign in with Google", the URL in the browser does not say your domain. It says something like:

https://yourapp.auth.us-east-1.amazoncognito.com

That is the default Cognito hosted UI URL. It works, but it looks like your users are leaving your app to sign in somewhere else. It does not look professional, and for security conscious users, it can feel suspicious.

The fix is to set up a custom domain so the login page shows your own URL, something like:

https://auth.yourdomain.com

This guide walks through the full setup step by step, and more importantly, it covers the three errors you will likely hit along the way.

Before You Start

Here is what you need to have in place before following this guide:

A Cognito User Pool already created with Google set up as a social identity provider. If you have not done that yet, set that up first and come back here.

A domain you own, managed in Route 53 (AWS's domain and DNS service). The subdomain you will be using for the login page, something like auth.yourdomain.com, should not be pointing anywhere yet.

Access to the Google Cloud Console where your OAuth app is configured.

Step 1: Create an ACM Certificate — But Only in us-east-1

This is the first place most developers go wrong, and it is the most confusing one because nothing in the Cognito console tells you about it until you are already stuck.

ACM stands for AWS Certificate Manager. It is the service AWS uses to create and manage SSL certificates. An SSL certificate is what makes your domain load over HTTPS (the secure version of HTTP) and shows the padlock icon in the browser.

You need to create a certificate for your custom domain, like auth.yourdomain.com. That part is straightforward. The gotcha is the region.

Even if your Cognito User Pool is in a different region, say ap-south-1 or eu-west-1, the ACM certificate must be created in us-east-1. No exceptions.

The reason is that Cognito uses CloudFront behind the scenes to serve the hosted UI. CloudFront is a global content delivery service, and it only accepts SSL certificates from the us-east-1 region. Your Cognito User Pool can be anywhere, but the certificate must always be in us-east-1.

Here is how to create it:

Go to the AWS Console and switch your region to US East (N. Virginia) which is us-east-1. You can change the region from the top right dropdown in the console.
Search for "Certificate Manager" and open ACM.
Click "Request a certificate".
Choose "Request a public certificate" and click Next.
Enter your domain name. Use the subdomain you want for login, for example auth.yourdomain.com.
Choose "DNS validation" as the validation method. This is the recommended option.
Click "Request". After requesting, ACM will give you a CNAME record that you need to add to your DNS to prove you own the domain. Go to Route 53, open your hosted zone, and add that CNAME record exactly as ACM shows it.

Once the record is in place, ACM will automatically verify it and mark the certificate as "Issued". This usually takes a few minutes but can take up to 30 minutes.

Do not move to the next step until the certificate status shows "Issued".

Step 2: Add the Custom Domain in Cognito

Now that the certificate is ready, go back to your Cognito User Pool. Make sure you are in the correct region where your User Pool lives, not us-east-1 unless that is where your pool is.

Open your User Pool in the Cognito console.
Go to the "App integration" tab.
Scroll down to "Domain" and click "Actions", then "Create custom domain".
Enter your custom domain, for example auth.yourdomain.com.
In the SSL certificate dropdown, select the ACM certificate you just created. It will only appear if the certificate is in us-east-1 and has a status of Issued.
Click "Create". After saving, Cognito will provision a CloudFront distribution for your domain. This can take anywhere from 15 to 40 minutes. You will see a CloudFront domain name appear once it is ready, something like d1234abcde.cloudfront.net. Copy this value. You will need it in the next step.

Step 3: Add the DNS Record in Route 53

This is the second place where things can go wrong.

Once Cognito gives you the CloudFront domain, you need to create a DNS record so that when someone visits auth.yourdomain.com, they are pointed to that CloudFront distribution.

The record type to use here is an Alias record, not a CNAME. Route 53 has a special Alias record type that is designed to point to AWS services like CloudFront. Using a regular CNAME at the root level of a subdomain can cause issues, so always use Alias when pointing to a CloudFront domain in Route 53.

Here is how to add it:

Open Route 53 in the AWS Console.
Go to "Hosted zones" and open your domain.
Click "Create record".
Set the record name to auth (Route 53 will append your domain automatically).
Set the record type to "A".
Toggle on "Alias".
In the "Route traffic to" dropdown, choose "Alias to CloudFront distribution".
Paste the CloudFront domain that Cognito gave you.
Click "Create records". DNS changes can take a few minutes to a few hours to fully propagate. You can check the status using a tool like MXToolbox or WhatsMyDNS by looking up your subdomain.

Step 4: Update the Redirect URI in Google Cloud Console

This is the third gotcha, and it is easy to miss because everything might look like it is working until a user actually tries to sign in with Google and gets an error.

When Google handles the OAuth login flow (OAuth is the protocol that lets users sign in with their Google account), it redirects the user back to a specific URL after they authenticate. That URL is called the redirect URI.

Before you set up the custom domain, Cognito's redirect URI was something like:

https://yourapp.auth.us-east-1.amazoncognito.com/oauth2/idpresponse

Now that you have a custom domain, the redirect URI has changed to:

https://auth.yourdomain.com/oauth2/idpresponse

Google will reject any redirect to a URI that is not explicitly listed as an authorized redirect URI in your OAuth app settings. If you do not update this, users will see a Google error page saying the redirect URI is not authorized.

Here is how to update it:

Go to the Google Cloud Console at console.cloud.google.com.
Open your project and navigate to "APIs and Services", then "Credentials".
Click on the OAuth 2.0 Client ID you created for Cognito.
Under "Authorized redirect URIs", click "Add URI".
Add your new redirect URI: https://auth.yourdomain.com/oauth2/idpresponse
Also update "Authorized JavaScript origins" if it was set to the old Cognito domain. Add https://auth.yourdomain.com.
Click "Save". You do not need to remove the old Cognito URI right away. Keep both for now while you test the new setup, and remove the old one once everything is confirmed working.

Step 5: Test the Full Flow

Once all the above steps are done and DNS has propagated, test the complete sign in flow from your app.

Open your app and trigger the Google login. The URL in the browser should now show auth.yourdomain.com instead of the Cognito domain. The user should be able to complete the Google sign in and land back in your app without any errors.

If you see a Google error about redirect URI mismatch, double check Step 4. If you see an SSL error, double check that the ACM certificate is in us-east-1 and is fully issued. If the DNS is not resolving, give it more time or check your Route 53 record.

Quick Summary of the Three Errors to Avoid

The ACM certificate must be in us-east-1 no matter what region your Cognito User Pool is in. Creating it in the wrong region means it will not appear in the Cognito domain setup and you will waste time wondering why.

The DNS record in Route 53 should be an Alias A record pointing to the CloudFront domain, not a plain CNAME. Using the wrong record type can cause resolution issues.

The redirect URI in Google Cloud Console must be updated to the new custom domain. Forgetting this means Google will block the login flow and your users will see an error.

Conclusion

Adding a custom domain to your Cognito login page is a small change that makes a big difference in how professional and trustworthy your app feels. The steps themselves are not complicated, but the three gotchas around the ACM region, DNS record type, and Google redirect URI are not obvious and not well documented in one place.

If you follow this guide in order and do not skip steps, you should have your custom domain up and running without the frustration of debugging silent failures.

Need Help?

If you are stuck at any of these steps or running into an error not covered here, feel free to reach out. Happy to help.

Email me at khantanseer43@gmail.com