DEV Community: Tanseer

Amazon Textract: Extract Data from Documents with AI

Tanseer — Wed, 26 Aug 2026 05:30:00 +0000

Turn scanned documents, forms, and tables into structured data your app can use, going well beyond plain text scanning. Stop seven in the AWS Hidden Gems series.

About this series

Most AWS learning stops after EC2, S3, IAM, and Lambda. But AWS has over two hundred services, and many of the most useful ones rarely appear in tutorials.

AWS Hidden Gems covers those underrated services you shouldn't ignore. Each article picks one, then explains why it exists, what it does, where it fits, and how to set it up from the console. Know the four basics above and you can follow along. Everything else gets explained as it comes up.

Today's service: Amazon Textract

Plenty of important data is trapped in documents: invoices, forms, receipts, IDs, contracts. Textract reads those documents and pulls out not just the words, but the structure. It knows which value belongs to which field on a form and how a table's rows and columns line up.

Why does this service exist?

Getting data out of documents has always been painful. Typing it in by hand is slow and error prone. Plain OCR, which stands for optical character recognition and means turning an image of text into machine readable text, gives you the words but throws away the layout. You get a wall of text with no idea that "Total: 42.00" means the total field holds 42.00, or which numbers sat in which table cell.

Textract is OCR plus document understanding. AWS trained models that recognize forms and tables, so instead of raw text you get field and value pairs and structured tables. That is the difference between reading a document and actually using it.

What is Amazon Textract?

Textract is a managed document analysis service. It reads text from images and PDFs, and it understands common document structures.

It can pull out:

Raw text, line by line and word by word
Form data as key and value pairs, like a field name and its answer
Tables, preserving rows and columns
Answers to specific questions you ask about the document, using its Queries feature
Specialized data from receipts and invoices, and from identity documents Short single images are processed instantly. Multi page PDFs stored in S3 are processed in the background, and Textract notifies you when the results are ready.

A real world problem

An accounts team receives hundreds of supplier invoices as PDFs every month. Someone opens each one and copies the invoice number, date, and total into their system by hand.

It is slow, and typos cause real accounting errors. They want the key fields pulled out of each invoice automatically and dropped into their database.

Textract does exactly this. Its invoice analysis returns the fields they care about from each PDF, so the data flows in without manual typing, and the mistakes that came with it disappear.

Real world use cases

Finance teams pull fields from invoices and receipts into accounting systems
Banks and lenders process loan and mortgage paperwork automatically
Healthcare digitizes intake forms and patient records
Insurance extracts data from claims documents to speed up processing
Governments and HR turn paper forms into searchable digital records
Any team replacing manual data entry from PDFs and scans The pattern is documents in, structured data out, with no one retyping.

Where it fits in AWS

Documents land in S3. A common pipeline: an upload triggers a Lambda function, the function calls Textract, and the structured results are stored in DynamoDB or passed on for more processing. For long PDFs, Textract sends a notification through SNS when the background job is done. Extracted text often flows next into Comprehend, the AWS text analysis service, to find meaning in it.

flowchart LR
    A[Document uploaded] --> B[S3 bucket]
    B -->|Upload event| C[Lambda function]
    C -->|Analyze document| D[Textract]
    D -->|Fields, tables, text| C
    C -->|Store structured data| E[DynamoDB]

Textract is the reader. Storage and triggers use familiar services, and Textract turns the document into data.

How the workflow runs

For a single image or short document, you send it to Textract and get structured results back right away: text, form pairs, and tables. For a multi page PDF in S3, you start a job, Textract processes it in the background, and it notifies you through SNS when the results are ready to fetch. You choose what to extract by asking for features, such as forms and tables, or by using the invoice or identity document analysis for those specific cases.

flowchart TD
    A[Document] --> B{Size?}
    B -->|Single image| C[Call Textract, get results instantly]
    B -->|Multi page PDF| D[Start a job]
    D --> E[Textract processes in background]
    E --> F[SNS notifies you]
    F --> G[Fetch structured results]

Setting it up in the AWS Console

The Textract console lets you try extraction on your own document before writing code.

Sign in to the AWS Console, search for Textract, and open it. Check the region in the top right corner.
In the left menu, open the Analyze Document demo.
Upload a document with some structure, such as a form or an invoice, or use a provided sample.
Turn on the features you want above the document, such as Forms and Tables, then let Textract analyze it.
Review the results tabs. The Forms tab shows key and value pairs it found, the Tables tab shows reconstructed tables, and the Raw Text tab shows every line. Notice how it links each field to its value.
To use Textract from your own code, create an IAM role with the AmazonTextractFullAccess policy, or a narrower policy allowing the Textract actions you need, plus s3:GetObject on the bucket holding your documents.
To confirm the setup, run the code in the next section on a document in your S3 bucket and check that the text comes back. Common mistakes: an access error usually means the IAM role lacks Textract permission or read access to the S3 bucket, so check both. If a scanned document extracts poorly, a clearer, higher resolution scan almost always improves the result.

Using it from code

This reads a document in S3 and prints its text, line by line.

import boto3

textract = boto3.client("textract")

response = textract.detect_document_text(
    Document={
        "S3Object": {"Bucket": "my-docs-bucket", "Name": "invoice.png"}
    }
)

for block in response["Blocks"]:
    if block["BlockType"] == "LINE":
        print(block["Text"])

For form fields and tables, call analyze_document instead, with FeatureTypes of ["FORMS", "TABLES"]. Its output links keys to their values through relationships, which is fiddly to walk by hand, so most people use the Textract Response Parser library (trp) to read the pairs and tables directly.

Pricing

Item	Detail
Text detection	Per page, cheapest option, plain text only
Forms or tables	Per page, higher rate, for structured data
Queries	Per page, for asking specific questions
Invoice or ID analysis	Per page, for those document types
Free tier	Pages per month for the first 3 months, across text and analysis

The AWS AI services family

AWS AI Services
├── Textract      text and structured data from documents
├── Rekognition   understanding of images and video
├── Comprehend    meaning and sentiment in text
├── Transcribe    speech to text
├── Translate     language translation
└── Polly         text to speech

Textract and Rekognition both read images, but for different goals. Rekognition, from earlier in this series, understands the scene in a photo. Textract understands the document on the page. They often pair with Comprehend, which analyzes the text once it is extracted.

Wrapping up

Textract reads documents the way you need software to: not just the words, but which value goes with which field and how tables are laid out. It replaces manual data entry from invoices, forms, and scans with a single API call. Next time data is stuck inside a PDF, you know how to get it out as structured data.

Series progress

You are on stop seven of AWS Hidden Gems.

AWS Elemental MediaConvert
Amazon IVS
Amazon Rekognition
Amazon Personalize
AWS AppSync
Amazon Timestream
Amazon Textract (you are here)
Amazon Kendra
AWS DataSync
AWS IoT Core Next up is Amazon Kendra, which brings AI powered natural language search to your own content.

Let's connect

Questions, corrections, or want to talk through where this fits in your own project? Reach me at khantanseer43@gmail.com.

Amazon Timestream: The Database Built for Time-Series Data

Tanseer — Mon, 24 Aug 2026 05:30:00 +0000

A serverless database made for data that arrives with a timestamp, like sensor readings, metrics, and events. Stop six in the AWS Hidden Gems series.

About this series

Most AWS learning stops after EC2, S3, IAM, and Lambda. But AWS has over two hundred services, and many of the most useful ones rarely appear in tutorials.

Today's service: Amazon Timestream

Some data is defined by when it happened: a temperature reading every second, a server's CPU usage over a day, a stock price tick by tick. This is time series data, and normal databases handle it poorly at scale. Timestream is a database built specifically for it.

Why does this service exist?

Time series data has an awkward shape. It arrives constantly and in huge volumes, you almost always query it by time ranges, and older data becomes less useful but you still want to keep it cheaply. General purpose databases struggle here. A relational database gets expensive and slow under a firehose of writes, and you end up over provisioning servers and hand building ways to expire old data.

Timestream is designed around this shape. It is serverless, so it scales writes automatically with no servers to size. It stores recent data in a fast tier and moves older data to a cheaper tier on its own. And its query language has time series functions built in, for things like filling gaps between readings.

What is Amazon Timestream?

Timestream is a serverless time series database. Time series data is a sequence of values recorded over time, each stamped with the moment it happened.

A record in Timestream has three parts:

Dimensions: labels that describe the source, like a device id or a region
Measures: the actual values, like temperature or CPU percentage
Time: when the measurement was taken It stores data in two tiers automatically. A memory tier holds recent data for fast queries, and a magnetic tier holds older data at lower cost. You set how long data stays in each with a retention policy, and Timestream moves data between them for you. You query it with SQL, the standard database query language, extended with functions made for time.

A real world problem

A company runs a few hundred temperature sensors across its warehouses, each reporting every ten seconds. That is millions of readings a day, forever.

They tried storing this in their existing relational database. Writes started to lag, storage costs climbed, and queries like average temperature per warehouse over the last hour got slow. The database was not built for this kind of load.

Moving the readings to Timestream fixes it. Writes keep up automatically, recent data stays fast to query, old data ages into cheap storage, and the time based queries they need are quick. Their database goes back to handling the business data it is good at.

Real world use cases

IoT platforms store sensor readings from thousands of devices reporting constantly
Operations teams keep server and application metrics for monitoring and alerting
Industrial systems track machine data to spot problems before a breakdown
Energy and utilities record meter readings over time for analysis and billing
Finance stores market data like prices and trades stamped to the moment
Apps log user activity events for later analysis by time The pattern is a high volume stream of timestamped values that you query by time.

Where it fits in AWS

Data usually flows in from many small producers. Devices send readings through IoT Core, or an app or stream writes records directly. Timestream stores them, and you query from a dashboard tool like Amazon Managed Grafana or Amazon QuickSight to see trends.

flowchart LR
    A[Sensors and devices] --> B[IoT Core]
    B -->|Route readings| C[Timestream]
    D[Apps and streams] -->|Write records| C
    C -->|SQL queries| E[Grafana or QuickSight dashboards]

Timestream is the store for the time series itself. Other services feed it and visualize it, and it focuses on ingesting and querying by time.

How the workflow runs

You create a database and a table, and set the table's retention: how long data stays in the fast memory tier and how long in the cheaper magnetic tier. Producers write records, each carrying dimensions, one or more measures, and a timestamp. Timestream keeps recent records in memory for fast access and ages older ones into magnetic storage on schedule. You run SQL queries over any time range, and Timestream reads from whichever tier holds that data.

flowchart TD
    A[Create database and table] --> B[Set memory and magnetic retention]
    B --> C[Producers write timestamped records]
    C --> D[Recent data in fast memory tier]
    C --> E[Older data in cheap magnetic tier]
    D --> F[SQL queries by time range]
    E --> F

Setting it up in the AWS Console

You will create a database and table, write a couple of records, and query them. This uses Timestream for LiveAnalytics, the serverless engine.

Sign in to the AWS Console, search for Timestream, and open it. Check the region in the top right corner.
In the left menu choose Databases, click Create database, select Standard database, give it a name, and create it.
Open your database, choose Tables, and click Create table. Name the table, then set the retention, for example keep 12 hours in the memory tier and 7 days in the magnetic tier. These control how long data stays fast versus cheap.
To add data quickly, open the Query editor from the left menu. Timestream expects most writes to come from code, but you can confirm the table exists and is queryable here.
Write a few records using the code in the next section, sending a device id as a dimension, a temperature as a measure, and the current time. This is the normal way data enters Timestream.
Back in the Query editor, run a query such as selecting all rows from your table ordered by time, and confirm your records appear. Timestream gives you sample queries you can adapt.
To validate a real time based query, try averaging your measure grouped by device over the last hour, and check the numbers match what you wrote. Common mistakes: a write that is rejected often has a timestamp outside the memory tier's retention window, so make sure you are writing recent times. An access error means the IAM role is missing Timestream write or query permission, so add it.

Using it from code

Timestream splits writing and querying into two clients. This writes a single temperature reading.

import boto3
import time

write_client = boto3.client("timestream-write")

now = str(int(time.time() * 1000))  # current time in milliseconds

write_client.write_records(
    DatabaseName="my-database",
    TableName="sensors",
    Records=[
        {
            "Dimensions": [{"Name": "device_id", "Value": "sensor-1"}],
            "MeasureName": "temperature",
            "MeasureValue": "22.5",
            "MeasureValueType": "DOUBLE",
            "Time": now,
        }
    ],
)

print("Wrote one reading")

To read the data back, use the timestream-query client and run a SQL statement like SELECT device_id, temperature, time FROM "my-database"."sensors" ORDER BY time DESC.

Pricing

Item	Detail
Writes	Per GB of data written
Memory tier storage	Per GB per hour, for recent fast data
Magnetic tier storage	Per GB per month, cheaper, for older data
Queries	Per GB of data scanned by the query
Free tier	Monthly write, storage, and query allowance for new accounts

The AWS database family

AWS Databases
├── Timestream    time series data like sensors and metrics
├── DynamoDB      key value and document data at any scale
├── Aurora/RDS    relational SQL databases
├── ElastiCache   in memory caching for speed
└── Neptune       graph data and relationships

AWS offers a database per job rather than one for everything. Timestream is the one for timestamped streams. DynamoDB is the general NoSQL workhorse, and Aurora or RDS handle relational data. Pick Timestream when time is the main axis of your data.

Wrapping up

Timestream is built for the exact shape of time series data: constant writes, time based queries, and old data you want to keep cheaply. It scales itself, tiers your data by age, and speaks SQL. Next time you are about to force sensor or metric data into a general database, reach for this instead.

Series progress

You are on stop six of AWS Hidden Gems.

AWS Elemental MediaConvert
Amazon IVS
Amazon Rekognition
Amazon Personalize
AWS AppSync
Amazon Timestream (you are here)
Amazon Textract
Amazon Kendra
AWS DataSync
AWS IoT Core Next up is Amazon Textract, which pulls text, forms, and tables out of scanned documents using AI.

Let's connect

Questions, corrections, or want to talk through where this fits in your own project? Reach me at khantanseer43@gmail.com.

AWS AppSync: The Easiest Way to Build Real-Time Apps

Tanseer — Wed, 19 Aug 2026 05:30:00 +0000

Build APIs that fetch exactly the data an app needs and push live updates as things change. Stop five in the AWS Hidden Gems series.

About this series

Most AWS learning stops after EC2, S3, IAM, and Lambda. But AWS has over two hundred services, and many of the most useful ones rarely appear in tutorials.

Today's service: AWS AppSync

Modern apps often need two things that are awkward to build: letting the client ask for exactly the data it wants in one request, and pushing live updates to the screen the moment data changes. AppSync gives you both. It is a managed service for building GraphQL APIs, with real time updates included.

Why does this service exist?

Two common API headaches led to AppSync. First, with traditional REST APIs, the client often calls several endpoints and gets back more data than it needs, which is slow on mobile. GraphQL fixes this by letting the client ask for precisely the fields it wants in a single request, but running a GraphQL server yourself means managing servers, connecting data sources, and writing a lot of glue.

Second, live updates. Showing new messages or a changing dashboard in real time usually means managing WebSocket connections, which are long lived two way connections between client and server. Doing that at scale is fiddly.

AppSync handles both. It runs the GraphQL API for you, connects it to your data with little code, and manages the real time connections so updates just arrive.

What is AWS AppSync?

AppSync is a managed GraphQL service. GraphQL is a query language for APIs where the client describes the exact data it wants, and the server returns just that, in one round trip.

You define a schema, which is the shape of your data and the operations allowed on it. You connect each part of the schema to a data source, which AppSync can talk to directly:

DynamoDB for fast NoSQL storage
Lambda for any custom logic
Relational databases, HTTP endpoints, and others AppSync supports three kinds of operations: queries to read data, mutations to change data, and subscriptions to receive live updates. When someone changes data through a mutation, AppSync automatically pushes that change to every client subscribed to it. Authentication is built in, with options including API keys and Amazon Cognito, the AWS user sign in service.

A real world problem

A team is building a shared task board, like a simple Trello. When one person moves a card, everyone else looking at the board should see it move immediately.

Building this the usual way means a REST API for the data plus a separate WebSocket layer to broadcast changes, with code to track who is connected and who cares about which board. That is a lot of plumbing for a small team.

With AppSync, they define the board and card types once, back them with DynamoDB, and add a subscription. Now a mutation to move a card is pushed to every open board automatically. The real time behavior comes almost for free.

Real world use cases

Chat and messaging apps deliver new messages instantly to everyone in a conversation
Collaborative tools like shared boards and documents sync changes across users live
Live dashboards update metrics on screen as new data arrives
Multiplayer game features share state like scores and moves between players
Mobile apps fetch lean, exact payloads to stay fast on slow connections
Apps that work offline sync their changes when the connection returns The pattern is flexible data access plus live updates, without building the real time layer yourself.

Where it fits in AWS

The client app talks to a single AppSync endpoint. AppSync routes each request to the right data source, most often DynamoDB for storage or Lambda for custom logic. Cognito handles who is allowed to do what. For live updates, clients hold a subscription connection to AppSync, and AppSync pushes changes to them.

flowchart LR
    A[Client app] -->|GraphQL query or mutation| B[AppSync API]
    B --> C[DynamoDB]
    B --> D[Lambda]
    B -->|Auth check| E[Cognito]
    B -->|Live updates| A

AppSync sits between your app and your data, giving one flexible endpoint and handling the real time push.

How the workflow runs

You define a GraphQL schema with your types and operations. You attach data sources and connect each field to an action, such as read this item from DynamoDB. The client sends a query or mutation, and AppSync runs the connected action and returns exactly the requested fields. When a mutation changes data, AppSync notifies every client subscribed to that change over their live connection.

flowchart TD
    A[Define GraphQL schema] --> B[Connect fields to data sources]
    B --> C[Client sends query or mutation]
    C --> D[AppSync runs it against the data source]
    D --> E[Return exactly the requested fields]
    D --> F[Push updates to subscribed clients]

Setting it up in the AWS Console

You will create a GraphQL API, let AppSync build a DynamoDB backed type, and test it live.

Sign in to the AWS Console, search for AppSync, and open it. Check the region in the top right corner.
Click Create API, choose GraphQL APIs, and pick the option to design from scratch. Give the API a name and create it.
Open the Schema page. Define a simple type and the operations for it, for example a Task type with an id and a title, and queries and mutations to create and list tasks.
Use the Create Resources shortcut on the schema. AppSync can generate a DynamoDB table and the resolvers, which are the small pieces that connect each GraphQL field to a table action, so you do not write them by hand.
Go to the Queries page, a built in tool for running operations against your API. Run a mutation to create a task, then a query to list tasks, and confirm your data comes back.
To see real time updates, open a subscription in the Queries page in one browser tab, then run a create mutation in another. The new task appears in the subscription tab immediately.
For your app to call the API, note the API URL and set an authorization mode under Settings. An API key is fine for testing, while Cognito is the usual choice for real users. Confirm a call works from your code using the URL and key. Common mistakes: an unauthorized error usually means the request is missing the API key or Cognito token the API expects, so check the authorization mode and that your client sends the right credential. If a field returns null, its resolver is probably not connected to a data source yet.

Using it from code

With AppSync, the code is your GraphQL schema and the operations clients run against it. Here is a small schema and the three operation types.

type Task {
  id: ID!
  title: String!
  done: Boolean!
}

type Query {
  listTasks: [Task]
}

type Mutation {
  addTask(title: String!): Task
}

type Subscription {
  onAddTask: Task
  @aws_subscribe(mutations: ["addTask"])
}

A client adds a task and, thanks to the subscription, every other client is notified without polling:

mutation {
  addTask(title: "Write the AppSync post") {
    id
    title
    done
  }
}

The @aws_subscribe line is what wires the subscription to the mutation, so any addTask pushes the new task to subscribers automatically.

Pricing

Item	Detail
Queries and mutations	Per million operations
Query and mutation rate	About $4.00 per million operations (US)
Real time updates	Per million updates pushed to clients
Connection time	Per million minutes clients stay connected
Free tier	250,000 operations and 250,000 real time updates per month for 12 months

The AWS API and app services family

AWS API and App Services
├── AppSync       managed GraphQL with real time updates
├── API Gateway   REST and WebSocket APIs you route yourself
├── Amplify       front end hosting and app building tools
└── Lambda        serverless functions behind your APIs

The main choice is AppSync versus API Gateway. AppSync is built around GraphQL and real time updates, and connects to data sources with little code. API Gateway is built around REST and WebSocket routing, giving you more manual control. For live, data driven apps, AppSync usually gets you there faster.

Wrapping up

AppSync gives you a flexible GraphQL API and real time updates without running a server or building a WebSocket layer. Define a schema, point it at your data, and changes flow to every client live. Next time an app needs live, precise data access, you know the service that skips the plumbing.

Series progress

You are on stop five of AWS Hidden Gems.

AWS Elemental MediaConvert
Amazon IVS
Amazon Rekognition
Amazon Personalize
AWS AppSync (you are here)
Amazon Timestream
Amazon Textract
Amazon Kendra
AWS DataSync
AWS IoT Core Next up is Amazon Timestream, a database built specifically for time series data like sensor readings and metrics.

Let's connect

Questions, corrections, or want to talk through where this fits in your own project? Reach me at khantanseer43@gmail.com.

Amazon Personalize: Build Recommendation Systems Without ML

Tanseer — Mon, 17 Aug 2026 05:30:00 +0000

The recommendation technology behind Amazon.com, packaged as a service you can use without a data science team. Stop four in the AWS Hidden Gems series.

About this series

Most AWS learning stops after EC2, S3, IAM, and Lambda. But AWS has over two hundred services, and many of the most useful ones rarely appear in tutorials.

Today's service: Amazon Personalize

"Customers who bought this also bought" and "recommended for you" feel simple as a user, but they run on serious machine learning. Amazon Personalize gives you that same recommendation technology as a managed service. You bring your data, it trains a private model, and your app asks it what to show each user.

Why does this service exist?

Good recommendations need machine learning, and building an ML recommender from scratch is a long road. You need data scientists, pipelines to clean and prepare data, model training and tuning, and infrastructure to serve predictions in real time as users browse. Most teams cannot staff that.

Personalize packages the recommendation systems Amazon developed over decades. It handles the data preparation, model training, and real time serving. You never see the algorithms or the servers. You give it your interaction history, and it gives you recommendations through an API.

What is Amazon Personalize?

Personalize is a managed recommendation service. A recommendation system predicts what a user is most likely to want next, based on their behavior and the behavior of similar users.

You give it three kinds of data:

Interactions: the events that happened, like a user viewing, clicking, or buying an item. This is the most important data.
Items: details about the things you recommend, like category or price.
Users: details about your users, like age group or location. From that, Personalize can power several things: personalized recommendations for each user, similar items to something a user is viewing, and a personalized reordering of a list you already have. It keeps recommendations fresh by taking in new activity as it happens, so suggestions shift as a user browses.

What Personalize does not do is give good results without enough data. It needs a reasonable history of interactions to learn from, so it suits products that already have some usage.

A real world problem

A streaming video app has a growing catalog and a home screen that shows the same popular titles to everyone. Users scroll past them because nothing feels relevant to their taste.

The team wants a "recommended for you" row that reflects what each person actually watches. Building that with in house machine learning would take months and specialists they do not have.

With Personalize, they upload their viewing history, choose a recommendation type, let it train, and call it from the app to fill that row per user. The home screen starts reflecting real taste, and watch time goes up.

Real world use cases

Streaming services build personalized "recommended for you" rows from viewing history
Online stores show product recommendations and "you might also like" suggestions
News and content sites reorder articles to match each reader's interests
Marketplaces surface relevant listings instead of the same popular few
Apps personalize search results and category pages per user
Marketing teams pick the most relevant items to feature in an email to each customer The pattern is showing each user the items they are most likely to want, drawn from their own behavior.

Where it fits in AWS

Your interaction data usually starts in your app's database or event stream. You export it as a CSV file into S3, and Personalize imports it to train. Your app sends live events to Personalize as users act, and calls it for recommendations when rendering a screen.

flowchart LR
    A[App activity: views, clicks, buys] --> B[CSV in S3]
    B -->|Import| C[Personalize trains a model]
    C --> D[Deployed recommender]
    E[Your app] -->|Get recommendations for a user| D
    E -->|Send new events| C

Personalize is the recommendation engine. S3 carries the training data in, and your app talks to the trained model over an API.

How the workflow runs

You start by creating a dataset group, a container for your data. You import your interactions data, and optionally items and users, from a CSV in S3. You pick a recipe, which is the algorithm for your goal, such as recommendations for a user or similar items. Personalize trains a model on your data, which takes a while. Once trained, you deploy it so your app can query it, then call it for recommendations in real time. As users keep acting, you send those events back so the model stays current.

flowchart TD
    A[Create dataset group] --> B[Import interaction data from S3]
    B --> C[Choose a recipe for your goal]
    C --> D[Personalize trains the model]
    D --> E[Deploy it for your app to query]
    E --> F[App gets recommendations per user]

Setting it up in the AWS Console

For this you need a CSV of interactions in S3 with at least three columns: a user id, an item id, and a timestamp.

Sign in to the AWS Console, search for Amazon Personalize, and open it. Check the region in the top right corner.
Click Create dataset group, give it a name, and choose the custom option so you control the steps directly.
Create the interactions dataset. Define a schema that matches your CSV columns (USER_ID, ITEM_ID, and TIMESTAMP), then start an import job pointing to your CSV file in S3. Personalize needs an IAM role that can read that S3 bucket, and the console can create this role for you.
Wait for the import to finish, then create a solution, which is a trained model. Pick a recipe such as User-Personalization for per user recommendations. Personalize trains on your data, and this step can take a while depending on data size.
When training finishes, create a campaign. A campaign is the deployed version of your model that your app calls for real time recommendations.
Test it in the console. On the campaign page, enter a user id and Personalize returns a ranked list of item ids recommended for that user.
To confirm the setup end to end, call the campaign from your code as shown next and check that it returns items for a real user id. Common mistakes: an import that fails almost always means the CSV columns do not match the schema you defined, so line them up exactly. If recommendations look random, the model may not have had enough interaction data, since Personalize needs a reasonable history to learn.

Using it from code

Once a campaign is deployed, getting recommendations is a single call with a user id.

import boto3

personalize = boto3.client("personalize-runtime")

response = personalize.get_recommendations(
    campaignArn="arn:aws:personalize:us-east-1:123456789012:campaign/my-campaign",
    userId="user-123",
    numResults=10,   # how many items to recommend
)

for item in response["itemList"]:
    print(item["itemId"], round(item.get("score", 0), 4))

To send new activity so recommendations stay fresh, use the put_events call on the personalize-events client as users view or buy items.

Pricing

Item	Detail
Data import	Per GB of data ingested
Training	Per hour of model training
Real time recommendations	Per throughput unit per hour while deployed
Batch recommendations	Per user or item processed in bulk jobs
Free tier	First 2 months include data processing, some training hours, and recommendation capacity

The AWS AI services family

AWS AI Services
├── Personalize   recommendations and personalization
├── Rekognition   image and video understanding
├── Textract      data extraction from documents
├── Comprehend    text analysis and sentiment
├── Transcribe    speech to text
└── Lex           chatbots and voice assistants

Like the rest of this family, Personalize hands you a hard machine learning capability through a simple API. Rekognition, from the last article, does the same for vision. Personalize does it for recommendations.

Wrapping up

Personalize turns your own usage data into recommendations, using Amazon's recommendation technology without asking you to build or understand the models. Next time a product needs "recommended for you" done well, you know the service that gets you there without a data science team.

Series progress

You are on stop four of AWS Hidden Gems.

AWS Elemental MediaConvert
Amazon IVS
Amazon Rekognition
Amazon Personalize (you are here)
AWS AppSync
Amazon Timestream
Amazon Textract
Amazon Kendra
AWS DataSync
AWS IoT Core Next up is AWS AppSync, the easiest way to build APIs that update in real time.

Let's connect

Questions, corrections, or want to talk through where this fits in your own project? Reach me at khantanseer43@gmail.com.

Amazon Rekognition: Add AI Vision to Your Applications

Tanseer — Wed, 12 Aug 2026 05:30:00 +0000

Give your app the ability to understand images and video, with no machine learning knowledge required. Stop three in the AWS Hidden Gems series.

About this series

Most AWS learning stops after EC2, S3, IAM, and Lambda. But AWS has over two hundred services, and many of the most useful ones rarely appear in tutorials.

Today's service: Amazon Rekognition

Your app can store and show images. Rekognition lets it understand them too: what objects are in a photo, what text appears, whether a face matches, or whether an image is unsafe. You call an API and get the answer back, with no models to train.

Why does this service exist?

Teaching a computer to understand images is one of the hardest problems in software. Traditionally it meant hiring machine learning engineers, collecting and labeling thousands of training images, renting GPUs, which are specialized chips for training models, and maintaining that model over time. That is out of reach for most teams.

Rekognition removes all of that. AWS trained the deep learning models already, on huge datasets, and exposes them through a simple API. You send an image, you get structured results back. The hard research and training is done, and you just make a call.

What is Amazon Rekognition?

Rekognition is a managed computer vision service. Computer vision means software that interprets images and video the way a person visually would.

It can:

Detect objects and scenes, returning labels like dog, car, or beach with a confidence score
Detect and analyze faces, including expressions and an estimated age range
Compare two faces, or search a face against a stored collection to find a match
Read text that appears inside an image, like a sign or a label
Flag unsafe or explicit content for moderation
Detect safety equipment such as helmets and masks in workplace photos
Analyze stored or live video for the same things across frames For images, results come back instantly. For video, Rekognition processes the file in the background and notifies you when it is done. If the built in labels are not enough, Custom Labels lets you train it to recognize your own specific objects from a small set of example images.

A real world problem

A marketplace app lets users upload photos of items they want to sell. Some upload blurry pictures, some pick the wrong category, and a few upload content that breaks the rules.

Reviewing every photo by hand does not scale past a few hundred a day. The team needs each upload checked automatically: is there actually a product in the photo, does it match the chosen category, and is the content safe to show.

Rekognition handles all three. One call returns labels to confirm the category, and another checks for unsafe content, so only clean, correctly tagged listings go live. No human in the loop for the routine cases.

Real world use cases

Marketplaces and social apps moderate uploaded images automatically for unsafe content
Retail tags product photos with labels so search and filtering work without manual data entry
Media companies find every clip a specific person or object appears in across a video library
Security and access systems match a face at a door against a collection of approved people
Manufacturing checks photos for required safety gear before allowing entry to a site
Apps make images searchable by reading the text printed inside them The pattern is turning raw pixels into structured data your app can act on.

Where it fits in AWS

Images and video usually live in S3. A common setup: a user uploads an image to S3, that upload triggers a Lambda function, the function calls Rekognition, and the results get stored in DynamoDB for your app to use. For faces, Rekognition keeps a collection, which is a searchable index of face data. For video, results are delivered through SNS, the AWS notification service, when the background job finishes.

flowchart LR
    A[User uploads image] --> B[S3 bucket]
    B -->|Upload event| C[Lambda function]
    C -->|Analyze image| D[Rekognition]
    D -->|Labels and results| C
    C -->|Store results| E[DynamoDB]

Rekognition is the vision brain you call from your own code. Everything around it, meaning storage, triggers, and results, uses services you already know.

How the workflow runs

For an image, the flow is one step: send the image to Rekognition, either its S3 location or the raw bytes, and it returns labels, faces, text, or moderation results as JSON. For video, it is three steps: start a job on a video in S3, Rekognition processes it in the background, and it sends a notification through SNS when the results are ready to fetch.

Face search adds a setup step. First you add known faces to a collection with IndexFaces. Then, to identify someone, you search a new face against that collection and get back the closest matches.

flowchart TD
    A[Image or video] --> B{Type?}
    B -->|Image| C[Call Rekognition, get results instantly]
    B -->|Video| D[Start a job]
    D --> E[Rekognition processes in background]
    E --> F[SNS notifies you]
    F --> G[Fetch results]

Setting it up in the AWS Console

The Rekognition console has built in demos, so you can try it on your own image before writing any code.

Sign in to the AWS Console, search for Rekognition, and open it. Check the region in the top right corner.
In the left menu under Demos, click Label detection. This is the feature that identifies objects and scenes.
Upload an image from your computer, or use one of the samples provided. Rekognition analyzes it right away.
Look at the results. On the right you see a list of labels, each with a confidence score from 0 to 100. The image on the left highlights where objects were found. Try a few different images to see how the labels change.
Explore the other demos in the left menu, such as Facial analysis, Text in image, and Content moderation, to see the range of what Rekognition returns.
To use Rekognition from your own code, create an IAM user or role with the AmazonRekognitionReadOnlyAccess policy for the detection APIs, plus s3:GetObject permission on the bucket holding your images. This lets your code call Rekognition and read the images it needs to analyze.
To confirm your setup works outside the console, run the code in the next section against an image in your S3 bucket and check that labels come back. Common mistakes: an access error usually means the IAM role is missing either Rekognition permission or read access to the S3 bucket, so check both. If labels seem wrong, remember the confidence score and filter out anything below a threshold like 80 for your app.

Using it from code

This calls Rekognition on an image stored in S3 and prints the labels it finds, with their confidence scores.

import boto3

rekognition = boto3.client("rekognition")

response = rekognition.detect_labels(
    Image={
        "S3Object": {"Bucket": "my-image-bucket", "Name": "photo.jpg"}
    },
    MaxLabels=10,        # return at most 10 labels
    MinConfidence=80,    # ignore anything the model is less than 80% sure about
)

for label in response["Labels"]:
    print(label["Name"], round(label["Confidence"], 1))

Swap detect_labels for detect_moderation_labels to check for unsafe content, or detect_text to read text in the image. The shape of the call stays the same.

Pricing

Item	Detail
Image analysis	Per image processed, per feature (labels, faces, text, moderation)
Image rate	About $1.00 per 1,000 images for common features (US, first tier)
Video analysis	Per minute of video processed
Face storage	Small monthly charge per 1,000 faces kept in a collection
Volume discounts	Lower per unit rates as monthly volume grows
Free tier	5,000 images per month for the first 12 months

The AWS AI services family

AWS AI Services
├── Rekognition   understands images and video
├── Textract      pulls text and data from documents
├── Transcribe    turns speech into text
├── Comprehend    finds meaning and sentiment in text
├── Polly         turns text into speech
└── Translate     translates between languages

These are the ready to use AI services, each solving one problem through a simple API with no model training. Rekognition is the one for vision. Textract, coming later in this series, is the one built specifically for documents.

Wrapping up

Rekognition gives your app eyes. Objects, faces, text, and unsafe content all come back from a single API call, with the machine learning already done for you. Next time a feature needs to understand an image, you know you do not have to build a vision model to get it.

Series progress

You are on stop three of AWS Hidden Gems.

AWS Elemental MediaConvert
Amazon IVS
Amazon Rekognition (you are here)
Amazon Personalize
AWS AppSync
Amazon Timestream
Amazon Textract
Amazon Kendra
AWS DataSync
AWS IoT Core Next up is Amazon Personalize, which builds recommendation systems using the same technology behind Amazon.com, again with no machine learning required.

Let's connect

Questions, corrections, or want to talk through where this fits in your own project? Reach me at khantanseer43@gmail.com.

Amazon IVS: Build Live Streaming Without the Complexity

Tanseer — Mon, 10 Aug 2026 05:30:00 +0000

Live video in your app without running a single streaming server. Stop two in the AWS Hidden Gems series.

About this series

Most AWS learning stops after EC2, S3, IAM, and Lambda. But AWS has over two hundred services, and many of the most useful ones rarely appear in tutorials.

Today's service: Amazon IVS

In the last article, MediaConvert handled video files that already existed. Amazon IVS, short for Interactive Video Service, is for the opposite case: video happening right now, like a live class, a product launch, or a gaming stream. It is the managed way to put live video into your app or website.

Why does this service exist?

Live streaming is genuinely hard to build. You need a server to receive the incoming video, something to transcode it into different qualities, a way to package it for players, a global network to deliver it, and a video player for viewers. On top of that, live audiences are spiky. Ten viewers one minute, ten thousand the next. Getting low latency, meaning a small delay between the camera and the viewer, takes real expertise.

Before IVS, teams stitched this together from separate services like MediaLive, MediaPackage, and a content delivery network, then built and tuned their own player. It worked, but it was a lot of moving parts.

IVS bundles all of it into one service. You create a channel, point your camera software at it, and share a playback URL. IVS handles ingest, transcoding, delivery, and scaling, and it gives you a ready made player. It is built on the same technology that runs Twitch.

What is Amazon IVS?

Amazon IVS is a fully managed live streaming service. It comes in two modes.

Low latency streaming is for broadcasting to an audience, from one host to thousands or millions of viewers, with a delay of only a few seconds. This is the classic live stream.

Real time streaming is for interactive video where the delay is under a third of a second. Think multiple hosts on screen together, audio rooms, or a guest joining a broadcast. It supports up to twelve hosts publishing at once.

Both give you SDKs for web, iOS, and Android, so you can broadcast and play video from inside your own app. IVS can also record streams to S3, add live chat, and carry timed metadata, which is data synced to a moment in the video for things like polls or product highlights.

What IVS does not do is process video files that already exist. That is MediaConvert, from the last article. IVS is only for live.

A real world problem

A fitness startup wants live workout classes in its app. An instructor streams from a studio, and members join from their phones at home.

Building this from scratch means running ingest servers, transcoding the video for different connection speeds, delivering it worldwide with a small delay, and writing a player for iOS and Android. The team is four people. None of that is their actual product.

With IVS, they create a channel, give the instructor a stream key, and drop the IVS player into their app pointed at the playback URL. The hard parts are handled, and they get back to building the fitness features that make the app worth using.

Real world use cases

Fitness and education run live classes with instructors streaming to members at home
Gaming and creator platforms build their own Twitch style streaming without owning the infrastructure
Retail and ecommerce run live shopping shows where hosts demo products and viewers buy in the moment
Events stream conferences, launches, and town halls to a large remote audience
Social apps add live audio rooms and multi host video using real time streaming
Auctions and betting need the tiny delay of real time streaming so no viewer sees the action late The common thread is live video inside an app, without a streaming team to run it.

Where it fits in AWS

The flow is simple. Broadcast software or the IVS broadcast SDK sends live video to an IVS channel using RTMPS, a secure protocol for pushing live streams. IVS transcodes the video into several qualities and delivers it through a playback URL. Viewers watch using the IVS player in your app. Optionally, IVS records the stream to S3 and sends stream events, like start and stop, to EventBridge.

flowchart LR
    A[Host camera and OBS] -->|RTMPS| B[IVS channel ingest]
    B --> C[IVS transcode and deliver]
    C -->|Playback URL| D[IVS player in your app]
    C -->|Optional recording| E[S3 bucket]
    C -->|Stream events| F[EventBridge]

IVS is the whole live pipeline in one service. You bring the camera and the app, and it handles everything in between.

How the workflow runs

You create a channel, which gives you three things: an ingest endpoint where video goes in, a stream key that authorizes your broadcast, and a playback URL where viewers watch. You put the ingest endpoint and stream key into your broadcast software, then start streaming. IVS ingests the video, transcodes it into multiple qualities on the fly, and serves it through the playback URL with only a few seconds of delay. Viewers open your app and the player loads the stream. When you stop broadcasting, the stream ends, and if recording is on, the file lands in S3.

flowchart TD
    A[Create channel] --> B[Get ingest endpoint, stream key, playback URL]
    B --> C[Start broadcasting to the ingest endpoint]
    C --> D[IVS transcodes and delivers]
    D --> E[Viewers watch through the player]
    C --> F[Stop broadcasting ends the stream]

Setting it up in the AWS Console

You will create a channel, then stream to it using OBS Studio, a free and popular broadcast tool.

Sign in to the AWS Console, search for IVS or Interactive Video Service, and open it. Check the region in the top right corner, since your channel lives in one region.
Click Create channel. For a first test, keep the default setup. The settings that matter are the channel type, which you can leave as Standard so IVS transcodes to several qualities, and the latency, which you can leave as Low. Give the channel a name and click Create channel.
On the channel page, IVS shows three things you need. Copy the Ingest server, which is an RTMPS address. Reveal and copy the Stream key, and keep it secret, since anyone who has it can stream to your channel. Copy the Playback URL as well.
Install and open OBS Studio. Go to Settings, then Stream. Set Service to Custom, paste the IVS ingest server into the Server field, and paste your stream key into the Stream Key field. Click OK.
In OBS, add a source so there is something to show, such as your webcam or a screen capture, then click Start Streaming.
Go back to the channel page in the IVS console. Within a few seconds the channel shows as Live with a preview of your video. You can also open the Playback URL in the IVS web player to watch it exactly as a viewer would.
To confirm everything works, check that the preview matches your camera and plays smoothly. Click Stop Streaming in OBS to end the stream.
To save your streams, edit the channel and turn on recording to an S3 bucket. IVS stores each stream there automatically, and the console sets up the permission it needs to write to the bucket. No IAM role is needed to broadcast, because the stream key is what authorizes you. Common mistakes: if OBS says it cannot connect, the ingest server or stream key is usually wrong, so paste them again carefully. If the console never shows Live, make sure OBS is actually streaming and that you picked the Custom service option rather than a preset like Twitch.

Going live from code

Two pieces of code make IVS practical: creating a channel on your backend, and showing the stream in your app.

First, create a channel with the AWS SDK. This returns the same ingest endpoint, stream key, and playback URL you saw in the console.

import boto3

ivs = boto3.client("ivs")

response = ivs.create_channel(
    name="my-live-channel",
    type="STANDARD",     # transcodes the stream into several qualities
    latencyMode="LOW",   # a few seconds of delay, good for most live streams
)

channel = response["channel"]
stream_key = response["streamKey"]["value"]

print("Ingest server:", channel["ingestEndpoint"])
print("Playback URL:", channel["playbackUrl"])
print("Stream key:", stream_key)   # keep this secret

Then, on the viewer side, load the playback URL in the IVS web player.

<script src="https://player.live-video.net/1.x/amazon-ivs-player.min.js"></script>
<video id="video-player" playsinline controls></video>
<script>
  const player = IVSPlayer.create();
  player.attachHTMLVideoElement(document.getElementById("video-player"));
  player.load("YOUR_PLAYBACK_URL");
  player.play();
</script>

To call create_channel, the backend's IAM role needs IVS permissions such as ivs:CreateChannel. The player needs nothing secret, since the playback URL is safe to share with viewers.

Pricing

Item	Detail
What you pay for	Input hours (video you send in) plus output hours (video sent to viewers)
Rate driver	Video quality, grouped as SD, HD, and Full HD
Input, HD	About $2.00 per streaming hour (US)
Output, HD	About $0.15 per viewer hour (US)
Cost example	1 hour HD stream watched by 100 viewers, about $2 input plus $15 output, near $17
Real time streaming	Billed separately, by participant minutes
Recording	Saved to S3, so normal S3 storage rates apply
Free tier	None for IVS streaming hours

The AWS live video family

AWS Live and Media Services
├── Amazon IVS      all in one live streaming with a built in player
├── MediaLive       broadcast grade live encoding you assemble yourself
├── MediaPackage    packaging and protection for live and on demand video
├── MediaConnect    reliable transport of live video between points
└── MediaConvert    file based transcoding for videos that already exist

The choice comes down to control versus speed. IVS is the fastest way to add live video to an app, with the player included. MediaLive with MediaPackage is the build it yourself route when you need fine control over a broadcast. MediaConvert, from the last article, is for finished files rather than live video.

Wrapping up

IVS turns live streaming from a big infrastructure project into a channel, a stream key, and a playback URL. You bring the camera and the app, and IVS handles ingest, transcoding, delivery, and the player. Next time you need live video in a product, you know the service that removes the hard parts.

Series progress

You are on stop two of AWS Hidden Gems.

AWS Elemental MediaConvert
Amazon IVS (you are here)
Amazon Rekognition
Amazon Personalize
AWS AppSync Next up is Amazon Rekognition, which adds AI vision to your apps so they can understand what is in an image or video.

Let's connect

Questions, corrections, or want to talk through where this fits in your own project? Reach me at khantanseer43@gmail.com.

AWS Elemental MediaConvert: The Video Service Every Developer Should Know

Tanseer — Mon, 03 Aug 2026 05:30:00 +0000

About this series

Most AWS learning stops after EC2, S3, IAM, and Lambda. But AWS has over two hundred services, and many of the most useful ones rarely appear in tutorials.

Today's service: MediaConvert

Almost every app has video in it, and every video hides one boring step: converting the raw file into formats that phones, laptops, TVs, and slow networks can all play. That is what MediaConvert does.

Why does this service exist?

Transcoding means rewriting a video into a different format, quality, or resolution. You need it because one file cannot serve every viewer. A weak mobile connection needs a small version. A fast home connection wants the crisp one.

Before AWS, teams ran FFmpeg (a free tool that converts almost any video) on their own server fleets. Those servers were expensive, sat idle between uploads, and needed constant patching. Getting broadcast quality output took specialist knowledge most teams did not have.

MediaConvert removes all of that. You hand it a file, it does the work on infrastructure you never manage, and you pay only for the minutes you process.

What is MediaConvert?

MediaConvert is a fully managed, file based video transcoding service. Fully managed means AWS runs the servers for you. File based means it works on complete video files that already exist, usually in S3, not on live streams.

What it does:

Changes the codec, the compression method, such as H.264 or the newer H.265
Changes resolution, so one 4K master becomes 1080p, 720p, and 480p
Adjusts bitrate, the data used per second and the main quality versus size lever
Adds captions, extra audio tracks, logo overlays, and thumbnails
Packages video for streaming with HLS and DASH, the formats behind adaptive bitrate, where the player switches quality as the network changes What it does not do is store or deliver your videos. Storage is the job of S3. Delivery is the job of CloudFront, a network that keeps copies of your files close to users.

A real world problem

An online course startup lets instructors upload lectures from their phones. Files arrive in every format and size. A student on wifi watches fine. A student on mobile data sees the giant 4K file stall every few seconds. Some older devices cannot play certain formats at all.

The startup cannot control how instructors record or how students watch. It can only fix the middle: convert every upload into a consistent set of streaming friendly versions. That is what MediaConvert is for.

Real world use cases

Streaming platforms turn studio master files into every quality level for phones, browsers, and TVs
Education platforms standardize instructor uploads so playback never depends on someone's recording setup
Healthcare processes recorded procedures and telehealth sessions into required formats for review and archiving
Finance and enterprise transcode earnings calls and training, captioned for accessibility and stored for the long term
Ecommerce converts product and review clips into fast loading versions that do not slow the page
Gaming and social apps normalize huge volumes of user uploaded clips into formats the app can play The pattern is always the same. Video comes in messy, and it leaves clean and ready to play.

Where it fits in AWS

A common setup: a video lands in S3, the upload triggers a Lambda function, Lambda starts a MediaConvert job, MediaConvert reads the file and writes converted versions back to S3, and CloudFront delivers them worldwide. MediaConvert reports progress through EventBridge, the AWS service for reacting to events, and it accesses S3 using an IAM role, a set of permissions a service borrows to act for you.

flowchart LR
    A[User uploads video] --> B[S3 input bucket]
    B -->|Upload event| C[Lambda function]
    C -->|Start job| D[MediaConvert]
    B -->|Reads source file| D
    D -->|Writes outputs| E[S3 output bucket]
    E --> F[CloudFront]
    F --> G[Viewers on any device]
    D -->|Job status| H[EventBridge]

MediaConvert is the processing engine in the middle. Everything around it is handled by services built for that job.

How the workflow runs

The unit of work is a job: one input file, the outputs you want, and the IAM role to use. A job is submitted, waits in a queue, gets processed on managed infrastructure, and writes its outputs to S3 when done. Status updates flow through EventBridge the whole time, and failures come back with an error message.

Two features save repetition. Presets store output settings, so you set 1080p H.264 once and reuse it. Job templates store a whole job's settings, so your standard outputs become a one line request.

flowchart TD
    A[Job submitted] --> B[Job waits in queue]
    B --> C[MediaConvert processes it]
    C --> D{Success?}
    D -->|Yes| E[Outputs written to S3]
    D -->|No| F[Job marked with error]

Setting it up in the AWS Console

Before you start, upload a short sample video to an S3 bucket. Note the bucket name and the folder where you want the output. Keep the file small so your first job finishes quickly.

Sign in to the AWS Console, type MediaConvert into the search bar at the top, and open it. Look at the region shown in the top right corner and make sure it matches the region of your S3 bucket. MediaConvert reads from and writes to S3, so keeping both in the same region avoids extra data transfer charges and keeps the job faster.
Click Create job. The page splits into two halves. The left panel is where you add the parts of your job, meaning the input and the output groups. The right panel is where you edit the settings for whatever part you have selected.
In the Input section on the left, paste the full S3 path to your sample video, for example s3://your-bucket-name/sample.mp4. This is the file MediaConvert will read. Check the path carefully, because a wrong path or a missing file extension is the most common reason a job fails right away.
In the left panel, find Output groups and click Add. MediaConvert asks what kind of group you want. Choose File group for this first test, which produces one converted file rather than a segmented streaming package, so it is the simplest way to confirm the flow works. Open the group and set its Destination to the S3 folder for your output, for example s3://your-bucket-name/output/.
Inside the output group there is an output entry. Select it and set three things on the right: the video codec to H.264, which every device supports, the resolution to 1280 by 720, and the audio settings left at their defaults. These few values are enough to produce a clean, playable file.
Scroll to the Job settings section and find the service access or IAM role option. MediaConvert needs a role that lets it read your input bucket and write to your output bucket. For a first job, pick the option that lets the console create this role for you. If you prefer to build it yourself, the role needs a trust policy allowing the MediaConvert service to assume it, plus s3:GetObject on the input bucket and s3:PutObject on the output bucket.
Leave the default on demand queue selected. You do not need reserved capacity to start.
Click Create at the bottom. The job appears with a status of Progressing. For a small file it changes to Complete within a minute or two. Open your output folder in S3, download the converted file, and play it to confirm the transcode worked. Common mistakes: a permissions error almost always means the IAM role cannot reach one of your buckets, so check that it covers both the input and the output. A job that fails right away usually means the input path is wrong, so confirm it exactly, including the file extension.

Starting a job with code

Clicking through the console is fine for learning, but in a real system you want jobs to start on their own. The usual pattern is a Lambda function that runs whenever a new video lands in S3 and tells MediaConvert to start a job.

One detail trips up newcomers. MediaConvert gives each account its own endpoint, which is its own address, so you fetch that endpoint once and build the client with it. To keep the code short, first save your settings from steps 4 and 5 as a job template in the console. The template holds the codec, resolution, and output destination, so the function only has to pass the input file.

import boto3

# MediaConvert has a per account endpoint, so fetch it once when the function loads.
default_client = boto3.client("mediaconvert")
endpoint = default_client.describe_endpoints()["Endpoints"][0]["Url"]
mediaconvert = boto3.client("mediaconvert", endpoint_url=endpoint)

def lambda_handler(event, context):
    # The S3 event tells us which file was just uploaded.
    record = event["Records"][0]["s3"]
    source = f"s3://{record['bucket']['name']}/{record['object']['key']}"

    response = mediaconvert.create_job(
        Role="arn:aws:iam::123456789012:role/MediaConvertRole",
        JobTemplate="my-video-template",
        Settings={
            "Inputs": [{"FileInput": source}]
        },
    )

    print("Started MediaConvert job:", response["Job"]["Id"])
    return {"jobId": response["Job"]["Id"]}

To make this run on its own, add an S3 trigger on your input bucket that invokes the function on new uploads, and give the function's role permission to call MediaConvert and to pass the MediaConvert role.

Pricing

Item	Detail
Billing unit	Per minute of output video, prorated per second
Billed on	Output, not input
Output math	One 10 minute video into 3 versions = 30 output minutes
Basic tier HD	About $0.015 per minute (US, on demand)
Basic tier SD	Lower than HD
Basic tier 4K	Higher than HD
Professional tier	Higher rate, for advanced broadcast features
Cost example	100 videos x 10 min x 2 HD versions = 2,000 min, about $30 per month
Free tier	Monthly allowance of basic tier output minutes for 12 months
Reserved pricing	Lower rate for large, predictable monthly volume

The AWS media services family

AWS Media Services
├── MediaConvert   file based transcoding for existing videos
├── MediaLive      real time encoding for live broadcasts
├── MediaPackage   packages and protects streams for delivery
├── MediaConnect   reliable transport of live video
├── MediaTailor    inserts personalized ads into streams
└── Amazon IVS     managed low latency interactive live streaming

The key split is file versus live. MediaConvert handles finished files. MediaLive and IVS handle video happening right now. If your video already exists, MediaConvert is the tool.

Wrapping up

MediaConvert takes video in any shape and produces clean, playable versions for every device and network, with no servers to run. Next time you build anything with video, you know the managed service that does the heavy lifting and how to wire it up.

Series progress

You are on stop one of AWS Hidden Gems.

AWS Elemental MediaConvert (you are here)
Amazon IVS
Amazon Rekognition
Amazon Personalize
AWS AppSync Next up is Amazon IVS, which picks up where MediaConvert stops, handling live and interactive video.

Let's connect

Questions, corrections, or want to talk through where this fits in your own project? Reach me at khantanseer43@gmail.com.

Terraform State Locking Without DynamoDB: S3 Native Locking Explained

Tanseer — Tue, 21 Jul 2026 04:53:34 +0000

A beginner friendly guide to S3 native state locking and why it makes your Terraform setup so much cleaner

Setting up remote state in Terraform used to come with a small annoyance. You could not just use an S3 bucket on its own. You also had to create a DynamoDB table sitting right next to it, purely to handle locking. Two AWS services for something that felt like one job.

That has changed. Starting with Terraform 1.10, S3 can handle state locking by itself. No DynamoDB table, no extra resource to provision, no extra permissions to hand out. In this post I will walk through what state locking is, why it needed DynamoDB in the first place, what changed under the hood, and the exact steps to switch to the simpler setup.

This is written for people who are still new to AWS and Terraform, so I will explain each term the first time it shows up.

A quick refresher on the Terraform state file

When you run Terraform, it needs to remember what it built. It keeps this record in a file called the state file, usually named terraform.tfstate. Think of it as a map that connects the resources written in your code to the real resources living in your AWS account.

If you work alone on your laptop, that file can just sit on your machine. But the moment a second person joins, everyone needs to share the same map. If two people have two different copies, they will fight over what exists and what does not.

The common fix is to store the state file in a shared location that the whole team points to. This shared location is called a backend. An S3 bucket is one of the most popular backends because it is cheap, durable, and simple. A bucket is just a storage container in S3 where you keep files, which AWS calls objects.

Why state locking matters

Here is the problem that locking solves.

Say two engineers on your team both run terraform apply at almost the same moment. Both runs read the current state file. Both make their changes. Both try to write the updated state back. One write lands on top of the other, and now your state file is scrambled. In the worst case Terraform gets confused about what already exists and starts creating duplicates or destroying things it should not touch.

State locking stops this. Before Terraform makes any change, it grabs a lock. While that lock is held, any other run that tries to start is told to wait. Once the first run finishes, it releases the lock and the next person can go. Only one operation writes at a time, so the state file stays clean.

The question is where that lock actually lives.

The old way: a DynamoDB table just for locks

For a long time, the S3 backend could store your state, but it could not lock it. S3 on its own had no simple way to say "only write this if nobody else is holding it." So Terraform borrowed a second service to keep track of locks: DynamoDB, which is a fast key value database from AWS.

The setup looked like this. You created a DynamoDB table with a single partition key called LockID. A partition key, also called a hash key, is just the main field DynamoDB uses to find a row. When Terraform started an operation, it wrote a lock item into that table. When it finished, it deleted the item. If a second run showed up and saw a lock item already there, it knew the state was busy.

The Terraform code for that table usually looked like this.

resource "aws_dynamodb_table" "terraform_locks" {
  name         = "terraform-locks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

And the backend block that tied it all together looked like this.

terraform {
  backend "s3" {
    bucket         = "my-terraform-state-bucket"
    key            = "global/s3/terraform.tfstate"
    region         = "ap-south-1"
    encrypt        = true
    dynamodb_table = "terraform-locks"
  }
}

Here key is the path to the state file inside the bucket, and encrypt = true tells S3 to encrypt the file at rest.

None of this is hard on its own. But notice how much you are carrying. You provision a whole extra service. You manage its name and region and make sure they match the backend config. You grant IAM permissions for it, where IAM is the AWS system that decides who is allowed to do what. And you have one more thing that can silently break, like a table created in the wrong region or a name typo that only shows up when a lock fails.

What actually changed

Two things had to line up for S3 to do locking on its own.

First, in August 2024, AWS added conditional writes to S3. This is the important piece. A conditional write lets you upload an object only if it does not already exist. Technically this works through an HTTP header called If-None-Match. In plain terms, you can now tell S3 "create this file, but only if it is not already there, otherwise refuse." That refuse behavior is exactly what a lock needs. If the lock file is already there, your write bounces, which means someone else has the lock.

Second, Terraform 1.10 (released in November 2024) added a new backend option called use_lockfile. When you turn it on, Terraform uses that S3 conditional write trick to manage locks directly in the same bucket as your state. DynamoDB is no longer part of the picture.

Terraform 1.11 went a step further and marked the old dynamodb_table argument as deprecated, which is the maintainers' way of saying the DynamoDB path is on its way out and you should move over.

One quick clarification so you do not get tripped up. S3 native state locking has nothing to do with a feature called S3 Object Lock. Object Lock is a compliance feature that stops files from being deleted or changed for a set period. That is not what we are using here. Terraform state locking uses conditional writes, and it works on a plain, normal S3 bucket with no special mode turned on.

How it works under the hood

When you enable use_lockfile and run an operation, Terraform creates a small lock file in the same bucket, right next to your state file. It reuses your state path and adds a .tflock extension.

So if your state file is at:

global/s3/terraform.tfstate

the lock file will appear at:

global/s3/terraform.tfstate.tflock

While an operation runs, that .tflock object exists. When the operation ends, Terraform deletes it. If someone else runs Terraform during that window, S3 rejects their conditional write because the lock file is already there, and Terraform shows them an error instead of letting two runs collide.

The error looks something like this.

Error: Error acquiring the state lock

Error message: operation error S3: PutObject, https response error
StatusCode: 412, RequestID: ..., HostID: ..., api error PreconditionFailed:
At least one of the pre-conditions you specified did not hold

Lock Info:
  ID:        3f6a1c9e-...-b2d4
  Path:      my-terraform-state-bucket/global/s3/terraform.tfstate.tflock
  Operation: OperationTypeApply
  Who:       sahil@laptop
  Created:   2026-07-21 09:14:22.5 +0000 UTC

Notice the StatusCode: 412 and PreconditionFailed. That 412 is the HTTP status S3 returns when a conditional write is refused because the object already exists. In other words, the lock is doing its job. Everything you need to identify who holds the lock is right there in the output.

Steps to set it up

Here is the full walkthrough for a fresh setup.

1. Check your Terraform version

use_lockfile needs Terraform 1.10.0 or newer. Check what you have.

terraform version

If you see anything older than 1.10, upgrade before continuing.

2. Create the S3 bucket for your state

If you already have a state bucket you can skip ahead. If not, create one with the AWS CLI, which is the command line tool for talking to AWS. Bucket names have to be globally unique across all of AWS, so pick something specific to you.

aws s3api create-bucket \
  --bucket my-terraform-state-bucket \
  --region ap-south-1 \
  --create-bucket-configuration LocationConstraint=ap-south-1

A small gotcha. Every region except us-east-1 needs that --create-bucket-configuration LocationConstraint line. If you are using us-east-1, drop that last line, because that region rejects it.

3. Turn on versioning

Versioning tells S3 to keep old copies of a file each time it changes. For a state file this is a safety net. If something ever corrupts your state, you can roll back to an earlier version.

aws s3api put-bucket-versioning \
  --bucket my-terraform-state-bucket \
  --versioning-configuration Status=Enabled

4. Write the backend block

Now the good part. In your Terraform project, add the backend block with use_lockfile = true. Notice there is no dynamodb_table line anywhere.

terraform {
  backend "s3" {
    bucket       = "my-terraform-state-bucket"
    key          = "global/s3/terraform.tfstate"
    region       = "ap-south-1"
    encrypt      = true
    use_lockfile = true
  }
}

That single use_lockfile = true line is the whole feature.

5. Set the IAM permissions

Because DynamoDB is gone, your permissions get shorter too. The role or user running Terraform now only needs S3 actions, and no DynamoDB actions at all. A minimal policy looks like this.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::my-terraform-state-bucket"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::my-terraform-state-bucket/global/s3/terraform.tfstate*"
    }
  ]
}

The trailing * on the resource is doing quiet but important work. It matches both the state file and the .tflock lock file, since the lock file shares the same path with .tflock tacked on. One rule covers both.

6. Initialize Terraform

Run init to wire everything up. Terraform reads your backend block and gets the bucket ready.

terraform init

7. Test that the lock works

Run a normal plan and apply.

terraform plan
terraform apply

While the apply is running, open the S3 console or run aws s3 ls on your bucket path. You will see a terraform.tfstate.tflock object appear during the run and vanish once it finishes. If you want to see the lock in action, open a second terminal and start another terraform apply while the first one is still going. The second one will hit that 412 lock error from earlier and refuse to run until the first is done. That is the whole point working exactly as designed.

Moving over if you already use DynamoDB

If you have an existing setup with a DynamoDB lock table, switching is easy and you do not have to do it all in one nervous jump.

The cautious approach is to keep both options set for a short transition period. You can list use_lockfile = true and dynamodb_table at the same time, and Terraform will acquire locks in both places. That way nothing breaks while you and your team catch up.

terraform {
  backend "s3" {
    bucket         = "my-terraform-state-bucket"
    key            = "global/s3/terraform.tfstate"
    region         = "ap-south-1"
    encrypt        = true
    dynamodb_table = "terraform-locks"
    use_lockfile   = true
  }
}

After changing the backend, run init again with the reconfigure flag so Terraform picks up the new settings.

terraform init -reconfigure

Once you are confident everyone is on Terraform 1.10 or newer and the S3 locking is working, remove the dynamodb_table line, run terraform init -reconfigure one more time, and then delete the DynamoDB table you no longer need.

When a lock gets stuck

Every so often a Terraform run crashes halfway, or someone closes their laptop mid apply. When that happens the .tflock file can be left behind even though no operation is actually running. The next person then gets the lock error for a lock nobody holds.

To clear it, grab the lock ID from the error output (the ID field) and run force unlock.

terraform force-unlock 3f6a1c9e-...-b2d4

Use this carefully. Only force unlock when you are sure no real Terraform operation is running anywhere, otherwise you are back to the exact race condition that locking was meant to prevent. If in doubt, ask your team before you unlock.

Why this genuinely reduces complexity

Pull back and look at what you removed. You went from two services to one. There is no DynamoDB table to create, name, tag, watch, or clean up. Your lock now lives in the same bucket as your state, so there is one place to look when you want to understand what is going on.

The bootstrap problem shrinks too. Every Terraform team hits the same chicken and egg puzzle at the start. To use remote state you first need the backend infrastructure to exist, but that infrastructure is itself something you want to manage. With the old approach that meant standing up a bucket and a table before anything else. Now it is just a bucket.

Your permissions shrink. No DynamoDB actions to grant means a shorter IAM policy and fewer things a reviewer has to reason about. Your Terraform code shrinks, because the whole aws_dynamodb_table resource can be deleted. And you cut out a category of quiet failures, like a lock table sitting in the wrong region or a name that does not match the backend config.

Same safety, fewer moving parts. That is a rare and welcome trade.

Wrapping up

State locking has always been one of those things you set up once, forget about, and only remember when it saves you from corrupting shared state. The mechanics used to lean on DynamoDB because S3 could not do the job alone. Now that S3 supports conditional writes, that reason is gone, and Terraform 1.10 gives you a one line switch to take advantage of it.

If you are starting a new project, reach for use_lockfile from day one. If you already have a DynamoDB table doing this work, plan a calm migration, keep both settings on for a bit, then retire the table. Your future self, staring at a much simpler backend block, will thank you.

Let us connect

If you have questions, ran into a snag while trying this, or just want to talk Terraform and AWS, reach out at khantanseer43@gmail.com. I am always happy to help someone get unstuck.

Build and Deploy a 3 Tier Serverless App on AWS with Terraform Modules

Tanseer — Tue, 14 Jul 2026 04:59:11 +0000

Introduction

Most tutorials show you how to create a Lambda function by clicking around the AWS console. That works once. Then you need a staging environment, and you click everything again. Then production. Then someone asks "wait, did staging have the 30 second timeout or was that only dev?" and nobody knows, because the infrastructure lives in memory and screenshots.

This post walks through a project where the entire 3 tier stack comes from a single terraform apply. Tier one is the frontend on Amplify, tier two is the API on Lambda and API Gateway, tier three is a MySQL database on RDS. Run the apply in the dev folder, you get a dev environment. Run it in prod, you get prod. Same code, zero clicking.

The app itself is a small todo application. That is on purpose. The app is just a passenger. The real work is the Terraform: reusable modules, three isolated environments, remote state stored in S3, and module outputs wired into module inputs so everything builds in the correct order.

The full source is here: github.com/TanseerS/serverless-web-application

Quick note on the word serverless before we start. It does not mean there are no servers. It means AWS manages the servers and you only bring code. You pay when the code runs, not while it sits idle.

What we are building

The flow is simple to trace. A user opens a React app hosted on AWS Amplify (a service that builds and hosts frontends straight from a GitHub repo). The app calls a REST API on API Gateway (the AWS service that gives your code an HTTPS URL). Every route on that API points to a single Lambda function (code that runs on demand without a server you manage). Inside the Lambda, an Express app handles the routing and talks to a MySQL database on RDS (managed relational databases on AWS).

Deployments are split in two. Push to GitHub and Amplify rebuilds the frontend on its own, since CI/CD comes free with Amplify. Change the backend code or any infrastructure, and terraform apply ships it. Terraform even re-zips the backend folder automatically whenever the source changes.

The repository layout

serverless-web-application/
├── frontend/            # React 18 + Vite app
├── backend/             # Express + Sequelize API, packaged for Lambda
├── assets/              # diagrams and screenshots
└── infrastructure/
    ├── bootstrap/       # run once, creates the S3 state bucket
    ├── modules/
    │   ├── database/    # RDS MySQL, security group, subnet group
    │   ├── backend/     # Lambda, API Gateway, IAM, code packaging
    │   └── frontend/    # Amplify app, branch, IAM
    └── environments/
        ├── dev/
        ├── staging/
        └── prod/

Two ideas make this layout work.

First, modules are reusable building blocks. The database module does not know or care whether it is building dev or prod. It takes a project and an environment variable and names everything ${var.project}-${var.environment}-..., so the same code stamps out todo-app-dev-rds, todo-app-staging-rds and todo-app-prod-rds.

Second, each environment folder is tiny. It looks up the default VPC with data sources, calls the three modules, and passes in values from its own terraform.tfvars file. Promoting a change from dev to prod means changing directory and applying again. You never edit module code to deploy an environment.

Step 1: solve the chicken and egg problem

Terraform keeps a state file, which is its record of everything it has created. By default that file sits on your laptop, which is fine until your laptop is not around or a teammate runs an apply and now there are two versions of the truth. The fix is remote state: store the file in an S3 bucket instead.

But here is the catch. The S3 bucket has to exist before Terraform can store state in it, and we want Terraform to create the bucket. Who creates the creator?

The answer is a small run once setup called the bootstrap pattern. The infrastructure/bootstrap/ folder is a tiny Terraform project with plain local state. Its only job is to create the state bucket. You run it once per AWS account and forget about it.

data "aws_caller_identity" "current" {}

resource "aws_s3_bucket" "this" {
  bucket = "${var.project}-statefile-${data.aws_caller_identity.current.account_id}"
}

Notice the bucket name. S3 bucket names are globally unique across every AWS account on the planet, so todo-app-statefile alone would probably be taken. The aws_caller_identity data source reads your AWS account ID at plan time and appends it, so the name is unique without hardcoding anything.

The bootstrap also turns on three protections for the bucket: versioning (every old copy of your state is kept, which is what saves you when an apply goes wrong), full public access blocking, and AES 256 encryption at rest.

Step 2: remote state with S3 native locking

With the bucket in place, every environment points at it through a backend.tf file. Same bucket, different key per environment:

terraform {
  backend "s3" {
    bucket       = "todo-app-statefile-<account-id>"
    key          = "dev/terraform.tfstate"
    region       = "ap-south-1"
    encrypt      = true
    use_lockfile = true
  }
}

Because dev, staging and prod each write to their own key, an apply in dev can never touch prod. They are fully isolated.

The use_lockfile = true line deserves attention. State locking stops two people from running apply at the same time and corrupting the state. For years the standard answer was a DynamoDB table just for the lock. Terraform 1.10 added native S3 locking: Terraform drops a lock object into the bucket itself while an apply runs. One less resource to create, pay for and explain.

Here is the bucket after all three environments have been initialized, one prefix per state file:

Step 3: the database module

The database module creates three things. A DB subnet group that tells RDS which subnets it can live in. A security group (a virtual firewall) allowing inbound MySQL on port 3306. And the RDS instance itself:

resource "aws_db_instance" "this" {
  identifier        = "${var.project}-${var.environment}-rds"
  db_name           = "TodoAppDb"
  engine            = "mysql"
  engine_version    = "8.0"
  allocated_storage = 20
  storage_type      = "gp3"
  instance_class    = "db.t3.micro"

  username = var.username
  password = var.password

  db_subnet_group_name   = aws_db_subnet_group.this.name
  vpc_security_group_ids = [aws_security_group.this.id]

  publicly_accessible = true
  skip_final_snapshot = true
}

MySQL 8.0 on a db.t3.micro with 20 GB of gp3 storage, which keeps it inside the free tier.

One honest admission. The security group currently allows 3306 from 0.0.0.0/0, meaning anywhere on the internet, and the instance is publicly accessible. That is a demo shortcut so the Lambda can reach the database without living inside the VPC. Production would put RDS in private subnets, place the Lambda inside the VPC, and scope the security group to the Lambda alone. The repo says this out loud in its trade offs section rather than pretending it is fine.

The module ends by outputting db_endpoint, db_name and db_port. Remember those. They are about to become someone else's inputs.

Step 4: the backend module, where most of the Terraform lives

This module packages the code, creates the Lambda, and builds the entire REST API. A few pieces are worth slowing down for.

Packaging code without a build script

data "archive_file" "this" {
  type        = "zip"
  source_dir  = "${path.module}/../../../backend"
  output_path = "${path.module}/../../build/function.zip"
}

Terraform zips the backend/ folder at plan time. The Lambda resource then sets source_code_hash = data.archive_file.this.output_base64sha256, which means the function only gets updated when the code actually changed. Edit a Terraform file and apply, the Lambda is untouched. Edit routes/tasks.js and apply, Terraform notices the hash changed and ships the new zip. No separate deploy pipeline for the backend at all.

The Lambda function

resource "aws_lambda_function" "this" {
  function_name    = "${var.project}-${var.environment}-lambda-function"
  handler          = "server.handler"
  runtime          = "nodejs22.x"
  timeout          = 30
  memory_size      = 512

  environment {
    variables = {
      DB_HOST     = var.db_endpoint
      DB_NAME     = var.db_name
      DB_PORT     = var.db_port
      DB_USER     = var.username
      DB_PASSWORD = var.password
    }
  }
}

Node.js 22, 512 MB of memory, 30 second timeout. The database connection details arrive as environment variables, and look where they come from: var.db_endpoint is the output of the database module. The environment folder wires them together like this:

module "backend" {
  source      = "../../modules/backend"
  db_endpoint = module.database.db_endpoint
  db_name     = module.database.db_name
  db_port     = module.database.db_port
  ...
}

This is the trick that makes the whole project build in the right order. Terraform sees that the backend needs an output from the database, so it builds the database first. Nobody wrote depends_on for that. The dependency graph falls out of the references themselves.

Eight API routes from one map

The API has four paths (/api, /api/health, /api/tasks, /api/tasks/{id}) and eight method and path combinations once you count the OPTIONS methods needed for CORS preflight. Writing eight method blocks and eight integration blocks by hand would be sixteen nearly identical resources. Instead, the routes live in one locals map:

locals {
  routes = {
    health_get    = { resource_id = aws_api_gateway_resource.health.id,     http_method = "GET" }
    tasks_get     = { resource_id = aws_api_gateway_resource.tasks.id,      http_method = "GET" }
    tasks_post    = { resource_id = aws_api_gateway_resource.tasks.id,      http_method = "POST" }
    task_put      = { resource_id = aws_api_gateway_resource.task_by_id.id, http_method = "PUT" }
    task_delete   = { resource_id = aws_api_gateway_resource.task_by_id.id, http_method = "DELETE" }
    # ...plus three OPTIONS entries
  }
}

resource "aws_api_gateway_method" "this" {
  for_each      = local.routes
  rest_api_id   = aws_api_gateway_rest_api.this.id
  resource_id   = each.value.resource_id
  http_method   = each.value.http_method
  authorization = "NONE"
}

The for_each loops over the map and stamps out one resource per entry. The integrations do the same, every one an AWS_PROXY integration pointing at the Lambda, which means API Gateway forwards the raw request and lets Express figure out the routing inside the function. Adding a new route later means adding one line to the map.

The silent API Gateway gotcha

Here is a bug that costs beginners hours. API Gateway keeps a deployed snapshot of your API separate from its configuration. You can change methods and integrations in Terraform, apply successfully, and API Gateway will keep serving the old snapshot without a single complaint. Everything looks green. Your change is just not live.

The fix is to force a redeploy whenever the routing changes:

resource "aws_api_gateway_deployment" "this" {
  rest_api_id = aws_api_gateway_rest_api.this.id

  triggers = {
    redeploy = sha1(jsonencode([
      aws_api_gateway_method.this,
      aws_api_gateway_integration.this,
    ]))
  }

  lifecycle {
    create_before_destroy = true
  }
}

The trigger is a hash of the routing config. Config changes, hash changes, Terraform creates a fresh deployment. The create_before_destroy lifecycle rule makes sure the new deployment exists before the old one goes away, so the stage never has a gap where the API is down.

Step 5: the frontend module

The Amplify app connects directly to the GitHub repo using a personal access token, with the build instructions written inline:

resource "aws_amplify_app" "this" {
  name         = "${var.project}-${var.environment}-amplify-app"
  repository   = var.repository
  access_token = var.github_access_token

  environment_variables = {
    VITE_API_URL = var.api_uri
  }
}

That VITE_API_URL line is my favorite detail in the project. The value comes from module.backend.api_invoke_url, the output of the backend module. So Terraform creates the API, learns its URL, and hands it to Amplify as a build time environment variable. The React app bakes it in during npm run build. The frontend learns its backend address from Terraform and nothing is ever hardcoded.

The module also adds a rewrite rule so deep links like /tasks/5 fall back to index.html instead of returning 404, which every single page app on Amplify needs, and a branch resource with auto build turned on. The branch stage is set with a conditional: var.environment == "prod" ? "PRODUCTION" : "DEVELOPMENT".

Step 6: the backend code that makes Express work in Lambda

A normal Express app ends with app.listen(3000). There is no port to listen on inside Lambda, so the app is wrapped instead:

const serverlessExpress = require('@vendia/serverless-express');
const app = express();
// routes...
exports.handler = serverlessExpress({ app });

The @vendia/serverless-express package translates the event API Gateway sends into the request object Express expects, and translates the response back. Your Express code does not change at all.

One more pattern worth stealing. The database table has to exist before the first query, but you do not want to create it on every request. The code caches the promise:

let dbReady = null;
app.use('/api/tasks', async (req, res, next) => {
  try {
    dbReady = dbReady || sequelize.sync();
    await dbReady;
    next();
  } catch (err) {
    dbReady = null; // don't cache a failed attempt
    res.status(500).json({ error: err.message });
  }
}, tasksRouter);

The first request in each Lambda container runs sequelize.sync() which creates the tasks table if needed. Every later request awaits the already resolved promise, which costs nothing. And if the sync fails, the cache resets so the next request tries again instead of being stuck with a failed promise forever. There is no manual schema step anywhere in the deployment.

Deploying it yourself

You need an AWS account with credentials configured, Terraform 1.15 or newer, and a GitHub personal access token that can read your fork and create webhooks.

First, bootstrap the state bucket, once per account:

cd infrastructure/bootstrap
terraform init
terraform apply

Second, edit backend.tf in each environment folder and set the bucket name to what the bootstrap printed. Backend blocks cannot take variables, so this one value is a literal.

Third, deploy an environment:

cd infrastructure/environments/dev
cp terraform.tfvars.example terraform.tfvars   # fill in DB password, repo URL, token, branch
terraform init
terraform plan
terraform apply

RDS takes a few minutes to come up. The result is 28 resources from one command:

Fourth, trigger the first frontend build. Amplify only builds on new commits, so push something to the configured branch after the apply. An empty commit does the job.

Finally, check it is alive:

curl https://<api-id>.execute-api.ap-south-1.amazonaws.com/dev/api/health
# {"status":"ok","database":"connected"}

What actually gets created

Everything below came from the same modules, applied once per environment folder. Three Lambda functions on Node.js 22:

Three regional REST APIs, each with a stage named after its environment:

A MySQL instance per environment:

And the Amplify apps, connected to GitHub and building on every push:

Notice what did not happen. Nobody opened the RDS console to click Create database three times. The staging environment is not "dev, but with some settings someone changed by hand last month." All three are provably identical because they came from identical code.

Honest trade offs

A few shortcuts were taken deliberately, and it is worth knowing what production would change. The open security group and public RDS would become private subnets with the Lambda inside the VPC. Database credentials would move from Lambda environment variables into Secrets Manager. The single AZ db.t3.micro with skip_final_snapshot would become Multi AZ with deletion protection. And the hardcoded bucket name in backend.tf would be handled with partial backend config through terraform init -backend-config.

The roadmap includes a GitHub Actions pipeline that runs terraform fmt, validate and plan on pull requests and applies on merge, plus tflint and checkov for static checks.

Conclusion

The todo app is forgettable. The patterns are not. The bootstrap trick for remote state, S3 native locking without a DynamoDB table, modules whose outputs feed the next module's inputs so the build order takes care of itself, for_each over a map instead of sixteen copy pasted resources, and a hash trigger that stops API Gateway from silently serving stale config. Every one of these shows up in real production Terraform, and this repo is a small enough place to learn them without drowning.

Clone it, deploy dev, then run terraform destroy when you are done exploring. RDS is the only always on cost here, and tearing it down is one command, the same way bringing it up was.

Repo: github.com/TanseerS/serverless-web-application

Contact

Questions, feedback, or stuck on a step? Reach me at khantanseer43@gmail.com. Happy to help.

Go Live in Minutes: Building Your First Live Stream with Amazon IVS

Tanseer — Tue, 07 Jul 2026 05:08:24 +0000

The same tech that powers Twitch, minus the servers, minus the headache

A few weeks ago I wanted to test a live streaming idea. My first thought was the usual one. I will need a media server, something like Wowza or a self hosted RTMP setup on EC2, then a CDN in front of it, then hours of tuning latency. Before I even opened the console I was already tired.

Then I found Amazon IVS, and honestly, the whole thing took less time than making tea.

If you are a developer who has never touched live streaming on AWS before, this post is for you. By the end of it you will know exactly what IVS is, why it is different from rolling your own setup, and how to go live in four steps.

What exactly is Amazon IVS

Amazon Interactive Video Service, or IVS for short, is a fully managed live streaming service from AWS. Fully managed means AWS runs everything behind the scenes. You do not create servers, you do not install streaming software on the cloud, and you do not worry about scaling.

Here is the part that made me sit up. IVS runs on the exact same infrastructure that powers Twitch. Twitch is one of the biggest live streaming platforms in the world, handling millions of concurrent viewers every single day. When you create an IVS channel, your tiny test stream rides on that same battle tested backbone.

So instead of you building a smaller, buggier version of Twitch infrastructure, AWS just hands you Twitch infrastructure as an API.

The killer feature: ultra low latency

If you have ever streamed with a typical setup, you know the pain. You wave at the camera, and viewers see the wave 15 to 30 seconds later. That delay is called latency, the time it takes for video to travel from your camera to the viewer's screen.

IVS brings that down to under 3 seconds.

Why does that matter? Because under 3 seconds, a stream stops feeling like a broadcast and starts feeling like a conversation. Viewers can react in chat and you can respond almost instantly.

IVS also lets you sync interactive elements with the video. Think live chat, polls, quizzes, or even shopping buttons that appear exactly when the host says "buy this now." This is done through something called timed metadata, small pieces of data you inject into the stream that arrive on the viewer's screen in sync with the video frame. For a marketplace or live commerce product, this is gold.

How is it serverless

The word serverless gets thrown around a lot, so let me be specific about what it means here.

There are no servers for you to manage. No EC2 instances, no patching, no capacity planning. You create a channel and AWS handles the rest.

It scales automatically. Whether 5 people show up to your stream or 5 million, you do nothing. The same channel handles both. There is no "upgrade your instance" moment at 2 AM when your stream suddenly goes viral.

You pay only for what you use. IVS pricing is based on two things: hours of video you send into the service (input) and hours of video delivered to viewers (output). No stream running means no bill running. This is the same pay as you go model that makes Lambda and other serverless tools so friendly for side projects.

Going live in 4 steps

This is the part that genuinely surprised me. Here is the entire setup.

Step 1: Create a channel

Log in to the AWS Console and search for Amazon IVS. Click Create channel, give it a name, and keep the default configuration for now. That is it. AWS provisions the channel in seconds.

Step 2: Copy your Ingest Server and Stream Key

Open your new channel and you will see two important values. The Ingest Server is the address your video gets sent to, something like rtmps://xxxxxx.global-contribute.live-video.net. The Stream Key is a secret token that proves the stream belongs to you. Treat the stream key like a password. Anyone who has it can broadcast on your channel.

Step 3: Paste them into OBS Studio

OBS Studio is free, open source broadcasting software and the de facto standard for streaming. Download it, open Settings, go to Stream, and choose Custom as the service. Paste the Ingest Server into the Server field and the Stream Key into the Stream Key field. Save.

Step 4: Click Start Streaming

Back on the main OBS screen, add a source (your webcam or screen), then hit Start Streaming. Switch to the AWS Console, open your channel, and you will see yourself live in the built in preview player within seconds.

Four steps. No infrastructure. You are now running on Twitch grade streaming systems.

Where to go from here

The console preview is just the beginning. IVS gives you a playback URL for every channel, and AWS provides a player SDK for web, iOS, and Android. That means you can drop your live stream into your own React app or mobile app with a few lines of code. Add the IVS chat feature and timed metadata, and you have a real interactive streaming product without touching a single server.

Conclusion

Live streaming used to be a project measured in weeks. With Amazon IVS it is a project measured in minutes. You get the infrastructure behind Twitch, latency under 3 seconds, automatic scaling from a handful of viewers to millions, and a bill that only grows when your stream does. If you have been putting off a live video feature because the plumbing felt scary, this is your sign to create a channel and press Start Streaming.

Let's connect

If you tried this and got stuck somewhere, or if you built something cool with IVS, I would love to hear about it. Reach out at khantanseer43@gmail.com.

Terraform guide to IAM Policies: The Two Cases You Need to Know

Tanseer — Wed, 01 Jul 2026 11:21:21 +0000

When to attach a ready made policy and when to write your own

When you first start writing IAM setup in Terraform, the examples online can feel like they contradict each other. One tutorial writes a long block called aws_iam_policy_document. Another just pastes a single line with a long string inside it. Both of them work, so which one is correct?

The answer is that both are correct. They simply solve two different problems. Once you can tell which situation you are in, IAM in Terraform stops feeling random and starts feeling obvious.

Before we get into the two cases, here are the words you will keep seeing:

IAM stands for Identity and Access Management. It is the AWS service that decides who is allowed to do what. An IAM role is an identity that an AWS service (like a Lambda function) takes on so it can act with certain permissions. An IAM policy is the document that lists which actions are allowed or denied. An ARN, short for Amazon Resource Name, is the unique address of a resource in AWS.

With that out of the way, let us look at the two cases.

Case 1: AWS Managed Policy (attach directly)

A managed policy is a policy that AWS already wrote and maintains for you. If AWS already provides the exact permissions you need, you do not create your own aws_iam_policy_document. You just point your role at the policy AWS already made.

For example, giving a Lambda function permission to write logs is a very common need, and AWS has a ready made policy for it:

resource "aws_iam_role_policy_attachment" "logs" {
  role       = aws_iam_role.this.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

Another common one is letting a Lambda function run inside a VPC (a Virtual Private Cloud, which is your own private network in AWS):

resource "aws_iam_role_policy_attachment" "vpc" {
  role       = aws_iam_role.this.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole"
}

Notice what is happening here. You are not writing any permissions yourself. You are taking an existing AWS managed policy, referring to it by its ARN, and attaching it to your role. That is the whole job.

Case 2: Your own custom permissions

Sometimes AWS does not have a managed policy that matches exactly what you need. Maybe you want a Lambda function to read and write files in one specific S3 bucket and nothing else. In that case you build the policy yourself, and it takes three small steps.

Step 1: Create the policy document

First you describe the permissions. This aws_iam_policy_document is a data source, which means it does not create anything in AWS on its own. It is a way to write your permission rules in clean Terraform instead of raw JSON.

data "aws_iam_policy_document" "s3_access" {
  statement {
    effect = "Allow"

    actions = [
      "s3:GetObject",
      "s3:PutObject"
    ]

    resources = [
      aws_s3_bucket.uploads.arn,
      "${aws_s3_bucket.uploads.arn}/*"
    ]
  }
}

Reading this in plain English: allow the actions of getting an object and putting an object, and only on the uploads bucket and everything inside it.

Step 2: Create the IAM policy

Now you turn that document into a real IAM policy that exists in AWS. The policy field pulls in the JSON produced by the document from Step 1.

resource "aws_iam_policy" "s3_access" {
  name   = "${var.project}-${var.environment}-s3-access"
  policy = data.aws_iam_policy_document.s3_access.json
}

Step 3: Attach it

Finally you attach the policy you just created to your role. This looks almost the same as Case 1, but the ARN now comes from your own policy instead of an AWS managed one.

resource "aws_iam_role_policy_attachment" "s3_access" {
  role       = aws_iam_role.this.name
  policy_arn = aws_iam_policy.s3_access.arn
}

The mental model

The easiest way to remember the difference is this. In Case 1 the policy already exists, so you skip straight to attaching. In Case 2 you have to build the policy first, then attach it.

The custom flow always moves in the same direction:

Policy Document
      │
      ▼
IAM Policy
      │
      ▼
Attach to Role

You write the document, the document becomes a policy, and the policy gets attached to the role. If you ever feel lost, just ask yourself one question: does AWS already have a policy for this? If yes, attach it directly. If no, build your own with the three steps above.

Conclusion

IAM in Terraform is not as confusing as it first looks. There are really only two situations. When AWS already provides the exact permissions, you attach a managed policy by its ARN in a single resource. When you need something specific that AWS does not offer, you create a policy document, turn it into an IAM policy, and attach that.

Keep that fork in mind and you will always know which approach the moment fits, instead of copying an example and hoping it is the right one.

Let us connect

If you have questions or want to share how you handle IAM in your own projects, reach out at khantanseer43@gmail.com. I am always happy to talk Terraform and AWS.

Lambda MicroVMs Are Kind of Insane: Tiny Virtual Machines That Boot in Milliseconds and Sleep When You Are Not Looking

Tanseer — Tue, 30 Jun 2026 04:53:28 +0000

A beginner friendly guide to the Firecracker powered microVMs behind AWS Lambda, the new suspend and resume superpower, and the big question everyone keeps asking: is this still serverless?

Introduction

Imagine a virtual computer so small and so fast that it starts up in less time than it takes you to blink. Now imagine thousands of them running on a single physical server, each one completely walled off from its neighbours, and each one quietly going to sleep the moment it has nothing to do, so that you stop paying for it.

That is not a science fiction pitch. That is roughly what is happening inside AWS Lambda today, and the newer microVM capabilities push the idea even further.

In this post we will start from zero. We will explain what a microVM even is, meet the tiny engine that makes the whole thing possible, walk through what happens when a single request comes in, and then look at the features that make people say this is a little bit crazy. By the end you will be able to answer the question that is dividing the cloud community: after all of this, is Lambda even serverless anymore?

Let us get into it.

So What Exactly Is a Lambda MicroVM?

A microVM is a tiny, lightweight virtual machine. It behaves like its own small computer with a strong security wall around it, but it throws away almost all the heavy baggage of a normal machine, so it starts in milliseconds and uses very little memory. Picture a tiny, fast, single purpose box that appears, does one job, and disappears.

Here is the part that surprises most people. Lambda has always run your code inside these microVMs. Every time your function runs, AWS quietly spins up a tiny isolated machine just for you. The recent excitement is about giving those microVMs new powers, such as the ability to pause when idle and to be reached directly over the internet. So when people say "Lambda MicroVMs," they are really talking about this next chapter of an idea that has been humming away under the hood for years.

Meet Firecracker, the Tiny Engine Doing All the Work

Powering all of this is Firecracker, a technology that AWS built and then released to the public as open source. AWS created it specifically to run microVMs safely at massive scale, and it is the reason a tiny machine can boot in around 125 milliseconds while still keeping the strong security wall of a real virtual machine. This same AWS made engine quietly runs underneath both Lambda and AWS Fargate.

How It Works: The Journey of a Single Request

Let us follow one request from start to finish so the architecture clicks.

You upload your code. This can be a simple zip file or a container image. You also choose how much memory the function gets, which in turn decides how much CPU power it receives.
Something triggers it. A user hits an endpoint, a file lands in storage, a message arrives on a queue, and Lambda decides your function needs to run.
A microVM appears. AWS spins up a Firecracker microVM with your chosen runtime and your code inside. This little box is called the execution environment. If a fresh one has to be created from scratch, that small delay is known as a cold start.
Your code runs. The environment initialises, your handler runs, and a response goes back.
The environment is kept around. Instead of throwing the microVM away immediately, Lambda often keeps it ready for a little while. If another request arrives soon, it reuses the same warm environment and skips the startup cost entirely. This is called a warm start, and it is why the second request to a function usually feels much faster than the first. So the architecture is really a fleet of these tiny machines being created, reused, and recycled constantly, all managed for you so you never touch a server.

Each One Gets Its Own Front Door: Dedicated HTTP Endpoints

An endpoint is just a web address that something can send a request to. HTTP is the everyday language of the web that browsers and apps use to talk to servers.

For a long time, if you wanted your Lambda function to be reachable over the web, you usually had to put another service in front of it to handle the incoming traffic. The newer approach gives a function its own dedicated HTTPS endpoint, a clean and direct web address that points straight at it. No extra plumbing in between.

The address looks something like this:

https://your-unique-id.lambda-url.your-region.on.aws

You can keep it open or lock it down so only authorised callers get through. For simple use cases, such as a webhook that some other service calls, or a small backend for a single page app, this is wonderfully direct. Your request lands on a private little microVM that exists just to answer you.

The Real Magic: Suspend and Resume

This is the feature that earns the word "crazy."

To understand it, you need one more idea: a snapshot. A snapshot is a complete saved picture of a running machine at a single moment in time, including everything sitting in its memory. If you take a snapshot and then restore it later, the machine wakes up exactly where it left off, as if no time had passed at all.

Firecracker can do this with microVMs. It can freeze a running microVM, save its full state, and bring it back to life almost instantly.

Now think about what that unlocks. Traditionally you had two bad choices. Either you kept a server running all the time so it was ready to respond instantly, and you paid for every idle second. Or you shut it down to save money and then suffered a slow start whenever traffic came back.

Suspend and resume gives you a third option. The microVM can stay paused while nothing is happening, and the headline benefit is simple: you are not paying for a fully running machine while it sits idle. When a request finally arrives, it wakes up quickly and carries on. You get the responsiveness of an always on server with a cost profile much closer to pay only for what you use.

That is the holy grail people have chased for years. Be instantly ready, but only pay when you are actually doing work.

Lambda MicroVM vs EC2 vs Classic Lambda

It helps to see where this sits between the two options most people already know.

EC2 is a rented virtual server that you control and manage yourself. Classic Lambda is short lived, event driven functions with a hard time limit. The microVM approach lands in the middle, borrowing the best of each.

Feature	EC2	Classic Lambda	Lambda MicroVM
What you manage	The whole server	Just your code	Just your code
How long it can run	As long as you like	Up to 15 minutes per run	Designed for longer and more stateful work than the 15 minute limit
Startup speed	Slow, minutes	Fast, with occasional cold starts	Fast, with quick resume from a paused state
Cost when idle	You keep paying while it runs	Nothing, it is not running	The big win is not paying for a fully running machine while idle
Scaling	You set it up	Automatic, scales to zero	Automatic, scales to zero
Best feeling	Total control	Pure simplicity	Control and simplicity together

The short version is this. EC2 gives you full control but you babysit it and pay for idle time. Classic Lambda is beautifully simple but tightly constrained. The microVM model tries to give you longer running, more flexible compute that still scales down to nothing when no one is using it.

What Is This Actually Good For?

A new tool is only exciting if it solves real problems. Here is where this approach shines.

It is great for long running jobs that do not fit neatly into a short window, such as heavier data processing or video work that would bump into the classic 15 minute ceiling.

It suits stateful sessions, meaning work where the machine needs to remember something between requests rather than starting fresh every single time.

It is a natural fit for AI and machine learning inference, where a large model can be expensive to load. You load it once, let the machine pause when traffic is quiet, and resume instantly when the next request comes, instead of paying to keep a big GPU style box running all day.

It works beautifully for interactive sandboxes and code execution, the kind of thing where each user needs their own safe, isolated little environment that spins up fast and tears down cleanly.

And it is useful for bursty traffic, where requests come in unpredictable spikes. You get instant readiness during the spike and near zero cost during the long quiet stretches in between.

When would you not bother? If your workload is genuinely tiny and event driven, classic Lambda is still the simplest possible answer. And if you need deep control over the operating system, custom networking, or hardware that must run nonstop, a traditional server like EC2 is still the right call.

What About the Price?

Cost is usually the deciding factor, so let us compare the three billing styles in plain terms. To follow along, you need one bit of vocabulary: a GB second is just one gigabyte of memory used for one second. It is the unit Lambda uses to measure how much work you actually consumed.

EC2 charges you for the time the server is switched on, whether or not anyone is using it. A box running quietly overnight with zero visitors still appears on your bill. You pay for capacity, not for usage.

Classic Lambda charges you in two parts: a tiny amount for each request, plus an amount for compute measured in those GB seconds. If nobody calls your function, it costs you nothing. You pay for usage, not for capacity.

The microVM model aims for the best of both. You get a more capable, longer lived machine, but the key promise is that you are not charged for a fully running VM while it is paused and idle. You pay mainly for the active time when it is actually doing something.

Important: cloud pricing changes often, and the exact rates for these options vary by region and configuration. Rather than trust any single number you read in a blog, including this one, always check the current AWS pricing pages before you make a decision.

So Is It Still Serverless?

Here is the fun part, and it is genuinely a matter of debate.

The word serverless never meant there were no servers. It meant you, the developer, never had to think about them. The classic definition came with a few expectations: your code is event driven and short lived, it scales automatically all the way down to zero, you never manage infrastructure, and you only pay for what you use.

Now look at what these microVMs offer. Long running. Able to hold state. Reachable through their own dedicated web address. Sitting paused and ready rather than vanishing completely. To some people that starts to look an awful lot like a small, managed server wearing a serverless costume.

The case for "yes, it is still serverless" is strong. You still never patch an operating system. It still scales to zero. You still pay for value rather than for idle capacity. By the spirit of the original promise, nothing has been broken.

The case for "no, the purity is gone" is also fair. The original idea of a tiny, stateless, event triggered function that lives for a moment and dies has clearly been stretched into something bigger and more persistent.

The honest answer is probably this. Serverless is becoming a spectrum rather than a strict box. On one end you have pure functions that flash in and out of existence. On the other you have long lived managed machines. These microVMs live somewhere in the comfortable middle, and they are quietly redefining what the word can mean. Maybe the better question is not "is this serverless," but "does this let me ship without babysitting infrastructure." If the answer is yes, the label matters a lot less than the freedom.

Wrapping Up

Lambda has always been powered by tiny, fast, isolated microVMs running on Firecracker. What is changing is how clever those microVMs are getting. They can now boot in a blink, answer the web directly, and most remarkably of all, doze off when idle so you are not paying for a machine that is doing nothing.

Whether you call it serverless, managed compute, or something brand new, one thing is clear. The line between a server and a function is getting blurrier, and that is a great problem for developers to have. The more the cloud handles for us, the more time we get to spend building the things we actually care about.

If you are new to AWS, the best next step is to try spinning up a simple function, give it a dedicated endpoint, and call it from your browser. Seeing your own tiny microVM answer back is the moment the whole idea finally clicks.

Let Us Connect

If you found this helpful, have a question, or spot something you want to discuss, I would love to hear from you. Reach out at khantanseer43@gmail.com.