DEV Community: Narendra | SRE

Automating AWS Infrastructure Deployment with Terraform

Narendra | SRE — Tue, 10 Nov 2026 16:21:51 +0000

Introduction

Manual cloud provisioning can be difficult to reproduce, review, and audit consistently. Infrastructure as Code (IaC) addresses this by describing the desired infrastructure in version-controlled configuration.

This project uses Terraform to deploy a reusable, security-conscious Amazon S3 bucket configuration on AWS. It demonstrates provider configuration, variables and outputs, reusable modules, Terraform state, controlled plan/apply/destroy workflows, AWS CLI verification, and automated Terraform checks with GitHub Actions.

The implementation focuses on practical infrastructure automation while keeping security controls and operational safety visible throughout the workflow.

Cost notice: This project provisions real AWS resources and is not guaranteed to be free. S3 storage, requests, and other account activity may incur charges. Review current AWS pricing and your account billing before deployment, and remove demonstration resources when they are no longer needed.

Implementation status: Terraform configuration deployed successfully; the six planned resources were created. Deployed S3 security settings verified using AWS CLI. Evidence command outputs and deployment screenshots captured. Infrastructure cleanup is Completed.

1. What we are building

Terraform creates an S3 bucket with:

A globally unique name generated using Terraform's random provider.
Versioning enabled.
Default server-side encryption using SSE-S3.
All four S3 Block Public Access settings enabled.
Bucket-owner-enforced object ownership.
Consistent project and environment tags.

The S3 configuration is encapsulated in a reusable Terraform module.

2. Architecture and deployment flow

Deployment Flow

Developer
   |
   v
Terraform root configuration
   |-- variables.tf
   |-- main.tf
   |-- outputs.tf
   |
   v
Reusable module: secure_s3_bucket
   |-- S3 bucket
   |-- Versioning
   |-- Default encryption
   |-- Block Public Access
   |-- Object ownership controls
   |
   v
AWS provider -> AWS account

Terraform state records managed resources.
GitHub Actions validates configuration; it does not deploy.

Architecture

3. Repository structure

article-06-terraform-infrastructure-deployment/
├── ARTICLE.md
├── README.md
├── .gitignore
├── .github/
│   └── workflows/
│       └── terraform-checks.yml
├── examples/
│   └── terraform.tfvars.example
├── images/
│   └── architecture.svg
├── evidence/
│   ├── README.md
│   ├── results/
│   │   ├── .gitkeep
│   │   └── [captured command outputs]
│   └── screenshots/
│       ├── [deployment and verification screenshots]
│       └── [GitHub Actions screenshot, if captured]
├── modules/
│   └── secure_s3_bucket/
│       ├── main.tf
│       ├── variables.tf
│       └── outputs.tf
├── scripts/
│   ├── terraform-check.sh
│   ├── plan.sh
│   ├── apply.sh
│   ├── verify.sh
│   ├── capture-evidence.sh
│   └── destroy.sh
└── terraform/
    ├── versions.tf
    ├── providers.tf
    ├── variables.tf
    ├── main.tf
    ├── outputs.tf
    ├── terraform.tfvars.example
    └── .terraform.lock.hcl

The .terraform.lock.hcl file is generated during initialization and should be committed to version control. Terraform working directories, state files, backup state files, local variable files, and generated plan files should remain excluded through .gitignore. Evidence files should contain genuine results and must be reviewed for sensitive information before committing.

4. Reusable Terraform module

The root configuration calls modules/secure_s3_bucket. The module separates the S3 resource implementation from environment-specific inputs. It accepts a bucket name and tags, then configures versioning, encryption, public-access blocking, and object ownership.

5. Variables and outputs

Variables allow the same configuration to be reused without editing resource definitions. Inputs include the bucket name prefix, AWS region, environment, and project name.

A random suffix is added to the bucket name to reduce the chance of a global S3 naming collision. Terraform records the generated value in state. Outputs expose the bucket name, ARN, and region for operators and verification scripts. Do not use outputs to expose secrets.

6. Terraform state management

Terraform state maps configuration resources to real infrastructure objects. Terraform uses it to calculate changes during subsequent plans.

This standalone demonstration uses the default local state file. State can contain sensitive infrastructure metadata:

Do not commit terraform.tfstate or backup state files.
Do not manually edit state files.
Protect and back up state.
For team or production use, prefer a protected remote backend with access controls, encryption, versioning, and locking.

This repository documents local state; it does not automatically create a remote backend. A remote backend requires a separately bootstrapped storage location and deliberate access-control design.

7. Deployment workflow

From the project root:

cp terraform/terraform.tfvars.example terraform/terraform.tfvars
terraform fmt -recursive
terraform -chdir=terraform init
./scripts/terraform-check.sh
./scripts/plan.sh

Review the plan before applying. It should contain only the intended S3 bucket and its security configuration. Do not approve changes you do not recognize.

./scripts/apply.sh

Terraform asks for confirmation. Read the proposed changes before approving.

8. Verify the deployed configuration

./scripts/verify.sh

The script uses Terraform outputs and AWS CLI read operations to check the bucket's public-access block, versioning, encryption, and ownership controls.

After deployment, a second Terraform plan should report no changes when the live infrastructure matches the configuration.

Report only settings that the verification commands actually confirm. A successful apply is not a substitute for checking the resulting configuration.

9. Automated checks with GitHub Actions

The included GitHub Actions workflow runs Terraform formatting, initialization, and validation when relevant files change. It does not apply infrastructure and does not require AWS credentials.

A production deployment workflow should use short-lived credentials through GitHub OIDC, a narrowly scoped IAM role, protected environments, and explicit approval gates rather than long-lived access keys.

10. Cost, safety, and cleanup

This project creates an S3 bucket and related S3 configuration resources. Charges may apply for storage, requests, and other AWS usage. Review current AWS pricing and your account billing. Consider setting a budget or billing alert.

After collecting and reviewing evidence, destroy the demonstration resources if they are no longer needed:

./scripts/destroy.sh

The bucket uses force_destroy = false. If it contains objects or versions, Terraform will not silently empty it. Inspect the bucket and remove only demonstration data you intentionally created, then run destroy again. Do not enable force deletion simply to make cleanup easier.

11. Limitations and next steps

Local state is used for a single-user demonstration; team use should adopt a protected remote backend.
GitHub Actions validates configuration but deliberately does not deploy.
This is a small S3 example, not a complete production landing zone.
It does not include organization-wide guardrails, centralized logging, or cross-account deployment.
Future improvements could add policy-as-code checks, environment promotion, remote-state bootstrapping, and an OIDC-based deployment workflow.

Conclusion

Terraform makes infrastructure changes reviewable and repeatable by describing desired state as code. Reusable modules, explicit variables and outputs, a deliberate state strategy, and a plan-before-apply workflow form a practical foundation for AWS infrastructure management.

Automation is more than running terraform apply: it includes reviewing changes, protecting state, verifying deployed settings, controlling permissions, understanding cost, and cleaning up responsibly.

Repository: [AWS Community Builder Repository]

From Wildcards to Least Privilege: Building a No-Infrastructure AWS IAM Policy Security Review Workflow

Narendra | SRE — Sun, 08 Nov 2026 12:14:04 +0000

Introduction

IAM policies can grant broader access than an application needs. A statement such as "Action": "*" combined with "Resource": "*" is a clear warning sign, while broad service permissions and sensitive IAM actions can also introduce unnecessary privilege.

This project compares an intentionally over-permissive IAM identity policy with a scoped S3 example. It uses three complementary review stages:

A local Python JSON checker that detects a deliberately limited set of risky patterns.
AWS IAM Access Analyzer policy validation.
AWS IAM Policy Simulator evaluations for selected action/resource pairs.

The workflow is review-only. It does not create IAM identities, attach policies, create an S3 bucket, or provision compute or storage infrastructure. The AWS API calls validate and simulate policy documents; they do not execute S3 operations.

Implementation Status: Implementation completed successfully. The Python IAM policy auditor, AWS IAM Access Analyzer validation, and IAM Policy Simulator tests have been executed, and the evidence outputs have been collected for review. The project is now moving into documentation finalization, screenshot collection, and GitHub publication.

Cost wording: This project provisions no AWS infrastructure. Do not interpret that as a guarantee that every account activity is free. Check current AWS pricing and your account's billing information independently.

1. The problem

A policy can be valid JSON and still grant excessive permissions. For example:

{
  "Effect": "Allow",
  "Action": "*",
  "Resource": "*"
}

This statement allows all actions on all resources within the scope of the identity policy, subject to the broader AWS authorization evaluation. Another pattern worth reviewing is a sensitive IAM action such as iam:PassRole allowed against every resource.

The challenge is to review policies before deployment without attaching an unsafe policy to a real identity or creating test infrastructure.

2. Architecture

The workflow has three stages:

Developer / DevOps engineer
           |
           v
    IAM policy JSON files
           |
     +-----+----------------+
     |                      |
     v                      v
Over-permissive         Scoped S3
example policy          example policy
     |                      |
     +----------+-----------+
                |
                v
       Stage 1: Local check
        Python JSON auditor
                |
                v
     Stage 2: Policy validation
       IAM Access Analyzer
          ValidatePolicy
                |
                v
      Stage 3: Authorization test
       IAM Policy Simulator
        SimulateCustomPolicy
                |
                v
       Review actual results
                |
                v
        Evidence and reporting

No IAM resources, buckets, or compute are created.

The stages answer different questions:

Local checker: Does the document contain any of the specific risky patterns selected for this project?
Access Analyzer: What findings does AWS return when it validates the identity policy?
Policy Simulator: What decision does the simulator return for the selected action/resource pairs and supplied policy?

These tools complement one another; none independently proves that a policy is safe for every workload.

3. Project files

policies/overprivileged-policy.json — deliberately unsafe teaching example. Do not attach it to a real identity.
policies/least-privilege-policy.json — scoped S3 permissions for a fictional bucket and telemetry prefix.
scripts/iam_policy_auditor.py — limited local JSON policy checker.
scripts/audit-policies.sh — runs the local checker.
scripts/validate-policies.sh — calls Access Analyzer ValidatePolicy for both documents.
scripts/simulate-policies.sh — evaluates three separate action/resource cases.
scripts/capture-evidence.sh — runs the checks and stores fresh command outputs.
scripts/run-local-checks.sh — runs the local checks, if included in the project version.
scripts/cleanup.sh — provides local evidence cleanup guidance.

4. Stage 1: Local static checks

The Python program parses the JSON document and inspects selected statement fields, including Effect, Action, and Resource. Its rules include patterns such as:

Allow with both Action: "*" and Resource: "*".
A service wildcard such as s3:* paired with Resource: "*".
Selected sensitive IAM actions allowed on Resource: "*".

Run the checker from the project root:

./scripts/audit-policies.sh

The deliberately over-permissive sample is expected to produce findings. The scoped sample is expected to pass this checker's limited rule set. A clean result means only that none of the implemented rules matched; it is not a full IAM security assessment. The checker does not infer application requirements or evaluate every IAM policy feature.

5. Stage 2: AWS policy validation

Run:

./scripts/validate-policies.sh

The script calls the Access Analyzer ValidatePolicy API for both policy documents. A representative command is:

aws accessanalyzer validate-policy \
  --policy-document file://policies/least-privilege-policy.json \
  --policy-type IDENTITY_POLICY \
  --output json

Review the actual findings returned by AWS. Finding types and messages are generated by AWS. Do not assume that every insecure-looking policy will return one particular finding, or that an empty findings list proves the policy is safe.

6. Stage 3: IAM Policy Simulator

The scoped policy is intended to allow s3:PutObject only for objects under the fictional bucket's telemetry/ prefix. It includes an explicit restriction for bucket deletion and does not grant object deletion.

Run:

./scripts/simulate-policies.sh

The script makes three separate simulator calls so each action is evaluated against its corresponding resource ARN.

Test	Action	Resource	Expected decision from the supplied policy
Intended operation	`s3:PutObject`	`arn:aws:s3:::article05-demo-bucket/telemetry/sample.json`	`allowed`
Explicit restriction	`s3:DeleteBucket`	`arn:aws:s3:::article05-demo-bucket`	`explicitDeny`
Not granted	`s3:DeleteObject`	`arn:aws:s3:::article05-demo-bucket/telemetry/sample.json`	`implicitDeny`

These are expected outcomes based on the sample policy and simulator inputs. Use the actual simulator output as the source of truth for the run and update the article if it differs.

The simulator does not send a real S3 request and does not prove that a real request would succeed. Actual authorization can depend on additional identity, resource, permissions-boundary, session, Organizations, and request-context controls.

7. Evidence collection

Run:

./scripts/capture-evidence.sh

Fresh command output is saved under evidence/results/. The capture script stores the local auditor output, Access Analyzer responses, and simulator decisions.

Local Python auditor findings for the over-permissive policy.

Access Analyzer findings for the over-permissive policy.

3. Access Analyzer response for the scoped policy.

4. Simulator output for the three action/resource cases.

8. Cost and safety considerations

This workflow does not provision AWS infrastructure, attach policies, or issue S3 data-plane requests. It invokes IAM policy validation and simulation APIs. Check current AWS pricing and your own account billing separately; this project does not guarantee that unrelated account activity has no cost.

Do not attach the over-permissive sample to a real IAM user, group, or role. The bucket ARN is fictional; no bucket is created or contacted.

9. Limitations and future work

Expand the local checker's test suite and rule coverage while keeping its limitations explicit.
Add unit tests for malformed documents, single-statement and array statements, wildcard actions, and sensitive permissions.
Integrate the checks into a pull-request workflow after reviewing false-positive and failure-handling behavior.
Add workload-specific requirements so a policy can be reviewed against what the application actually needs.
Evaluate permissions boundaries, resource policies, session policies, and Organizations controls in an appropriate review or simulation context.

Conclusion

Least privilege is not achieved by finding one wildcard and replacing it mechanically. It requires understanding the workload's intended actions and resources, checking policy structure, and testing representative authorization cases. A local checker, Access Analyzer validation, and Policy Simulator provide three useful review layers before a policy is deployed—without creating test infrastructure for this demonstration.

Repository: [ AWS Community Builder Repository]

Preventing Public Exposure Before It Happens: Event-Driven AWS Security Guardrails with Terraform, Lambda, and EventBridge

Narendra | SRE — Thu, 05 Nov 2026 16:18:11 +0000

Implementation Status: The guardrail was deployed in AWS and validated using EventBridge, Lambda, CloudTrail, EC2 Security Groups, S3, DynamoDB, and SNS.

Introduction

Most cloud security tooling tells you about a mistake after it has already happened. A scanner runs on a schedule, a finding appears in a dashboard, and someone eventually opens a ticket. In the meantime, a security group that allows SSH from 0.0.0.0/0 is reachable from the entire internet, and a bucket whose public access block has been removed can be opened to the internet with a single ACL or bucket policy change.

In this article I build a guardrail that reacts to the API call itself rather than to the next scan. When someone opens a sensitive port to the world or weakens the public access block on an S3 bucket, a Lambda function is invoked within seconds. It either reverts the change or, in dry-run mode, only alerts. Every decision is written to an audit table, and the team receives an email. The entire stack is provisioned with Terraform and designed to run inside the AWS Free Tier.

The complete project, including Terraform code, the remediator, test scripts, and sanitized evidence, is in my GitHub repository, linked at the end.

1. The Problem

Two misconfigurations cause a disproportionate share of avoidable cloud exposure:

Administrative ports open to the internet. SSH (22) and RDP (3389) opened to 0.0.0.0/0 are routinely targeted by automated scanners within minutes of appearing.
Public S3 buckets. Buckets become public when the Block Public Access settings are removed, usually to make a quick test work and then forgotten.

Both are easy to introduce. A single CLI command or console click is enough. Both are also easy to miss: the change looks reasonable in isolation, and nobody is watching the API stream.

Detection alone has a structural weakness. A scheduled scan has a detection window measured in hours, while the change itself takes effect in seconds. Auto-remediation closes that window, but it introduces its own risk. A guardrail that silently breaks a legitimate workflow is worse than none. The design therefore has to answer three questions: how to act quickly, how to avoid acting wrongly, and how to prove afterwards what happened.

2. Architecture

The flow has six steps:

An engineer or an automation pipeline calls AuthorizeSecurityGroupIngress, DeleteBucketPublicAccessBlock, or PutBucketPublicAccessBlock.
CloudTrail records the call. Its management events reach the default EventBridge bus as AWS API Call via CloudTrail, so no additional trail is required for detection.
An EventBridge rule matches only the three violation-relevant event names, and invokes the remediator Lambda with up to three retries.
The Lambda claims the event ID in DynamoDB, evaluates the change, and either remediates or simulates the remediation depending on the dry-run setting.
The outcome is written to the findings table, and an alert is published to SNS and delivered by email.
Any event that fails every retry is parked in an SQS dead-letter queue for inspection.

The services were chosen for cost as much as capability. I avoided AWS Config and Security Hub, which are powerful but billed per rule evaluation or finding, and I avoided customer-managed KMS keys, which carry a monthly charge. Lambda, EventBridge rules on the default bus, DynamoDB on-demand, SNS email, SQS, and CloudWatch Logs with 14-day retention all fit within the Free Tier for a workload of this size.

3. Design Decisions

Event-driven rather than polling. The guardrail reacts to the API call, so it acts within seconds and does not pay for periodic full-account scans.

Revoke the exact rule, narrowed to the world. The remediator does not wipe the security group. It removes only the authorized rule, and only the 0.0.0.0/0 or ::/0 entries within it. A rule that also allows a private CIDR keeps its private entries. This keeps the blast radius of the correction as small as the violation.

Idempotent processing. EventBridge delivers at least once, so the same event can arrive twice. The remediator writes the event ID to DynamoDB with a conditional PutItem. A second delivery fails the condition and is recorded as a duplicate. If processing crashes midway, the claim is deleted before the error is re-raised, so the retry can run to completion instead of being wrongly skipped.

Loop safety. Remediation itself produces CloudTrail events. Revoking an ingress rule emits RevokeSecurityGroupIngress, which the rule does not match. Restoring a public access block emits PutBucketPublicAccessBlock, which the rule does match, but the handler first checks compliance and does nothing if the bucket is already fully blocked. The system converges instead of looping.

Dry-run by default. The first deployment sets DRY_RUN=true. It records what it would have changed and sends alerts, but changes nothing. This lets a team observe real traffic for days before enforcing. Enforcement is a single Terraform variable.

Exemptions by tag. Some resources legitimately need public access, such as a static website bucket or a bastion host. A resource tagged guardrail-exempt=true is skipped and recorded as exempt. The tradeoff is that anyone who can apply tags can grant an exemption, so tag-write permissions become a security-sensitive privilege. I discuss how to govern that in the limitations section.

Least privilege. Each IAM statement maps to a specific API call in the handler. Describe calls are the only ones that cannot be scoped by resource, and they are isolated in their own statement. The one known gap is the S3 statement, which covers all buckets. Narrowing it with a tag condition is planned future work, and I've documented it in the code rather than hiding it.

4. Implementation

4.1 Detection rule

The EventBridge pattern matches the CloudTrail event envelope on source service and event name:

event_pattern = jsonencode({
  "detail-type" = ["AWS API Call via CloudTrail"]
  detail = {
    eventSource = ["ec2.amazonaws.com", "s3.amazonaws.com"]
    eventName   = concat(["AuthorizeSecurityGroupIngress"], local.bucket_events)
  }
})

The target carries both a retry policy and a dead-letter queue. These two settings are what turn "the Lambda failed once" into "the event was retried for an hour and then preserved for inspection."

retry_policy {
  maximum_event_age_in_seconds = 3600
  maximum_retry_attempts       = 3
}

dead_letter_config {
  arn = aws_sqs_queue.dlq.arn
}

4.2 Revoking only the violating rule

The security group handler inspects each authorized permission in the CloudTrail record and builds the smallest revoke request that undoes the violation:

def _revoke_rule(perm, v4, v6):
    rule = {"IpProtocol": perm["ipProtocol"]}
    if "fromPort" in perm:
        rule["FromPort"] = perm["fromPort"]
        rule["ToPort"] = perm["toPort"]
    if v4:
        rule["IpRanges"] = [{"CidrIp": cidr} for cidr in v4]
    if v6:
        rule["Ipv6Ranges"] = [{"CidrIpv6": cidr} for cidr in v6]
    return rule

The revoke must match the original protocol and port range exactly, which is why the handler copies them from the event rather than reconstructing them from the configured sensitive ports.

4.3 Claiming each event once

def _claim(event_id):
    try:
        findings.put_item(
            Item={"event_id": event_id, "status": "PROCESSING"},
            ConditionExpression="attribute_not_exists(event_id)",
        )
        return True
    except ClientError as err:
        if err.response["Error"]["Code"] == "ConditionalCheckFailedException":
            return False
        raise

The wrapper that calls this function releases the claim on failure:

try:
    outcome = evaluate(detail)
except Exception:
    findings.delete_item(Key={"event_id": event_id})
    raise

Without that release, a transient API error would leave the event permanently marked as processing, and every retry would be skipped.

4.4 Infrastructure

Everything is defined in five Terraform files: versions.tf pins the AWS and archive providers, provider.tf applies default tags, variables.tf exposes the dry-run switch, sensitive port list, and alert address, main.tf declares the resources, and outputs.tf exposes names needed by the test and evidence scripts. The handler is packaged from lambda/handler.py by an archive data source, so a code change produces a new source_code_hash and Terraform updates the function on the next apply.

5. Testing the Guardrail

scripts/simulate-violations.sh exercises three scenarios:

Open SSH on a normal security group. This should be a violation.
Open SSH on a security group tagged guardrail-exempt=true. This should be allowed.
Delete the public access block on a new bucket. This should be a violation.

I ran the test in dry-run mode first, confirmed that the alerts and audit records were correct and nothing was changed, and then set dry_run = false and reran it.

Scenario	Expected in dry-run	Expected in enforce mode	Observed
1. Open SSH, untagged SG	`would_revoke` recorded, alert sent, rule still present	rule revoked, alert sent

The time from API call to revocation was [insert measured seconds from CloudTrail eventTime to the revoke call in the logs]. This is the window in which the exposure existed, and it is the number that matters most when comparing this approach with scheduled scanning.

6. Evidence

The evidence/ folder contains sanitized outputs from the run. Account IDs are replaced with 111122223333.

File	What it shows
`terraform-plan-summary.json`	The resources the stack creates
`remediator-logs-sanitized.json`	Lambda decisions, including duplicate and exempt paths
`cloudtrail-authorize-ingress-sanitized.json`	The originating API call for scenario 1
`cloudtrail-delete-public-access-block-sanitized.json`	The originating API call for scenario 3
`sg-after-remediation-sanitized.json`	The security group after enforcement
`screenshots/`	Console views of the alert email, DynamoDB items, and bucket settings

7. Cost

On an account used only for this project, the workload should stay within the Free Tier: a handful of Lambda invocations, a few dozen DynamoDB writes, a few email notifications, and a small amount of log data. I verified this against the Billing dashboard after running the simulation, and the observed charges were [insert observed charges, or "zero"]. Costs can change, so check your own account's Free Tier usage before leaving the stack running.

8. Limitations and Future Work

The guardrail is reactive, not preventive. There is a window of seconds between the API call and the correction. A Service Control Policy can prevent some of these calls outright, and I would layer one on top in a production setting.

It does not find existing exposure. The guardrail only sees new API calls. A one-time sweep of existing security groups and buckets would complement it.

Exemptions need governance. Because exemption is a tag, the control is only as strong as the permission to write tags. A production version should restrict tag changes on guarded resources, or require an approval record.

Bucket policies and ACLs are out of scope. This rule covers the public access block only. A bucket made public through a bucket policy or ACL would need additional rules.

The S3 IAM statement is broader than ideal. It covers all buckets. A tag-based condition would narrow it to buckets the team owns.

Conclusion

Public exposure is one of the most common and most preventable cloud misconfigurations. Scanning finds it eventually, but the exposure starts the moment the API call succeeds. This project closes that gap with an event-driven guardrail that acts within seconds, revokes only what violates policy, records every decision, and fails safely.

The design choices that matter most are not the services but the behaviors around them: dry-run before enforcement, idempotent processing, a dead-letter queue, narrowly scoped revocation, and an explicit exemption path. Those choices are what make an automated security control safe to run in a real account.

GitHub repository: [AWS-Community-Builder Repository]

Detecting and Remediating AWS Infrastructure Drift Using Terraform and CloudTrail

Narendra | SRE — Wed, 04 Nov 2026 14:14:17 +0000

Implementation Status: The demonstration was deployed and validated using AWS EC2, CloudTrail and Security groups.

Introduction

Infrastructure as Code (IaC) has become an important part of modern cloud infrastructure management. Tools such as Terraform allow engineering teams to provision, configure, and manage cloud resources consistently through declarative configuration files.

However, managing infrastructure through code does not guarantee that every change will always happen through code.

In real-world environments, administrators or engineers may manually modify AWS resources through the AWS Management Console or AWS CLI. These changes can cause the actual infrastructure to differ from the configuration maintained in Terraform.

This situation is known as infrastructure drift.

In this article, I'll demonstrate how to introduce infrastructure drift deliberately in AWS, detect it using Terraform, investigate the manual modification using AWS CloudTrail, and restore the desired configuration using Terraform.

We'll use an Amazon EC2 instance and its tags as a simple but practical example.

1. Understanding Infrastructure Drift

Infrastructure drift occurs when the actual configuration of a cloud resource differs from the desired configuration defined in Infrastructure as Code.

Consider an EC2 instance managed through Terraform with the following tags:

tags = {
  Name        = "Terraform infrastructure drift"
  Environment = "development"
  ManagedBy   = "Terraform"
}

Now imagine an administrator manually changes the Environment tag through the AWS Console:

Environment = production

The actual AWS resource now differs from the Terraform configuration.

Terraform still declares:

Environment = development

But AWS reports:

Environment = production

This difference is infrastructure drift.

Drift can create configuration inconsistencies, unexpected infrastructure behavior, security risks, and difficulties during infrastructure maintenance.

2. Architecture and Technology Stack

The demonstration uses the following components:

Terraform: Infrastructure provisioning, drift detection, and remediation.
Amazon EC2: The AWS resource used for the drift experiment.
Amazon VPC security group: Controls network access to the EC2 instance.
AWS CloudTrail: Records the API activity associated with the manual change.
AWS CLI: Inspects AWS resources and retrieves audit events.

The implementation is performed in the AWS Asia Pacific (Hyderabad) Region (ap-south-2).

Architecture flow:

Terraform provisions an EC2 instance. A manual modification changes its Environment tag. Terraform detects the difference, CloudTrail provides an audit record of the modification, and Terraform restores the declared configuration.

3. Provisioning Infrastructure Using Terraform

The first step is to define the required AWS infrastructure using Terraform.

The demonstration provisions:

One Amazon EC2 instance.
One security group.
An Amazon Linux 2023 AMI discovered dynamically through a Terraform data source.

The EC2 instance uses the t3.micro instance type.

AMI discovery

Instead of hardcoding an AMI ID, Terraform uses an AWS data source to discover a suitable Amazon Linux 2023 image.

data "aws_ami" "amazon_linux" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["al2023-ami-2023.*-x86_64"]
  }

  filter {
    name   = "architecture"
    values  = ["x86_64"]
  }

  filter {
    name   = "virtualization-type"
    values  = ["hvm"]
  }
}

EC2 resource

The EC2 instance is configured with the discovered AMI and the selected instance type.

resource "aws_instance" "drift_demo" {
  ami                    = data.aws_ami.amazon_linux.id
  instance_type          = var.instance_type
  vpc_security_group_ids = [aws_security_group.drift_demo.id]

  tags = {
    Name        = var.project_name
    Environment = "development"
    ManagedBy   = "Terraform"
  }
}

The Environment tag is important because we'll intentionally modify it outside Terraform.

Terraform initialization and validation

Before provisioning, I initialized and validated the Terraform configuration.

terraform init
terraform fmt -recursive
terraform validate

I generated an execution plan:

terraform plan -out=tfplan

After reviewing the plan, I deployed the infrastructure:

terraform apply tfplan

The deployment created the EC2 instance and security group.

4. Establishing the Initial Infrastructure State

Before introducing drift, I verified the actual AWS resource configuration.

I retrieved the EC2 instance ID:

$instanceId = terraform output -raw instance_id

I then inspected the EC2 tags using AWS CLI:

aws ec2 describe-tags `
  --filters "Name=resource-id,Values=$instanceId" `
  --region ap-south-2 `
  --query "Tags[].{Key:Key,Value:Value}" `
  --output table

The initial configuration was:

Tag	Value
Name	Terraform infrastructure drift
Environment	development
ManagedBy	Terraform

I also ran:

terraform plan

Terraform reported:

No changes. Your infrastructure matches the configuration.

This established our baseline: the actual AWS infrastructure matched the declared Terraform configuration.

5. Introducing Infrastructure Drift

To simulate a real-world manual infrastructure change, I modified the Environment tag directly through the AWS Management Console.

I navigated to:

EC2 → Instances → Select instance → Tags → Manage tags.

I changed:

Environment = development

to:

Environment = production

I did not modify the Terraform configuration.

This is important: the difference was introduced directly in AWS, not through Terraform.

I verified the updated tag using AWS CLI.

aws ec2 describe-tags `
  --filters "Name=resource-id,Values=$instanceId" `
  --region ap-south-2 `
  --query "Tags[].{Key:Key,Value:Value}" `
  --output table

The actual AWS configuration now showed:

Tag	Value
Name	Terraform infrastructure drift
Environment	production
ManagedBy	Terraform

The Terraform configuration still declared Environment = development.

We had successfully introduced infrastructure drift.

6. Detecting Infrastructure Drift Using Terraform

The next step was to determine whether Terraform could identify the difference.

I executed:

terraform plan

Terraform refreshed its view of the managed resource and compared the observed infrastructure with the declared configuration.

The plan identified an in-place update to the EC2 instance tags.

The relevant difference was:

Environment = production -> development

The ~ symbol in Terraform's plan output indicates that the resource is being updated in place.

To preserve the execution plan, I generated a saved plan:

terraform plan -out=drift-plan

I also exported the plan in JSON format:

terraform show -json drift-plan |
  Out-File -Encoding utf8 ..\evidence\terraform-plan-drift.json

This demonstrated that Terraform had detected the difference between the actual AWS resource and the desired configuration.

7. Investigating Manual Changes Using AWS CloudTrail

Detecting drift is important, but understanding how the change occurred is equally important.

For this, I used AWS CloudTrail.

When EC2 tags are created or modified through AWS APIs, CloudTrail records the CreateTags management event.

I queried the CloudTrail event history:

aws cloudtrail lookup-events `
  --lookup-attributes AttributeKey=EventName,AttributeValue=CreateTags `
  --max-results 50 `
  --region ap-south-2 `
  --output json

I inspected the event associated with the EC2 instance.

The event provides information such as:

Event name and time.
AWS Region.
Identity that performed the operation.
Resource affected.
Requested tag values.

The CloudTrail record provided an audit trail for the manual infrastructure modification.

By comparing the initial EC2 tag evidence with the CloudTrail event and the drifted configuration, I could establish what changed and investigate the corresponding API activity.

8. Remediating Infrastructure Drift

After identifying the difference, I reviewed the Terraform execution plan.

The proposed change was:

Environment = production -> development

Since the desired configuration was development, I proceeded with the reviewed plan.

terraform apply drift-plan

Terraform applied the proposed update and restored the Environment tag.

I verified the result using AWS CLI:

aws ec2 describe-tags `
  --filters "Name=resource-id,Values=$instanceId" `
  --region ap-south-2 `
  --query "Tags[].{Key:Key,Value:Value}" `
  --output table

The Environment tag was restored to:

Environment = development

The EC2 instance itself was not replaced. The correction was an in-place tag update.

9. Final Validation

To verify that the infrastructure was back in the desired state, I executed:

terraform plan

Terraform reported:

No changes. Your infrastructure matches the configuration.

This confirmed that the observed AWS configuration matched the Terraform configuration after remediation.

10. Key Learnings

Through this implementation, I gained practical experience with several important infrastructure management concepts.

1. Infrastructure drift

Manual changes outside Infrastructure as Code can cause differences between desired and actual infrastructure.

2. Terraform state and refresh

Terraform uses its state and refresh behavior to understand managed resources and compare them with the declared configuration.

3. Drift detection

terraform plan helps identify differences and presents proposed changes for review.

4. CloudTrail auditing

CloudTrail provides an audit trail of AWS API activity, helping engineers investigate infrastructure modifications.

5. Controlled remediation

Terraform does not automatically correct every detected difference. Engineers must review the proposed changes and explicitly apply them.

6. Infrastructure governance

Teams should establish clear processes for infrastructure changes and minimize unmanaged manual modifications.

Conclusion

Infrastructure drift is a common challenge in cloud environments where infrastructure can be modified through multiple access paths.

In this demonstration, I provisioned an EC2 instance using Terraform, deliberately modified its tag through the AWS Console, detected the resulting drift using Terraform, investigated the API activity using AWS CloudTrail, and restored the desired configuration.

The final Terraform plan confirmed that the infrastructure matched the declared configuration again.

This exercise demonstrates how Terraform and CloudTrail can complement each other: Terraform helps identify and correct configuration differences, while CloudTrail provides an audit trail for investigating AWS API activity.

The complete Terraform implementation and sanitized evidence are available in my GitHub repository.

GitHub repository: [AWS Community Builders repository]

Secure AWS Automation with GitHub Actions and OpenID Connect (OIDC)

Narendra | SRE — Sat, 31 Oct 2026 14:47:51 +0000

Secure AWS Automation with GitHub Actions and OpenID Connect (OIDC)

Level: Intermediate
Focus: AWS IAM, temporary credentials, OpenID Connect, GitHub Actions, least privilege.

Example code: aws-community-builders repository

Implementation status:
Created the GitHub Actions OIDC identity provider in AWS IAM.
Created a dedicated S3 test bucket.
Configured an IAM role with an exact GitHub OIDC subject condition.
Configured short-lived AWS credentials using GitHub Actions.
Successfully assumed the IAM role through OIDC.
Successfully accessed the permitted reports/ prefix.
Verified that access to private/ was denied.
Captured Terraform, IAM, CloudTrail and GitHub Actions evidence.

Introduction

CI/CD pipelines often need access to cloud resources: publishing an artifact, deploying an application, or updating infrastructure. A common shortcut is to create an IAM user's access key and store it as a CI/CD secret. These credentials are reusable until rotated or revoked. If exposed through a compromised runner, workflow, or accidental logging, they may be used outside the pipeline.

AWS supports a different pattern: a workload proves its identity using OpenID Connect (OIDC), then requests temporary AWS credentials for an IAM role. This article explains that pattern using GitHub Actions and AWS Security Token Service (AWS STS).

The objective is not just to remove access keys. It is to establish a narrowly scoped trust relationship between a specific repository and workflow context and one AWS role, then grant that role only the permissions the workflow needs.

Figure 1. GitHub Actions obtains a short-lived role session through OIDC instead of using a stored long-lived AWS access key.

1. Why long-lived access keys are risky

Static IAM user access keys can remain valid for a long time. They may be copied into workflows, developer machines, or secret stores. Rotation becomes an operational task, and every copy must be updated safely.

Common risks include accidental commits, credentials printed by workflows, compromised runners, keys remaining active after a project ends, and broad permissions increasing the impact of exposure.

Secret storage is important for credentials that must exist, but it does not change the fact that a static access key is reusable. Where workload federation is supported, short-lived credentials are usually a better fit.

2. The building blocks

Component	Responsibility
GitHub Actions	Runs the automation workflow.
GitHub OIDC issuer	Issues a signed identity token for the workflow context.
AWS IAM OIDC provider	Establishes trust in the token issuer and audience.
IAM role trust policy	Defines which federated identities may assume the role.
AWS STS	Validates the token and issues temporary credentials.
IAM permissions policy	Defines what the assumed role can do in AWS.

Two policy types must be understood separately:

Trust policy: Who can assume this role?
Permissions policy: What can the role do after it is assumed?

A restrictive trust policy does not compensate for an overly broad permissions policy, and vice versa.

3. How the OIDC flow works

Figure 2. AWS validates the token issuer and audience, evaluates the role trust conditions, and STS issues a temporary session.

A GitHub Actions job starts.
The workflow is allowed to request an OIDC token by declaring id-token: write.
GitHub issues a signed token describing the workflow identity.
The AWS credentials action presents the token to AWS STS using AssumeRoleWithWebIdentity.
AWS validates the token against the configured GitHub OIDC provider and evaluates the IAM role trust policy.
If authorized, STS returns temporary credentials.
The workflow uses those credentials to call permitted AWS APIs. The session expires according to the role session duration.

id-token: write permits the job to request an OIDC token; it does not itself grant AWS permissions. AWS authorization still depends on the role trust policy and attached permissions.

4. Configure the GitHub OIDC provider in AWS

An IAM OIDC identity provider must exist in the AWS account before a role can trust it.

Provider URL: https://token.actions.githubusercontent.com
Client ID (audience): sts.amazonaws.com

Create the provider in IAM using the AWS Console or your approved infrastructure-as-code process. Confirm current AWS and GitHub documentation for provider setup and certificate/thumbprint handling rather than copying an old thumbprint from an example.

The Terraform example in this project accepts the ARN of an already-created GitHub OIDC provider as an input. This keeps provider lifecycle management separate from the example role.

Do not create duplicate providers for the same issuer in one account; reuse an existing provider where appropriate.

5. Understand GitHub's OIDC subject (`sub`) claim

One of the most important parts of configuring GitHub Actions OIDC is understanding the sub (subject) claim.

AWS uses this claim to determine which GitHub repository and workflow context are permitted to assume an IAM role.

Historically, the default subject format for a workflow running from the main branch was:

repo:OWNER/REPOSITORY:ref:refs/heads/main

However, GitHub introduced immutable OIDC subject claims to strengthen the security of federated identity.

5.1 Legacy subject format

Repositories created before July 15, 2026 generally use the original name-based subject format unless they explicitly opt in to immutable subject claims.

Example:

repo:octo-org/octo-repo:ref:refs/heads/main

This format identifies the repository using its owner and repository names.

5.2 New immutable subject format

Repositories created after July 15, 2026 automatically use an immutable subject format by default.

Example:

repo:octo-org@123456/octo-repo@456789:ref:refs/heads/main

The format is:

repo:OWNER@OWNER-ID/REPOSITORY@REPOSITORY-ID:ref:refs/heads/BRANCH

The new format includes:

The repository owner's name and immutable ID.
The repository name and immutable ID.
The branch context.

The purpose is to prevent identity ambiguity if a repository or organization is renamed, transferred, deleted, or recreated under a previously used name.

5.3 Which repositories use which format?

Repository situation	Default subject format
Created before July 15, 2026	Legacy format, unless opted in
Created after July 15, 2026	Immutable format
Renamed after July 15, 2026	Immutable format
Transferred after July 15, 2026	Immutable format
Existing repository explicitly opted in	Immutable format

This distinction is important because a trust policy that expects the legacy format will not match a token containing the new immutable format.

For example, the following two values are different:

repo:octo-org/octo-repo:ref:refs/heads/main

repo:octo-org@123456/octo-repo@456789:ref:refs/heads/main

AWS IAM evaluates an exact StringEquals condition against the actual claim. If the values do not match, role assumption fails.

Important: Do not assume which format your repository uses based only on its name or your workflow YAML. Verify the actual token subject before configuring the trust policy.

6. Verify the actual OIDC subject claim

Before creating the IAM trust policy, inspect the sub claim issued by your GitHub Actions workflow.

This is particularly important for repositories created after July 15, 2026, or repositories that have opted in to immutable subject claims.

Add a temporary diagnostic step to a GitHub Actions workflow:

- name: Inspect GitHub OIDC subject
  shell: bash
  env:
    ACTIONS_ID_TOKEN_REQUEST_URL: ${{ env.ACTIONS_ID_TOKEN_REQUEST_URL }}
    ACTIONS_ID_TOKEN_REQUEST_TOKEN: ${{ env.ACTIONS_ID_TOKEN_REQUEST_TOKEN }}
  run: |
    set -euo pipefail

    TOKEN_RESPONSE=$(curl -sS \
      -H "Authorization: Bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
      "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=sts.amazonaws.com")

    TOKEN=$(echo "$TOKEN_RESPONSE" | jq -r '.value')

    if [ -z "$TOKEN" ] || [ "$TOKEN" = "null" ]; then
      echo "Unable to retrieve OIDC token"
      exit 1
    fi

    python3 -c '
    import base64
    import json
    import sys

    token = sys.argv[1]
    payload = token.split(".")[1]

    payload += "=" * (-len(payload) % 4)

    claims = json.loads(
        base64.urlsafe_b64decode(payload)
    )

    print("OIDC subject:", claims.get("sub"))
    print("OIDC audience:", claims.get("aud"))
    ' "$TOKEN"

    unset TOKEN TOKEN_RESPONSE

This diagnostic step:

Requests a GitHub OIDC token with the AWS STS audience.
Decodes the JWT payload locally.
Displays only the sub and aud claims.
Does not print the complete token.

The token is a temporary credential. Never publish it, save it in build artifacts, or print it in workflow logs.

After identifying the subject, use the exact value when configuring the AWS IAM trust policy. Remove the diagnostic step when it is no longer required.

7. Restrict the role trust policy

For a workflow running from the main branch, the trust policy must match the subject format actually issued by GitHub.

Example A: Legacy subject format

For a repository using the legacy format:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
          "token.actions.githubusercontent.com:sub": "repo:octo-org/octo-repo:ref:refs/heads/main"
        }
      }
    }
  ]
}

Example B: Immutable subject format

For a repository using the new immutable format:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
          "token.actions.githubusercontent.com:sub": "repo:octo-org@123456/octo-repo@456789:ref:refs/heads/main"
        }
      }
    }
  ]
}

The account ID, owner name, owner ID, repository name, and repository ID above are illustrative values. Replace them with your actual values.

Use only the trust policy that matches your repository's actual subject claim. These are alternatives, not two policies that should automatically be combined.

Terraform implementation

If you manage the IAM role using Terraform, make the complete expected subject an input rather than assuming every repository uses the old format.

For example:

variable "github_oidc_subject" {
  description = "Exact GitHub OIDC sub claim for the trusted workflow"
  type        = string

  validation {
    condition     = length(trimspace(var.github_oidc_subject)) > 0
    error_message = "The GitHub OIDC subject must not be empty."
  }
}

Use that input in the role trust policy:

data "aws_iam_policy_document" "github_actions_trust" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = [var.github_oidc_provider_arn]
    }

    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:aud"
      values   = ["sts.amazonaws.com"]
    }

    condition {
      test     = "StringEquals"
      variable = "token.actions.githubusercontent.com:sub"
      values   = [var.github_oidc_subject]
    }
  }
}

Example input for a legacy repository:

github_oidc_subject = "repo:octo-org/octo-repo:ref:refs/heads/main"

Example input for a repository using immutable claims:

github_oidc_subject = "repo:octo-org@123456/octo-repo@456789:ref:refs/heads/main"

This approach makes the trust relationship explicit and avoids silently assuming an outdated subject format.

Branches and GitHub Environments

The subject format also depends on the workflow context.

For example, a workflow using a GitHub Environment may have a subject similar to:

repo:octo-org/octo-repo:environment:production

For repositories using immutable subjects, the owner and repository IDs are also included.

When using environments, inspect the actual sub claim and update the trust policy accordingly. Configure environment protection rules, such as required reviewers and deployment restrictions.

Do not copy a branch-based trust policy unchanged into an environment-based workflow.

Avoid broad conditions such as repo:OWNER/* unless every repository in that scope is intentionally trusted. A wildcard can allow an unintended repository to assume the role.

8. Grant only the permissions the workflow needs

The example role has a deliberately small read-only S3 policy for one bucket and the reports/ prefix:

s3:ListBucket on the specific bucket, constrained by the s3:prefix condition.
s3:GetObject on objects under reports/ in that bucket.

It does not grant bucket creation, object writes, deletion, IAM administration, or access to every S3 bucket.

For a real pipeline, start with the exact API operations required. Scope resource ARNs wherever the service supports resource-level permissions.

Some actions require "Resource": "*"; document why and consider applicable condition keys.

Do not attach AdministratorAccess just to make a workflow succeed.

9. Configure the GitHub Actions workflow

The sample workflow is in .github/workflows/oidc-demo.yml. It runs on pushes to main and can also be started manually.

Its important permissions are:

permissions:
  contents: read
  id-token: write

The workflow configures AWS credentials using a role ARN and Region, then runs:

aws sts get-caller-identity

This confirms which AWS identity the job is using. It then lists the approved S3 prefix as a permissions demonstration.

Store the role ARN and bucket name as GitHub Actions repository variables. The role ARN is an identifier, not a secret. The workflow must not contain an AWS access key ID or secret access key.

For production, pin third-party GitHub Actions to a reviewed full commit SHA rather than relying only on a mutable major-version tag. Review action updates through your normal dependency-management process.

10. Security considerations

OIDC reduces the need for long-lived credentials, but it does not make a pipeline automatically secure.

Restrict the subject to the required repository and branch or protected environment.
Protect the trusted branch with reviews and required status checks.
Review workflow changes: someone who can modify a trusted workflow may change what the role does.
Use least privilege for AWS actions and resources.
Grant id-token: write only to jobs that need federation.
Protect sensitive environments with required reviewers and deployment rules.
Pin third-party actions to reviewed immutable references.
Never print tokens, credentials, or sensitive deployment output.
Use CloudTrail to review role-assumption and API activity.
Choose a session duration that is sufficient but not unnecessarily long.
Verify the repository's actual OIDC subject format before configuring the trust policy.
Revisit the trust policy if the repository is renamed, transferred, or changes its OIDC subject configuration.

A compromised workflow running in a trusted repository and branch may still obtain the role. OIDC establishes workload identity; repository governance and workflow security remain essential.

11. Troubleshooting

Symptom	Possible cause	What to check
`Not authorized to perform sts:AssumeRoleWithWebIdentity`	Trust policy does not match token claims	Provider ARN, audience, exact `sub` condition.
`No OpenIDConnect provider found`	Provider missing or ARN/issuer incorrect	Confirm provider exists in the same AWS account.
Audience mismatch	Trust policy expects another audience	Confirm `sts.amazonaws.com` is configured.
Subject mismatch	Legacy and immutable subject formats differ	Inspect the actual token's `sub` claim and update the trust policy.
Branch mismatch	Workflow runs from a different branch	Verify the actual branch context.
Environment mismatch	Workflow uses a GitHub Environment	Check the environment-based subject and trust condition.
S3 `AccessDenied`	Role assumed, but permissions are insufficient	Check attached policy, bucket policy, prefix, and organization controls.
Workflow cannot request OIDC token	`id-token: write` missing or overridden	Check workflow/job permissions.

Run:

aws sts get-caller-identity

after configuring credentials.

If it succeeds, role assumption worked. An access-denied error on a later S3 command is a separate permissions issue.

12. Cost considerations

OIDC federation does not require a continuously running compute service. However, AWS API usage and the resources your workflow operates can incur charges. GitHub Actions usage is subject to GitHub's plan and runner limits.

Check current pricing and usage policies before scaling automation.

13. Validation checklist

This project has been deployed or tested. In a dedicated non-production account:

Validation	Result
GitHub OIDC authentication	PASS
IAM role assumption	PASS
Approved S3 prefix access	PASS
Unauthorized prefix access	PASS — AccessDenied
CloudTrail verification	COMPLETED
Resource cleanup	COMPLETED

14. When should you use this pattern?

Use OIDC federation when an external workload such as GitHub Actions needs AWS access and supports issuing a trusted identity token.

Examples include:

CI pipelines that publish artifacts.
Application deployment workflows.
Infrastructure automation running Terraform against AWS.
Automated testing that needs temporary access to AWS resources.

The trust conditions and permissions should differ by workload. A build job that uploads an artifact should not automatically receive the same role as a production deployment job.

Conclusion

Long-lived access keys are not the only way for automation to authenticate to AWS. With OIDC federation, GitHub Actions can present a signed workload identity to AWS STS and receive temporary credentials for a narrowly scoped IAM role.

The strongest design combines a precise trust policy, least-privilege permissions, protected source branches, reviewed workflow changes, and CloudTrail visibility.

GitHub's introduction of immutable OIDC subject claims makes it especially important to verify the actual subject before configuring AWS IAM. A trust policy must match the identity GitHub really issues, not simply a format copied from an older example.

OIDC reduces credential-management risk, but it does not replace secure CI/CD practices.

The central lesson is to treat the pipeline as an identity with a defined trust boundary: decide exactly which workload may assume the role, what it may do, and how its activity will be reviewed.

References

Building a Serverless Event-Driven Processing Pipeline with Amazon S3, Amazon SQS, and AWS Lambda

Narendra | SRE — Sat, 31 Oct 2026 14:38:47 +0000

Building a Serverless Event-Driven Processing Pipeline with Amazon S3, Amazon SQS, and AWS Lambda

Level: Intermediate

Focus: Event-driven architecture, asynchronous processing, reliability, security, and observability

Example code: aws-community-builders repository

Implementation status: This reference implementation has been deployed and validated in an AWS learning environment in the Asia Pacific (Hyderabad) Region ( ap-south-2 ).

The implementation was provisioned using Terraform and tested with real S3 uploads, SQS message delivery, Lambda processing, failure handling, retries, dead-letter queue routing, and
CloudWatch monitoring. Validation evidence is available in the accompanying GitHub repository.

This is a learning reference implementation, not a production-ready solution. Review the security, reliability, and cost considerations before adapting it for production.

Originally published on AWS Community Builder

Introduction

Imagine you operate an application where users upload invoices, images, reports, or CSV files. Each upload needs to trigger some work: validate metadata, extract information, transform data, or notify another system.

A tightly coupled implementation can make the upload path slower and harder to operate. If processing is temporarily unavailable, the upload workflow may also be affected. A more resilient approach is to separate the act of accepting a file from the work that follows it.

In this article, we'll explore an event-driven pattern built with Amazon Simple Storage Service (Amazon S3), Amazon Simple Queue Service (Amazon SQS), and AWS Lambda. We'll also cover a dead-letter queue, IAM permissions, encryption, monitoring, retries, and idempotency.

Figure 1. A file-created event is queued, processed asynchronously by Lambda, and observed through CloudWatch. Repeatedly failing messages can be moved to a dead-letter queue.

1. The problem we're solving

A simple file upload workflow often starts with one question: what should happen after a file arrives?

For a small prototype, an application might upload a file and then call a processing function directly. This can be acceptable, but it creates a tighter dependency between the upload path and the processing operation. Slow processing, transient errors, or sudden bursts of uploads can complicate the user experience and recovery process.

The design goal is to make file storage and file processing separate responsibilities. We want to:

Accept files without requiring the processing job to finish immediately.
Buffer events when uploads arrive faster than they can be processed.
Retry temporary failures.
Isolate repeatedly failing messages for investigation.
Limit permissions to what each component needs.
Make activity and failures visible to operators.

2. Architecture overview

The main components are:

Component	Responsibility
Amazon S3	Stores uploaded objects and emits object-created notifications.
Amazon SQS processing queue	Buffers notifications so processing can happen asynchronously.
AWS Lambda	Polls the queue through an event source mapping and handles messages in batches.
Amazon SQS dead-letter queue (DLQ)	Holds messages that could not be processed after the configured retry limit.
AWS CloudWatch Logs	Stores Lambda execution logs for troubleshooting.
AWS IAM	Defines which actions the Lambda execution role can perform.

The flow is deliberately asynchronous. S3 publishes an event notification to SQS; Lambda polls SQS using an event source mapping. This differs from the model where S3 invokes Lambda directly.

Why put SQS between S3 and Lambda?

SQS acts as a buffer. If a burst of uploads creates more work than Lambda can process immediately, messages can wait in the queue. The queue also gives the consumer a retry mechanism and a place to route messages that repeatedly fail.

A queue does not make processing exactly-once. Standard SQS queues provide at-least-once delivery, so a message may be delivered more than once. The consumer should be designed to tolerate duplicates.

3. End-to-end event flow

Figure 2. Upload, event notification, queueing, Lambda polling, processing, and logging in sequence.

Let's walk through the request lifecycle:

A user uploads an object to S3. The bucket stores the object. The bucket can be configured to send object-created notifications to a designated SQS queue.
S3 publishes a notification. The queue policy must allow the expected S3 bucket to send messages. Restrict the policy to the intended source bucket and account where applicable.
SQS retains the message. The message stays in the queue until the consumer handles it successfully or the redrive policy sends it to the DLQ.
Lambda polls the queue. The Lambda event source mapping retrieves batches of messages and invokes the function.
The function validates and processes each record. In a real implementation, processing might read object metadata, parse a file, or submit a downstream task. The example handler logs object details as a small demonstration rather than performing expensive file transformations.
Successful records are acknowledged. Lambda/SQS integration deletes successfully processed messages after invocation handling.
Failures are retried. With partial batch response support enabled, the function can report individual failed message identifiers so successful records in the same batch do not have to be retried unnecessarily.
Logs support investigation. Operators can review Lambda logs and queue metrics to understand processing volume and failures.

A practical note about payloads

S3 event notifications normally contain bucket and object-key details. Object keys in event records are URL-encoded; decode them before using them as literal keys. If the application needs to read the object body, the Lambda execution role also needs narrowly scoped S3 read access. Don't grant object access merely because the event contains a bucket and key.

4. Handling failures: retries and a dead-letter queue

Failures are part of normal distributed-system operation. A downstream dependency may time out, a file may be malformed, or the function may encounter a bug.

Figure 3. Failed messages can be retried by SQS/Lambda integration and moved to a dead-letter queue after the configured receive limit.

A dead-letter queue helps separate messages that need attention from the main processing backlog. The queue's redrive policy defines the maximum receive count before a message is moved to the DLQ.

When a message lands in the DLQ:

Inspect the body and message attributes to understand the failure.
Correlate the message with Lambda logs using timestamps and application identifiers.
Determine whether the issue was transient, data-specific, or caused by a code defect.
Fix the underlying problem before redriving messages to the source queue.
Avoid blindly replaying a large DLQ into an unhealthy downstream service.

Partial batch failures

Lambda can receive multiple SQS messages in one invocation. Without partial batch response handling, one failed record can cause the whole batch to be retried. When ReportBatchItemFailures is enabled, the function returns the message identifiers for records that failed, allowing successfully processed records to be acknowledged separately.

The handler must return the correct SQS message identifiers for failures. This improves retry efficiency but does not remove the need for idempotency.

5. Idempotency: preparing for duplicate delivery

For standard queues, at-least-once delivery means duplicate processing is possible. For example, a function might complete a side effect but fail before the message is acknowledged. The same message can then be delivered again.

An idempotent operation produces the same effective result when repeated with the same input. Useful techniques include:

Use a stable business identifier or a combination of bucket, object key, object version, and event metadata to identify the work item.
Store processing status in a durable data store when the business operation requires deduplication.
Use conditional writes or unique constraints to prevent duplicate side effects.
Make downstream APIs idempotent when they support idempotency keys.
Consider S3 versioning if the processing semantics depend on a particular object version.

The simple example in the repository demonstrates event handling and partial batch failure structure. It is not a complete business-level deduplication system.

6. Security design: least privilege and protected storage

Figure 4. Each service has a specific permission boundary: S3 may publish to the queue, Lambda may poll the queue and write logs, and bucket access remains private by default.

Security should be designed into the workflow rather than added after it works.

S3 bucket controls

The Terraform example configures S3 Block Public Access, server-side encryption, and versioning. These are useful baseline controls, but they do not replace a complete data-protection policy. For production, consider retention requirements, lifecycle rules, access logging where appropriate, and whether a customer-managed KMS key is required.

Queue policy

The queue policy should permit s3.amazonaws.com to send messages only from the intended bucket and account context. Avoid a broad policy that allows arbitrary sources to publish messages to the queue.

Lambda execution role

The Lambda role should contain only the permissions required for the function to operate: write its CloudWatch logs, receive/delete messages, inspect queue attributes, and access other resources only when the application requires them. The example should be reviewed carefully before production use, including the exact actions and resource ARNs.

Encryption

S3 server-side encryption protects stored objects. SQS encryption can be configured separately; if using a customer-managed KMS key, ensure the relevant principals have the necessary key permissions. Encryption configuration does not, by itself, determine who is authorized to read a resource.

Keep credentials out of code

Use an appropriate AWS credential provider or role-based access rather than embedding access keys in source files. Never commit real credentials, account identifiers that you intend to keep private, customer data, or secrets into the repository.

7. Observability and operational signals

CloudWatch Logs are the starting point for inspecting function execution, but logs alone are not enough to operate a queue-based workflow.

Monitor signals such as:

ApproximateNumberOfMessagesVisible: messages waiting to be processed.
ApproximateAgeOfOldestMessage: how long the oldest queued message has been waiting.
NumberOfMessagesSent / NumberOfMessagesReceived: useful for understanding queue activity, with the usual caveat that these are approximate queue metrics.
Lambda errors and throttles: indicators that processing is failing or capacity is constrained.
DLQ visible message count: a useful alert signal for messages that need investigation.

For a production deployment, define alarms using the service metrics that matter to the workload, set retention for logs, and document the response procedure when alerts fire. Avoid logging full file contents or sensitive values.

8. Cost considerations

The architecture avoids managing long-running servers, but serverless does not mean free. Charges may arise from S3 storage and requests, SQS requests, Lambda invocations and duration, CloudWatch log ingestion and storage, data transfer, and optional encryption or monitoring features.

Before running the example:

Check the current AWS Free Tier terms and eligibility for your account. Offerings and eligibility rules can change.
Use a dedicated test bucket and a small number of test objects.
Configure log retention rather than keeping logs indefinitely by default.
Delete test resources after validating the workflow.
Review AWS Billing and Cost Management and consider a budget alert. A budget alert is not a hard spending cap and may arrive after charges have accrued.

This article intentionally does not promise a zero-cost deployment. Actual costs depend on your account eligibility, Region, request volume, payload sizes, logging, and usage duration.

9. Implementation and Validation

The companion repository contains the Terraform implementation, Python Lambda handler, and supporting validation evidence.

The infrastructure was deployed in AWS using Terraform.
The deployment created 14 resources successfully.

Deployment validation

The following activities were performed:

Provisioned the infrastructure using Terraform.
Uploaded a test object to the S3 bucket.
Verified S3 event notification delivery through SQS.
Confirmed successful Lambda execution.
Inspected Lambda execution logs in CloudWatch.
Tested intentional processing failures.
Verified SQS retry behaviour.
Verified failed-message delivery to the DLQ.
Inspected CloudWatch DLQ alarm configuration and state.

Evidence

The repository includes:

Terraform deployment outputs.
Terraform infrastructure state.
Successful Lambda processing logs.
Failed processing logs.
SQS processing queue attributes.
DLQ attributes.
Lambda event source mapping configuration.
CloudWatch alarm information.
AWS Console screenshots.

See the evidence/ directory for the corresponding outputs.

Cleanup

After completing validation and collecting the required evidence,
the test infrastructure was removed using Terraform.

This prevents unnecessary ongoing resource charges.

The example should be deployed in a controlled AWS account.
Always review the Terraform plan and estimated costs before applying it.

10. When would I use this architecture?

This pattern is useful when a storage event should start background work and the system benefits from buffering and retry handling. Examples include:

Validating uploaded CSV files.
Starting image or document processing.
Running metadata extraction jobs.
Kicking off asynchronous data-import workflows.
Separating ingestion from downstream processing in internal tools.

It may be unnecessary for a tiny workflow where a direct S3-to-Lambda integration is sufficient. It also isn't a universal solution for long-running orchestration, strict ordering, or exactly-once business effects. Those requirements need additional design choices.

Conclusion

Combining Amazon S3, Amazon SQS, and AWS Lambda provides a practical way to build asynchronous, event-driven processing systems.

This architecture separates object storage from processing, provides message buffering, supports retries, and isolates repeatedly failing messages through a dead-letter queue.

The implementation also demonstrates why security, observability, and operational validation are important parts of cloud architecture.

By deploying and testing this architecture with Terraform, we can move beyond understanding individual AWS services and learn how they work together in a real cloud environment.

The complete implementation and validation evidence are available
in the companion GitHub repository.

References

Author note before publication: Add your own short paragraph about why you chose this pattern and what you learned while reviewing the implementation. Once you've actually tested it, update the implementation-status note with the specific checks you completed and any limitations you observed. Do not claim deployment or test results before doing them.

What Really Happens When You Type a URL in Your Browser?

Narendra | SRE — Sat, 23 May 2026 11:03:23 +0000

You type a URL. Press Enter. A beautiful website appears in milliseconds. But behind that single action, an entire chain of networking, security, and system-level operations takes place.

If you're learning DevOps, SRE, Backend Engineering, Networking, or System Design - this is one of the most important flows to understand.

🌍 Example

Suppose you type:

🌍https://google.com

what happens internally?
Let's break it down step-by-step.

⚡ High-Level Flow

Enter URL
↓
Browser Cache Check
↓
DNS Resolution
↓
TCP Handshake
↓
TLS/SSL Handshake
↓
HTTP Request Sent
↓
Load Balancer / Server
↓
Application Processing
↓
HTTP Response
↓
Browser Rendering

1. Browser Checks Cache First

Before contacting any server, the browser checks whether it already knows the IP address.
It checks:

✔ Browser cache
✔ OS DNS cache
✔ Router cache
✔ ISP cache
If found:

Use cache IP address

This avoids unnecessary DNS lookups and makes websites load faster.

2. DNS Resolution Happens

Humans remember names. Computer communicates using IP addresses.

So the browser asks:

What is the IP address of google.com?

The DNS server responds with something like:

142.250.xxx.xxx

This process is called DNS Resolution.

3. TCP Connection is Established

Now the browser knows the server IP.

Next step:

🤝 TCP 3-Way Handshake

Client → SYN
Server → SYN-ACK
Client → ACK

Now a reliable connection is established.

📌 Why TCP?

TCP guarantees:

✔ Reliable delivery
✔ Ordered packets
✔ Error checking
✔ Retransmission

This is extremely important for web applications.

4. TLS/SSL Handshake (HTTPS)

Since the URL starts with:

https://

A secure encrypted connection must be created.

During TLS handshake:

✔ Certificate validation
✔ Encryption negotiation
✔ Session key exchange
✔ Secure communication setup

Now communication becomes encrypted.

5. Browser Sends HTTP Request

Now the browser sends:

GET / HTTP/1.1
Host: google.com

The request also contains:

✔ Headers
✔ Cookies
✔ Authentication tokens
✔ User-Agent
✔ Compression support

6. Request Travels Across the Internet

The request moves through multiple layers:

Browser

→ Operating System
→ Router
→ ISP
→ Internet
→ CDN / Load Balancer
→ Web Server

In modern production systems:

Requests usually hit CDN or Load Balancer first

7. Server Processes the Request

Now the backend application starts working.

Possible operations:

✔ Run application logic
✔ Authenticate user
✔ Query databases
✔ Read cache
✔ Call microservices
✔ Fetch files

The server then prepares a response.

8. Server Sends HTTP Response

Example response:

HTTP/1.1 200 OK

The server may return:

✔ HTML
✔ CSS
✔ JavaScript
✔ Images
✔ JSON data

9. Browser Renders the Webpage

Now the browser starts rendering.

Internally it:

✔ Parses HTML
✔ Builds DOM tree
✔ Downloads CSS
✔ Executes JavaScript
✔ Paints pixels on screen

Finally:

🎉 The webpage appears

🎯 Interview Keywords You Must Know

DNS

TCP Handshake
TLS Handshake
HTTP Request/Response
Caching
CDN
Load Balancer
Browser Rendering
Latency
Microservices

🔥 If You Found This Useful

Follow for more content on:

DevOps
SRE
Linux
Kubernetes
Networking
System Design
Production Engineering

Happy Learning

DEV Community: Narendra | SRE

Automating AWS Infrastructure Deployment with Terraform

Introduction

1. What we are building

2. Architecture and deployment flow

Deployment Flow

Architecture

3. Repository structure

4. Reusable Terraform module

5. Variables and outputs

6. Terraform state management

7. Deployment workflow

8. Verify the deployed configuration

9. Automated checks with GitHub Actions

10. Cost, safety, and cleanup

11. Limitations and next steps

Conclusion

From Wildcards to Least Privilege: Building a No-Infrastructure AWS IAM Policy Security Review Workflow

Introduction

1. The problem

2. Architecture

3. Project files

4. Stage 1: Local static checks

5. Stage 2: AWS policy validation

6. Stage 3: IAM Policy Simulator

7. Evidence collection

8. Cost and safety considerations

9. Limitations and future work

Conclusion

Preventing Public Exposure Before It Happens: Event-Driven AWS Security Guardrails with Terraform, Lambda, and EventBridge

Introduction

1. The Problem

2. Architecture

3. Design Decisions

4. Implementation

4.1 Detection rule

4.2 Revoking only the violating rule

4.3 Claiming each event once

4.4 Infrastructure

5. Testing the Guardrail

6. Evidence

7. Cost

8. Limitations and Future Work

Conclusion

Detecting and Remediating AWS Infrastructure Drift Using Terraform and CloudTrail

Introduction

1. Understanding Infrastructure Drift

2. Architecture and Technology Stack

3. Provisioning Infrastructure Using Terraform

AMI discovery

EC2 resource

Terraform initialization and validation

4. Establishing the Initial Infrastructure State

5. Introducing Infrastructure Drift

6. Detecting Infrastructure Drift Using Terraform

7. Investigating Manual Changes Using AWS CloudTrail

8. Remediating Infrastructure Drift

9. Final Validation

10. Key Learnings

Conclusion

Secure AWS Automation with GitHub Actions and OpenID Connect (OIDC)

Secure AWS Automation with GitHub Actions and OpenID Connect (OIDC)

Introduction

1. Why long-lived access keys are risky

2. The building blocks

3. How the OIDC flow works

4. Configure the GitHub OIDC provider in AWS

5. Understand GitHub's OIDC subject (sub) claim

5.1 Legacy subject format

5.2 New immutable subject format

5.3 Which repositories use which format?

6. Verify the actual OIDC subject claim

7. Restrict the role trust policy

Example A: Legacy subject format

Example B: Immutable subject format

Terraform implementation

Branches and GitHub Environments

8. Grant only the permissions the workflow needs

9. Configure the GitHub Actions workflow

10. Security considerations

5. Understand GitHub's OIDC subject (`sub`) claim