DEV Community: Samuel Obeng

I Built a Website That Fails Over from AWS to Azure: A Terraform Walkthrough (and Every Error I Hit)

Samuel Obeng — Thu, 05 Nov 2026 10:03:01 +0000

I built this while working through a project module from Lucy Wang's advance cloud project course. This write-up is my own: my own Terraform files, updated for current provider versions, plus the real errors I hit and how I fixed them. Credit to the course for the project idea.

Repo: samobeng14Dev/aws_azure_multicloud_deployment_terraforms

Why I built this

I'm building a cloud and DevOps portfolio.
I kept reading about "disaster recovery" without ever seeing it work.
So I built a site where one cloud can go down and visitors still get the page.
I wrote down every place I got stuck, because those moments taught me the most.

What this project shows

Skill	Where it shows up
Infrastructure as Code	Terraform with two providers (AWS and Azure), staged applies, a provider alias, `for_each`, `dynamic` blocks
AWS	S3 static hosting, CloudFront, ACM certificates, Route 53 health checks and failover, IAM
Azure	Storage static website, resource providers, CLI authentication
Networking	DNS delegation, CNAME rules, TTLs, failover timing
Troubleshooting	14 problems diagnosed and fixed (see the troubleshooting table)
Security	No secrets in the repo, git-ignored state, credentials via CLI profiles

What you will build

A weather app served from two clouds at once:

Primary: AWS, with S3 static hosting behind CloudFront (HTTPS)
Secondary: Azure Blob Storage static website
Failover: Route 53 health checks check each site every 30 seconds. After 3 failed checks, DNS answers with the other cloud
Infrastructure: all of it in Terraform

You will learn:

Terraform with two providers
S3 static hosting and CloudFront with an ACM certificate
Azure Blob static websites
Route 53 failover, and how to test it

Before you start

Prerequisites

An AWS account and an Azure subscription
A registered domain (I used Namecheap)
Terraform, AWS CLI and Azure CLI
Basic command-line comfort. I was new to Terraform when I started.

Setup used in this post


AWS region	`eu-north-1` (the CloudFront certificate must be in `us-east-1`)
Azure region	`North Europe`
Shell	Windows PowerShell
Terraform	>= 1.5
Cost	Small if you destroy it quickly. The running costs are the Route 53 hosted zone (about $0.50/month) and two health checks. The domain costs extra.

Warning: this creates billable resources. Finish the cleanup in Step 10.

Names used (replace with your own)

Thing	Name
S3 bucket	`weather-tracker-<name>-<4 digits>` (globally unique)
Azure storage account	`weather<name><4 digits>` (3-24 characters, lowercase letters and digits only)
Azure resource group	`rg-weather-static-website`
Domain	`<your-domain>`
Failover hostname	`www.<your-domain>`
AWS CLI profile	`weather`

Project structure

aws_azure_multicloud_deployment_terraforms/
├── .gitignore
├── README.md
├── docs/            architecture.png, troubleshooting.md, screenshots/
├── website/         index.html, script.js, styles.css, assets/
└── terraform/
    ├── .terraform.lock.hcl      (commit this)
    ├── versions.tf  providers.tf  variables.tf  outputs.tf
    ├── aws_s3.tf  aws_cloudfront.tf  azure_storage.tf  dns_failover.tf
    ├── terraform.tfvars.example (commit)
    └── terraform.tfvars         (NEVER commit)

.gitignore

.terraform/
*.tfstate
*.tfstate.*
terraform.tfvars
secrets/
.env

Always run Terraform from the terraform/ folder. In the project root it fails with "No configuration files".

Step 0: Prerequisites

0.1 Install the tools

terraform version
aws --version
az version

Run the three commands above. Each should print a version.
Download and install any tool that is missing, then run it again.

0.2 AWS identity (no keys in code)

Create an IAM user for this project.
Attach these managed policies:
- AmazonS3FullAccess
- CloudFrontFullAccess
- AmazonRoute53FullAccess
- AWSCertificateManagerFullAccess
Create an access key:
- Sign in to the AWS Management Console and go to IAM → Users.
- Select the user and open Security credentials.
- Under Access keys, click Create access key and choose Command Line Interface (CLI).
- Copy the Access Key ID and Secret Access Key. AWS shows the secret only once.
Configure a named profile. When prompted, enter the key pair, your default region and an output format:

aws configure --profile weather
$env:AWS_PROFILE = "weather"
aws sts get-caller-identity

Check the output of aws sts get-caller-identity. The user name must be the IAM user you just created.

These broad managed policies are fine for a throwaway learning account. For anything real, write a least-privilege policy.

Important: aws configure without --profile overwrites your default credentials, which can silently affect your other projects. A named profile keeps each project's credentials separate.

My experience: I use a separate IAM user and CLI profile for every portfolio project. The first time I ran Terraform, it deployed successfully even though I had put no credentials in my code, and I wasn't sure which identity it was using. Now I run aws sts get-caller-identity before every terraform apply.

0.3 Azure identity

Terraform needs two things for Azure: a signed-in account and a subscription to deploy into.

az login
$env:ARM_SUBSCRIPTION_ID = (az account show --query id -o tsv)
az provider register --namespace Microsoft.Storage --wait
az provider show --namespace Microsoft.Storage --query registrationState -o tsv

What each command does:

az login opens a browser window and signs you in.
The second command stores your subscription ID in ARM_SUBSCRIPTION_ID, which the Azure provider reads.
The last two register the Microsoft.Storage resource provider. New subscriptions don't always have it enabled, and Azure can't create storage accounts without it. The final command should print Registered.

Important: environment variables last only for the current PowerShell window. In every new window, set the subscription ID again before running Terraform:
$env:ARM_SUBSCRIPTION_ID = (az account show --query id -o tsv)

Why: from AzureRM provider v4 onward, Terraform requires an explicit subscription ID. Without it, init can succeed and then planning or deploying fails because the provider doesn't know which subscription to use.

My experience: this was one of the easiest steps to overlook. Terraform initialised fine, but deployments failed because the Azure provider had no subscription context. Now I check ARM_SUBSCRIPTION_ID as soon as I open a new terminal.

0.4 How does Terraform know who I am?

	AWS	Azure
Identity	IAM user	My Entra ID login
Credential	Long-lived access key	Short-lived token from `az login`
Stored in	`~/.aws/credentials`	The Azure CLI's token cache
In the repo?	Never	Never

AWS: Terraform uses the credentials of the selected profile. AWS tools search in this order: environment variables, named profiles in ~/.aws/credentials, ~/.aws/config, then SSO.
Azure: Terraform reuses the session created by az login.
Result: no access keys, passwords or tokens in any Terraform file.

The routine:

Authenticate once with each CLI.
Let Terraform use those credentials.
Check your identity before every apply.

My experience: before I understood this, I searched my Terraform files for the "hidden" credentials it was using. There were none. Terraform was simply using the identities I had already signed in with.

0.5 Domain

Buy a domain (I used Namecheap).
Turn on WHOIS privacy and two-factor authentication.
Verify your contact email (the ICANN verification) within 15 days, or the domain can be suspended.

0.6 Account limits to expect

A new AWS account can fail CloudFront creation with "Your account must be verified before you can add new CloudFront resources".
Open a free Support case (Account and billing) early, and never paste your account ID in public.

My experience: this hit me at Step 4 and was my longest delay. I opened a case, kept building other parts while I waited, and it was approved. On a new account, open the case on day one.

Step 1: Provider setup

Four files set up Terraform itself. Create them in terraform/.

versions.tf pins the Terraform and provider versions.

terraform {
  required_version = ">= 1.5"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 5.0"
    }
    azurerm = {
      source = "hashicorp/azurerm"
      # Pin the MAJOR version you actually installed (check: terraform version).
      # Unpinned, a new major release changed resource syntax under me.
      version = "~> 5.0"
    }
  }
}

Pin the major version of azurerm that you actually installed. An unpinned provider changed its syntax under me (see Step 3).

providers.tf configures both clouds.

# Providers for the multi-cloud deployment.
# AWS hosts the S3 and CloudFront primary site, while Azure hosts the failover website.
provider "aws" {
  region = var.aws_region
}

provider "azurerm" {
  features {}
}

# CloudFront certificates must be requested in us-east-1 because CloudFront is a global service.
provider "aws" {
  alias  = "us_east_1"
  region = "us-east-1"
}

Neither provider has credentials in it (see 0.4).
The second AWS provider, with the alias us_east_1, exists only for the CloudFront certificate.

variables.tf holds the values you can change.

# Shared configuration values used across the AWS and Azure deployment resources.
# These values control the target regions, regional naming constraints, custom-domain setup,
# and the Route53 health check used for failover.

variable "aws_region" {
  description = "AWS region for the S3 bucket."
  type        = string
  default     = "eu-north-1"
}

variable "bucket_name" {
  description = "Globally unique S3 bucket name for the website."
  type        = string
}

variable "azure_location" {
  description = "Azure region for the resource group and storage account."
  type        = string
  default     = "North Europe"
}

variable "azure_storage_account_name" {
  description = "Globally unique Azure storage account name (3-24 chars, lowercase letters and digits only)."
  type        = string

  validation {
    condition     = can(regex("^[a-z0-9]{3,24}$", var.azure_storage_account_name))
    error_message = "Use 3-24 characters: lowercase letters and digits only, no hyphens."
  }
}

variable "domain_name" {
  description = "Your registered domain, without www."
  type        = string
}

variable "website_config_filename" {
  description = "Website config file to upload: config.js.example or your local config.js."
  type        = string
  default     = "config.js.example"

  validation {
    condition     = contains(["config.js", "config.js.example"], var.website_config_filename)
    error_message = "Set website_config_filename to either config.js or config.js.example."
  }
}

variable "enable_custom_domain" {
  description = "Stage switch: false = default CloudFront certificate, true = www custom domain with ACM certificate."
  type        = bool
  default     = false
}

variable "primary_health_check_path" {
  description = "Path the primary health check requests. Use a missing path to simulate an AWS outage."
  type        = string
  default     = "/"
}

bucket_name and azure_storage_account_name must be globally unique. The Azure name is validated: 3-24 lowercase letters and digits.
website_config_filename chooses which config file is uploaded: config.js.example (default) or your local config.js. Everything uploaded to S3 and Azure is publicly readable, so never put a real secret in either.
enable_custom_domain switches between the stage 1 and stage 2 setups (Steps 5 and 7).
primary_health_check_path is how I simulate an outage in Step 9.

terraform.tfvars is git-ignored. Keep a placeholder copy named terraform.tfvars.example.

aws_region                 = "eu-north-1"
bucket_name                = "weather-tracker-<name>-<4digits>"
azure_storage_account_name = "weather<name><4digits>"
domain_name                = "<your-domain>"
enable_custom_domain       = false

Step 2: S3 static website (primary origin)

aws_s3.tf creates the bucket and uploads the site.

# AWS S3 hosts the primary static website used by CloudFront.
# This file creates the public website bucket, enables website hosting, and uploads the
# static frontend files with the correct content types for browsers to render them properly.

locals {
  # Keep the website root files explicit; assets are discovered recursively below.
  site_dir       = "${path.module}/../website"
  config_js_path = "${local.site_dir}/${var.website_config_filename}"

  site_files = {
    "index.html" = "text/html"
    "styles.css" = "text/css"
    "script.js"  = "application/javascript"
    "config.js"  = "application/javascript"
  }

  site_file_sources = {
    for name in keys(local.site_files) :
    name => name == "config.js" ? local.config_js_path : "${local.site_dir}/${name}"
  }

  # Common static asset types; unknown extensions fall back to binary content.
  mime_types = {
    png  = "image/png"
    jpg  = "image/jpeg"
    jpeg = "image/jpeg"
    gif  = "image/gif"
    svg  = "image/svg+xml"
    webp = "image/webp"
    ico  = "image/x-icon"
  }
}

# Create the S3 bucket that hosts the static website content.
resource "aws_s3_bucket" "weather_app" {
  bucket = var.bucket_name
}

# Configure the bucket as a static website so it exposes the index document at the public endpoint.
resource "aws_s3_bucket_website_configuration" "site" {
  bucket = aws_s3_bucket.weather_app.id

  index_document {
    suffix = "index.html"
  }
}

# Allow public reads required for a static website served from S3.
resource "aws_s3_bucket_public_access_block" "public_access" {
  bucket                  = aws_s3_bucket.weather_app.id
  block_public_acls       = false
  block_public_policy     = false
  ignore_public_acls      = false
  restrict_public_buckets = false
}

# Grant anonymous visitors read access to website objects without requiring AWS credentials.
resource "aws_s3_bucket_policy" "public_read" {
  bucket     = aws_s3_bucket.weather_app.id
  depends_on = [aws_s3_bucket_public_access_block.public_access]

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Sid       = "PublicReadGetObject"
      Effect    = "Allow"
      Principal = "*"
      Action    = "s3:GetObject"
      Resource  = "${aws_s3_bucket.weather_app.arn}/*"
    }]
  })
}

# Upload the main HTML, CSS, and JavaScript files with the correct MIME types.
resource "aws_s3_object" "site_files" {
  for_each     = local.site_files
  bucket       = aws_s3_bucket.weather_app.id
  key          = each.key
  source       = local.site_file_sources[each.key]
  etag         = filemd5(local.site_file_sources[each.key])
  content_type = each.value
}

# Upload static assets under website/assets while keeping the same relative path structure.
resource "aws_s3_object" "site_assets" {
  for_each = fileset("${local.site_dir}/assets", "**")
  bucket   = aws_s3_bucket.weather_app.id
  key      = "assets/${each.value}"
  source   = "${local.site_dir}/assets/${each.value}"
  etag     = filemd5("${local.site_dir}/assets/${each.value}")

  # Derive the file extension from each path so nested assets also receive a correct MIME type.
  content_type = lookup(local.mime_types, lower(reverse(split(".", each.value))[0]), "application/octet-stream")
}

What it does:

Creates the bucket and turns on static website hosting
Allows public reads, because the bucket is a public website
Uploads the site files and assets with the correct content types

Before you apply:

Check the website JavaScript for hardcoded API keys. Everything in website/ becomes public.

Run:

cd <project>\terraform
terraform init
terraform validate
terraform plan
terraform apply
aws s3 ls s3://<bucket>/ --recursive

Verify:

The site opens at http://<bucket>.s3-website.eu-north-1.amazonaws.com. S3 website endpoints are HTTP only.
The bucket lists your site files.

What went wrong for me:

"No configuration files." I ran terraform init in the project root, twice. Terraform only reads .tf files from the current folder, so I now open my terminal directly in terraform/.

403 on s3:CreateBucket. My IAM user existed (authentication worked) but nothing allowed S3 (authorization failed). Attaching AmazonS3FullAccess and waiting a minute fixed it.

Step 3: Azure static website (secondary)

azure_storage.tf creates the resource group, the storage account, and uploads the same site to the $web container.

# Azure Blob Storage hosts the secondary site used during Route53 failover.
# This configuration enables static website hosting and uploads the built site files into the
# Azure-managed "$web" container that serves the application.

resource "azurerm_resource_group" "rg" {
  name     = "rg-weather-static-website"
  location = var.azure_location
}

resource "azurerm_storage_account" "storage" {
  name                     = var.azure_storage_account_name
  resource_group_name      = azurerm_resource_group.rg.name
  location                 = azurerm_resource_group.rg.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
  account_kind             = "StorageV2"

  # Static website hosting and public traffic are enabled for the failover endpoint.
  public_network_access = "Enabled"

  custom_domain {
    name          = "www.${var.domain_name}"
    use_subdomain = true # indirect (asverify) validation
  }

  depends_on = [aws_route53_record.azure_asverify]
}

# Enabling static website hosting creates the special "$web" container that serves the site.
resource "azurerm_storage_account_static_website" "site" {
  storage_account_id = azurerm_storage_account.storage.id
  index_document     = "index.html"
}

locals {
  # Resource Manager ID of the Azure-managed "$web" container used for static hosting.
  web_container_id = "${azurerm_storage_account.storage.id}/blobServices/default/containers/$web"
}

resource "azurerm_storage_blob" "site_files" {
  for_each = local.site_files

  name                 = each.key
  storage_container_id = local.web_container_id
  type                 = "Block"
  content_type         = each.value
  source               = local.site_file_sources[each.key]
  content_md5          = filemd5(local.site_file_sources[each.key])

  depends_on = [azurerm_storage_account_static_website.site]
}

resource "azurerm_storage_blob" "site_assets" {
  for_each = fileset("${local.site_dir}/assets", "**")

  name                 = "assets/${each.value}"
  storage_container_id = local.web_container_id
  type                 = "Block"
  content_type         = lookup(local.mime_types, lower(reverse(split(".", each.value))[0]), "application/octet-stream")
  source               = "${local.site_dir}/assets/${each.value}"
  content_md5          = filemd5("${local.site_dir}/assets/${each.value}")

  depends_on = [azurerm_storage_account_static_website.site]
}

What it does:

Creates the resource group and storage account
Turns on static website hosting, which creates the special $web container
Uploads the same files as S3
The custom_domain block lets Azure accept www.<your-domain>. It is explained in Step 8.

Heads-up: this file references the azure_asverify record from dns_failover.tf (Step 5), so Terraform can't plan it alone. Either create the Step 5 file first, or comment out the custom_domain block and the depends_on until Step 8.

Run (with ARM_SUBSCRIPTION_ID set and az login done):

terraform init -upgrade
terraform validate
terraform plan
terraform apply
terraform output azure_website_url

Verify:

Open the URL. HTTPS works on the Azure endpoint.
In the Azure portal: Storage accounts → your account → Data storage → Containers → $web.
The URL is also under Data management → Static website.

What went wrong for me:

MissingSubscriptionRegistration for Microsoft.Storage. A fresh subscription doesn't have every provider enabled. The az provider register command in Step 0.3 fixes it.

My code stopped validating. I hadn't pinned azurerm, so I got the newest major release (5.x). The inline static_website block was removed and blobs now take a storage_container_id. I read the docs for the version I actually had, rewrote the file (the version above), and now pin ~> 5.0.

"Connection may have been reset" right after the storage account was created. Its DNS name hadn't resolved yet. I waited a minute and re-ran terraform apply.

Step 4: CloudFront (primary, stage 1)

aws_cloudfront.tf puts HTTPS and caching in front of S3.

# CloudFront distribution for the static website.
# This layer serves the site globally over HTTPS, caches static content at the edge,
# and optionally uses a custom domain certificate when the domain is enabled.

resource "aws_cloudfront_distribution" "site" {
  enabled             = true
  default_root_object = "index.html"
  price_class         = "PriceClass_100"

  # Attach the custom domain only when the user has enabled domain hosting.
  aliases = var.enable_custom_domain ? ["www.${var.domain_name}"] : []

  # The S3 website endpoint is configured for website hosting and only supports HTTP.
  # CloudFront therefore uses an HTTP origin connection while still serving HTTPS to viewers.
  origin {
    domain_name = aws_s3_bucket_website_configuration.site.website_endpoint
    origin_id   = "s3-website"

    custom_origin_config {
      http_port              = 80
      https_port             = 443
      origin_protocol_policy = "http-only"
      origin_ssl_protocols   = ["TLSv1.2"]
    }
  }

  # Serve the site with a simple, cache-friendly policy for static content.
  default_cache_behavior {
    target_origin_id       = "s3-website"
    viewer_protocol_policy = "redirect-to-https"
    allowed_methods        = ["GET", "HEAD"]
    cached_methods         = ["GET", "HEAD"]
    cache_policy_id        = "658327ea-f89d-4fab-a63d-7e88639e58f6" # Managed-CachingOptimized
  }

  # No geo-blocking restrictions are applied for this deployment.
  restrictions {
    geo_restriction {
      restriction_type = "none"
    }
  }

  # Use the default CloudFront certificate when no custom domain is configured.
  dynamic "viewer_certificate" {
    for_each = var.enable_custom_domain ? [] : [1]
    content {
      cloudfront_default_certificate = true
    }
  }

  # Use the ACM certificate and custom TLS settings when a custom domain is configured.
  dynamic "viewer_certificate" {
    for_each = var.enable_custom_domain ? [1] : []
    content {
      acm_certificate_arn      = aws_acm_certificate_validation.site[0].certificate_arn
      ssl_support_method       = "sni-only"
      minimum_protocol_version = "TLSv1.2_2021"
    }
  }
}

What it does:

Uses the S3 website endpoint as the origin. That endpoint speaks only HTTP, so CloudFront connects over HTTP.
Redirects viewers from HTTP to HTTPS
Uses the default CloudFront certificate until you enable the custom domain (Step 7)
Takes 5-15 minutes to deploy

What went wrong for me: apply returned 403 "Your account must be verified before you can add new CloudFront resources". Terraform can't fix this. I opened the Support case from Step 0.6, built other parts while I waited, and the distribution created normally once AWS approved it.

Step 5: Route 53 zone, health checks and failover records

dns_failover.tf

# DNS and failover configuration for the global website.
# This file creates the hosted zone, requests the public certificate, validates it,
# and routes traffic between the primary AWS endpoint and the secondary Azure endpoint.

locals {
  site_host = "www.${var.domain_name}"
}

# Hosted zone for the custom domain used by the site.
resource "aws_route53_zone" "main" {
  name = var.domain_name
}

# ACM certificate for the CloudFront custom domain. This certificate must be issued in us-east-1
# because CloudFront is a global AWS service and expects certificates in that region.
resource "aws_acm_certificate" "site" {
  count             = var.enable_custom_domain ? 1 : 0
  provider          = aws.us_east_1
  domain_name       = local.site_host
  validation_method = "DNS"

  lifecycle {
    create_before_destroy = true
  }
}

# Create the DNS validation records required to verify the ACM certificate.
resource "aws_route53_record" "cert_validation" {
  for_each = var.enable_custom_domain ? {
    for dvo in aws_acm_certificate.site[0].domain_validation_options : dvo.domain_name => {
      name   = dvo.resource_record_name
      type   = dvo.resource_record_type
      record = dvo.resource_record_value
    }
  } : {}

  zone_id         = aws_route53_zone.main.zone_id
  name            = each.value.name
  type            = each.value.type
  records         = [each.value.record]
  ttl             = 60
  allow_overwrite = true
}

# Wait for the ACM certificate to be valid before using it in CloudFront.
resource "aws_acm_certificate_validation" "site" {
  count                   = var.enable_custom_domain ? 1 : 0
  provider                = aws.us_east_1
  certificate_arn         = aws_acm_certificate.site[0].arn
  validation_record_fqdns = [for r in aws_route53_record.cert_validation : r.fqdn]
}

# Health checks allow Route53 to fail over automatically when the primary endpoint becomes unavailable.
resource "aws_route53_health_check" "primary" {
  type              = "HTTPS"
  fqdn              = aws_cloudfront_distribution.site.domain_name
  port              = 443
  resource_path     = var.primary_health_check_path
  request_interval  = 30
  failure_threshold = 3

  tags = { Name = "weather-primary-aws" }
}

resource "aws_route53_health_check" "secondary" {
  type              = "HTTPS"
  fqdn              = azurerm_storage_account.storage.primary_web_host
  port              = 443
  resource_path     = "/"
  request_interval  = 30
  failure_threshold = 3

  tags = { Name = "weather-secondary-azure" }
}

# Route53 failover records share the same host name and type, but are distinguished by their set identifier.
# The primary record points to the AWS CloudFront distribution and the secondary record points to Azure.
resource "aws_route53_record" "primary" {
  zone_id         = aws_route53_zone.main.zone_id
  name            = local.site_host
  type            = "CNAME"
  ttl             = 60
  records         = [aws_cloudfront_distribution.site.domain_name]
  set_identifier  = "primary"
  health_check_id = aws_route53_health_check.primary.id

  failover_routing_policy {
    type = "PRIMARY"
  }
}

resource "aws_route53_record" "secondary" {
  zone_id         = aws_route53_zone.main.zone_id
  name            = local.site_host
  type            = "CNAME"
  ttl             = 60
  records         = [azurerm_storage_account.storage.primary_web_host]
  set_identifier  = "secondary"
  health_check_id = aws_route53_health_check.secondary.id

  failover_routing_policy {
    type = "SECONDARY"
  }
}

# Lets Azure verify that we own www.<domain> before it accepts requests for that name.
resource "aws_route53_record" "azure_asverify" {
  zone_id = aws_route53_zone.main.zone_id
  name    = "asverify.www.${var.domain_name}"
  type    = "CNAME"
  ttl     = 60
  records = ["asverify.${var.azure_storage_account_name}.blob.core.windows.net"]
}

What it does:

Creates the hosted zone for your domain
Requests the ACM certificate in us-east-1 and validates it by DNS (used in stage 2)
Creates one health check per cloud: HTTPS, every 30 seconds, 3 failures
Creates two www CNAME records, PRIMARY (CloudFront) and SECONDARY (Azure)
Creates the asverify record Azure uses to verify domain ownership (Step 8)

A design lesson: failover records must have the same name and type. The version I started from used different names, which isn't failover at all. Azure can't be an alias target, so both records are CNAMEs on www. The trade-off: the bare domain can't take part, because a CNAME can't sit at the root.

outputs.tf

# Deployment endpoints and DNS values exported for use after Terraform apply.
# These outputs provide the direct URLs for each platform and the DNS details needed
# to complete the custom domain setup.

output "s3_website_url" {
  description = "Direct S3 static website endpoint over HTTP."
  value       = "http://${aws_s3_bucket_website_configuration.site.website_endpoint}"
}

output "azure_website_url" {
  description = "Azure static website endpoint served over HTTPS."
  value       = azurerm_storage_account.storage.primary_web_endpoint
}

output "azure_website_host" {
  description = "Azure website hostname used as the secondary failover target."
  value       = azurerm_storage_account.storage.primary_web_host
}

output "cloudfront_url" {
  description = "Primary site URL served through CloudFront over HTTPS."
  value       = "https://${aws_cloudfront_distribution.site.domain_name}"
}

output "route53_name_servers" {
  description = "Name servers to configure at the registrar for the custom domain."
  value       = aws_route53_zone.main.name_servers
}

output "site_url" {
  description = "Custom domain URL for the production website after DNS is configured."
  value       = "https://www.${var.domain_name}"
}

Run stage 1 (enable_custom_domain = false):

terraform validate
terraform plan
terraform apply
terraform output cloudfront_url
terraform output route53_name_servers

Verify:

cloudfront_url opens the app over HTTPS.
route53_name_servers prints four nameservers. You need them in Step 6.
In Route 53, both health checks show Healthy.

Step 6: Point the domain at Route 53

In Namecheap: Domain List → Manage → Nameservers → Custom DNS.
Enter the four awsdns nameservers. No trailing dot, no spaces.
Save, then check:

Resolve-DnsName <your-domain> -Type NS -Server 8.8.8.8

Wait until it lists the awsdns names. This can take up to about 30 minutes.
Do not start stage 2 before this works.
Every destroy and re-apply creates a new zone with new nameservers, so repeat this step each time.

What went wrong for me:

After switching nameservers I kept seeing my registrar's parking page. Google's resolver (-Server 8.8.8.8) already had the right answer, so it was cached DNS on my own PC. ipconfig /flushdns and a private window fixed it.

I also had to verify my domain contact email (ICANN) at the registrar, or the domain could be suspended.

Step 7: Stage 2, custom domain with HTTPS

Make sure the IAM user has AWSCertificateManagerFullAccess.
Set enable_custom_domain = true in terraform.tfvars.
Run:

terraform apply

What happens:

Terraform requests the certificate in us-east-1
It creates the DNS validation record and waits for validation (2-10 minutes)
It updates CloudFront with the alias and certificate (5-15 minutes)

Verify:

https://www.<your-domain> loads with a padlock.
If you still see an old page, run ipconfig /flushdns.

What went wrong for me:

RequestCertificate failed with AccessDenied. I had given my IAM user S3, CloudFront and Route 53 permissions but forgot ACM. I attached AWSCertificateManagerFullAccess and re-ran.

CloudFront only accepts certificates from us-east-1, even though my infrastructure is in eu-north-1. That is why providers.tf has a second, aliased AWS provider.

Step 8: Make Azure accept the custom domain

Without this, http://www.<your-domain> returns 400 InvalidUri when DNS points at Azure. Azure rejects requests whose Host header it doesn't know.

In the code above, two things fix it:

The azure_asverify CNAME in dns_failover.tf proves you own the name.
The custom_domain block and depends_on in azure_storage.tf register www.<your-domain> on the storage account.

If you commented them out in Step 3, uncomment them now, then:

terraform plan
terraform apply

Azure checks the asverify record when it applies the custom domain, so complete Step 6 first.
Before applying, plan should show the storage account updated in place, not replaced.
Limit: this works over HTTP only. HTTPS with a custom domain on a plain storage account shows a certificate warning. The production fix is Azure Front Door.

What went wrong for me: opening the failover hostname while it pointed at Azure first returned 400 InvalidUri. After the custom domain was registered with the asverify record and the custom_domain block, the Azure site loaded on www.<your-domain>.

Step 9: Test the failover

Before. In Route 53 → Health checks, both are Healthy. Then query one of your four nameservers, which avoids local caching:

   (Resolve-DnsName www.<domain> -Type CNAME -Server <one-awsdns-nameserver>).NameHost

Expect the CloudFront name.

Break the primary to fake an outage, in the AWS console:
- Go to Route 53 → Health checks and select weather-primary-aws.
- Click Edit and change Path from / to /does-not-exist.
- Save. That path doesn't exist, so the check starts failing.

The Terraform alternative does the same thing from code:

   terraform apply -var="primary_health_check_path=/does-not-exist"

Watch the health check. Refresh the Health checks page. Allow 2-3 minutes (3 failures x 30 seconds). The primary changes from Healthy to Unhealthy.
Confirm which cloud is answering. Your browser can't tell you, because both copies look the same. Ask DNS instead:

   1..12 | ForEach-Object { (Resolve-DnsName www.<domain> -Type CNAME -Server <one-awsdns-nameserver>).NameHost; Start-Sleep 20 }

Once the primary is Unhealthy, expect the answer to change from the CloudFront name to the Azure hostname.

Fail back. Edit the health check path back to / (or run terraform apply without the -var). After 1-3 minutes the primary is Healthy again.

What I observed: after I changed the health check path in the Route 53 console, the primary check went from Healthy to Unhealthy, and the site stayed available. That’s what failover is meant to do: visitors don’t see an outage.

A limit: DNS failover is recovery in minutes, not instant. Detection takes a few health-check intervals, and clients can cache the old answer for up to the record's TTL (60 seconds here).

Step 10: Clean up

terraform destroy

CloudFront is slow to delete (10-15 minutes).

Then verify:

terraform state list
aws s3 ls
aws cloudfront list-distributions --query "DistributionList.Items[].Id"
az group list -o table

Also check:

The ACM certificate in us-east-1
Route 53 hosted zones and health checks

Terraform does not remove:

The domain itself. Turn off auto-renew at Namecheap if you don't want to keep it.
The domain's nameservers. Point them back at your registrar's default DNS.
The IAM user and its access keys. Deactivate or delete them.
Local terraform.tfstate (git-ignored)

Check Billing and Cost Management the next day.

Troubleshooting: every error I hit

Symptom	Cause	Fix
`No configuration files`	Ran Terraform in the project root	`cd` into `terraform/`
403 on `s3:CreateBucket`	IAM user has no S3 permission (authorization, not authentication)	Attach `AmazonS3FullAccess`, wait a minute
Terraform acts as an unexpected user	`aws configure` overwrote `default`, or stale `AWS_*` variables	Use named profiles and `$env:AWS_PROFILE`; check `aws sts get-caller-identity`
Azure: subscription ID required	azurerm v4+ needs it	`$env:ARM_SUBSCRIPTION_ID = (az account show --query id -o tsv)`
Azure: `MissingSubscriptionRegistration`	Resource provider not registered on a new subscription	`az provider register --namespace Microsoft.Storage --wait`
`static_website` not expected / `storage_container_id` required	azurerm 5.x syntax changed; version not pinned	Use `azurerm_storage_account_static_website` and `storage_container_id`; pin `~> 5.0`
Azure: connection reset after creating the account	New account's DNS name not resolving yet	Wait a minute, re-run apply
CloudFront 403: account must be verified	New-account restriction	Support case (Account and billing), wait
ACM `RequestCertificate` AccessDenied	IAM user lacks ACM permission	Attach `AWSCertificateManagerFullAccess`
Certificate not offered for CloudFront	Created outside `us-east-1`	Use the `aws.us_east_1` provider alias for ACM
Failover records do nothing	Primary and secondary had different name or type	Same name and type (CNAME on `www`) for both
Browser shows registrar parking page after nameserver change	Cached DNS on my PC	`ipconfig /flushdns`, private window, check with `-Server 8.8.8.8`
Azure `400 InvalidUri` on the custom domain	Azure rejects an unregistered Host header	Step 8 (`asverify` record plus `custom_domain`)
Domain contact alert at the registrar	ICANN contact email not verified	Verify the email at the registrar

Security checklist

No keys, account IDs, subscription IDs or Support case IDs in code, docs or screenshots
terraform.tfvars, .terraform/ and *.tfstate* are git-ignored. .terraform.lock.hcl is committed.
Credentials come from CLI profiles and az login, not from Terraform variables
Check the website files for hardcoded API keys before committing, because everything in website/ is public
WHOIS privacy and two-factor authentication on the registrar account
Delete or deactivate unused IAM access keys after the project

Known limits

Failover is not instant. Detection takes a few health-check intervals, then clients can cache the old answer for up to the TTL.
Only www fails over. A CNAME can't sit at the bare domain.
Azure custom domain is HTTP only on a plain storage account.
State is local. That's fine for a demo. Production uses a remote backend with locking.

What I learned

Terraform is only as powerful as the identity behind it. Know your credential chain before you apply.
Read the docs for the version you installed. An unpinned provider changed its syntax under me.
New cloud accounts have hidden walls (verification, provider registration). Find them on day one.
A failover diagram isn't a failover. I tested it by breaking the primary health check and watching what happened.

What I'd do next

Azure Front Door in front of the Azure site, for HTTPS on a custom domain
Remote state with locking
OIDC federation from CI, so no long-lived keys exist
Refactor into Terraform modules

Let's connect

I'm actively building cloud and DevOps projects and I'm open to opportunities. If you're hiring, or you're learning Terraform and got stuck where I did, reach out.

GitHub: samobeng14Dev
LinkedIn: samobeng14dev

Built from a module in Lucy Wang's advanced cloud project course, with my own changes and write-up.

Building a Near Real-Time Stock Analytics Pipeline on AWS (and the Silent Bug That Hid My Data)

Samuel Obeng — Thu, 05 Nov 2026 10:02:39 +0000

Building a Near Real-Time Stock Data Pipeline on AWS: What I Learned and Debugged

Tech stack: Python, boto3, Amazon Kinesis Data Streams, AWS Lambda, DynamoDB, Amazon S3, AWS Glue, Amazon Athena, Amazon SNS, IAM, CloudWatch

Repository: amazon-kinesis-stock-analyst-project

Introduction

I wanted to understand how data moves through a cloud pipeline, from an external API to a streaming service, through serverless processing, into storage, and finally into analytics and notifications.

So I built a stock market data pipeline using Python and AWS.

The Python script sends stock price records to Amazon Kinesis Data Streams. AWS Lambda processes the incoming records, storing processed data in DynamoDB and raw data in Amazon S3. AWS Glue and Amazon Athena make the archived data available for SQL queries, while DynamoDB Streams, another Lambda function, and Amazon SNS provide the infrastructure for trend-based email alerts.

This is a near-real-time pipeline, not a real-time trading system. Records are sent every 30 seconds and processed in small batches.

Credit: I built this project while taking a project course by Lucy Wang. The course provided the project brief and starter code, including the Lambda code. This post isn't a walkthrough. It's my field notes on the changes I made, what broke, and how I debugged it. If you want the guided version, I recommend the course.

The Architecture

Here's how the data moves through the system:

Stock API
    |
    v
Python Producer
    |
    v
Kinesis Data Streams
    |
    v
Processing Lambda
    |
    +------> DynamoDB (processed data)
    |
    +------> S3 (raw data)
                 |
                 v
              AWS Glue
                 |
                 v
             Amazon Athena
             (SQL analytics)

DynamoDB Streams
    |
    v
Alert Lambda
    |
    v
Amazon SNS
    |
    v
Email notification

Each service has a specific responsibility:

Service	Purpose
Kinesis Data Streams	Receives incoming stock price records.
AWS Lambda	Processes records, calculates metrics, and identifies potential anomalies.
DynamoDB	Stores processed records for fast lookups.
Amazon S3	Stores raw records for historical analysis.
AWS Glue	Catalogs the data stored in S3.
Amazon Athena	Queries the archived data using SQL.
DynamoDB Streams	Captures changes to DynamoDB items.
Lambda + SNS	Supports trend detection and email notifications.
CloudWatch	Provides logs for monitoring and debugging.

I built and tested the project on Windows using PowerShell and a Python virtual environment, with my AWS resources configured in eu-north-1.

What I Changed Beyond the Starter Code

Since I planned to publish the project publicly, I made several changes to the configuration and security setup.

1. Moved configuration out of the code

The sample script contained hardcoded configuration values, including a placeholder stream name.

I changed it to read settings from environment variables loaded from a local .env file.

For example:

STREAM_NAME = os.getenv(
    "KINESIS_STREAM_NAME",
    "stock-market-stream",
)

AWS_REGION = os.getenv("AWS_REGION")

I also added a .env.example file to document the expected configuration without exposing my local values.

The actual .env file is excluded from version control through .gitignore.

2. Kept AWS credentials out of the repository

The Python script contains no AWS access keys. Instead, boto3 retrieves credentials through the AWS CLI configuration.

I also restricted the producer's IAM permissions to writing records to the intended Kinesis stream rather than granting unnecessary access to other services.

3. Reduced unnecessary logging

The original script printed the full Kinesis response, including request metadata.

I reduced the output to useful information, such as the stock symbol, price, and shard identifier. This made the logs easier to read and safer to share when troubleshooting.

4. Added secret-protection measures

Before publishing the repository, I added:

A .gitignore entry for .env.
Secret scanning.
GitHub push protection.

These safeguards help reduce the risk of accidentally committing credentials. They complement, rather than replace, appropriate IAM permissions and careful review of commits.

The Bug That Cost Me an Hour

This was the most valuable part of the project for me.

My Python script printed a success message every 30 seconds:

INFO Sent AAPL price=330.32 shard=shardId-000000000002

But nothing appeared downstream.

DynamoDB had no processed items.
CloudWatch had no expected Lambda log streams.
Kinesis Data Viewer showed no records in the intended stream.
The Lambda event source mapping reported no records processed.

Everything looked correct at first, so I started debugging from the middle of the pipeline.

Step 1: Tested the processing Lambda independently

I sent a manually constructed test event shaped like a Kinesis event to the processing Lambda.

The function returned a successful response, but that alone wasn't enough. My handler caught some errors internally, meaning an invocation could appear successful even when processing had failed.

I checked the logs and verified that the test record appeared in DynamoDB and that the expected file was written to S3.

That gave me evidence that the processing and storage path could work independently.

Step 2: Checked the Lambda trigger

Next, I checked the event source mapping connecting Kinesis to Lambda.

It was enabled and attached to the intended stream. The status indicated that no records had been processed.

Combined with the empty Kinesis Data Viewer, this pointed me toward the ingestion side rather than immediately blaming the Lambda function.

My question changed from:

Why isn't Lambda running?

to:

Are my records reaching the Kinesis stream that Lambda is reading?

Step 3: Investigated the Python producer

I went back to the producer and checked its environment.

I found this error:

ModuleNotFoundError: No module named 'dotenv'

The python-dotenv package wasn't installed in my virtual environment.

My dependency list had been generated using pip freeze, but it didn't include the package the script required.

There was another problem: the import was wrapped in an except ImportError block that silently ignored the failure.

As a result, the script continued without loading the .env file.

That meant the intended AWS region wasn't being loaded from the project's configuration. Boto3 could instead use the region configured through the AWS CLI or another applicable AWS configuration source.

The producer could therefore appear healthy while sending records somewhere other than the stream I was monitoring.

Step 4: Fixed the configuration and verified the pipeline

I installed the missing dependency, restarted the producer, and monitored the pipeline in sequence.

My verification checklist was:

Confirm records appeared in Kinesis.
Confirm the processing Lambda received records.
Confirm processed items appeared in DynamoDB.
Confirm raw data was written to S3.

This helped me isolate the problem instead of changing multiple AWS resources unnecessarily.

The biggest lesson: A successful message from the producer doesn't prove that the intended consumer received the data.

Technical Decisions I Can Explain

Raw data in S3, processed data in DynamoDB

I used S3 for raw records and DynamoDB for processed records.

S3 provides economical object storage for historical data, while DynamoDB supports fast item-level lookups. Keeping the raw data also provides a useful starting point for historical analysis.

Athena instead of maintaining a query database

AWS Glue and Athena allowed me to query the data stored in S3 using SQL without maintaining a continuously running database.

Athena charges according to the amount of data scanned, so query design and data layout still matter when controlling costs.

`Decimal` instead of `float` for DynamoDB

Python's float type isn't supported by boto3's DynamoDB serializer for numeric attributes.

I used Decimal for numeric values written to DynamoDB to avoid serialization errors and handle decimal values appropriately.

Small batches for a low-volume stream

The producer sends records every 30 seconds, and the Lambda event source mapping uses a small batch size.

A batch size of two doesn't guarantee that Lambda runs exactly once every minute. Actual invocation timing also depends on polling, record arrival, and the event source mapping's configuration.

The goal was to keep the pipeline simple while learning how batching affects processing latency and invocation overhead.

Kinesis on-demand mode

I chose on-demand mode so I wouldn't need to determine a fixed shard capacity upfront.

However, on-demand doesn't mean free. Costs depend on the pricing model and usage, so unused resources still need to be cleaned up.

What I Learned About the Data

Repeated prices don't necessarily mean a broken pipeline

During testing, I noticed the same stock price appearing repeatedly.

The script fetched daily stock bars rather than continuously changing market prices. Sending the same daily value every 30 seconds didn't make the underlying data update every 30 seconds.

The US market was also closed during my test, contributing to the repeated values.

This taught me an important distinction: ingestion frequency is not the same as data freshness.

Trend alerts need meaningful price movement

The alert logic compares a five-record moving average with a 20-record moving average.

A crossover can indicate a potential trend change, but repeated or nearly identical prices provide little movement for testing this logic.

A better test would use data containing meaningful price variation and known crossover scenarios, allowing the alert conditions to be validated deliberately.

Timestamp formats matter

I also identified a potential issue with the alert function's time-window logic.

The processing path writes timestamps in a format like this:

2026-10-02T12:00:00Z

The alert function constructs its cutoff using a space between the date and time instead of the T.

When timestamps are compared as strings, inconsistent formats can produce incorrect time-window results. The exact impact depends on the stored values and comparison logic.

A safer implementation would use consistent UTC timestamps and compare normalized values or actual datetime objects.

It's a reminder that a pipeline can successfully move data while still having problems with the logic applied to that data.

Verifying the Pipeline

The following screenshots provide evidence of the main stages of the project.

1. Kinesis Data Streams

This is where I checked whether the producer's records were reaching the intended stream.

2. DynamoDB

The processed records should appear in DynamoDB after the processing Lambda handles incoming events.

3. Amazon S3

The raw records are stored in S3 for historical analysis.

4. Amazon Athena

Athena provides a way to query the archived data using SQL.

These screenshots show different parts of the system. Together, they help readers follow the data from ingestion to storage and analytics.

Cleanup and Cost Awareness

After completing the project, I removed the AWS resources to avoid unnecessary ongoing charges.

My cleanup covered the producer and the AWS resources used by the pipeline, including Kinesis, Lambda, DynamoDB, SNS, S3, Glue, CloudWatch log groups, and associated IAM roles.

The important lesson is that stopping an application doesn't automatically remove the cloud resources it uses.

Before deleting resources, preserve any evidence or data you want to keep and check for dependencies. After cleanup, verify that the intended resources have actually been removed.

What I'd Improve Next

If I extended the project, I'd focus on six areas:

Use fresher market data. Introduce a suitable source of more frequent price updates so the trend logic can be tested with meaningful variation.
Fix and test the alert time window. Normalize timestamps and create test cases for expected alert and non-alert scenarios.
Apply least-privilege IAM to both Lambda functions. Limit each function to the actions and resources it actually needs.
Externalize remaining configuration. Move bucket names and SNS topic ARNs into environment-specific configuration.
Introduce Terraform. Make the infrastructure reproducible and simplify setup and teardown.
Improve failure handling. Configure suitable retries, monitoring, and failure destinations or dead-letter queues where supported.

These improvements would make the project easier to reproduce, maintain, and extend.

What I Learned

This project taught me more than how to connect AWS services. It taught me how to investigate a distributed system when one component appears healthy but the overall pipeline isn't behaving as expected.

My biggest takeaways:

Trace data end to end. Verify the producer, stream, consumer, and storage independently.
Make configuration failures visible. Missing dependencies and configuration should never fail silently.
Test components in isolation. A controlled Lambda test can help rule out entire sections of a pipeline.
Understand the data, not just the infrastructure. Frequent ingestion doesn't automatically mean fresh data.
Build with security and cost in mind. Restrict permissions, protect credentials, and clean up unused resources.

I'm continuing to build my practical AWS skills, and this project gave me a clearer understanding of how streaming, serverless processing, storage, analytics, and event-driven notifications fit together.

Explore the Project

GitHub: amazon-kinesis-stock-analyst-project
LinkedIn: samobeng14dev

Technologies used: Python, boto3, Amazon Kinesis Data Streams, AWS Lambda, DynamoDB, DynamoDB Streams, Amazon S3, AWS Glue, Amazon Athena, Amazon SNS, IAM, CloudWatch, Git, and GitHub.

What Building My First AWS CI/CD Pipeline Taught Me (and Where It Broke)

Samuel Obeng — Thu, 05 Nov 2026 10:01:07 +0000

From Git Push to AWS Deployment: My Docker CI/CD Pipeline for the 2048 Game

Tech stack: Docker, AWS CodePipeline, AWS CodeBuild, Amazon ECR, Amazon ECS Fargate, IAM, Amazon S3, GitHub

Project repository: aws-ecs-ci-cd-2048-game

Introduction

For a while, my Docker experience stopped at my laptop. I could build an image and run a container locally, but I wanted to understand what happens when a team needs to ship changes automatically.

So I built a CI/CD pipeline on AWS where a GitHub push triggers a build, publishes a Docker image, and deploys the updated application to Amazon ECS Fargate.

I used the open-source 2048 game because the application itself is relatively simple. That let me focus on the infrastructure, the deployment process, and the problems that came up along the way.

The goal: push code, let the pipeline do the work, and verify that the change is live without manually rebuilding or deploying the application.

Credit: I built this project while taking project-based cloud course by Lucy Wang. The course supplied the project brief and starter files. This post isn't a walkthrough. It's my field notes on what went wrong, what I changed, and what I learned. If you want the guided version, I recommend the course.

The Shape of the System

GitHub Push
    |
    v
AWS CodePipeline
    |
    v
AWS CodeBuild
    |
    | Build and tag Docker image
    v
Amazon ECR
    |
    | Pull image and deploy
    v
Amazon ECS Fargate
    |
    v
Browser

Here's what each service does:

Service	Responsibility
GitHub	Stores the application source code and triggers the workflow when changes are pushed.
AWS CodePipeline	Orchestrates the source, build, and deployment stages.
AWS CodeBuild	Builds the Docker image and prepares the deployment configuration.
Amazon ECR	Stores the versioned Docker images in a private container registry.
Amazon ECS Fargate	Runs the application container without requiring me to manage the underlying servers.
IAM	Controls which AWS resources and actions the pipeline can access.
Amazon S3	Supports storage of pipeline artifacts.

I built and tested the project on Windows using PowerShell, with the AWS resources deployed in eu-north-1.

The architecture is straightforward, but getting all the pieces to work together taught me quite a bit.

What Was Given vs. What I Added

The course provided the project brief, starter application, and sample build configuration. I made several changes to improve the configuration and understand how the deployment process worked.

1. Removed hardcoded AWS account details

The original build configuration included my AWS account ID and region directly in the file.

That wasn't something I wanted to leave in a public repository. I changed the configuration to retrieve the account ID during the build and construct the ECR registry URI dynamically.

For example, these commands can be used in a CodeBuild pre_build phase:

pre_build:
  commands:
    - ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
    - REGISTRY=$ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com

The account ID now comes from the AWS identity used by the build environment rather than being hardcoded in the repository.

This also made the configuration easier to reuse across environments.

2. Added commit-based Docker image tags

Initially, the latest tag seemed sufficient. But there's a problem with relying on it alone: the tag can point to a different image after every build.

If something breaks after deployment, I want to know exactly which version of the source code produced the image.

I added a tag derived from the Git commit hash:

- IMAGE_TAG=$(echo ${CODEBUILD_RESOLVED_SOURCE_VERSION:-local} | cut -c 1-7)

For Git sources where CodeBuild exposes the commit hash, this produces a short identifier that connects the image to a specific revision.

For example, an image might be tagged with a value such as a1b2c3d.

Now, instead of relying only on a moving latest tag, I can identify a build by its source commit. With the corresponding image retained in ECR and the deployment configuration referencing that tag, I can redeploy an earlier image if necessary.

3. Simplified the image URI used for deployment

I also changed how the deployment configuration was assembled.

The sample configuration combined parts of the image address in multiple places, making it easier to accidentally duplicate the repository name.

I used a single variable for the full image URI and reused it when generating the deployment file.

This made the build configuration easier to follow and reduced the chance of deploying an incorrectly constructed image address.

4. Kept a troubleshooting log

I documented the errors I encountered instead of treating them as isolated annoyances.

That turned out to be one of the most useful parts of the project. Several failures had nothing to do with the application itself.

The Errors That Taught Me the Most

1. Docker couldn't reach its engine

The first error looked like this:

ERROR: failed to connect to the docker API
at npipe:////./pipe/dockerDesktopLinuxEngine

My first instinct was that something was wrong with Docker.

It turned out that Docker Desktop wasn't running.

The docker command is a client that communicates with the Docker engine. Having the CLI available doesn't mean the engine is running.

I started Docker Desktop and verified the connection:

docker info

Once the engine was available, I could continue building and running containers.

Lesson: Check whether the underlying service is running before troubleshooting the commands that depend on it.

2. PowerShell didn't recognize `aws`

Next, PowerShell returned:

aws : The term 'aws' is not recognized as the name of a cmdlet...

The AWS CLI was either not installed correctly or wasn't available in the terminal's current environment.

After installing AWS CLI v2, I reopened PowerShell and checked the installation:

aws --version

Reopening the terminal ensured it picked up the updated environment.

Lesson: A tool can be installed on your machine without being available in a terminal session that was already open.

3. AWS didn't know who I was

The next error was:

NoCredentials: Unable to locate credentials

The AWS CLI was installed, but the local environment didn't have credentials configured for the commands I was running.

For this learning project, I configured an IAM user's access key using:

aws configure

Then I verified which AWS identity the CLI was using:

aws sts get-caller-identity

That command became one of my first checks whenever I started working on a new AWS project.

For a local learning environment, credentials need to be configured securely. Access keys should never be committed to Git or exposed in screenshots, and the identity should have only the permissions it needs. For longer-term development, temporary credentials through an appropriate role or IAM Identity Center are preferable where available.

Lesson: Before troubleshooting an AWS permission error, confirm which identity is actually making the request.

4. The wall inside the pipeline

This was the most frustrating error because it happened after I had connected the AWS services and configured the pipeline.

When the pipeline reached the build stage, it failed before any of my build commands ran:

Error calling startBuild: Cannot have more than 0 builds in queue for the account
(Error Code: AccountLimitExceededException)

At first, I wondered whether my build configuration or IAM permissions were wrong.

But the error pointed to something different: an account-level CodeBuild limit.

My account's applicable build quota was zero, so CodeBuild couldn't accept the build request. This wasn't an application bug or a Dockerfile problem.

How to approach this error: Check the relevant AWS CodeBuild quotas in Service Quotas and determine whether the account needs an eligible quota increase. Once the account limit has been addressed, retry the pipeline and inspect the next failure, if any.

This was a useful reminder that an AWS account can have service limits that prevent a technically correct configuration from running.

Lesson: Read the error type before changing your code. An AccountLimitExceededException points you toward an account or service quota, not automatically toward your build file.

Decisions I Can Explain

These are the choices that helped me understand the architecture rather than simply reproduce a set of instructions.

Deploy manually before automating

I deployed the application before relying on the pipeline.

That established a working baseline: the ECS service, task definition, networking, and container configuration needed to be functional before automation could update them.

When a pipeline fails, knowing that the underlying application already works makes debugging much easier.

Keep Nginx in the foreground

The 2048 application is served as a web application through Nginx.

A container needs its main process to remain running. If Nginx starts in daemon mode and exits from the container's perspective, the container can stop even though the application files are present.

For this setup, the Nginx process needs to remain in the foreground, commonly using:

CMD ["nginx", "-g", "daemon off;"]

This allows the container runtime to monitor the main process properly.

Enable privileged mode for Docker builds in CodeBuild

My CodeBuild environment needed to build Docker images.

Running Docker builds inside CodeBuild can require privileged mode so the build environment can access the Docker daemon, depending on the selected build environment and configuration.

My initial symptom was another Docker daemon connection error.

This taught me that a Dockerfile can be perfectly valid while the environment attempting to build it is not configured to support Docker.

Privileged mode also has security implications, so it should be enabled only when needed and combined with appropriate IAM permissions and build isolation.

Match the container names in the deployment configuration

The container name referenced by the deployment configuration needs to match the container name in the ECS task definition.

If the names differ, the deployment action may be unable to identify the container image that needs updating.

It's a small detail, but it illustrates how deployment automation depends on consistent configuration across services.

Choose Fargate over servers I manage

I used ECS Fargate because I wanted to focus on container deployment without managing EC2 instances.

The trade-off is cost: running tasks and associated infrastructure can continue to incur charges. Serverless container management doesn't mean the workload is free.

Proof It Worked

The most satisfying moment was seeing the entire process work without manually intervening in the deployment.

I changed the page title to include my name, committed the change, and pushed it to GitHub.

CodePipeline picked up the change, CodeBuild built a new Docker image, and the deployment process updated the ECS service to run the new version.

When I opened the address for the running task, the updated title was there.

Pipeline execution

Updated application

That was the point where the architecture clicked for me. The change started in GitHub and made its way to the running application through the pipeline.

The screenshots show the successful pipeline execution and the updated page. Together, they provide evidence that the automated deployment worked.

Cleanup: The Part You Shouldn't Skip

After testing the pipeline, I removed the AWS resources to avoid unnecessary ongoing charges.

My cleanup covered the pipeline, ECS service and cluster, CodeBuild project, ECR repository, artifact storage, log groups, IAM roles, GitHub connection, and the local IAM user's access key.

The exact deletion order depends on resource dependencies. Generally, remove or disconnect resources that depend on other resources before deleting those dependencies. For example, stop the ECS service and its tasks before deleting the cluster, and ensure S3 buckets and ECR repositories are empty if deletion requires it.

I also verified the resources I intended to remove rather than assuming that deleting the pipeline automatically removed everything it used.

A pipeline can be temporary. Its costs don't necessarily stop when you stop working on it.

Cleanup is part of building responsibly in the cloud.

What I'd Improve Next

The project works as a learning exercise, but there are several ways I'd make it more production-ready.

Add an Application Load Balancer. This would provide a stable entry point instead of relying on a task's public address.
Configure HTTPS and a custom domain. This would provide a more appropriate endpoint for a publicly accessible application.
Introduce Terraform. Infrastructure as code would make the AWS resources reproducible and simplify setup and teardown.
Add an automated test stage. Tests should run before an image is promoted to deployment.
Apply least-privilege IAM. The learning setup used broader managed policies than I'd want for production. I'd restrict each role to the actions and resources it actually needs.
Improve deployment and rollback controls. I'd make the deployed image version explicit and define a reliable way to return to a previously working version.

These changes would make the project more secure, repeatable, and maintainable.

What I Learned

This project taught me that building a CI/CD pipeline is about more than connecting AWS services.

My biggest takeaways were:

Verify the toolchain first. My initial errors were setup problems, not application bugs.
Read the error before changing the code. Account limits, missing credentials, and configuration mismatches can look like application failures.
Avoid hardcoding account-specific details. Keep configuration reusable and secrets out of public repositories.
Test components before connecting them. A working Docker build, a valid ECS deployment, and a functioning pipeline are separate things to verify.
Understand what happens after a Git push. Knowing how images are built, stored, tagged, and deployed makes troubleshooting much easier.
Treat cleanup as part of the project. AWS resources can continue to cost money after the application is no longer needed.

The biggest shift for me was moving from running containers locally to understanding how a source-code change can become a deployed application through an automated workflow.

That's the kind of practical cloud engineering experience I wanted to gain from this project.

Explore the Project

GitHub repository: aws-ecs-ci-cd-2048-game
LinkedIn: samobeng14dev

Skills demonstrated: Docker, Amazon ECR, Amazon ECS Fargate, AWS CodeBuild, AWS CodePipeline, IAM, Amazon S3, VPCs, security groups, Git, and GitHub.

DEV Community: Samuel Obeng

I Built a Website That Fails Over from AWS to Azure: A Terraform Walkthrough (and Every Error I Hit)

Why I built this

What this project shows

What you will build

Before you start

Step 0: Prerequisites

0.1 Install the tools

0.2 AWS identity (no keys in code)

0.3 Azure identity

0.4 How does Terraform know who I am?

0.5 Domain

0.6 Account limits to expect

Step 1: Provider setup

Step 2: S3 static website (primary origin)

Step 3: Azure static website (secondary)

Step 4: CloudFront (primary, stage 1)

Step 5: Route 53 zone, health checks and failover records

Step 6: Point the domain at Route 53

Step 7: Stage 2, custom domain with HTTPS

Step 8: Make Azure accept the custom domain

Step 9: Test the failover

Step 10: Clean up

Troubleshooting: every error I hit

Security checklist

Known limits

What I learned

What I'd do next

Let's connect

Building a Near Real-Time Stock Analytics Pipeline on AWS (and the Silent Bug That Hid My Data)

Building a Near Real-Time Stock Data Pipeline on AWS: What I Learned and Debugged

Introduction

The Architecture

What I Changed Beyond the Starter Code

1. Moved configuration out of the code

2. Kept AWS credentials out of the repository

3. Reduced unnecessary logging

4. Added secret-protection measures

The Bug That Cost Me an Hour

Step 1: Tested the processing Lambda independently

Step 2: Checked the Lambda trigger

Step 3: Investigated the Python producer

Step 4: Fixed the configuration and verified the pipeline

Technical Decisions I Can Explain

Raw data in S3, processed data in DynamoDB

Athena instead of maintaining a query database

Decimal instead of float for DynamoDB

Small batches for a low-volume stream

Kinesis on-demand mode

What I Learned About the Data

Repeated prices don't necessarily mean a broken pipeline

Trend alerts need meaningful price movement

Timestamp formats matter

Verifying the Pipeline

1. Kinesis Data Streams

2. DynamoDB

3. Amazon S3

4. Amazon Athena

Cleanup and Cost Awareness

What I'd Improve Next

What I Learned

Explore the Project

What Building My First AWS CI/CD Pipeline Taught Me (and Where It Broke)

From Git Push to AWS Deployment: My Docker CI/CD Pipeline for the 2048 Game

Introduction

The Shape of the System

What Was Given vs. What I Added

1. Removed hardcoded AWS account details

2. Added commit-based Docker image tags

3. Simplified the image URI used for deployment

4. Kept a troubleshooting log

The Errors That Taught Me the Most

1. Docker couldn't reach its engine

2. PowerShell didn't recognize aws

3. AWS didn't know who I was

4. The wall inside the pipeline

Decisions I Can Explain

Deploy manually before automating

Keep Nginx in the foreground

Enable privileged mode for Docker builds in CodeBuild

`Decimal` instead of `float` for DynamoDB

2. PowerShell didn't recognize `aws`