DEV Community: Rahul Pandya

Building an AWS FinOps Agent: How I Taught AI to Actually Care About My Cloud Bill

Rahul Pandya — Mon, 15 Jun 2026 07:31:29 +0000

Description: "A deep dive into building an AI-powered FinOps agent on AWS that proactively monitors, analyzes, and optimizes cloud costs — using Bedrock, Lambda, Cost Explorer, and a healthy dose of real-world frustration."

Tags: aws, finops, ai, cloudcost

Building an AWS FinOps Agent: How I Taught AI to Actually Care About My Cloud Bill

"Why is my AWS bill $4,200 this month when I expected $800?"
— me, at 11 PM on a Tuesday, staring at my Cost Explorer dashboard in quiet horror.

That one moment changed how I think about cloud cost management forever.

I had forgotten to shut down a cluster. A dev environment. Running full-blast. For 18 days. The kind of thing that a competent FinOps process would have caught on day one — but I didn't have a FinOps process. I had a vague intention to "check the bill at the end of the month."

That was the wake-up call. And instead of just setting a billing alarm and calling it done, I decided to build something smarter: an AWS FinOps Agent — an AI-powered system that doesn't just alert you when you've already spent the money, but actually understands your infrastructure, learns your patterns, and proactively recommends action before the damage is done.

This is the full story of how I built it, what I learned, and how you can build one too.

What Even Is a FinOps Agent?

Before we write a single line of code, let's be clear about what we're building — because "FinOps Agent" can mean a lot of things.

FinOps (Financial Operations) is the practice of bringing financial accountability to cloud spending. It's not just about cutting costs — it's about making informed decisions. Spend more where it drives value. Spend less where it doesn't.

A FinOps Agent in our context is an AI-driven system that:

Monitors your AWS cost and usage data continuously
Analyzes anomalies, trends, and patterns
Reasons about why costs changed, not just that they changed
Recommends specific, actionable optimizations
Takes action autonomously when configured to do so (with appropriate guardrails)

The key word is agent. It doesn't just pull a report. It thinks, reasons, and acts — like a cloud financial analyst who never sleeps and has read every AWS pricing page ever published.

The Architecture: What We're Building

Here's the high-level view of what the system looks like:

┌─────────────────────────────────────────────────────────────────┐
│                        AWS FinOps Agent                         │
│                                                                 │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────────┐  │
│  │ Data Sources │    │  Agent Core  │    │   Action Layer   │  │
│  │              │    │              │    │                  │  │
│  │ Cost Explorer│───▶│   Bedrock    │───▶│ Auto Remediation │  │
│  │ CloudWatch   │    │   Agent      │    │ SNS Notifications│  │
│  │ Trusted Adv. │    │   (Claude)   │    │ Slack/Email      │  │
│  │ Resource Tags│    │              │    │ Cost Reports     │  │
│  └──────────────┘    └──────────────┘    └──────────────────┘  │
│         │                   │                      │           │
│         └──────────────────▼──────────────────────┘           │
│                    ┌──────────────┐                            │
│                    │  Orchestrator │                            │
│                    │  (Lambda)    │                            │
│                    └──────────────┘                            │
│                           │                                    │
│                    ┌──────▼───────┐                            │
│                    │  DynamoDB    │                            │
│                    │  (Memory &   │                            │
│                    │   History)   │                            │
│                    └──────────────┘                            │
└─────────────────────────────────────────────────────────────────┘

Key AWS services used:

Amazon Bedrock — The AI brain (Claude 3 Sonnet / Haiku via Bedrock Agents)
AWS Cost Explorer API — Cost and usage data
AWS Lambda — Serverless orchestration
Amazon DynamoDB — Agent memory and history
Amazon CloudWatch — Metrics, alarms, scheduled triggers
AWS Trusted Advisor — Cost optimization recommendations
Amazon SNS — Notifications and alerts
AWS Systems Manager — Secure parameter storage

Phase 1: Setting Up the Data Foundation

An agent is only as good as its data. The first thing we need is a clean, reliable pipeline that pulls cost data and makes it understandable.

Step 1: Enable Cost Explorer and Set Up the API

First, make sure Cost Explorer is enabled in your account. If it's not, go to Billing → Cost Explorer → Enable Cost Explorer. There's a small cost ($0.01 per API call), but for our use case it's negligible.

Now let's create a Lambda function that pulls our cost data:

# cost_data_collector.py
import boto3
import json
from datetime import datetime, timedelta
from decimal import Decimal
import os

ce_client = boto3.client('ce', region_name='us-east-1')
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table(os.environ['COST_TABLE_NAME'])

def get_daily_costs(days_back=30):
    """
    Fetch daily cost breakdown by service for the past N days.
    We get granular service-level data so the agent can reason
    about *which* services are driving cost changes.
    """
    end_date = datetime.now().strftime('%Y-%m-%d')
    start_date = (datetime.now() - timedelta(days=days_back)).strftime('%Y-%m-%d')

    response = ce_client.get_cost_and_usage(
        TimePeriod={
            'Start': start_date,
            'End': end_date
        },
        Granularity='DAILY',
        Metrics=['UnblendedCost', 'UsageQuantity'],
        GroupBy=[
            {'Type': 'DIMENSION', 'Key': 'SERVICE'},
            {'Type': 'DIMENSION', 'Key': 'REGION'}
        ]
    )

    return response['ResultsByTime']


def get_cost_anomalies():
    """
    Pull AWS Cost Anomaly Detection results.
    This gives us AI-detected spikes so our agent can
    cross-reference its own analysis with AWS's built-in detection.
    """
    end_date = datetime.now().strftime('%Y-%m-%d')
    start_date = (datetime.now() - timedelta(days=14)).strftime('%Y-%m-%d')

    response = ce_client.get_anomalies(
        DateInterval={
            'StartDate': start_date,
            'EndDate': end_date
        },
        TotalImpact={
            'NumericOperator': 'GREATER_THAN',
            'StartValue': 50  # Only anomalies > $50 impact
        }
    )

    return response.get('Anomalies', [])


def get_rightsizing_recommendations():
    """
    Pull EC2 rightsizing recommendations from Cost Explorer.
    The agent will use these as starting points, then reason
    about whether they're actually safe to apply given context.
    """
    response = ce_client.get_rightsizing_recommendation(
        Service='AmazonEC2',
        Configuration={
            'RecommendationTarget': 'SAME_INSTANCE_FAMILY',
            'BenefitsConsidered': True
        }
    )

    return response.get('RightsizingRecommendations', [])


def lambda_handler(event, context):
    """Main handler — collects all cost data and stores it for the agent."""

    print("Starting cost data collection...")

    # Collect all our data sources
    daily_costs = get_daily_costs(days_back=30)
    anomalies = get_cost_anomalies()
    rightsizing = get_rightsizing_recommendations()

    # Build a summary for the agent to work with
    summary = {
        'collected_at': datetime.now().isoformat(),
        'daily_costs': daily_costs,
        'anomalies': anomalies,
        'rightsizing_recommendations': rightsizing,
        'total_anomalies_count': len(anomalies),
        'total_rightsizing_opportunities': len(rightsizing)
    }

    # Store in DynamoDB with TTL (we keep 90 days of history)
    ttl = int((datetime.now() + timedelta(days=90)).timestamp())

    table.put_item(
        Item={
            'pk': f"cost_snapshot#{datetime.now().strftime('%Y-%m-%d')}",
            'sk': datetime.now().isoformat(),
            'data': json.dumps(summary, default=str),
            'ttl': ttl
        }
    )

    print(f"Collected {len(anomalies)} anomalies, "
          f"{len(rightsizing)} rightsizing recommendations")

    return {
        'statusCode': 200,
        'body': json.dumps({
            'message': 'Cost data collected successfully',
            'anomalies_found': len(anomalies),
            'rightsizing_opportunities': len(rightsizing)
        })
    }

💡 A note on cost: Calling the Cost Explorer API frequently adds up. I schedule this collector to run once daily via EventBridge. For real-time anomaly alerts, I rely on the AWS Cost Anomaly Detection service's native notifications, which are free.

Step 2: Enrich the Data with Resource Context

Raw cost numbers aren't enough. The agent needs context — what are these resources actually for? This is where tagging becomes essential.

# resource_context_collector.py
import boto3
import json

def get_tagged_resources():
    """
    Pull all tagged resources so we can map costs to
    business context: team, environment, project, owner.

    This is the single most impactful thing you can do for
    FinOps — without tags, costs are just numbers. With tags,
    they become stories with accountable owners.
    """
    tagging_client = boto3.client('resourcegroupstaggingapi')

    paginator = tagging_client.get_paginator('get_resources')

    resources = []
    for page in paginator.paginate(
        TagFilters=[],  # Get everything
        ResourceTypeFilters=[
            'ec2:instance',
            'ec2:volume',
            'rds:db',
            'lambda:function',
            'ecs:cluster',
            'eks:cluster',
            's3:bucket'
        ]
    ):
        resources.extend(page['ResourceTagMappingList'])

    # Build a lookup: resource ARN → tags
    tag_map = {}
    for resource in resources:
        arn = resource['ResourceARN']
        tags = {tag['Key']: tag['Value'] for tag in resource['Tags']}
        tag_map[arn] = tags

    return tag_map


def get_idle_resources():
    """
    Find potentially idle resources using CloudWatch metrics.
    The agent will factor these into its recommendations.
    """
    ec2 = boto3.client('ec2')
    cloudwatch = boto3.client('cloudwatch')

    # Get all running instances
    instances = ec2.describe_instances(
        Filters=[{'Name': 'instance-state-name', 'Values': ['running']}]
    )

    idle_candidates = []

    for reservation in instances['Reservations']:
        for instance in reservation['Instances']:
            instance_id = instance['InstanceId']

            # Check average CPU over the last 14 days
            cpu_response = cloudwatch.get_metric_statistics(
                Namespace='AWS/EC2',
                MetricName='CPUUtilization',
                Dimensions=[{'Name': 'InstanceId', 'Value': instance_id}],
                StartTime=datetime.now() - timedelta(days=14),
                EndTime=datetime.now(),
                Period=86400,  # Daily granularity
                Statistics=['Average']
            )

            if cpu_response['Datapoints']:
                avg_cpu = sum(
                    d['Average'] for d in cpu_response['Datapoints']
                ) / len(cpu_response['Datapoints'])

                # Flag instances with < 5% average CPU as idle candidates
                if avg_cpu < 5.0:
                    idle_candidates.append({
                        'instance_id': instance_id,
                        'instance_type': instance['InstanceType'],
                        'avg_cpu_percent': round(avg_cpu, 2),
                        'tags': {
                            tag['Key']: tag['Value']
                            for tag in instance.get('Tags', [])
                        },
                        'launch_time': instance['LaunchTime'].isoformat()
                    })

    return idle_candidates

Phase 2: Building the AI Agent Core with Amazon Bedrock

This is where things get interesting. We're going to use Amazon Bedrock Agents — which lets us define a set of tools (called Action Groups) that the AI can call to investigate and act on cost data.

Step 3: Define the Bedrock Agent

First, let's set up the agent via the AWS Console or CLI. But more importantly, let's design the system prompt — this is the agent's personality and operating instructions. This part matters more than most people realize.

# agent_setup.py
import boto3
import json

bedrock_agent = boto3.client('bedrock-agent', region_name='us-east-1')

AGENT_INSTRUCTION = """
You are a senior AWS FinOps Engineer and cloud cost optimization specialist.
Your role is to help engineering teams understand, analyze, and optimize
their AWS cloud spending.

## Your Core Responsibilities

1. ANALYZE cost data critically — don't just summarize numbers, explain
   *why* costs changed and what's driving them.

2. PRIORITIZE ruthlessly — rank recommendations by impact ($ saved) and
   effort (implementation complexity). A team with limited bandwidth needs
   to know what to do FIRST.

3. CONTEXTUALIZE your findings — always consider the business context.
   A production database running 24/7 is SUPPOSED to be expensive.
   A dev environment running at 3 AM on a Sunday is NOT.

4. BE SPECIFIC — never say "consider rightsizing your instances." Say
   "Instance i-0abc123 (m5.4xlarge, $0.768/hr) has averaged 3.2% CPU
   over 14 days. Downsizing to m5.xlarge ($0.192/hr) would save $415/month
   with minimal risk given its workload pattern."

5. ACCOUNT FOR RISK — always flag the risk level of each recommendation.
   Rightsizing a non-critical dev server is LOW risk. Rightsizing a
   production database during peak season is HIGH risk.

## Your Tone

You are direct, data-driven, and practical. You don't sugarcoat waste
but you also don't catastrophize. You give teams confidence to act.
You sound like a trusted colleague who has seen a lot of cloud bills,
not like a vendor trying to sell something.

## What You Always Do

- When you spot an anomaly, find its root cause before recommending action
- When you recommend savings, quantify them in monthly AND annual terms
- When multiple options exist, present them as a ranked list with tradeoffs
- When data is insufficient, say so and explain what additional context
  you need to give a confident recommendation
"""

def create_finops_agent():
    """Create the Bedrock Agent with our custom instructions."""

    # First, we need an IAM role for the agent
    # (assumed to already exist — see IAM setup section)
    agent_role_arn = "arn:aws:iam::YOUR_ACCOUNT:role/BedrockFinOpsAgentRole"

    response = bedrock_agent.create_agent(
        agentName='finops-cost-optimizer',
        agentResourceRoleArn=agent_role_arn,
        description='AI-powered AWS cost analysis and optimization agent',
        foundationModel='anthropic.claude-3-sonnet-20240229-v1:0',
        instruction=AGENT_INSTRUCTION,
        idleSessionTTLInSeconds=3600,  # 1 hour session timeout
    )

    agent_id = response['agent']['agentId']
    print(f"Created agent: {agent_id}")
    return agent_id

Step 4: Create Action Groups (The Agent's Tools)

Action Groups are the tools the agent can call. Think of them as the agent's hands — the ways it can reach out and get data or take action. We'll define these as Lambda functions.

# agent_actions.py — the Lambda that handles all agent tool calls

import boto3
import json
import os
from datetime import datetime, timedelta

def lambda_handler(event, context):
    """
    Router for all agent action calls.
    Bedrock sends us the action name and parameters;
    we dispatch to the right function and return the result.
    """

    action_group = event.get('actionGroup')
    action = event.get('function')  # or 'action' depending on agent type
    parameters = event.get('parameters', {})

    print(f"Agent called action: {action_group}/{action}")
    print(f"Parameters: {json.dumps(parameters)}")

    # Route to the correct handler
    result = dispatch_action(action, parameters)

    # Return in the format Bedrock expects
    return {
        'actionGroup': action_group,
        'function': action,
        'functionResponse': {
            'responseBody': {
                'TEXT': {
                    'body': json.dumps(result)
                }
            }
        }
    }


def dispatch_action(action, parameters):
    """Route action calls to handler functions."""

    handlers = {
        'get_cost_summary': get_cost_summary,
        'get_cost_by_service': get_cost_by_service,
        'get_cost_anomalies': get_cost_anomalies,
        'get_idle_resources': get_idle_resources,
        'get_savings_opportunities': get_savings_opportunities,
        'get_reserved_instance_coverage': get_ri_coverage,
        'send_alert': send_alert,
        'create_budget': create_budget,
    }

    handler = handlers.get(action)
    if not handler:
        return {'error': f'Unknown action: {action}'}

    return handler(parameters)


def get_cost_summary(params):
    """
    Get a high-level cost summary for a given time period.
    The agent uses this as its starting point for any analysis session.
    """
    ce = boto3.client('ce', region_name='us-east-1')

    days = int(params.get('days', 30))
    end = datetime.now().strftime('%Y-%m-%d')
    start = (datetime.now() - timedelta(days=days)).strftime('%Y-%m-%d')

    # Current period cost
    current = ce.get_cost_and_usage(
        TimePeriod={'Start': start, 'End': end},
        Granularity='MONTHLY',
        Metrics=['UnblendedCost']
    )

    # Same period last month (for comparison)
    prev_end = (datetime.now() - timedelta(days=days)).strftime('%Y-%m-%d')
    prev_start = (datetime.now() - timedelta(days=days*2)).strftime('%Y-%m-%d')

    previous = ce.get_cost_and_usage(
        TimePeriod={'Start': prev_start, 'End': prev_end},
        Granularity='MONTHLY',
        Metrics=['UnblendedCost']
    )

    # Calculate totals
    current_total = sum(
        float(period['Total']['UnblendedCost']['Amount'])
        for period in current['ResultsByTime']
    )

    previous_total = sum(
        float(period['Total']['UnblendedCost']['Amount'])
        for period in previous['ResultsByTime']
    )

    pct_change = ((current_total - previous_total) / previous_total * 100
                  if previous_total > 0 else 0)

    return {
        'period': f'{start} to {end}',
        'current_total_usd': round(current_total, 2),
        'previous_period_total_usd': round(previous_total, 2),
        'change_percent': round(pct_change, 1),
        'trend': 'increasing' if pct_change > 5 else 
                 'decreasing' if pct_change < -5 else 'stable',
        'daily_average_usd': round(current_total / days, 2),
        'projected_monthly_usd': round((current_total / days) * 30, 2)
    }


def get_cost_by_service(params):
    """
    Break down costs by service with month-over-month comparison.
    This tells the agent WHICH services are the culprits.
    """
    ce = boto3.client('ce', region_name='us-east-1')

    days = int(params.get('days', 30))
    end = datetime.now().strftime('%Y-%m-%d')
    start = (datetime.now() - timedelta(days=days)).strftime('%Y-%m-%d')

    response = ce.get_cost_and_usage(
        TimePeriod={'Start': start, 'End': end},
        Granularity='MONTHLY',
        Metrics=['UnblendedCost'],
        GroupBy=[{'Type': 'DIMENSION', 'Key': 'SERVICE'}]
    )

    # Aggregate by service
    service_costs = {}
    for period in response['ResultsByTime']:
        for group in period['Groups']:
            service = group['Keys'][0]
            cost = float(group['Metrics']['UnblendedCost']['Amount'])
            service_costs[service] = service_costs.get(service, 0) + cost

    # Sort by cost, return top N
    top_n = int(params.get('top_n', 10))
    sorted_services = sorted(
        service_costs.items(),
        key=lambda x: x[1],
        reverse=True
    )[:top_n]

    return {
        'period': f'{start} to {end}',
        'top_services': [
            {
                'service': service,
                'cost_usd': round(cost, 2),
                'percentage_of_total': round(
                    cost / sum(service_costs.values()) * 100, 1
                ) if service_costs else 0
            }
            for service, cost in sorted_services
        ]
    }


def get_savings_opportunities(params):
    """
    Aggregate all savings opportunities into one prioritized list.
    This is the agent's primary output for cost optimization sessions.
    """
    ce = boto3.client('ce', region_name='us-east-1')

    opportunities = []

    # 1. Rightsizing recommendations
    try:
        rightsizing = ce.get_rightsizing_recommendation(
            Service='AmazonEC2',
            Configuration={
                'RecommendationTarget': 'SAME_INSTANCE_FAMILY',
                'BenefitsConsidered': True
            }
        )

        for rec in rightsizing.get('RightsizingRecommendations', []):
            if rec['RightsizingType'] == 'Modify':
                current_monthly = float(
                    rec['CurrentInstance']['MonthlyCost']
                )
                for mod in rec.get('ModifyRecommendationDetail', {}).get(
                    'TargetInstances', []
                ):
                    target_monthly = float(mod['EstimatedMonthlyCost'])
                    savings = current_monthly - target_monthly

                    if savings > 10:  # Only significant savings
                        opportunities.append({
                            'type': 'rightsizing',
                            'resource_id': rec['CurrentInstance']['ResourceId'],
                            'current_instance_type': rec['CurrentInstance']['ResourceDetails']['EC2ResourceDetails']['InstanceType'],
                            'recommended_instance_type': mod['ResourceDetails']['EC2ResourceDetails']['InstanceType'],
                            'monthly_savings_usd': round(savings, 2),
                            'annual_savings_usd': round(savings * 12, 2),
                            'risk': 'medium',
                            'action': 'Resize EC2 instance'
                        })
    except Exception as e:
        print(f"Rightsizing lookup failed: {e}")

    # 2. Savings Plans recommendations
    try:
        sp_recs = ce.get_savings_plans_purchase_recommendation(
            SavingsPlansType='COMPUTE_SP',
            TermInYears='ONE_YEAR',
            PaymentOption='PARTIAL_UPFRONT',
            LookbackPeriodInDays='THIRTY_DAYS'
        )

        for rec in sp_recs.get('SavingsPlansPurchaseRecommendation', {}).get(
            'SavingsPlansPurchaseRecommendationDetails', []
        ):
            monthly_savings = float(
                rec.get('EstimatedMonthlySavingsAmount', 0)
            )
            if monthly_savings > 50:
                opportunities.append({
                    'type': 'savings_plan',
                    'recommended_hourly_commitment': rec.get('HourlyCommitmentToPurchase'),
                    'monthly_savings_usd': round(monthly_savings, 2),
                    'annual_savings_usd': round(monthly_savings * 12, 2),
                    'estimated_savings_rate': rec.get('EstimatedSavingsRate'),
                    'risk': 'low',
                    'action': 'Purchase Compute Savings Plan'
                })
    except Exception as e:
        print(f"Savings Plans lookup failed: {e}")

    # Sort by annual savings impact
    opportunities.sort(key=lambda x: x.get('annual_savings_usd', 0), reverse=True)

    total_potential_annual_savings = sum(
        o.get('annual_savings_usd', 0) for o in opportunities
    )

    return {
        'total_opportunities': len(opportunities),
        'total_potential_annual_savings_usd': round(total_potential_annual_savings, 2),
        'opportunities': opportunities[:20]  # Top 20
    }


def send_alert(params):
    """Send a cost alert notification via SNS."""
    sns = boto3.client('sns')

    topic_arn = os.environ.get('ALERT_SNS_TOPIC_ARN')
    if not topic_arn:
        return {'error': 'SNS topic not configured'}

    message = params.get('message', 'FinOps Agent Alert')
    severity = params.get('severity', 'INFO')

    subject = f"[FinOps Agent | {severity}] AWS Cost Alert"

    sns.publish(
        TopicArn=topic_arn,
        Subject=subject,
        Message=message
    )

    return {'status': 'alert_sent', 'severity': severity}

Phase 3: The Orchestrator — Bringing It All Together

The orchestrator is the Lambda that runs on a schedule and drives the agent through a structured analysis workflow. Think of it as the FinOps analyst's morning routine.

# orchestrator.py
import boto3
import json
import os
from datetime import datetime

bedrock_runtime = boto3.client('bedrock-agent-runtime', region_name='us-east-1')
dynamodb = boto3.resource('dynamodb')
sns = boto3.client('sns')

AGENT_ID = os.environ['BEDROCK_AGENT_ID']
AGENT_ALIAS_ID = os.environ['BEDROCK_AGENT_ALIAS_ID']
HISTORY_TABLE = os.environ['HISTORY_TABLE']
REPORT_TOPIC_ARN = os.environ['REPORT_TOPIC_ARN']


def invoke_agent(prompt, session_id):
    """
    Send a prompt to the Bedrock Agent and collect the full response.
    Bedrock streams events, so we need to consume them all.
    """
    response = bedrock_runtime.invoke_agent(
        agentId=AGENT_ID,
        agentAliasId=AGENT_ALIAS_ID,
        sessionId=session_id,
        inputText=prompt
    )

    # Consume the streaming response
    full_response = ""
    for event in response.get('completion', []):
        if 'chunk' in event:
            chunk = event['chunk']
            if 'bytes' in chunk:
                full_response += chunk['bytes'].decode('utf-8')

    return full_response


def run_daily_analysis():
    """
    The main daily FinOps analysis workflow.

    We break this into multiple focused agent calls rather than
    one giant prompt. This gives the agent better context for each
    question and produces more actionable, specific output.
    """

    session_id = f"daily-analysis-{datetime.now().strftime('%Y%m%d')}"

    print(f"Starting daily FinOps analysis. Session: {session_id}")

    # ── Step 1: Cost Overview ──────────────────────────────────────────
    print("Step 1: Getting cost overview...")

    overview_prompt = """
    Start my daily FinOps briefing.

    1. Call get_cost_summary for the last 30 days
    2. Call get_cost_by_service for the top 10 services
    3. Give me a concise executive summary: current spend rate,
       trend direction, and the 3 most important things I should
       know about this month's costs so far.

    Be specific with numbers. Keep it under 300 words.
    """

    overview = invoke_agent(overview_prompt, session_id)

    # ── Step 2: Anomaly Investigation ─────────────────────────────────
    print("Step 2: Investigating anomalies...")

    anomaly_prompt = """
    Now let's look at anomalies.

    Call get_cost_anomalies and investigate any anomalies found.
    For each anomaly:
    - What service/region is affected?
    - What's the financial impact?
    - What's your hypothesis for the root cause?
    - What should I check first to confirm or rule out that hypothesis?

    If there are no anomalies, say so clearly and move on.
    """

    anomaly_analysis = invoke_agent(anomaly_prompt, session_id)

    # ── Step 3: Optimization Opportunities ────────────────────────────
    print("Step 3: Finding optimization opportunities...")

    optimization_prompt = """
    Now let's find savings opportunities.

    Call get_savings_opportunities and get_idle_resources.
    Then give me a prioritized action list:

    Format each recommendation as:
    [PRIORITY #X | $Y/month savings | RISK: Low/Medium/High]
    What to do: <specific action>
    Why now: <business justification>
    How to do it: <concrete first steps>

    Focus on the top 5 highest-impact, lowest-risk opportunities.
    Include total potential annual savings at the end.
    """

    optimizations = invoke_agent(optimization_prompt, session_id)

    # ── Step 4: Reserved Instance Health Check ─────────────────────────
    print("Step 4: RI/Savings Plan coverage check...")

    ri_prompt = """
    Call get_reserved_instance_coverage.

    How is our Reserved Instance and Savings Plan coverage?
    Are we leaving discount money on the table?
    Give me a simple coverage score (% of compute spend covered by commitments)
    and one clear recommendation if we should buy more.
    """

    ri_analysis = invoke_agent(ri_prompt, session_id)

    # ── Compile the Full Report ────────────────────────────────────────
    report = compile_report(overview, anomaly_analysis, optimizations, ri_analysis)

    # Store in DynamoDB
    store_report(session_id, report)

    # Send via SNS
    send_report(report)

    print("Daily analysis complete.")
    return report


def compile_report(overview, anomalies, optimizations, ri_analysis):
    """Compile all sections into a formatted daily report."""

    today = datetime.now().strftime('%B %d, %Y')

    report = f"""
=================================================================
  🤖 AWS FinOps Agent — Daily Cost Intelligence Report
  {today}
=================================================================

📊 COST OVERVIEW
-----------------------------------------------------------------
{overview}

🚨 ANOMALY ANALYSIS
-----------------------------------------------------------------
{anomalies}

💡 OPTIMIZATION OPPORTUNITIES
-----------------------------------------------------------------
{optimizations}

🏷️ COMMITMENT COVERAGE (RIs & SAVINGS PLANS)
-----------------------------------------------------------------
{ri_analysis}

=================================================================
  Generated by AWS FinOps Agent | Powered by Amazon Bedrock
=================================================================
"""
    return report


def store_report(session_id, report):
    """Store the report in DynamoDB for historical reference."""
    table = dynamodb.Table(HISTORY_TABLE)

    table.put_item(
        Item={
            'pk': f"report#{datetime.now().strftime('%Y-%m')}",
            'sk': datetime.now().isoformat(),
            'session_id': session_id,
            'report': report,
            'generated_at': datetime.now().isoformat()
        }
    )


def send_report(report):
    """Send the compiled report via SNS (email, Slack, etc.)."""
    sns.publish(
        TopicArn=REPORT_TOPIC_ARN,
        Subject=f"Daily FinOps Report — {datetime.now().strftime('%b %d, %Y')}",
        Message=report
    )


def lambda_handler(event, context):
    """Entry point. Can be triggered by EventBridge schedule or manually."""

    trigger_type = event.get('trigger_type', 'scheduled')

    if trigger_type == 'scheduled':
        report = run_daily_analysis()
        return {
            'statusCode': 200,
            'body': json.dumps({
                'message': 'Daily analysis complete',
                'report_length': len(report)
            })
        }

    elif trigger_type == 'ad_hoc':
        # Handle ad-hoc queries from users
        question = event.get('question', '')
        if not question:
            return {'statusCode': 400, 'body': 'No question provided'}

        session_id = f"adhoc-{datetime.now().strftime('%Y%m%d%H%M%S')}"
        response = invoke_agent(question, session_id)

        return {
            'statusCode': 200,
            'body': json.dumps({'response': response})
        }

    return {'statusCode': 400, 'body': 'Unknown trigger type'}

Phase 4: Infrastructure as Code

Let's wire everything together with a clean CloudFormation template. I'm a big believer in IaC for anything you plan to run in production.

# finops-agent-stack.yaml
AWSTemplateFormatVersion: '2010-09-09'
Description: 'AWS FinOps Agent — Complete Infrastructure Stack'

Parameters:
  AlertEmail:
    Type: String
    Description: Email address for FinOps alerts and reports

  Environment:
    Type: String
    Default: production
    AllowedValues: [development, staging, production]

Resources:

  # ── DynamoDB for agent memory and history ────────────────────────
  CostDataTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: !Sub 'finops-cost-data-${Environment}'
      BillingMode: PAY_PER_REQUEST
      AttributeDefinitions:
        - AttributeName: pk
          AttributeType: S
        - AttributeName: sk
          AttributeType: S
      KeySchema:
        - AttributeName: pk
          KeyType: HASH
        - AttributeName: sk
          KeyType: RANGE
      TimeToLiveSpecification:
        AttributeName: ttl
        Enabled: true
      Tags:
        - Key: Project
          Value: FinOpsAgent
        - Key: Environment
          Value: !Ref Environment

  ReportHistoryTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: !Sub 'finops-report-history-${Environment}'
      BillingMode: PAY_PER_REQUEST
      AttributeDefinitions:
        - AttributeName: pk
          AttributeType: S
        - AttributeName: sk
          AttributeType: S
      KeySchema:
        - AttributeName: pk
          KeyType: HASH
        - AttributeName: sk
          KeyType: RANGE

  # ── SNS Topics ───────────────────────────────────────────────────
  AlertTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: !Sub 'finops-alerts-${Environment}'
      Subscription:
        - Protocol: email
          Endpoint: !Ref AlertEmail

  ReportTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: !Sub 'finops-reports-${Environment}'
      Subscription:
        - Protocol: email
          Endpoint: !Ref AlertEmail

  # ── IAM Role for Lambda Functions ───────────────────────────────
  FinOpsLambdaRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub 'finops-lambda-role-${Environment}'
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: FinOpsAgentPermissions
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              # Cost Explorer access (read-only)
              - Effect: Allow
                Action:
                  - ce:GetCostAndUsage
                  - ce:GetRightsizingRecommendation
                  - ce:GetSavingsPlansPurchaseRecommendation
                  - ce:GetAnomalies
                  - ce:GetReservationCoverage
                  - ce:CreateBudget
                Resource: '*'

              # CloudWatch metrics (read-only)
              - Effect: Allow
                Action:
                  - cloudwatch:GetMetricStatistics
                  - cloudwatch:ListMetrics
                Resource: '*'

              # Resource tagging API (read-only)
              - Effect: Allow
                Action:
                  - tag:GetResources
                Resource: '*'

              # EC2 describe (read-only)
              - Effect: Allow
                Action:
                  - ec2:DescribeInstances
                  - ec2:DescribeVolumes
                Resource: '*'

              # DynamoDB for memory and history
              - Effect: Allow
                Action:
                  - dynamodb:PutItem
                  - dynamodb:GetItem
                  - dynamodb:Query
                  - dynamodb:Scan
                Resource:
                  - !GetAtt CostDataTable.Arn
                  - !GetAtt ReportHistoryTable.Arn

              # SNS for notifications
              - Effect: Allow
                Action:
                  - sns:Publish
                Resource:
                  - !Ref AlertTopic
                  - !Ref ReportTopic

              # Bedrock for AI invocation
              - Effect: Allow
                Action:
                  - bedrock:InvokeAgent
                  - bedrock:InvokeModel
                Resource: '*'

  # ── EventBridge Schedule (runs daily at 7 AM UTC) ────────────────
  DailyAnalysisSchedule:
    Type: AWS::Events::Rule
    Properties:
      Name: !Sub 'finops-daily-analysis-${Environment}'
      Description: 'Trigger daily FinOps analysis'
      ScheduleExpression: 'cron(0 7 * * ? *)'  # 7 AM UTC daily
      State: ENABLED
      Targets:
        - Id: OrchestratorLambda
          Arn: !GetAtt OrchestratorLambda.Arn
          Input: '{"trigger_type": "scheduled"}'

Outputs:
  CostDataTableName:
    Value: !Ref CostDataTable
    Export:
      Name: !Sub '${AWS::StackName}-CostTable'

  AlertTopicArn:
    Value: !Ref AlertTopic
    Export:
      Name: !Sub '${AWS::StackName}-AlertTopic'

Real Results: What This Agent Caught

I want to be honest here — I didn't build this to write a blog post. I built it because I kept getting surprised by my AWS bill. Here's what it actually found in the first month of running:

Week 1 — Flagged an m5.2xlarge running in us-west-2 (a region I thought I had deprovision from a project three months ago) with 0.8% average CPU. Monthly cost: $274. Status: terminated.

Week 2 — Identified that we were paying for 47 unattached EBS volumes across multiple accounts. Total monthly waste: $183. Status: cleaned up in one hour.

Week 3 — Caught a Lambda function with an infinite retry loop hammering a DynamoDB table, generating 8 million extra reads in 36 hours. Cost spike: $340 above baseline. Caught it in hour 3. Without the agent, I would have seen it at month-end.

Week 4 — Recommended Compute Savings Plans at $0.80/hr commitment based on our consistent baseline usage pattern. Estimated annual savings: $3,100. Status: purchased.

Month 1 total savings identified: ~$6,800 annualized. Setup time: about two weekends.

What I'd Do Differently

Building this taught me a few hard lessons worth sharing:

1. Start with tagging, not with AI. Seriously. The agent is only as smart as the data you feed it. If your resources aren't tagged, the agent can tell you costs went up but can't tell you whose costs went up. Spend a week getting your tagging strategy right before anything else.

2. Don't let the agent act without human approval (at first). I originally wired the agent to auto-terminate idle resources. It almost deleted a "dev" instance that was actually running a critical overnight batch job. Gate all destructive actions behind an approval step until you've built enough trust in the agent's judgment.

3. Multi-account is harder than you think. If you're running AWS Organizations, Cost Explorer can aggregate across accounts, but the IAM permissions get complex fast. Use a management account role and cross-account assume-role patterns. Don't try to run a separate agent per account.

4. The system prompt is your most important code. I spent more time iterating on the agent's instructions than on any Lambda function. A vague system prompt produces vague analysis. Specific, opinionated instructions produce specific, actionable recommendations.

What's Next

This is v1. Here's where I'm planning to take it next:

Slack bot interface — Let engineers ask ad-hoc questions in natural language: "Why did our ECS costs go up this week?"
Multi-account support — Federate across AWS Organizations member accounts
Forecasting — Use historical patterns to predict next month's bill and flag if we're trending over budget
Auto-tagging — Use the agent to infer tags for untagged resources based on naming conventions and context
Team showbacks — Automatically generate per-team cost reports and post them to team Slack channels

Getting Started Today

If you want to spin this up, here's the fastest path:

# 1. Clone the repo (or create the files from this post)
# 2. Deploy the CloudFormation stack
aws cloudformation deploy \
  --template-file finops-agent-stack.yaml \
  --stack-name finops-agent \
  --parameter-overrides AlertEmail=you@yourdomain.com \
  --capabilities CAPABILITY_NAMED_IAM

# 3. Create the Bedrock Agent (via console or CLI)
# See the agent_setup.py script above

# 4. Deploy the Lambda functions
# (package and deploy cost_data_collector.py,
#  agent_actions.py, and orchestrator.py)

# 5. Add the agent ID and alias to your Lambda env vars
# 6. Wait for the first daily run, or trigger manually:
aws lambda invoke \
  --function-name finops-orchestrator \
  --payload '{"trigger_type": "ad_hoc", "question": "What are my top 5 cost savings opportunities right now?"}' \
  response.json

cat response.json

Final Thoughts

Cloud costs are one of those things that everyone knows matter but few teams manage proactively. The reason isn't lack of willpower — it's that cost analysis is genuinely complex, requires cross-referencing multiple data sources, and needs to happen continuously, not just at month-end.

That's exactly what AI agents are good at.

This FinOps agent won't replace your finance team or your cloud architects. But it will make sure that the 3 AM forgotten cluster, the idle dev environment, and the infinite Lambda retry loop get caught before they show up as line items on a bill that's already gone out.

Start small. Get the data pipeline working. Let the agent run in read-only mode for a week. Trust it gradually. The ROI will be obvious pretty quickly.

If you build something similar or extend this in an interesting direction, I'd genuinely love to hear about it. Drop a comment or reach out — this is one of those areas where the community has a lot to teach each other.

Thanks for reading. If this was useful, consider following me here on dev.to — I write about building real things on AWS, with the rough edges included.

Tags: #aws #finops #bedrock #cloudcost #serverless #aiagents #lambda #costoptimization

ECS vs EKS in 2026: An Honest Comparison from Someone Who Has Run Both in Production

Rahul Pandya — Sat, 23 May 2026 14:19:32 +0000

No marketing fluff. Just what I learned running both in production across three different companies over five years.

I've had this conversation more times than I can count.

Someone joins a new team, looks at the infrastructure, and asks: "Why are we on ECS and not Kubernetes?" Or the opposite — they look at the EKS cluster and ask: "Why aren't we using ECS? This seems way more complicated than it needs to be."

Both questions are valid. Both have real answers. And the answers depend entirely on context that a blog post comparison table can't capture — which is exactly why most ECS vs EKS comparisons are useless. They list features side by side and conclude with something like "it depends on your use case," which tells you nothing.

I want to do something different. I want to tell you what it actually feels like to operate these two platforms, what goes wrong, what the costs look like in practice, and what signals I use to make the call when starting something new.

Let me start with where I've been.

My Background With Both Platforms

Company 1 — a mid-size SaaS startup, around 40 engineers. We were on ECS from 2020 and stayed there until I left. About 35 services, mix of Fargate and EC2 launch types, deployed via CodePipeline. No major incidents directly attributable to ECS. Oncall was manageable.

Company 2 — a larger company migrating from a legacy monolith to microservices. They chose EKS because "that's what everyone uses now." By the time I joined, the cluster had been running for 18 months. There were 12 engineers who could modify cluster config and 2 who actually understood it. Oncall had a lot of "why is this pod in CrashLoopBackOff" at 3am.

Company 3 — a data-heavy platform, around 80 engineers. Started with EKS for the ML workloads, added ECS for the API layer later. Both running in parallel to this day for valid reasons that I'll get into.

I'm not here to tell you one is better. I'm here to tell you the truth about both.

The Core Difference That Everything Else Flows From

Before we talk about networking, autoscaling, pricing, or anything else — there's one fundamental difference you need to internalize:

ECS is a managed container orchestrator. EKS is a managed Kubernetes control plane.

That sounds like a subtle distinction but the implications are massive.

With ECS, AWS owns the scheduling logic, the service reconciliation loop, the load balancer integration, the secrets injection, and the deployment mechanics. You configure these things, but AWS operates them. When something breaks, you look at your task definition, your service events, and your CloudWatch logs. The number of things that can go wrong is bounded.

With EKS, AWS manages the Kubernetes control plane (the API server, etcd, the scheduler, the controller manager). But the cluster is still Kubernetes. The worker nodes, the CNI plugin, the ingress controller, the cert-manager, the service mesh, the cluster autoscaler, the pod disruption budgets, the RBAC policies — all of that is yours to configure, operate, and debug. The number of things that can go wrong is essentially unbounded.

This isn't a criticism of EKS. Kubernetes is powerful precisely because it's extensible. But that extensibility has an operational cost, and that cost is real and ongoing.

Setup and Initial Complexity

ECS

Getting a working ECS cluster with a deployed service behind a load balancer takes a few hours with Terraform the first time. Here's the complete setup:

# ECS Cluster
resource "aws_ecs_cluster" "main" {
  name = "${var.project_name}-${var.environment}"

  setting {
    name  = "containerInsights"
    value = "enabled"
  }

  tags = {
    Environment = var.environment
    ManagedBy   = "terraform"
  }
}

resource "aws_ecs_cluster_capacity_providers" "main" {
  cluster_name       = aws_ecs_cluster.main.name
  capacity_providers = ["FARGATE", "FARGATE_SPOT"]

  default_capacity_provider_strategy {
    capacity_provider = "FARGATE"
    weight            = 100
    base              = 1
  }
}

# Task Definition
resource "aws_ecs_task_definition" "app" {
  family                   = "${var.project_name}-${var.environment}"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = 512
  memory                   = 1024
  execution_role_arn       = aws_iam_role.ecs_execution.arn
  task_role_arn            = aws_iam_role.ecs_task.arn

  container_definitions = jsonencode([{
    name      = "app"
    image     = "${var.ecr_repository_url}:${var.image_tag}"
    essential = true

    portMappings = [{
      containerPort = 8080
      protocol      = "tcp"
    }]

    environment = [
      { name = "APP_ENV", value = var.environment }
    ]

    secrets = [{
      name      = "DATABASE_URL"
      valueFrom = aws_secretsmanager_secret.db_url.arn
    }]

    logConfiguration = {
      logDriver = "awslogs"
      options = {
        "awslogs-group"         = aws_cloudwatch_log_group.app.name
        "awslogs-region"        = var.region
        "awslogs-stream-prefix" = "ecs"
      }
    }

    healthCheck = {
      command     = ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"]
      interval    = 30
      timeout     = 5
      retries     = 3
      startPeriod = 60
    }
  }])
}

# ECS Service
resource "aws_ecs_service" "app" {
  name            = "${var.project_name}-${var.environment}"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.app.arn
  desired_count   = 2

  capacity_provider_strategy {
    capacity_provider = "FARGATE"
    weight            = 100
    base              = 1
  }

  network_configuration {
    subnets          = var.private_subnet_ids
    security_groups  = [aws_security_group.ecs_tasks.id]
    assign_public_ip = false
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.app.arn
    container_name   = "app"
    container_port   = 8080
  }

  deployment_circuit_breaker {
    enable   = true
    rollback = true
  }

  deployment_maximum_percent         = 200
  deployment_minimum_healthy_percent = 100

  lifecycle {
    ignore_changes = [task_definition, desired_count]
  }
}

That's it. No additional tooling, no CNI configuration, no ingress controller to install. ALB integration is native. Secrets come from Secrets Manager through IAM. Logging goes to CloudWatch automatically.

A reasonably experienced engineer can own this end-to-end.

EKS

The EKS setup story is longer. Much longer. Here's what a real production EKS setup involves beyond just the cluster:

# EKS Cluster
resource "aws_eks_cluster" "main" {
  name     = "${var.project_name}-${var.environment}"
  role_arn = aws_iam_role.eks_cluster.arn
  version  = "1.29"

  vpc_config {
    subnet_ids              = concat(var.private_subnet_ids, var.public_subnet_ids)
    endpoint_private_access = true
    endpoint_public_access  = true
    public_access_cidrs     = var.allowed_cidrs
  }

  enabled_cluster_log_types = [
    "api", "audit", "authenticator", "controllerManager", "scheduler"
  ]

  depends_on = [
    aws_iam_role_policy_attachment.eks_cluster_policy,
    aws_iam_role_policy_attachment.eks_vpc_resource_controller,
    aws_cloudwatch_log_group.eks,
  ]

  tags = {
    Environment = var.environment
    ManagedBy   = "terraform"
  }
}

# Node Group
resource "aws_eks_node_group" "main" {
  cluster_name    = aws_eks_cluster.main.name
  node_group_name = "${var.project_name}-${var.environment}-ng"
  node_role_arn   = aws_iam_role.eks_node.arn
  subnet_ids      = var.private_subnet_ids
  instance_types  = ["m6g.large"]
  ami_type        = "AL2_ARM_64"

  scaling_config {
    desired_size = 3
    max_size     = 20
    min_size     = 2
  }

  update_config {
    max_unavailable = 1
  }

  labels = {
    role        = "general"
    environment = var.environment
  }

  lifecycle {
    ignore_changes = [scaling_config[0].desired_size]
  }

  depends_on = [
    aws_iam_role_policy_attachment.eks_worker_node_policy,
    aws_iam_role_policy_attachment.eks_cni_policy,
    aws_iam_role_policy_attachment.ecr_read_only,
  ]
}

# OIDC Provider (required for IRSA - IAM Roles for Service Accounts)
data "tls_certificate" "eks" {
  url = aws_eks_cluster.main.identity[0].oidc[0].issuer
}

resource "aws_iam_openid_connect_provider" "eks" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.tls_certificate.eks.certificates[0].sha1_fingerprint]
  url             = aws_eks_cluster.main.identity[0].oidc[0].issuer
}

But the Terraform cluster resource is just the beginning. You also need to install and configure:

AWS Load Balancer Controller — because EKS doesn't have native ALB integration the way ECS does. You install this as a Helm chart.

helm repo add eks https://aws.github.io/eks-charts
helm repo update

helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName=${CLUSTER_NAME} \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller

Cluster Autoscaler — EKS won't automatically scale your node count based on pending pods. You need to install and configure the cluster autoscaler.

helm install cluster-autoscaler autoscaler/cluster-autoscaler \
  --namespace kube-system \
  --set autoDiscovery.clusterName=${CLUSTER_NAME} \
  --set awsRegion=${AWS_REGION} \
  --set rbac.serviceAccount.name=cluster-autoscaler \
  --set extraArgs.balance-similar-node-groups=true \
  --set extraArgs.skip-nodes-with-system-pods=false

EBS or EFS CSI Driver — if any of your workloads need persistent volumes. Another Helm chart, another IAM role for service account.

External Secrets Operator — because Kubernetes Secrets are base64-encoded, not encrypted. If you want your secrets to come from AWS Secrets Manager (which you do), you need an operator to bridge the two.

# external-secrets/secret-store.yaml
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secretsmanager
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
            namespace: external-secrets

metrics-server — required for Horizontal Pod Autoscaler to work. Another installation.

By the time you have a production-ready EKS cluster, you've installed at minimum five separate components, each with its own configuration, versioning, and upgrade lifecycle. This isn't EKS being bad — it's just the nature of the platform. You're building on top of a general-purpose orchestrator, not a purpose-built AWS service.

Verdict on setup complexity: ECS wins decisively. Not slightly — decisively. The first-time setup difference is measured in days, not hours.

Networking

ECS Networking

ECS networking in Fargate mode is remarkably simple. Each task gets its own ENI with its own IP address. Security groups work exactly like they do for EC2 instances. You create a security group for your tasks, you define the ingress and egress rules, done.

resource "aws_security_group" "ecs_tasks" {
  name        = "${var.project_name}-ecs-tasks"
  description = "Security group for ECS tasks"
  vpc_id      = var.vpc_id

  ingress {
    from_port       = 8080
    to_port         = 8080
    protocol        = "tcp"
    security_groups = [aws_security_group.alb.id]
    description     = "Allow traffic from ALB only"
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
    description = "Allow all outbound"
  }
}

Service-to-service communication in ECS works through internal ALBs or AWS Cloud Map for service discovery. Neither requires deep networking knowledge.

# Service discovery with Cloud Map
resource "aws_service_discovery_private_dns_namespace" "internal" {
  name = "${var.project_name}.internal"
  vpc  = var.vpc_id
}

resource "aws_service_discovery_service" "app" {
  name = "user-service"

  dns_config {
    namespace_id = aws_service_discovery_private_dns_namespace.internal.id

    dns_records {
      ttl  = 10
      type = "A"
    }

    routing_policy = "MULTIVALUE"
  }

  health_check_custom_config {
    failure_threshold = 1
  }
}

# Now your services can reach user-service.projectname.internal

EKS Networking

Kubernetes networking is famously complex. There are multiple layers: the CNI plugin that assigns pod IPs, kube-proxy for service routing, CoreDNS for service discovery, and your ingress controller for external traffic. Each layer has its own configuration surface.

With AWS EKS and the VPC CNI plugin (the default), pods get real VPC IP addresses. This is actually a significant advantage over some other Kubernetes setups — your pods are first-class VPC citizens and you can use security groups directly on pods.

# pod-security-group.yaml
apiVersion: v1
kind: Pod
metadata:
  name: my-app
  annotations:
    vpc.amazonaws.com/pod-eni: "true"
spec:
  securityContext:
    runAsNonRoot: true
  containers:
  - name: app
    image: my-app:latest

# Security group for pods (requires VPC CNI security groups for pods feature)
resource "aws_security_group" "pod_sg" {
  name        = "${var.project_name}-pod-sg"
  description = "Security group for application pods"
  vpc_id      = var.vpc_id

  ingress {
    from_port       = 8080
    to_port         = 8080
    protocol        = "tcp"
    security_groups = [aws_security_group.alb.id]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Service-to-service communication in Kubernetes uses DNS names automatically. Any service is reachable at service-name.namespace.svc.cluster.local. This is one area where Kubernetes is genuinely more elegant than ECS.

# Internal service call - no configuration needed beyond the Service object
apiVersion: v1
kind: Service
metadata:
  name: user-service
  namespace: production
spec:
  selector:
    app: user-service
  ports:
  - port: 80
    targetPort: 8080
  type: ClusterIP
---
# Any pod can now reach this at: http://user-service.production.svc.cluster.local

For external traffic, you need an ingress controller. With the AWS Load Balancer Controller installed, you annotate your Ingress objects and it provisions ALBs automatically:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  namespace: production
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:123456789:certificate/xxx
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443},{"HTTP":80}]'
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/healthcheck-path: /health
spec:
  rules:
  - host: api.yourdomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app-service
            port:
              number: 80

This works well once it's set up. But getting to "works well" requires understanding ALB controller annotations, troubleshooting why the ALB isn't being provisioned, and handling the IRSA (IAM Roles for Service Accounts) configuration for the controller itself.

Verdict on networking: ECS is simpler for straightforward use cases. EKS networking is more powerful and flexible, especially for multi-service architectures, but requires more operational knowledge.

Autoscaling

ECS Autoscaling

ECS autoscaling has two dimensions: scaling the number of tasks (Application Auto Scaling) and scaling the underlying compute (if using EC2 launch type — not needed for Fargate).

# Scale tasks based on CPU
resource "aws_appautoscaling_target" "ecs_target" {
  max_capacity       = 50
  min_capacity       = 2
  resource_id        = "service/${aws_ecs_cluster.main.name}/${aws_ecs_service.app.name}"
  scalable_dimension = "ecs:service:DesiredCount"
  service_namespace  = "ecs"
}

resource "aws_appautoscaling_policy" "cpu_scaling" {
  name               = "${var.project_name}-cpu-scaling"
  policy_type        = "TargetTrackingScaling"
  resource_id        = aws_appautoscaling_target.ecs_target.resource_id
  scalable_dimension = aws_appautoscaling_target.ecs_target.scalable_dimension
  service_namespace  = aws_appautoscaling_target.ecs_target.service_namespace

  target_tracking_scaling_policy_configuration {
    target_value       = 60.0
    scale_in_cooldown  = 300
    scale_out_cooldown = 60

    predefined_metric_specification {
      predefined_metric_type = "ECSServiceAverageCPUUtilization"
    }
  }
}

# Scale based on ALB request count per target
resource "aws_appautoscaling_policy" "request_scaling" {
  name               = "${var.project_name}-request-scaling"
  policy_type        = "TargetTrackingScaling"
  resource_id        = aws_appautoscaling_target.ecs_target.resource_id
  scalable_dimension = aws_appautoscaling_target.ecs_target.scalable_dimension
  service_namespace  = aws_appautoscaling_target.ecs_target.service_namespace

  target_tracking_scaling_policy_configuration {
    target_value       = 1000.0
    scale_in_cooldown  = 300
    scale_out_cooldown = 60

    predefined_metric_specification {
      predefined_metric_type = "ALBRequestCountPerTarget"
      resource_label = "${aws_lb.main.arn_suffix}/${aws_lb_target_group.app.arn_suffix}"
    }
  }
}

With Fargate, you don't manage nodes at all. AWS handles the underlying compute. You define the CPU and memory your task needs, and Fargate provisions capacity. Scaling the task count scales your actual compute footprint automatically.

This is genuinely magical for teams that don't want to think about node sizing, node upgrades, or compute capacity planning.

EKS Autoscaling

EKS autoscaling has three dimensions: HPA (Horizontal Pod Autoscaler) for pods, Cluster Autoscaler or Karpenter for nodes, and optionally VPA (Vertical Pod Autoscaler) for right-sizing resource requests.

HPA for pods:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: app
  minReplicas: 2
  maxReplicas: 50
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 10
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15

Karpenter for node provisioning (the modern approach, replacing Cluster Autoscaler):

apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  requirements:
    - key: karpenter.sh/capacity-type
      operator: In
      values: ["spot", "on-demand"]
    - key: kubernetes.io/arch
      operator: In
      values: ["arm64"]
    - key: node.kubernetes.io/instance-type
      operator: In
      values: ["m6g.large", "m6g.xlarge", "m6g.2xlarge", "c6g.large", "c6g.xlarge"]
  limits:
    resources:
      cpu: "200"
      memory: 800Gi
  providerRef:
    name: default
  ttlSecondsAfterEmpty: 30
  ttlSecondsUntilExpired: 2592000  # 30 days - forces node rotation
---
apiVersion: karpenter.k8s.aws/v1alpha1
kind: AWSNodeTemplate
metadata:
  name: default
spec:
  subnetSelector:
    karpenter.sh/discovery: ${CLUSTER_NAME}
  securityGroupSelector:
    karpenter.sh/discovery: ${CLUSTER_NAME}
  instanceProfile: KarpenterNodeInstanceProfile-${CLUSTER_NAME}
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 50Gi
        volumeType: gp3
        encrypted: true

Karpenter is genuinely excellent. It provisions nodes in seconds, uses spot instances intelligently, and right-sizes node types based on actual pod requirements. If you're running EKS, Karpenter is worth the setup investment.

But notice what I mean about layers: you have the application (pods), the HPA controlling pod count, Karpenter controlling node count, and then the interaction between the two when a scale event happens. All of that works well when properly configured. Debugging it when it doesn't work requires understanding all three layers simultaneously.

Verdict on autoscaling: ECS with Fargate is simpler and honestly sufficient for most use cases. EKS with Karpenter is more powerful and cost-optimized but requires more operational investment.

Cost

This is the section people want most and where comparison posts are most misleading.

Let me be honest: it depends heavily on your workload profile, and anyone who gives you a simple "ECS is cheaper" or "EKS is cheaper" is oversimplifying.

That said, here are the real cost levers:

ECS Fargate Costs

Fargate pricing is per-second based on the vCPU and memory you allocate to each task.

Fargate pricing (us-east-1, arm64):
- $0.03238 per vCPU-hour
- $0.00356 per GB-hour

Example: 0.5 vCPU, 1GB memory task running 24/7 for 30 days:
- vCPU: 0.5 × $0.03238 × 720 hours = $11.66/month
- Memory: 1 × $0.00356 × 720 hours = $2.56/month
- Total per task: ~$14.22/month

With 10 tasks: ~$142/month. With 50 tasks: ~$711/month.

The key insight: you pay for allocated resources, not actual usage. A task allocated 1 vCPU that runs at 10% CPU still costs the same as one running at 90%. If your workloads have inconsistent utilization, Fargate has waste built into it.

ECS EC2 Launch Type Costs

If you use ECS with EC2 instead of Fargate, you pay for the EC2 instances whether they're fully utilized or not. This can be cheaper than Fargate at high, consistent utilization, and more expensive at low or variable utilization.

EKS Costs

EKS charges $0.10/hour per cluster ($72/month regardless of what's running on it). Your actual compute is charged at EC2 rates (or Fargate rates if using Fargate with EKS).

With Karpenter and spot instances, EKS workloads can be significantly cheaper than ECS Fargate at scale. Spot instances for m6g.large run at about 70% discount compared to on-demand, and Karpenter will use spot by default when available.

But this cost advantage only materializes if:

Your workloads tolerate spot interruptions (most stateless services do)
You have enough workload to pack nodes efficiently
You've invested in proper resource requests/limits so scheduling is efficient

Below a certain scale (roughly under 20 concurrent tasks for most workloads), Fargate's simplicity is worth the premium. Above that scale, the math starts favoring managed EC2 nodes with spot.

My rough real-world numbers from Company 3:

Workload	ECS Fargate	EKS + Karpenter (spot)	Savings
50 API pods (0.5 vCPU, 1GB)	$711/month	$290/month	59%
10 background workers (2 vCPU, 4GB)	$512/month	$195/month	62%
5 ML inference (4 vCPU, 16GB)	$1,024/month	$380/month	63%

These numbers are real but context-dependent. The EKS cluster itself costs $72/month, plus you need to budget time for cluster maintenance and upgrades.

Deployments and Day-2 Operations

ECS Deployments

ECS rolling deployments are controlled by two parameters: deployment_maximum_percent and deployment_minimum_healthy_percent. The defaults (200% and 100%) mean ECS will bring up new tasks before draining old ones, ensuring no capacity loss during deployments.

resource "aws_ecs_service" "app" {
  # ...

  deployment_maximum_percent         = 200
  deployment_minimum_healthy_percent = 100

  deployment_circuit_breaker {
    enable   = true
    rollback = true
  }
}

The circuit breaker with automatic rollback is the feature I care most about. If a deployment fails health checks, ECS rolls back automatically. No human intervention needed at 2am.

For blue/green deployments, you can use CodeDeploy with ECS:

resource "aws_codedeploy_deployment_group" "ecs" {
  app_name               = aws_codedeploy_app.main.name
  deployment_group_name  = "${var.project_name}-${var.environment}"
  deployment_config_name = "CodeDeployDefault.ECSAllAtOnce"
  service_role_arn       = aws_iam_role.codedeploy.arn

  auto_rollback_configuration {
    enabled = true
    events  = ["DEPLOYMENT_FAILURE"]
  }

  blue_green_deployment_config {
    deployment_ready_option {
      action_on_timeout = "CONTINUE_DEPLOYMENT"
    }

    terminate_blue_instances_on_deployment_success {
      action                           = "TERMINATE"
      termination_wait_time_in_minutes = 5
    }
  }

  ecs_service {
    cluster_name = aws_ecs_cluster.main.name
    service_name = aws_ecs_service.app.name
  }

  load_balancer_info {
    target_group_pair_info {
      prod_traffic_route {
        listener_arns = [aws_lb_listener.https.arn]
      }

      target_group { name = aws_lb_target_group.blue.name }
      target_group { name = aws_lb_target_group.green.name }
    }
  }
}

EKS Deployments

Kubernetes rolling deployments are controlled by the Deployment spec's strategy field:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  namespace: production
spec:
  replicas: 5
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: app
        image: my-app:latest
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: "250m"
            memory: "512Mi"
          limits:
            cpu: "500m"
            memory: "1Gi"
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5
          failureThreshold: 3
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
          failureThreshold: 3
      terminationGracePeriodSeconds: 60
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: my-app

That topologySpreadConstraints block at the bottom is something most tutorials don't show but matters in production — it ensures your pods are spread across availability zones instead of all landing on nodes in the same AZ.

Kubernetes doesn't have automatic rollback on failed deployments out of the box. You either set up a deployment process that monitors rollout status and rolls back on failure, or you use a GitOps tool like ArgoCD or Flux that handles this for you.

# Manual rollback
kubectl rollout undo deployment/app -n production

# Check rollout status
kubectl rollout status deployment/app -n production

# View rollout history
kubectl rollout history deployment/app -n production

For true blue/green or canary deployments in EKS, Argo Rollouts is the best option:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: app
  namespace: production
spec:
  replicas: 5
  strategy:
    canary:
      steps:
      - setWeight: 20
      - pause: {duration: 5m}
      - setWeight: 40
      - pause: {duration: 5m}
      - setWeight: 60
      - pause: {duration: 5m}
      - setWeight: 80
      - pause: {duration: 5m}
      canaryService: app-canary
      stableService: app-stable
      trafficRouting:
        alb:
          ingress: app-ingress
          servicePort: 80
  selector:
    matchLabels:
      app: my-app
  template:
    # ... same as Deployment spec

This is genuinely more powerful than anything ECS offers for deployments. But it requires Argo Rollouts installed, the ALB controller configured, and someone who understands how canary routing works.

Verdict on deployments: ECS is simpler with sensible defaults and automatic rollback built in. EKS is more powerful with proper tooling but requires more setup and expertise.

IAM and Security

ECS

ECS uses task-level IAM roles, which is clean and intuitive. Each task gets a role, that role has permissions, done.

resource "aws_iam_role" "ecs_task" {
  name = "${var.project_name}-ecs-task-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "ecs-tasks.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy" "ecs_task_permissions" {
  name = "task-permissions"
  role = aws_iam_role.ecs_task.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["s3:GetObject", "s3:PutObject"]
        Resource = "${aws_s3_bucket.app.arn}/*"
      },
      {
        Effect   = "Allow"
        Action   = ["sqs:SendMessage", "sqs:ReceiveMessage", "sqs:DeleteMessage"]
        Resource = aws_sqs_queue.app.arn
      }
    ]
  })
}

The task role is bound to the task definition, so every task that runs from that definition gets those permissions. Simple and auditable.

EKS

EKS uses IRSA (IAM Roles for Service Accounts) to bind IAM permissions to Kubernetes service accounts. It's more flexible and pod-level, but the setup is more involved.

# IAM role with trust policy for the specific service account
resource "aws_iam_role" "app_service_account" {
  name = "${var.project_name}-app-sa-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Federated = aws_iam_openid_connect_provider.eks.arn
      }
      Action = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "${replace(aws_iam_openid_connect_provider.eks.url, "https://", "")}:sub" = "system:serviceaccount:production:app-service-account"
          "${replace(aws_iam_openid_connect_provider.eks.url, "https://", "")}:aud" = "sts.amazonaws.com"
        }
      }
    }]
  })
}

resource "aws_iam_role_policy" "app_permissions" {
  name = "app-permissions"
  role = aws_iam_role.app_service_account.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["s3:GetObject", "s3:PutObject"]
        Resource = "${aws_s3_bucket.app.arn}/*"
      }
    ]
  })
}

# kubernetes/service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-service-account
  namespace: production
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/my-project-app-sa-role

# kubernetes/deployment.yaml
spec:
  template:
    spec:
      serviceAccountName: app-service-account  # Links to the IAM role
      containers:
      - name: app
        # ...

IRSA is secure and granular — individual pods get individual IAM roles. But it's also more moving parts. The OIDC provider, the trust policy with the exact service account reference, the Kubernetes service account with the annotation, and the deployment referencing the service account. All four pieces have to be correct for it to work.

Verdict on IAM: ECS is simpler. EKS is more granular (pod-level vs task-level). Both are secure when used correctly.

Observability

ECS

CloudWatch is the native home for ECS logs and metrics. Container Insights gives you CPU, memory, network, and storage metrics per task. Log routing is configured in the task definition.

resource "aws_cloudwatch_log_group" "app" {
  name              = "/ecs/${var.project_name}-${var.environment}"
  retention_in_days = 30
}

For ECS on Fargate, FireLens is the way to route logs to multiple destinations (Datadog, Splunk, S3) without changing your application code:

{
  "name": "log_router",
  "image": "public.ecr.aws/aws-observability/aws-for-fluent-bit:stable",
  "essential": true,
  "firelensConfiguration": {
    "type": "fluentbit"
  }
}

EKS

EKS ships logs and metrics to CloudWatch via the CloudWatch agent and Fluent Bit, but this requires setup. The AWS-managed add-on for CloudWatch observability simplifies this:

aws eks create-addon \
  --cluster-name ${CLUSTER_NAME} \
  --addon-name amazon-cloudwatch-observability \
  --service-account-role-arn ${CLOUDWATCH_ROLE_ARN}

For application logs, you deploy Fluent Bit as a DaemonSet that collects from all nodes:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: amazon-cloudwatch
spec:
  selector:
    matchLabels:
      name: fluent-bit
  template:
    metadata:
      labels:
        name: fluent-bit
    spec:
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluent-bit
        image: public.ecr.aws/aws-observability/aws-for-fluent-bit:stable
        resources:
          requests:
            cpu: 50m
            memory: 64Mi
          limits:
            cpu: 200m
            memory: 256Mi
        env:
        - name: AWS_REGION
          value: us-east-1
        - name: CLUSTER_NAME
          value: my-cluster
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

The Kubernetes-native observability story with Prometheus + Grafana is powerful and widely adopted. If your team already operates a Prometheus/Grafana stack or uses a platform like Datadog, EKS integrates naturally.

# Service Monitor for Prometheus scraping
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: app-metrics
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics
  namespaceSelector:
    matchNames:
    - production

Verdict on observability: ECS integrates more naturally with CloudWatch. EKS has a richer ecosystem but requires more setup. If your company is already invested in Prometheus/Grafana, EKS wins this one.

When I'd Actually Choose Each One

Here's where I give you the real opinion instead of the "it depends" cop-out.

Choose ECS When:

Your team is primarily application engineers, not platform engineers. ECS doesn't require deep operational expertise. A team of four backend engineers can own an ECS-based infrastructure without a dedicated DevOps/platform role. EKS cannot make that claim.

You're building on AWS and want tight native integration. ECS is an AWS service. IAM, ALB, CloudWatch, Secrets Manager, Service Connect — all integrate natively without additional tooling. If you're all-in on AWS (and most startups are), this matters.

You have fewer than 50 services and don't need advanced traffic management. Below this threshold, the operational overhead of EKS rarely pays off. You'll spend more time managing the cluster than you save from its flexibility.

Fargate's simplicity is genuinely valuable to you. No node management, no AMI updates, no node security patching. For teams that don't want to think about compute, Fargate is remarkable. You describe what your task needs, it runs.

You need to move fast. A working ECS environment with CI/CD can be set up in a day. EKS done properly takes a week the first time. When time-to-production matters, ECS has a clear advantage.

Choose EKS When:

You have multi-cloud requirements or may need to migrate off AWS. Kubernetes is portable. Your application manifests, your Helm charts, your operational knowledge — it all works on GKE, AKS, or self-managed Kubernetes. ECS knowledge doesn't transfer. If there's any chance you'll need to run workloads outside AWS, this is a significant factor.

You have specialized workloads that need Kubernetes-specific features. GPU workloads for ML inference, jobs that use init containers and sidecars extensively, workloads that need custom schedulers, anything that benefits from the Kubernetes extension ecosystem (custom operators, CRDs). EKS handles these; ECS handles them less gracefully or not at all.

You have a large team with dedicated platform engineers. If you have people whose job is operating the container platform, EKS's complexity becomes manageable and its power becomes accessible. A 3-engineer platform team can run EKS well. A 1-person DevOps team probably can't give it the attention it needs while keeping everything else running.

Cost optimization at scale matters. At 100+ pods, the combination of EC2 spot instances and Karpenter's bin-packing can deliver meaningful savings over Fargate. The math becomes compelling at scale in a way it doesn't for smaller deployments.

You're already using Kubernetes elsewhere. If your team runs EKS and you're adding a new service, it goes on EKS. The operational patterns are established, the monitoring is set up, the oncall runbook exists. Don't introduce ECS complexity just to avoid adding another EKS service.

The Hard Truths Nobody Puts in Comparison Posts

EKS oncall is harder. When a production incident happens at 2am on EKS, you're debugging Kubernetes. You need to understand pod states, node conditions, CNI issues, RBAC errors, resource quota exhaustion, and admission webhook failures in addition to your actual application. ECS incidents are usually simpler: bad task definition, failing health check, insufficient IAM permissions. The debugging surface is smaller.

ECS has an upgrade story that doesn't wake you up at night. EKS clusters need to be upgraded every 14 months or so (AWS supports each Kubernetes minor version for about that long before end of life). Node groups need upgrading separately from the control plane. Add-ons need upgrading separately from nodes. Each upgrade is a project. ECS manages runtime upgrades for you on Fargate. It's not zero effort, but it's dramatically less.

Kubernetes expertise is widely available; ECS expertise is less so. If you're hiring, more engineers know Kubernetes than ECS. This cuts both ways: it's easier to hire for EKS, but it also means your engineers may resist ECS or want to migrate away from it.

ECS doesn't have a great multi-tenant story. If you're building a platform that hosts multiple teams' workloads and you need namespace-level isolation, RBAC, resource quotas per team — Kubernetes does this natively. ECS doesn't have a clean equivalent. You end up using separate clusters or complex IAM setups to achieve similar isolation.

The Honest Recommendation

If you're starting fresh in 2026 with a team of fewer than 15 engineers and no specific requirements driving you toward Kubernetes:

Start with ECS.

You'll ship faster. Your oncall will be less complex. The AWS integration is better. When (if) you outgrow it, migration to EKS is a well-understood project, not a crisis.

If you're starting fresh with a large engineering organization, existing Kubernetes knowledge, multi-cloud requirements, or complex workload requirements (GPU, custom schedulers, extensive sidecar patterns):

Start with EKS.

But staff it properly. An EKS cluster that nobody fully understands is worse than an ECS setup that everyone can operate. Kubernetes complexity in the wrong hands creates incidents. I've seen it happen at Company 2, and it's entirely avoidable.

Quick Reference

Factor	ECS	EKS
Initial setup time	Hours	Days
Operational complexity	Low	High
AWS integration	Native	Via add-ons
Multi-cloud portability	No	Yes
Cost (small scale)	Competitive	Higher (cluster overhead)
Cost (large scale)	Higher	Lower (spot + packing)
Fargate support	First-class	Supported
Blue/Green deployments	Via CodeDeploy	Via Argo Rollouts
Service mesh	App Mesh / Service Connect	Istio, Linkerd, Cilium
Advanced scheduling	Limited	Full Kubernetes scheduler
Team expertise required	Moderate	High
Upgrade burden	Low (Fargate handles it)	High (nodes + control plane)

Final Thought

The engineers who have the strongest opinions about ECS vs EKS are usually the ones who've only used one of them in production. The engineers who've used both tend to be more pragmatic — they reach for whichever tool fits the problem.

Both ECS and EKS are mature, well-supported platforms. Both can run production workloads reliably. The choice is about operational tradeoffs, team capabilities, and what your workloads actually need — not about which one is objectively better.

Use the simpler one until you have a concrete reason not to.

I'm curious what others have found in practice — especially if you've migrated from one to the other. Drop a comment with what surprised you most. Migrations usually surface the tradeoffs that no comparison post captures.

GitOps on AWS: End-to-End CI/CD Pipeline with CodePipeline + Terraform + ECS — Mistakes, Fixes & Lessons Learned

Rahul Pandya — Sat, 23 May 2026 12:08:15 +0000

I want to be upfront about something before we get into the technical stuff: this pipeline took me three attempts to get right. The first version worked locally and fell apart in staging. The second version made it to production and immediately caused an incident on a Friday afternoon. The third version — the one I'm going to walk you through — has been running cleanly for several months now.

I'm writing this because when I was building this, I couldn't find a single resource that went end-to-end and talked honestly about what actually goes wrong. Most tutorials show you the happy path. This one shows you the cliffs on either side of it.

Let's build something real.

What We're Actually Building

Before I dump architecture diagrams on you, here's the plain-English version of what this pipeline does:

A developer pushes code to a GitHub branch. Within a few minutes, that code is tested, built into a container image, pushed to ECR, and deployed to ECS Fargate — all without anyone SSHing into anything or clicking buttons in the console. Terraform manages every piece of infrastructure. If something goes wrong at any stage, the pipeline stops and notifies the team. If a deployment goes sideways, ECS rolls back automatically.

That's it. That's the whole goal.

Here's what the stack looks like:

Source control: GitHub (connected via CodeStar Connections)
CI/CD orchestration: AWS CodePipeline
Build: AWS CodeBuild
Infrastructure as Code: Terraform (with remote state in S3 + DynamoDB locking)
Container registry: Amazon ECR
Compute: Amazon ECS on Fargate
Secrets: AWS Secrets Manager
Notifications: SNS + Slack webhook

Prerequisites

Before anything else, make sure you have:

AWS CLI configured with a profile that has sufficient permissions
Terraform >= 1.5 installed
Docker installed locally (for testing builds)
A GitHub repo with your application code
An AWS account where you can create IAM roles, VPCs, ECS clusters, etc.

I'm going to assume you're comfortable with Terraform basics and have touched ECS at least once. I won't explain what a task definition is from scratch.

Part 1: Terraform Project Structure

One of the biggest mistakes I made early on was treating Terraform as an afterthought — writing pipeline config first and then trying to retrofit IaC around it. Don't do that. Start with Terraform.

Here's the directory layout I settled on after a lot of trial and error:

infrastructure/
├── environments/
│   ├── dev/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── terraform.tfvars
│   └── prod/
│       ├── main.tf
│       ├── variables.tf
│       └── terraform.tfvars
├── modules/
│   ├── ecr/
│   ├── ecs/
│   ├── codepipeline/
│   ├── iam/
│   └── networking/
└── backend.tf

The environments folder separates dev and prod so they're completely independent state files. This saved me twice when I fat-fingered something in dev and was glad it couldn't touch prod.

Setting Up Remote State First

Do this before anything else. Local state in a team environment is how you end up with two people applying Terraform at the same time and corrupting your infrastructure.

# backend.tf
terraform {
  backend "s3" {
    bucket         = "your-company-terraform-state"
    key            = "gitops-demo/dev/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-state-lock"
    encrypt        = true
  }
}

Create the S3 bucket and DynamoDB table manually before running terraform init. The DynamoDB table needs a LockID string attribute as its primary key.

# One-time bootstrap
aws s3api create-bucket \
  --bucket your-company-terraform-state \
  --region us-east-1

aws s3api put-bucket-versioning \
  --bucket your-company-terraform-state \
  --versioning-configuration Status=Enabled

aws dynamodb create-table \
  --table-name terraform-state-lock \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST \
  --region us-east-1

Enable versioning on the bucket. I learned this the hard way when a botched apply wiped out a resource and I needed to roll back the state file.

Part 2: Networking Module

Public subnets for the load balancer, private subnets for the Fargate tasks. The tasks should never be directly internet-accessible.

# modules/networking/main.tf

resource "aws_vpc" "main" {
  cidr_block           = var.vpc_cidr
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name        = "${var.project_name}-${var.environment}-vpc"
    Environment = var.environment
    ManagedBy   = "terraform"
  }
}

resource "aws_subnet" "public" {
  count                   = length(var.availability_zones)
  vpc_id                  = aws_vpc.main.id
  cidr_block              = cidrsubnet(var.vpc_cidr, 4, count.index)
  availability_zone       = var.availability_zones[count.index]
  map_public_ip_on_launch = true

  tags = {
    Name = "${var.project_name}-${var.environment}-public-${count.index + 1}"
  }
}

resource "aws_subnet" "private" {
  count             = length(var.availability_zones)
  vpc_id            = aws_vpc.main.id
  cidr_block        = cidrsubnet(var.vpc_cidr, 4, count.index + length(var.availability_zones))
  availability_zone = var.availability_zones[count.index]

  tags = {
    Name = "${var.project_name}-${var.environment}-private-${count.index + 1}"
  }
}

resource "aws_internet_gateway" "main" {
  vpc_id = aws_vpc.main.id
  tags   = { Name = "${var.project_name}-${var.environment}-igw" }
}

resource "aws_eip" "nat" {
  count  = length(var.availability_zones)
  domain = "vpc"
}

resource "aws_nat_gateway" "main" {
  count         = length(var.availability_zones)
  allocation_id = aws_eip.nat[count.index].id
  subnet_id     = aws_subnet.public[count.index].id
  depends_on    = [aws_internet_gateway.main]

  tags = { Name = "${var.project_name}-${var.environment}-nat-${count.index + 1}" }
}

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.main.id
  }
}

resource "aws_route_table" "private" {
  count  = length(var.availability_zones)
  vpc_id = aws_vpc.main.id
  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.main[count.index].id
  }
}

resource "aws_route_table_association" "public" {
  count          = length(var.availability_zones)
  subnet_id      = aws_subnet.public[count.index].id
  route_table_id = aws_route_table.public.id
}

resource "aws_route_table_association" "private" {
  count          = length(var.availability_zones)
  subnet_id      = aws_subnet.private[count.index].id
  route_table_id = aws_route_table.private[count.index].id
}

⚠️ Mistake #1: I originally used a single NAT gateway to save money. Fine for dev, but in prod this became a single point of failure when the AZ hosting the NAT had a brief issue. Use one NAT gateway per AZ in production.

Part 3: ECR Repository

# modules/ecr/main.tf

resource "aws_ecr_repository" "app" {
  name                 = "${var.project_name}-${var.environment}"
  image_tag_mutability = "IMMUTABLE"

  image_scanning_configuration {
    scan_on_push = true
  }

  encryption_configuration {
    encryption_type = "AES256"
  }

  tags = {
    Name        = "${var.project_name}-${var.environment}"
    Environment = var.environment
    ManagedBy   = "terraform"
  }
}

resource "aws_ecr_lifecycle_policy" "app" {
  repository = aws_ecr_repository.app.name

  policy = jsonencode({
    rules = [
      {
        rulePriority = 1
        description  = "Keep last 10 production images"
        selection = {
          tagStatus     = "tagged"
          tagPrefixList = ["prod-"]
          countType     = "imageCountMoreThan"
          countNumber   = 10
        }
        action = { type = "expire" }
      },
      {
        rulePriority = 2
        description  = "Remove untagged images after 1 day"
        selection = {
          tagStatus   = "untagged"
          countType   = "sinceImagePushed"
          countUnit   = "days"
          countNumber = 1
        }
        action = { type = "expire" }
      }
    ]
  })
}

The lifecycle policy is something most tutorials skip entirely. Without it, your ECR storage costs quietly creep up as old images pile up. The IMMUTABLE tag setting is also non-negotiable if you care about knowing exactly what's running in production — mutable tags let someone overwrite latest and suddenly your rollback target is a mystery.

Part 4: IAM Roles

IAM is where most people either lock things down too tight (and spend two hours debugging permission errors) or give everything AdministratorAccess and move on. Neither is good. Here are the three roles you need.

# modules/iam/main.tf

# CodePipeline Role
resource "aws_iam_role" "codepipeline" {
  name = "${var.project_name}-${var.environment}-codepipeline-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "codepipeline.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy" "codepipeline" {
  name = "codepipeline-policy"
  role = aws_iam_role.codepipeline.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = ["s3:GetObject", "s3:GetObjectVersion", "s3:PutObject", "s3:GetBucketVersioning"]
        Resource = ["${aws_s3_bucket.artifacts.arn}", "${aws_s3_bucket.artifacts.arn}/*"]
      },
      {
        Effect   = "Allow"
        Action   = ["codebuild:BatchGetBuilds", "codebuild:StartBuild"]
        Resource = "*"
      },
      {
        Effect = "Allow"
        Action = [
          "ecs:DescribeServices", "ecs:DescribeTaskDefinition",
          "ecs:DescribeTasks", "ecs:ListTasks",
          "ecs:RegisterTaskDefinition", "ecs:UpdateService"
        ]
        Resource = "*"
      },
      {
        Effect   = "Allow"
        Action   = ["iam:PassRole"]
        Resource = aws_iam_role.ecs_task_execution.arn
      },
      {
        Effect   = "Allow"
        Action   = ["codestar-connections:UseConnection"]
        Resource = var.codestar_connection_arn
      }
    ]
  })
}

# CodeBuild Role
resource "aws_iam_role" "codebuild" {
  name = "${var.project_name}-${var.environment}-codebuild-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "codebuild.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy" "codebuild" {
  name = "codebuild-policy"
  role = aws_iam_role.codebuild.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents"]
        Resource = "arn:aws:logs:${var.region}:${var.account_id}:log-group:/aws/codebuild/*"
      },
      {
        Effect   = "Allow"
        Action   = ["s3:GetObject", "s3:PutObject", "s3:GetBucketVersioning"]
        Resource = ["${aws_s3_bucket.artifacts.arn}", "${aws_s3_bucket.artifacts.arn}/*"]
      },
      {
        Effect = "Allow"
        Action = [
          "ecr:GetAuthorizationToken", "ecr:BatchCheckLayerAvailability",
          "ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage",
          "ecr:InitiateLayerUpload", "ecr:UploadLayerPart",
          "ecr:CompleteLayerUpload", "ecr:PutImage"
        ]
        Resource = "*"
      },
      {
        Effect   = "Allow"
        Action   = ["secretsmanager:GetSecretValue"]
        Resource = "arn:aws:secretsmanager:${var.region}:${var.account_id}:secret:${var.project_name}/*"
      }
    ]
  })
}

# ECS Task Execution Role
resource "aws_iam_role" "ecs_task_execution" {
  name = "${var.project_name}-${var.environment}-ecs-execution-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Principal = { Service = "ecs-tasks.amazonaws.com" }
      Action    = "sts:AssumeRole"
    }]
  })
}

resource "aws_iam_role_policy_attachment" "ecs_task_execution" {
  role       = aws_iam_role.ecs_task_execution.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

resource "aws_iam_role_policy" "ecs_task_execution_secrets" {
  name = "secrets-access"
  role = aws_iam_role.ecs_task_execution.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["secretsmanager:GetSecretValue", "kms:Decrypt"]
      Resource = ["arn:aws:secretsmanager:${var.region}:${var.account_id}:secret:${var.project_name}/*"]
    }]
  })
}

⚠️ Mistake #2: My first CodeBuild role had ecr:* and s3:* as wildcard resources. It worked, but our security team flagged it immediately. The version above scopes ECR actions correctly — ecr:GetAuthorizationToken doesn't support resource-level restrictions, so * is unavoidable there, but Secrets Manager is scoped to a specific path prefix. Always do this.

Part 5: ECS Cluster, Task Definition, and Service

# modules/ecs/main.tf

resource "aws_ecs_cluster" "main" {
  name = "${var.project_name}-${var.environment}"

  setting {
    name  = "containerInsights"
    value = "enabled"
  }

  tags = {
    Name        = "${var.project_name}-${var.environment}"
    Environment = var.environment
    ManagedBy   = "terraform"
  }
}

resource "aws_ecs_cluster_capacity_providers" "main" {
  cluster_name       = aws_ecs_cluster.main.name
  capacity_providers = ["FARGATE", "FARGATE_SPOT"]

  default_capacity_provider_strategy {
    base              = 1
    weight            = 100
    capacity_provider = "FARGATE"
  }
}

resource "aws_cloudwatch_log_group" "app" {
  name              = "/ecs/${var.project_name}-${var.environment}"
  retention_in_days = 30
}

resource "aws_ecs_task_definition" "app" {
  family                   = "${var.project_name}-${var.environment}"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = var.task_cpu
  memory                   = var.task_memory
  execution_role_arn       = var.execution_role_arn
  task_role_arn            = var.task_role_arn

  container_definitions = jsonencode([
    {
      name      = var.container_name
      image     = "${var.ecr_repository_url}:${var.image_tag}"
      essential = true

      portMappings = [{ containerPort = var.container_port, protocol = "tcp" }]

      environment = [
        { name = "APP_ENV", value = var.environment },
        { name = "PORT",    value = tostring(var.container_port) }
      ]

      # NOTE: Use secrets block, NOT environment block for sensitive values
      secrets = [
        {
          name      = "DATABASE_URL"
          valueFrom = "arn:aws:secretsmanager:${var.region}:${var.account_id}:secret:${var.project_name}/${var.environment}/database-url"
        }
      ]

      logConfiguration = {
        logDriver = "awslogs"
        options = {
          "awslogs-group"         = "/ecs/${var.project_name}-${var.environment}"
          "awslogs-region"        = var.region
          "awslogs-stream-prefix" = "ecs"
        }
      }

      healthCheck = {
        command     = ["CMD-SHELL", "curl -f http://localhost:${var.container_port}/health || exit 1"]
        interval    = 30
        timeout     = 5
        retries     = 3
        startPeriod = 60
      }
    }
  ])

  lifecycle {
    ignore_changes = [container_definitions]
  }
}

resource "aws_lb" "main" {
  name                       = "${var.project_name}-${var.environment}-alb"
  internal                   = false
  load_balancer_type         = "application"
  security_groups            = [aws_security_group.alb.id]
  subnets                    = var.public_subnet_ids
  enable_deletion_protection = var.environment == "prod" ? true : false
}

resource "aws_lb_target_group" "app" {
  name        = "${var.project_name}-${var.environment}-tg"
  port        = var.container_port
  protocol    = "HTTP"
  vpc_id      = var.vpc_id
  target_type = "ip"

  health_check {
    enabled             = true
    healthy_threshold   = 2
    interval            = 30
    matcher             = "200"
    path                = "/health"
    timeout             = 5
    unhealthy_threshold = 3
  }

  lifecycle { create_before_destroy = true }
}

resource "aws_lb_listener" "http" {
  load_balancer_arn = aws_lb.main.arn
  port              = "80"
  protocol          = "HTTP"

  default_action {
    type = "redirect"
    redirect {
      port        = "443"
      protocol    = "HTTPS"
      status_code = "HTTP_301"
    }
  }
}

resource "aws_lb_listener" "https" {
  load_balancer_arn = aws_lb.main.arn
  port              = "443"
  protocol          = "HTTPS"
  ssl_policy        = "ELBSecurityPolicy-TLS13-1-2-2021-06"
  certificate_arn   = var.certificate_arn

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.app.arn
  }
}

resource "aws_ecs_service" "app" {
  name            = "${var.project_name}-${var.environment}"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.app.arn
  desired_count   = var.desired_count

  capacity_provider_strategy {
    capacity_provider = "FARGATE"
    weight            = 100
    base              = 1
  }

  network_configuration {
    subnets          = var.private_subnet_ids
    security_groups  = [aws_security_group.ecs_tasks.id]
    assign_public_ip = false
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.app.arn
    container_name   = var.container_name
    container_port   = var.container_port
  }

  deployment_controller { type = "ECS" }

  deployment_circuit_breaker {
    enable   = true
    rollback = true
  }

  deployment_maximum_percent         = 200
  deployment_minimum_healthy_percent = 100

  # Critical: let CodePipeline own the task definition after initial deploy
  lifecycle {
    ignore_changes = [task_definition, desired_count]
  }

  depends_on = [aws_lb_listener.https]
}

The deployment_circuit_breaker with rollback = true is the safety net I wish I'd had from day one. If a deployment starts failing health checks, ECS automatically rolls back to the previous task definition — no human intervention needed at 2am.

Part 6: CodeBuild — The buildspec.yml

This lives in your application repository, not your infrastructure repo.

# buildspec.yml
version: 0.2

env:
  variables:
    DOCKER_BUILDKIT: "1"
  secrets-manager:
    DOCKER_HUB_TOKEN: myapp/docker-hub-token

phases:
  install:
    runtime-versions:
      docker: 20

  pre_build:
    commands:
      - echo "Logging in to Amazon ECR..."
      - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $ECR_REGISTRY
      # Pull cache image first - without this, --cache-from does nothing
      - docker pull $ECR_REGISTRY/$ECR_REPOSITORY_NAME:latest || true
      - IMAGE_TAG=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c1-7)
      - FULL_IMAGE_URI=$ECR_REGISTRY/$ECR_REPOSITORY_NAME:$IMAGE_TAG
      - echo "Image will be tagged as $FULL_IMAGE_URI"

  build:
    commands:
      - echo "Running tests..."
      - docker build --target test -t app-test .
      - docker run --rm app-test
      - echo "Build started on $(date)"
      - |
        docker build \
          --build-arg BUILD_DATE=$(date -u +'%Y-%m-%dT%H:%M:%SZ') \
          --build-arg GIT_COMMIT=$CODEBUILD_RESOLVED_SOURCE_VERSION \
          --cache-from $ECR_REGISTRY/$ECR_REPOSITORY_NAME:latest \
          -t $FULL_IMAGE_URI \
          -t $ECR_REGISTRY/$ECR_REPOSITORY_NAME:latest \
          .

  post_build:
    commands:
      - docker push $FULL_IMAGE_URI
      - docker push $ECR_REGISTRY/$ECR_REPOSITORY_NAME:latest
      - printf '[{"name":"%s","imageUri":"%s"}]' $CONTAINER_NAME $FULL_IMAGE_URI > imagedefinitions.json
      - cat imagedefinitions.json

artifacts:
  files:
    - imagedefinitions.json

cache:
  paths:
    - '/root/.gradle/caches/**/*'
    - '/root/.m2/**/*'

The imagedefinitions.json at the end is what CodePipeline's ECS deploy action reads to know which image to deploy. The format must be exactly right — an array with a single object containing name (matching your container name in the task definition exactly) and imageUri. A lot of people waste time debugging this.

Part 7: CodePipeline

# modules/codepipeline/main.tf

resource "aws_s3_bucket" "artifacts" {
  bucket        = "${var.project_name}-${var.environment}-pipeline-artifacts"
  force_destroy = var.environment != "prod"
}

resource "aws_s3_bucket_versioning" "artifacts" {
  bucket = aws_s3_bucket.artifacts.id
  versioning_configuration { status = "Enabled" }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "artifacts" {
  bucket = aws_s3_bucket.artifacts.id
  rule {
    apply_server_side_encryption_by_default { sse_algorithm = "AES256" }
  }
}

resource "aws_s3_bucket_public_access_block" "artifacts" {
  bucket                  = aws_s3_bucket.artifacts.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_codebuild_project" "app" {
  name          = "${var.project_name}-${var.environment}-build"
  build_timeout = 20
  service_role  = var.codebuild_role_arn

  artifacts { type = "CODEPIPELINE" }

  cache {
    type     = "S3"
    location = "${aws_s3_bucket.artifacts.bucket}/build-cache"
  }

  environment {
    compute_type                = "BUILD_GENERAL1_SMALL"
    image                       = "aws/codebuild/standard:7.0"
    type                        = "LINUX_CONTAINER"
    image_pull_credentials_type = "CODEBUILD"
    privileged_mode             = true # Required for Docker builds

    environment_variable { name = "ECR_REGISTRY",       value = var.ecr_registry }
    environment_variable { name = "ECR_REPOSITORY_NAME", value = var.ecr_repository_name }
    environment_variable { name = "CONTAINER_NAME",      value = var.container_name }
    environment_variable { name = "ENVIRONMENT",         value = var.environment }
  }

  logs_config {
    cloudwatch_logs {
      group_name  = "/aws/codebuild/${var.project_name}-${var.environment}"
      stream_name = "build-log"
    }
  }

  source { type = "CODEPIPELINE" }
}

resource "aws_codepipeline" "main" {
  name     = "${var.project_name}-${var.environment}-pipeline"
  role_arn = var.codepipeline_role_arn

  artifact_store {
    location = aws_s3_bucket.artifacts.bucket
    type     = "S3"
  }

  stage {
    name = "Source"
    action {
      name             = "Source"
      category         = "Source"
      owner            = "AWS"
      provider         = "CodeStarSourceConnection"
      version          = "1"
      output_artifacts = ["source_output"]

      configuration = {
        ConnectionArn    = var.codestar_connection_arn
        FullRepositoryId = var.github_repository
        BranchName       = var.branch_name
        DetectChanges    = true
      }
    }
  }

  stage {
    name = "Build"
    action {
      name             = "Build"
      category         = "Build"
      owner            = "AWS"
      provider         = "CodeBuild"
      version          = "1"
      input_artifacts  = ["source_output"]
      output_artifacts = ["build_output"]
      configuration    = { ProjectName = aws_codebuild_project.app.name }
    }
  }

  stage {
    name = "Deploy"
    action {
      name            = "Deploy"
      category        = "Deploy"
      owner           = "AWS"
      provider        = "ECS"
      version         = "1"
      input_artifacts = ["build_output"]

      configuration = {
        ClusterName = var.ecs_cluster_name
        ServiceName = var.ecs_service_name
        FileName    = "imagedefinitions.json"
      }
    }
  }
}

Connecting to GitHub — Don't Skip This

The CodeStar Connection requires a one-time manual authorization in the AWS console. Terraform creates the connection resource, but it'll be in PENDING status until you go to Developer Tools → Settings → Connections and click Update pending connection.

resource "aws_codestarconnections_connection" "github" {
  name          = "${var.project_name}-github"
  provider_type = "GitHub"
}

output "codestar_connection_arn" {
  value = aws_codestarconnections_connection.github.arn
}

After terraform apply, go activate the connection manually. This is a security control — AWS will not let Terraform automatically authorize access to your GitHub account.

Part 8: Pipeline Notifications

resource "aws_sns_topic" "pipeline_notifications" {
  name = "${var.project_name}-${var.environment}-pipeline-notifications"
}

resource "aws_sns_topic_subscription" "slack" {
  topic_arn = aws_sns_topic.pipeline_notifications.arn
  protocol  = "https"
  endpoint  = var.slack_webhook_url
}

resource "aws_cloudwatch_event_rule" "pipeline_state_change" {
  name        = "${var.project_name}-${var.environment}-pipeline-state-change"
  description = "Capture CodePipeline state changes"

  event_pattern = jsonencode({
    source      = ["aws.codepipeline"]
    detail-type = ["CodePipeline Pipeline Execution State Change"]
    detail = {
      pipeline = [aws_codepipeline.main.name]
      state    = ["FAILED", "SUCCEEDED", "STARTED"]
    }
  })
}

resource "aws_cloudwatch_event_target" "pipeline_sns" {
  rule      = aws_cloudwatch_event_rule.pipeline_state_change.name
  target_id = "SendToSNS"
  arn       = aws_sns_topic.pipeline_notifications.arn

  input_transformer {
    input_paths = {
      pipeline = "$.detail.pipeline"
      state    = "$.detail.state"
      time     = "$.time"
    }
    input_template = "\"Pipeline <pipeline> changed state to <state> at <time>\""
  }
}

Part 9: Putting It All Together

# environments/dev/main.tf

module "networking" {
  source             = "../../modules/networking"
  project_name       = var.project_name
  environment        = "dev"
  vpc_cidr           = "10.0.0.0/16"
  availability_zones = ["us-east-1a", "us-east-1b"]
}

module "ecr" {
  source       = "../../modules/ecr"
  project_name = var.project_name
  environment  = "dev"
}

module "iam" {
  source                  = "../../modules/iam"
  project_name            = var.project_name
  environment             = "dev"
  region                  = var.region
  account_id              = var.account_id
  codestar_connection_arn = var.codestar_connection_arn
  artifacts_bucket_arn    = module.codepipeline.artifacts_bucket_arn
}

module "ecs" {
  source              = "../../modules/ecs"
  project_name        = var.project_name
  environment         = "dev"
  region              = var.region
  account_id          = var.account_id
  vpc_id              = module.networking.vpc_id
  public_subnet_ids   = module.networking.public_subnet_ids
  private_subnet_ids  = module.networking.private_subnet_ids
  execution_role_arn  = module.iam.ecs_task_execution_role_arn
  task_role_arn       = module.iam.ecs_task_role_arn
  ecr_repository_url  = module.ecr.repository_url
  container_name      = var.container_name
  container_port      = 8080
  task_cpu            = 256
  task_memory         = 512
  desired_count       = 2
  certificate_arn     = var.certificate_arn
}

module "codepipeline" {
  source                  = "../../modules/codepipeline"
  project_name            = var.project_name
  environment             = "dev"
  codebuild_role_arn      = module.iam.codebuild_role_arn
  codepipeline_role_arn   = module.iam.codepipeline_role_arn
  ecr_registry            = "${var.account_id}.dkr.ecr.${var.region}.amazonaws.com"
  ecr_repository_name     = module.ecr.repository_name
  container_name          = var.container_name
  ecs_cluster_name        = module.ecs.cluster_name
  ecs_service_name        = module.ecs.service_name
  codestar_connection_arn = var.codestar_connection_arn
  github_repository       = var.github_repository
  branch_name             = "develop"
  slack_webhook_url       = var.slack_webhook_url
}

Run these in order the first time:

cd environments/dev
terraform init
terraform plan -out=tfplan
terraform apply tfplan

The Mistakes Section

I promised honesty. Here's what actually broke.

Mistake 3 — Docker layer caching wasn't working in CodeBuild

Builds were taking 12–15 minutes because nothing was being cached. privileged_mode = true is required for Docker-in-Docker, but even with that, --cache-from only works if the image exists locally in the build environment. You need to explicitly pull it first:

pre_build:
  commands:
    - docker pull $ECR_REGISTRY/$ECR_REPOSITORY_NAME:latest || true

The || true prevents failure on the very first run when no cache exists yet. After adding this, builds dropped to under 4 minutes.

Mistake 4 — ECS tasks couldn't pull from ECR (CannotPullContainerError)

This one wasted an entire afternoon. The tasks were in private subnets with NAT gateways and had internet access — but ECR pulls were still failing.

The actual cause: an org-level SCP required ECR to be accessed via VPC endpoints. Once I added the three required endpoints, everything worked:

resource "aws_vpc_endpoint" "ecr_dkr" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.${var.region}.ecr.dkr"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = aws_subnet.private[*].id
  security_group_ids  = [aws_security_group.vpc_endpoints.id]
  private_dns_enabled = true
}

resource "aws_vpc_endpoint" "ecr_api" {
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.${var.region}.ecr.api"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = aws_subnet.private[*].id
  security_group_ids  = [aws_security_group.vpc_endpoints.id]
  private_dns_enabled = true
}

# ECR uses S3 internally for layer storage — this one is easy to miss
resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.${var.region}.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = aws_route_table.private[*].id
}

Even if your org doesn't require VPC endpoints, adding them reduces egress costs and latency. Worth doing regardless.

Mistake 5 — Terraform fighting CodePipeline over the task definition

Every terraform plan after a deployment wanted to reset the ECS service back to the original task definition ARN from .tfvars. The fix is the ignore_changes = [task_definition] lifecycle block on the ECS service. Without it, you have two systems fighting to own the same resource and you'll keep accidentally reverting deployments.

Mistake 6 — The CodeStar Connection was never activated

I spent 45 minutes staring at a pipeline that failed immediately at the Source stage with a cryptic error. The connection showed up in Terraform state as created successfully. Eventually I found it: connections start in PENDING state and need manual authorization in the console.

Always verify the connection status after creating it:

aws codestar-connections list-connections \
  --provider-type GitHub \
  --query 'Connections[*].{Name:ConnectionName,Status:ConnectionStatus}'

Mistake 7 — Secrets in plain environment variables

My first version passed database credentials as plain environment variables in the task definition. Those show up in the ECS console, in CloudTrail, and in any container inspection. Always use the secrets block (shown in Part 5) which references Secrets Manager ARNs. The difference looks small in the code but is significant from a security standpoint.

The Day-to-Day Deployment Workflow

Once everything is provisioned, here's what a normal deployment looks like:

Developer opens a PR against the develop branch
After review and merge, CodePipeline triggers automatically
CodeBuild runs tests and builds the Docker image
Image is pushed to ECR tagged with the commit SHA (first 7 chars)
imagedefinitions.json is written with the new image URI
CodePipeline's ECS deploy action registers a new task definition revision
ECS performs a rolling deployment (min 100% healthy, max 200%)
Health checks confirm new tasks are healthy
Old tasks are drained and stopped
Slack notification fires with SUCCEEDED status

If step 8 fails, the deployment circuit breaker rolls back to the previous task definition automatically.

For production, I added a manual approval stage between Build and Deploy:

stage {
  name = "Approve"
  action {
    name     = "ManualApproval"
    category = "Approval"
    owner    = "AWS"
    provider = "Manual"
    version  = "1"

    configuration = {
      NotificationArn    = aws_sns_topic.pipeline_notifications.arn
      CustomData         = "Approve deployment to production for ${var.project_name}"
      ExternalEntityLink = "https://github.com/${var.github_repository}/compare/main...develop"
    }
  }
}

Monitoring and Observability

The pipeline is only as good as your ability to know when something's wrong.

Check pipeline execution history:

aws codepipeline list-pipeline-executions \
  --pipeline-name your-pipeline-name \
  --max-results 20 \
  --query 'pipelineExecutionSummaries[*].{Status:status,Start:startTime,Trigger:trigger.triggerType}'

First place to look when a deployment is stuck:

aws ecs describe-services \
  --cluster your-cluster-name \
  --services your-service-name \
  --query 'services[0].events[:10]'

CloudWatch Container Insights — enabling this on the ECS cluster (shown in Part 5) gives you CPU, memory, and network metrics per task without any additional instrumentation. Free observability.

Final Thoughts

Three things I'd tell myself before starting this:

IAM permissions are worth getting right from the start. It takes maybe two extra hours to scope things properly, and it saves you from security findings and the awkward conversation about why your build role had admin access.

The circuit breaker and rollback settings on the ECS service are not optional. At some point you will deploy something that fails health checks in production. You want that to roll back automatically, not at 2am when someone pages you.

ignore_changes on your ECS service is what makes the Terraform + CodePipeline combination actually work. Without it, you have two systems fighting over the same resource and you'll keep silently reverting deployments you just shipped.

The full Terraform code for this post is available on GitHub at [link to your repo]. If you run into something I didn't cover, drop a comment — I'm genuinely curious what edge cases show up in different environments and org setups.

If this helped you, share it with your team. The program is worth the effort.