DEV Community

Chem Patrick Edward
Chem Patrick Edward

Posted on Edited on

DevSecOps on AWS: Deploying a Spring PetClinic App to ECS with Zero Internet Egress

#aws #terraform #devops #security

Deploying Spring PetClinic to ECS with Terraform, GitHub Actions, Trivy, and SonarCloud, with no internet access from the private subnets.

Repo: https://github.com/EddieByte/devsecops-ecs-pipeline

Time to recreate: about 45 minutes · Cost while running: about $60–80/month · Level: intermediate AWS/Terraform

What this is, and why I built it

I wanted to see how close I could get to a production-grade AWS deployment without cutting the security corners most tutorials take: no static credentials, no public compute, no manual DB migrations. This is what it took.

The problem. Most "deploy to AWS" tutorials get an app running and stop there. To stay short, they quietly skip the parts that make a deployment safe to run for real: where the credentials live, what the servers can reach, and what gets checked before code ships. Teams that do care about those parts rarely have one end-to-end example they can read, run, and tear down in an afternoon. This project is my attempt to fill that gap.

Here is what typically goes wrong, and what this deployment does about it:

Common problem What this deployment does
Long-lived AWS keys stored in CI GitHub OIDC gives short-lived credentials, with trust scoped to your org and repo
Compute with a route to the internet (data exfiltration, unvetted downloads) Private subnets with no internet route; AWS services reached only through VPC endpoints
Vulnerable code or images reaching production SonarCloud analysis and a Trivy image scan run before anything deploys
Secrets in code or config files Database credentials live in Secrets Manager
Manual database setup that drifts between environments Flyway runs versioned migrations at startup
Failed deployments that hang or retry forever ECS deployment circuit breaker
Hand-built infrastructure and forgotten bills Everything is in Terraform, with a teardown script

If you've built ECS deployments before and want a hardened baseline, or you're learning DevSecOps and want something concrete to run, this is for you.

The goal was a pipeline where code goes from a git tag to a running container without any workload ever touching the public internet, and where security checks are part of the pipeline, not an afterthought.

A note on "air-gapped": here it means the workloads have no route to the internet. They still reach AWS services privately through PrivateLink, so it isn't fully disconnected.

By the end of this deployment, you'll have:

  • A GitHub Actions pipeline: test → scan → build → deploy
  • Code quality checks (SonarCloud) and image vulnerability scanning (Trivy)
  • ECS on EC2 in private subnets, behind an ALB, with RDS for the database
  • Deployments authenticated with OIDC, so there are no AWS access keys in GitHub
  • Everything created and destroyed with Terraform

The architecture 

graph LR
  A["git tag dev/v*"] --> B["TEST: mvn + SonarCloud"]
  B --> C["BUILD & PUBLISH: Docker + Trivy + ECR"]
  C --> D["DEPLOY: render task def + ECS rolling update"]
  D --> E["ALB"] --> F["ECS on EC2 (private subnets)"] --> G[("RDS")]

Architecture diagram: a GitHub Actions pipeline tests, scans, and deploys PetClinic to ECS in private AWS subnets with no internet route

The full picture: pipeline on top, private AWS environment below.

Three decisions shape everything else:

1. No NAT gateway, so VPC endpoints. The instances have no route to the internet. They reach ECR, S3, Secrets Manager, CloudWatch, and SSM through PrivateLink endpoints (9 interface endpoints in total). The workloads can only talk to AWS services you've explicitly allowed.

VPC console endpoints list showing the endpoints used instead of a NAT gateway

The endpoints that replace the NAT gateway.

2. OIDC instead of access keys. GitHub Actions gets short-lived credentials by assuming an IAM role. The trust policy is scoped to your specific org and repo, so a leaked workflow file is useless anywhere else.

3. Security checks inside the pipeline. SonarCloud runs with the tests, and Trivy scans the image before it's pushed to ECR. If either fails, nothing deploys.

Before you start

Tool Version
Terraform >= 1.11.0
AWS CLI >= 2.x
Docker Desktop >= 24.x
JDK 17 + Maven 3.9 17 / 3.9
Git, PowerShell 7+ any / >= 7.x

You'll also need an AWS account with admin access (or a scoped policy for EC2, ECS, ECR, RDS, VPC, IAM, Secrets Manager, S3, KMS, CloudWatch, SSM, Auto Scaling) and a free SonarCloud account with a project, token, and organisation.

What it costs

Resource Spec Monthly
EC2 (x2) t3.medium $30
RDS db.t3.micro, 20 GiB $15
ALB 1 $18
VPC Endpoints (x9) Interface, 2 AZs $14
ECR + CloudWatch small $1
Total $60–80

⚠️ Run the teardown at the end of this post. Leaving it up for a month is the expensive mistake.

Build it

Step 1: Clone and authenticate 

git clone https://github.com/EddieByte/devsecops-ecs-pipeline.git
cd devsecops-ecs-pipeline

aws configure                     # or: aws sso login --profile <profile>
aws sts get-caller-identity       # confirm you're in the right account
Enter fullscreen mode Exit fullscreen mode

Step 2: Bootstrap the Terraform state bucket

Terraform can't create the bucket that stores its own state, so a small script does it once per account. The bucket is versioned, encrypted, and blocks public access.

.\scripts\bootstrap_s3.ps1
Enter fullscreen mode Exit fullscreen mode

The bucket name comes from your account ID (petclinic-tfstate-<ACCOUNT_ID>), and the script checks that backend.tf matches it.

Step 3: Configure your variables

Create terraform/terraform.tfvars (it's git-ignored, so don't commit it):

github_org      = "<your-github-user-or-org>"
github_repo     = "devsecops-ecs-pipeline"
github_org_id   = "<numeric owner ID>"
github_repo_id  = "<numeric repo ID>"
db_password     = "<strong password, min 8 chars>"

# Optional
# aws_region        = "us-east-1"
# ec2_instance_type = "t3.medium"
Enter fullscreen mode Exit fullscreen mode

Why numeric IDs? The OIDC trust policy matches on your org and repo IDs, not just their names. If they're wrong, the pipeline fails later with an AssumeRoleWithWebIdentity error (see "What broke" below). Here is how to get them:

Option 1: GitHub CLI

gh api repos/<owner>/<repo> --jq '.id'          # github_repo_id
gh api repos/<owner>/<repo> --jq '.owner.id'    # github_org_id
Enter fullscreen mode Exit fullscreen mode

Option 2: No GitHub CLI

Open https://api.github.com/repos/<owner>/<repo> in your browser. The repo's "id" is github_repo_id, and "owner": { "id": ... } is github_org_id.

Step 4: Provision the infrastructure 

cd terraform
terraform init
terraform plan     # expect about 57 resources
terraform apply
Enter fullscreen mode Exit fullscreen mode

This takes 8–12 minutes, and RDS accounts for about 7 of them. Keep the outputs handy: alb_dns_name, ecr_repository_url, and github_actions_role_arn.

Terminal output of terraform apply showing resources added and the outputs

A clean apply, with the outputs you'll need next.

Step 5: Create the database secret (manual, don't skip)

The app reads its DB credentials from Secrets Manager. Use the same password you put in terraform.tfvars:

aws secretsmanager create-secret `
  --name "dev/app/rds" `
  --region "us-east-1" `
  --secret-string "{""RDS_USER"":""admin"",""RDS_PASS"":""<your-password>""}"
Enter fullscreen mode Exit fullscreen mode

Step 6: Add the GitHub secrets

Go to your repo → Settings → Secrets and variables → Actions and add:

Secret Value
AWS_ROLE_TO_ASSUME output github_actions_role_arn
AWS_REGION us-east-1
REGISTRY <ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com (the ECR URL without /petclinic)
SONAR_TOKEN SonarCloud → My Account → Security
SONAR_PROJECT_KEY SonarCloud → Project Information
SONAR_ORGANIZATION SonarCloud → Organisation

Step 7: Trigger the pipeline 

cd ..
git tag dev/v1.0.0
git push origin dev
git push origin dev/v1.0.0
Enter fullscreen mode Exit fullscreen mode

Go to the Actions tab. Three jobs run in sequence:

  1. TEST: mvn clean test plus SonarCloud analysis
  2. BUILD_AND_PUBLISH: Docker build, Trivy scan, push to ECR
  3. DEPLOY: three steps. It downloads the live task definition, uses amazon-ecs-render-task-definition to swap in the new image tag, then deploys the result as a rolling update. Rendering into the live definition means you only change the image and keep everything else Terraform set up.

GitHub Actions run with the test, build and publish, and deploy jobs all passing

TEST, BUILD_AND_PUBLISH, and DEPLOY, all passing.

Trivy scan output in the build and publish job log

Trivy scans the image before it reaches ECR.

SonarCloud dashboard showing the quality gate result

The SonarCloud quality gate for the same commit.

Prove it works

Don't just trust the green checkmarks. Check each layer.

Check 1: Targets are healthy behind the ALB 

aws elbv2 describe-target-health `
  --target-group-arn $(aws elbv2 describe-target-groups --names petclinic-tg `
    --query "TargetGroups[0].TargetGroupArn" --output text) `
  --query "TargetHealthDescriptions[].{Target:Target.Id,State:TargetHealth.State}" `
  --output table
Enter fullscreen mode Exit fullscreen mode

ALB target group showing both targets healthy

Both ECS instances are healthy behind the ALB.

Check 2: The app is up 

$alb = terraform -chdir=terraform output -raw alb_dns_name
Invoke-WebRequest -Uri "http://$alb/actuator/health" | Select-Object -ExpandProperty Content
# {"status":"UP"}
Enter fullscreen mode Exit fullscreen mode

Open http://<alb_dns_name>/ and you should see the PetClinic UI with owners, pets, and vets.

Spring PetClinic page loaded from the ALB URL (1 of 3)

Spring PetClinic page loaded from the ALB URL (2 of 3)

Spring PetClinic page loaded from the ALB URL (3 of 3)

The app running behind the ALB.

Check 3: Flyway ran the migrations 

$stream = aws logs describe-log-streams --log-group-name "/ecs/petclinic-cluster" `
  --order-by LastEventTime --descending --query "logStreams[0].logStreamName" --output text

aws logs get-log-events --log-group-name "/ecs/petclinic-cluster" --log-stream-name $stream `
  --query "events[*].message" --output text | Select-String -Pattern "flyway|migration" -CaseSensitive:$false
Enter fullscreen mode Exit fullscreen mode

You should see Successfully applied 2 migrations to schema 'petclinic'.

CloudWatch log lines showing Flyway applying two migrations

Flyway applying both migrations on startup.

Check 4: The air-gap is real

This is the check that matters most. Connect through SSM (no SSH keys or open ports):

$instance = aws ec2 describe-instances `
  --filters "Name=tag:aws:autoscaling:groupName,Values=petclinic-ecs-asg" "Name=instance-state-name,Values=running" `
  --query "Reservations[0].Instances[0].InstanceId" --output text
aws ssm start-session --target $instance
Enter fullscreen mode Exit fullscreen mode

Then run these two commands inside the session:

curl --max-time 5 https://google.com
# Expected: Connection timed out. No internet route.

curl -s https://<ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com/v2/ -o /dev/null -w "%{http_code}"
# Expected: 401. ECR is reachable over PrivateLink, just not authenticated.
Enter fullscreen mode Exit fullscreen mode

Test 1: curl to google.com times out after 5000 milliseconds, proving there is no internet route

Test 2: the ECR endpoint returns 401, so it is reachable over PrivateLink

Test 1 times out, Test 2 returns 401: no route to the internet, but ECR is reachable over PrivateLink.

A timeout on Google and a 401 from ECR show that the instance can't reach the internet but can reach the AWS services it needs.

What broke (and how I fixed it)

This is the part I'd have wanted before starting. Each of these cost me time.

1. Tasks stuck with RESOURCE:MEMORY

  • Cause: I started with t3.small. The task needs 2048 MiB, and a t3.small can't fit it alongside the ECS agent and OS.
  • Fix: Use t3.medium or larger.

2. i/o timeout pulling images from ECR

  • Cause: ECR stores image layers in S3, so pulling an image also needs S3 access. The ECS security group was missing an egress rule for the S3 prefix list.
  • Fix: Allow egress to the S3 prefix list from the ECS security group.
  • Note: This one is easy to miss in an air-gapped setup.

3. Not authorized to perform sts:AssumeRoleWithWebIdentity

  • Cause: The OIDC trust policy didn't match the token GitHub issued, because the org/repo IDs were wrong.
  • Fix: Set the correct github_org_id and github_repo_id in terraform.tfvars and re-apply.

4. Unknown database 'petclinic'

  • Cause: RDS was created with db_name = "accounts", so the app's database didn't exist.
  • Fix: Add createDatabaseIfNotExist=true to the JDBC URL so Flyway can run against a fresh database.

5. Pipeline hanging for 5+ minutes on deploy

  • Cause: A bad task kept retrying forever.
  • Fix: Enable the deployment circuit breaker (deployment_circuit_breaker { enable = true, rollback = false }) so ECS gives up on a failing deployment instead of hanging.
  • Why rollback = false: On a first deploy there's no previous task definition to revert to, and the old tasks (if any) keep running. Once you have a stable version, rollback = true lets ECS revert automatically.

6. ResourceNotFoundException on secrets

  • Cause: I forgot to create dev/app/rds.
  • Fix: Create it in Step 5. It's manual on purpose, so the password never ends up in Terraform state.

More detail is in docs/troubleshooting.md.

Tear it down

Run the drain script before terraform destroy. ECS managed termination protection blocks instances from terminating while tasks are running, and skipping the script costs you a 20-minute timeout.

.\scripts\teardown.ps1          # scales service and ASG to 0, removes scale-in protection

cd terraform
terraform destroy               # RDS deletion is the slow part

aws secretsmanager delete-secret --secret-id "dev/app/rds" `
  --force-delete-without-recovery --region "us-east-1"
Enter fullscreen mode Exit fullscreen mode

 

Terraform Destroy deprovisions all resources successfully

Optional: Delete the state bucket

Only do this if you're done with the project entirely.

$bucket = "petclinic-tfstate-$(aws sts get-caller-identity --query Account --output text)"

# Delete all object versions, then delete markers (versioning is on)
aws s3api list-object-versions --bucket $bucket `
  --query "Versions[*].{Key:Key,VersionId:VersionId}" --output json | `
  ConvertFrom-Json | ForEach-Object {
    aws s3api delete-object --bucket $bucket --key $_.Key --version-id $_.VersionId | Out-Null
  }

aws s3api list-object-versions --bucket $bucket `
  --query "DeleteMarkers[*].{Key:Key,VersionId:VersionId}" --output json | `
  ConvertFrom-Json | ForEach-Object {
    aws s3api delete-object --bucket $bucket --key $_.Key --version-id $_.VersionId | Out-Null
  }

aws s3api delete-bucket --bucket $bucket --region us-east-1
Enter fullscreen mode Exit fullscreen mode

What I took away from this deployment

  • Air-gapped isn't free. Every AWS service the workload touches needs an endpoint and the right security group rules. Plan the list first.
  • OIDC is worth the setup. No keys to rotate or leak, but the trust policy has to be exact.
  • Verify, don't assume. The timeout-on-Google test is a two-minute check that proves the whole design.
  • Teardown is part of the design. Write the destroy path while you build, not after.

Next on the list: HTTPS on the ALB with ACM and Route 53, and splitting dev and prod into separate AWS accounts. The pipeline already supports prod/v* tags. It just needs somewhere to deploy to.

If you recreate this and get stuck, open an issue on the repo. If you find a better way to do any of it, I'd like to hear that too.

Repo: https://github.com/EddieByte/devsecops-ecs-pipeline

Top comments (0)