Luqman Bello for AWS Community Builders

Posted on Nov 24

Designing an Internal Developer Platform (IDP) on AWS Using Backstage, EKS & Terraform

#kubernetes #terraform #aws #devops

Introduction — Why IDPs Are Becoming Mandatory

If you've worked in engineering long enough, you already know the truth nobody wants to say out loud:

Most developers don’t want to touch YAML, Terraform, Kubernetes manifests, or IaC pipelines. They just want their service to run.

And honestly… can we blame them?
Between debugging a failing pod, hunting for IAM permissions, or figuring out why Terraform refuses to plan, developer energy gets drained on infrastructure instead of innovation.

That’s where Internal Developer Platforms (IDPs) come in.

IDPs are the “self-service layer” that removes friction.
The best IDPs allow developers to:

scaffold services
spin up infrastructure
deploy to multiple environments
observe their workloads
and do all this without knowing how the underlying platform works

Today, you’ll learn how to build such a platform using:

✔ Backstage (developer portal)
✔ Terraform (self-service IaC)
✔ ArgoCD (GitOps)
✔ Amazon EKS (runtime platform)
✔ AWS native services like IAM, KMS, Secrets Manager

This is the same pattern used at modern engineering orgs and unicorns.

So let’s build a real one, the right way.

The Four Layers of a Modern IDP

Every successful platform has the same 4 layers:

1. Developer Experience Layer  → Backstage
2. Platform Orchestration      → Terraform + GitOps (ArgoCD)
3. Runtime Infrastructure      → Amazon EKS + networking
4. Shared Services             → Observability, security, secrets, CI

Let’s break them down.

1. Developer Experience Layer — Backstage

Backstage gives you:

💠 Software Catalog

A single source of truth for all services:

Ownership
Deployments
Documentation
Tech stack
Status

💠 Golden Paths (Software Templates)

This is the real magic.

Instead of telling developers:

“Hey, here’s the Confluence page with our recommended architecture. Follow it.”

You give them a self-service template that does everything:

Create a Git repo
Generate CI/CD pipelines
Create Terraform infrastructure
Register service in Backstage
Deploy to EKS
Wire up monitoring & alerts

With one click.

2. Platform Orchestration — Terraform + GitOps

This is where your platform’s power comes from.

🟦 Terraform modules

These represent reusable “building blocks”:

microservice module
VPC module
RDS module
DynamoDB table module
S3 bucket module
EKS namespace module

Developers never write Terraform.
They fill in a simple form in Backstage → Backstage generates Terraform config → GitOps takes over → ArgoCD deploys it.

🟥 GitOps (ArgoCD)

This ensures:

everything is deployed from Git
automatic rollbacks
drift detection
clear audit trails
separation between build (CI) and deploy (CD)

Git is the source of truth.
ArgoCD is the engine.

3. Runtime Infrastructure — Amazon EKS

This is where workloads actually run.

Your EKS architecture includes:

Multi-tenant namespaces
IRSA (IAM Roles for Service Accounts)
Karpenter for fast autoscaling
Network Policies
Pod Security Standards
Ingress controller (ALB/Nginx/Ambassador)
Internal service mesh (optional)

EKS is your “runtime engine”, Backstage is the steering wheel, and GitOps is the autopilot.

4. Shared Services Layer

This includes all the “invisible” but necessary pieces:

✔ Secrets

AWS Secrets Manager + IRSA.

✔ Observability

Prometheus
Grafana
Tempo / Loki / OTel Collector

✔ Security

KMS encryption
Audit logs
IAM boundaries
Registry scanning

✔ CI Pipelines

GitHub Actions or GitLab CI.
CI builds → GitOps deploys.

Full Architecture flow diagram

🧬 Golden Path Template Example (Backstage)

Here’s a simplified template:

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: aws-eks-service
  title: "Create a Service on EKS"
  description: "Scaffold a microservice with Terraform + GitOps"
spec:
  owner: platform-team
  parameters:
    - title: "Service Info"
      properties:
        name:
          type: string
          description: Name of the service
        owner:
          type: string
          description: Team name

  steps:
    - id: template
      name: Create Repo from Template
      action: fetch:template

    - id: terraform
      name: Terraform Module
      action: publish:github
      input:
        repoUrl: "{{ parameters.name }}"
        files:
          - terraform/main.tf

    - id: register
      name: Register in Backstage Catalog
      action: catalog:register

📦 Terraform Module Example (EKS Namespace)

module "namespace" {
  source = "git::https://github.com/org/platform-modules.git//eks-namespace"

  name      = var.service_name
  team      = var.team_name
  iam_roles = var.iam_roles
}

🔁 GitOps Flow (ArgoCD App)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myservice
spec:
  source:
    repoURL: https://github.com/org/myservice
    path: deploy/
  destination:
    namespace: myservice
    server: https://kubernetes.default.svc
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Real-World Challenges & Lessons Learned

No one tells you these things until you experience them:

Templates need constant updating
Teams will bypass your platform unless you make it genuinely delightful
Backstage plugin versioning can be chaotic
Keep the IDP simple: don’t over-engineer
EKS upgrades require strict compatibility planning
Terraform module contracts must be stable and versioned

But once it works, developer onboarding drops from weeks to minutes.

Final Thoughts — The Future of IDPs

The next evolution of IDPs will include:

AI-powered troubleshooting
Auto-generated infrastructure from natural language
Predictive autoscaling
Auto-remediation
Learning-based deployment risk scoring
AI-assisted documentation & code generation

And you’ll be ahead of the curve because you’re already building the foundation today.

DEV Community