From Idea to Infra: Building Scalable Systems with Kubernetes, Terraform & Cloud (Detailed)

Introduction: Scale the System You Actually Have

An MVP has two jobs: help you learn whether the product is useful, and leave you a workable path when demand grows. Building a distributed platform before you understand the workload can make both jobs harder.

The difficult part is deciding which constraints deserve attention now. A slow query, an unreliable deployment, and a queue that never catches up are different problems. They do not all need Kubernetes, more services, or another cloud.

This guide follows six phases from architecture to operations. It uses an illustrative analytics API and a separate AdTech design exercise. These are engineering examples, not claims about a client deployment or measured company results. The earlier unsourced company performance table has been removed.

The code is a starting point with explicit prerequisites. Local syntax checks are useful, but they do not prove cloud permissions, networking, recovery, or production capacity. Do not apply infrastructure examples to an account without reviewing their cost and access requirements.

Phase 1: Turn Product Requirements into Constraints

Before drawing service boxes, write down:

• The critical user operation and its acceptable latency/error rate.
• Expected steady and burst traffic, payload sizes, and data retention.
• Which work must finish synchronously and which can be queued.
• What must remain correct during retries, duplicate events, and partial failure.
• Who will operate the system, including deployments and incidents.

A modular monolith can support substantial traffic. Microservices become useful when there is a concrete reason for independent deployment, ownership, scaling, or isolation. They also introduce network failures, distributed tracing, version compatibility, and data-consistency work. Team size alone is not a reliable decision rule.

An illustrative analytics application

Start with one API, a relational database, object storage for uploads, and a worker queue for long-running analysis. Give background jobs stable identifiers and make retries safe for the operations they perform. Add a streaming platform when you need its ordering, retention, replay, or consumer model, not merely because the architecture diagram looks more scalable.

Client → API → database
           ↓
      job queue → workers → object storage / results

The first scaling question might be whether workers need more capacity. Splitting authentication into a new service would not address that bottleneck.

Phase 2: Make Infrastructure Changes Reviewable

Terraform is useful because infrastructure changes can be versioned, compared, and reviewed. It does not make a configuration correct by itself.

Keep environment roots distinct from reusable modules:

infra/
  modules/
    network/
    database/
  environments/
    dev/
    production/
k8s/
  deployment.yaml
  service.yaml
  hpa.yaml

Use separate state and credentials for environments that need isolation. CLI workspaces are a state-selection mechanism; they are not an access-control boundary.

For an S3 backend, provision the state bucket separately, enable versioning, restrict access, and configure locking. HashiCorp now documents S3 lockfiles through use_lockfile; DynamoDB-based locking is deprecated. State and saved plans can contain sensitive values, so treat their storage and retention accordingly.

An EKS module boundary with explicit inputs

This fragment targets the EKS module's v21 interface. It pins the module so the example is reviewable. Before real deployment, review release/security changes and choose supported Kubernetes and add-on versions for your region. The existing VPC, private subnets, DNS, routing, egress, and operator/CI access are prerequisites, not resources created by this fragment.

terraform {
  required_version = ">= 1.10, < 2.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

variable "region" {
  type = string
}

variable "kubernetes_version" {
  type = string
}

variable "vpc_id" {
  type = string
}

variable "private_subnet_ids" {
  type = list(string)
}

provider "aws" {
  region = var.region
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "21.0.0"

  name                    = "analytics-example"
  kubernetes_version      = var.kubernetes_version
  vpc_id                  = var.vpc_id
  subnet_ids              = var.private_subnet_ids
  endpoint_private_access = true
  endpoint_public_access  = false

  addons = {
    vpc-cni = {
      before_compute = true
    }
    kube-proxy = {}
    coredns    = {}
  }

  eks_managed_node_groups = {
    workers = {
      instance_types = ["m6i.large"]
      min_size       = 2
      desired_size   = 2
      max_size       = 6
    }
  }
}

The node-group bounds are illustrative capacity settings, not a benchmark. Setting a maximum does not install a node autoscaler. Define how node capacity will grow, how workloads spread across failure domains, and how upgrades are tested. Configure EKS access entries and Kubernetes permissions deliberately; creating a cluster is not the same as granting a deployment runner access to it.

For a new local validation directory, terraform init -backend=false downloads modules/providers without initializing remote state, and terraform validate checks configuration consistency. These commands do not provision a cluster. A useful plan still requires real reviewed inputs, provider access, and the intended backend. Apply only the reviewed plan through your controlled release process.

Phase 3: Make Kubernetes Objects Agree

Three objects need to point at the same workload: the Deployment owns matching Pods, the Service selects those Pods, and the HPA targets the Deployment.

The following files assume a pre-existing production namespace. Replace example.invalid/analytics-api:replace-me with your tested image, preferably pinned by digest. The application must listen on port 8080 and implement /readyz and /livez. These placeholders deliberately cannot be mistaken for a supplied application image.

k8s/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
  namespace: production
spec:
  selector:
    matchLabels:
      app: api-server
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: api-server
    spec:
      containers:
        - name: api
          image: example.invalid/analytics-api:replace-me
          ports:
            - name: http
              containerPort: 8080
          resources:
            requests:
              cpu: "500m"
              memory: "256Mi"
            limits:
              cpu: "1"
              memory: "512Mi"
          readinessProbe:
            httpGet:
              path: /readyz
              port: http
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /livez
              port: http
            initialDelaySeconds: 20
            periodSeconds: 10

The Deployment selector must match its Pod template labels. That required relationship was missing from the earlier example. CPU and memory values here are starting values to measure, not production sizing advice.

Readiness controls whether a Pod receives Service traffic; liveness can trigger container restarts. Design the endpoints accordingly. Making liveness fail whenever a shared database is unavailable can turn a dependency outage into a restart storm. Use a startup probe if initialization needs a separate allowance. See the Kubernetes probe documentation.

k8s/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: api-server
  namespace: production
spec:
  selector:
    app: api-server
  ports:
    - name: http
      port: 80
      targetPort: http
  type: ClusterIP

The Service makes the application reachable inside the cluster. It does not create public ingress, DNS, or TLS.

k8s/hpa.yaml:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

CPU utilization is measured against CPU requests, and the cluster needs a working resource-metrics API, commonly supplied by Metrics Server. The Deployment omits replicas because the HPA manages that value. Without an HPA, set an explicit replica count instead.

HPA scales Pods, not databases or node capacity. Load-test the full path and watch connection pools, queue age, downstream limits, and scheduling failures. More replicas can overload a dependency faster.

Phase 4: Show the Authentication and Deployment Steps

A fresh CI job does not inherit another job's checked-out files or Kubernetes context. A deployment step needs both, plus an authorized identity and a reachable API endpoint.

Keep infrastructure provisioning and application rollout as separate reviewed workflows. The example below deploys an application to an existing EKS cluster. It does not create the cluster or automatically apply Terraform on every push.

Prerequisites:

• A runner labeled eks-deploy with AWS CLI v2 and a compatible kubectl, plus network access to the private EKS endpoint. Restrict that runner to trusted deployment workflows.
• GitHub's AWS OIDC provider and a least-privilege deployment role. Match the trust policy to the repository/environment and its actual subject-claim format.
• An EKS access entry and Kubernetes authorization for that role in production. AWS authentication alone is insufficient.
• A GitHub production environment with appropriate deployment protection rules, and environment variables AWS_REGION, AWS_DEPLOY_ROLE_ARN, and EKS_CLUSTER_NAME.
• The three checked-in manifests above with a real image and tested health endpoints. Registry access must work from the cluster.

name: Deploy existing EKS application
on:
  workflow_dispatch:

permissions:
  contents: read
  id-token: write

concurrency:
  group: production-application
  cancel-in-progress: false

jobs:
  deploy:
    environment: production
    runs-on: [self-hosted, linux, eks-deploy]
    timeout-minutes: 15
    env:
      AWS_REGION: ${{ vars.AWS_REGION }}
      EKS_CLUSTER_NAME: ${{ vars.EKS_CLUSTER_NAME }}
    steps:
      - uses: actions/checkout@v4

      - name: Assume deployment role through OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ vars.AWS_DEPLOY_ROLE_ARN }}
          aws-region: ${{ vars.AWS_REGION }}

      - name: Create a job-specific Kubernetes context
        shell: bash
        run: |
          set -euo pipefail
          export KUBECONFIG="$RUNNER_TEMP/eks-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}.config"
          trap 'rm -f "$KUBECONFIG"' EXIT
          aws eks update-kubeconfig \
            --region "$AWS_REGION" \
            --name "$EKS_CLUSTER_NAME" \
            --kubeconfig "$KUBECONFIG" \
            --alias article-production
          kubectl --context article-production -n production apply --dry-run=server -f k8s/
          kubectl --context article-production -n production apply -f k8s/
          kubectl --context article-production -n production rollout status deployment/api-server --timeout=180s

GitHub documents the OIDC permissions and trust configuration, and AWS documents update-kubeconfig and its prerequisites. id-token: write permits requesting an OIDC token; it does not itself grant AWS or Kubernetes access. The example uses readable action version tags; production workflows should pin reviewed commit SHAs and maintain them.

Server-side dry-run exercises admission and validation against the target cluster, but rollout readiness is still not an end-to-end application test. Add a smoke test, watch error/latency changes, and define a rollback decision. A failed rollout command reports failure; it does not automatically restore the previous release.

Choose one owner for application reconciliation

Direct CI deployment and GitOps are alternatives for this workload. With Argo CD automated sync, have CI publish the image and update the desired manifest revision; let Argo CD reconcile the application. Do not also have an unrelated CI job continually apply different desired state to the same objects.

Git history supports review and rollback, but it cannot undo an incompatible database migration or restore lost data. Helm can package reusable manifests, but rendered output still needs validation. Keep plaintext secrets out of Git; if using SOPS, control decryption permissions and validate the encrypted files as part of the workflow.

Phase 5: Treat Recovery as a Tested Procedure

Multi-cloud adds dependencies and operational work. It is not automatically a disaster-recovery plan. A second Kubernetes cluster is of little use if the application cannot reach its database, decrypt data, obtain credentials, or serve the right DNS name.

For an illustrative GKE-to-EKS recovery design, write down:

Dependency	What the recovery test must establish
Compute	Compatible images, capacity, configuration, and network policies
Database	An engine-compatible restore or replication path, tested for consistency
Object storage	Replication lag, version history, access permissions, and deletion behavior
Identity and secrets	Working authentication and decrypt permissions in the recovery environment
Traffic	Health checks, DNS behavior, TLS, and safe failover/failback

Aurora backups and BigQuery exports are not interchangeable database restore mechanisms. Choose a recovery procedure for the actual engine and workload. Likewise, copying objects nightly leaves a possible data-loss window and does not by itself prove recoverability.

Define RPO as the acceptable data-loss window and RTO as the acceptable restoration time. Restore into an isolated environment, run consistency and application checks, and record elapsed time and failure points. A backup job succeeding is only the beginning of that evidence.

Phase 6: Observe User Outcomes and Saturation

Begin with request rate, errors, latency distributions, and saturation. Add queue age and retry/dead-letter counts for asynchronous work. Trace IDs should connect an incoming request to the relevant jobs and downstream calls.

Prometheus, Grafana, and OpenTelemetry are useful options. Choose retention and cardinality deliberately; adding a user ID or request ID as a metric label can make the monitoring system itself expensive.

Here is a complete illustrative Prometheus alert-rule file, not a Grafana provisioning document. It assumes your counter is named http_requests_total and has service and status labels. Adjust the metric names, exclusions, traffic floor, and threshold to your service.

groups:
  - name: api-example
    rules:
      - alert: ApiHighErrorRatio
        expr: |
          (
            sum(rate(http_requests_total{service="api",status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total{service="api"}[5m]))
          ) > 0.05
          and
          sum(rate(http_requests_total{service="api"}[5m])) > 1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "API error ratio exceeds the illustrative threshold"

The traffic floor avoids treating a single low-volume failure as the same situation as a sustained incident. Missing metrics still need their own alert; no data is not proof of health. See Prometheus alerting-rule syntax and validate with promtool check rules in your monitoring project.

For an SLO, first define the eligible operation, what counts as good, and the measurement window. An illustrative request-based availability objective of 99.95% permits a bad-request fraction of 0.05%. Track budget consumption over multiple windows; a fixed 5% error alert is not a complete error-budget policy.

Design Exercise: An AdTech Platform Under Bursty Load

Consider a hypothetical bidding API with campaign management, asynchronous analytics, and fraud checks. This section is a design exercise, not an account of Magnite's architecture or results.

Separate latency-critical decisions from work that can finish later. A cached campaign record may accelerate the read path, but budget updates still need concurrency control. A local check like BudgetLeft > bidPrice cannot prevent two simultaneous requests from overspending the same budget. Decide where reservation, expiry, and reconciliation happen, and test retries and races.

For the bidding workload, use the same Deployment selector/Pod-label relationship shown earlier, with app: bidding-engine. Add measured CPU and memory requests before using utilization-based autoscaling. Benchmark the full request path rather than asserting a latency based on the programming language or Redis alone.

The evidence worth collecting is concrete:

Question	Test and record
Does the API meet its latency target?	Traffic shape, payload distribution, p50/p95/p99, error rate, warm-up, and test duration
Does autoscaling help?	Burst arrival, scale-up lag, queue age, pending Pods, and downstream saturation
Are budget decisions correct?	Concurrent requests, duplicate delivery, timeout/retry cases, and reconciliation mismatches
Can a release be recovered?	Failed rollout, rollback time, schema compatibility, and user-visible impact
Is the design affordable?	Cost per defined workload unit, including idle capacity and observability

Publish before/after numbers only when you can explain the workload, infrastructure, measurement window, and the change that produced them. A table of impressive figures without that context is not engineering evidence.

A Release Checklist That Earns Its Place

1. Validate configuration and render the exact artifacts that will be deployed.
2. Review the infrastructure plan, identities, network reachability, and cost implications.
3. Exercise application health checks, dependency failure, and representative load in a suitable test environment.
4. Verify deployment, smoke-test behavior, and observe the service against its acceptance criteria.
5. Practice recovery, including data and configuration, before depending on it during an incident.

Treat infrastructure as a product: versioned, tested, documented, and owned. Add complexity when a measured requirement justifies it. The strongest architecture is one the team can explain, operate, and recover, with evidence for the claims it makes.

→ Connect with me on LinkedIn