Terraform Apply Succeeded. Your EKS Platform Isn’t Ready Yet

How splitting AWS infrastructure and Kubernetes platform provisioning into two Terraform states made my deployment dependencies explicit.

A green terraform apply tells you Terraform finished its work. It does not tell you the cluster can run your workloads, and when two lifecycles share one apply, that gap is where deployments break.

Creating an EKS cluster and deploying onto it have different lifecycles. I split them into two Terraform roots with separate state, put readiness gates between the stages, and reversed the order on teardown. Here is why, how, and where it falls short.

Two Terraform roots, two states. The infrastructure root owns the AWS resources and exports outputs; a readiness gate sits between it and the platform root.

The problem: two lifecycles in one apply

I built a centralized logging platform on Amazon EKS in eu-west-2. Elasticsearch and Kibana run through Elastic Cloud on Kubernetes (ECK), Filebeat collects logs from an Order API, Vault handles secrets, and ingress-nginx with cert-manager and Route 53 provides TLS and DNS.

The deployment has two layers:

• Infrastructure: the VPC, the EKS cluster and its node pools, IAM, KMS, ECR and S3.
• Platform: namespaces, Helm releases, ECK resources, Filebeat, Vault and the application. All of it needs a working Kubernetes API.

The order is obvious. The catch is that Terraform configures providers before it plans resources. The Kubernetes, Helm and kubectl providers need a cluster endpoint, CA data and credentials, and none of those exist until the cluster does. Resource dependencies order resources, but they do not make a provider wait for a cluster.

The design: two roots, two states

Each environment (dev, sit, prod) has the same two roots, backed by shared modules:

terraform/
├── modules/
│   ├── infrastructure/        # VPC, EKS, addons, IAM, S3, KMS, ECR
│   └── logging-platform/      # Kubernetes resources and Helm releases
├── infrastructure/environments/{dev,sit,prod}
└── platform/environments/{dev,sit,prod}

Every root has its own state in S3, with a DynamoDB table for locking:

enterprise-k8s-logging/<env>/infrastructure/terraform.tfstate
enterprise-k8s-logging/<env>/platform/terraform.tfstate

The infrastructure root exports the cluster name, endpoint, CA data, region and the IAM, S3, KMS and ECR values the platform needs. The platform root reads them with terraform_remote_state, and it is the only root that configures the Kubernetes-side providers. The infrastructure root never declares a Kubernetes provider in the state that creates EKS, so a clean deployment needs no terraform apply -target.

The token command is defined once and shared by the Kubernetes, Helm and kubectl providers:

locals {
  eks_auth_args = [
    "eks", "get-token",
    "--cluster-name", data.terraform_remote_state.infrastructure.outputs.cluster_name,
    "--region", data.terraform_remote_state.infrastructure.outputs.aws_region,
  ]
}

provider "kubernetes" {
  host                   = data.terraform_remote_state.infrastructure.outputs.cluster_endpoint
  cluster_ca_certificate = base64decode(data.terraform_remote_state.infrastructure.outputs.cluster_certificate_authority_data)
  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args        = local.eks_auth_args
  }
}

The Helm and kubectl providers use the same host, CA and exec block. Because the provider calls aws eks get-token when it needs credentials, no short-lived token is stored in configuration or state. The trade-off is that the aws CLI must be available wherever Terraform runs, which matters once this moves into CI.

The platform root also configures a Docker provider with an ECR authorization token, so building and pushing the application image happens in the platform layer too.

Making the order explicit

deploy-dev.sh encodes the sequence. The infrastructure apply finishes completely before the platform root is even initialized, so the platform providers never see a cluster that does not exist yet.

set -euo pipefail

terraform -chdir="$infra_dir" init -backend-config="$infra_backend_file"
terraform -chdir="$infra_dir" plan -out=tfplan
terraform -chdir="$infra_dir" apply tfplan

cluster_name="$(terraform -chdir="$infra_dir" output -raw cluster_name)"
aws_region="$(terraform -chdir="$infra_dir" output -raw aws_region)"

aws eks wait cluster-active --name "$cluster_name" --region "$aws_region"
aws eks update-kubeconfig --name "$cluster_name" --region "$aws_region"

# Gate 1: block until every node has registered and is Ready
kubectl wait --for=condition=Ready node --all --timeout=300s

terraform -chdir="$platform_dir" init -backend-config="$platform_backend_file"
terraform -chdir="$platform_dir" plan -out=tfplan
terraform -chdir="$platform_dir" apply tfplan

# Gate 2: block until Elasticsearch reports healthy
kubectl wait --for=jsonpath='{.status.health}'=green \
  elasticsearch/<es-name> -n elastic-system --timeout=900s

With set -euo pipefail, a timeout stops the script with a non-zero exit instead of carrying on against a half-ready cluster.

The gates matter because each check proves something different. aws eks wait cluster-active proves the control plane is up. kubectl wait on nodes proves compute has joined. The Elasticsearch check proves the application tier works. A bare kubectl get nodes proves none of these, because it prints and moves on. My first version of the script did exactly that, and the gates above are what I added after noticing.

Teardown runs in reverse

destroy-dev.sh initializes and destroys the platform state first, then the infrastructure state. Destroying the cluster first would leave the platform state trying to talk to an API that no longer exists.

The script keeps Terraform’s interactive approval prompt on destroy. That is deliberate for a dev environment, and CI would need to change it.

Identity: Pod Identity where it works

I used EKS Pod Identity instead of IRSA for cert-manager, the EBS CSI driver and Vault’s own AWS access. There is no OIDC provider federation to manage, no ServiceAccount annotations, and the trust policies are simpler.

I deliberately did not use it for Elasticsearch’s S3 snapshots. At the time of writing, both federated-identity mechanisms ran into open upstream bugs with the bundled repository-s3 plugin, which showed up for me as an S3 AccessDenied on snapshots. Instead, snapshots use Elastic’s documented fallback: a narrowly scoped IAM user whose credentials are stored in the Elasticsearch keystore.

It is a trade-off rather than a clean win: static credentials need rotation and careful scoping, which is exactly what Pod Identity avoids elsewhere in this platform.

The runtime architecture

Runtime architecture: ingress-nginx and cert-manager at the edge, Filebeat feeding a dedicated Elasticsearch data pool, Vault-injected credentials, and nightly snapshots to S3.

Traffic enters through ingress-nginx, with TLS issued by cert-manager using Route 53 DNS-01. The Order API runs three replicas and writes JSON logs to stdout. A Filebeat DaemonSet ships them to a three-node data pool (es-data), coordinated by a separate three-node master pool (es-master). Kibana runs two replicas. Vault injects credentials into the Order API and Filebeat, and a nightly snapshot lifecycle policy writes to an S3 bucket that moves to Glacier after 30 days and expires after 365.

Because the platform layer owns all of this, it can be destroyed and rebuilt without touching the VPC or the cluster.

Migrating an existing single state

If you already run everything in one state, you cannot just apply the new roots. Back up the old state with terraform state pull, then use terraform state mv to move AWS addresses into the infrastructure state and Kubernetes, Helm and kubectl addresses into the platform state. Generate the address list from terraform state list instead of guessing, and do not apply until terraform plan shows no unintended creates. In my layout the module prefix changed for infrastructure addresses (module.logging_platform became module.infrastructure) but not for platform ones.

Single root, two roots, or GitOps?

The right structure follows the dependency graph, not fashion. This is how I think about the three common options:

Approach	Best when	Watch out for
One root, one state	Small or short-lived environments where cluster and workloads change together	Providers configured from resources that do not exist yet; awkward on cluster replacement and destroy
Two roots, two states (this post)	The cluster and the platform on top of it have different lifecycles and you want an explicit gate between them	The output contract, two backends, and sequencing in a script or pipeline
Terraform for infrastructure, GitOps for the platform (Argo CD or Flux)	Many teams or clusters, and you want in-cluster state continuously reconciled	A second toolchain, and bootstrapping the GitOps controller itself

What the split costs

The infrastructure outputs are now a contract: rename one and the platform root breaks. You also take on:

• Two backends and two state locks to configure and secure.
• Least-privilege access to remote state, and no secrets in outputs.
• The aws CLI as a dependency of every provider that uses exec auth.
• Coordination whenever a change replaces the cluster.

The split does not fix networking, IAM, storage or scheduling problems, and it is not the only option. A single root can be fine for small or short-lived environments, and some teams move the platform layer to GitOps with Argo CD or Flux. Choose the structure that matches your real dependency graph.

What I would harden next

• Run both stages in CI with a plan on every pull request, and require approval before any prod apply.
• Authenticate CI to AWS with short-lived OIDC credentials instead of stored access keys.
• Pin provider and module versions so a plan means the same thing next month.
• Run a scheduled terraform plan per root to catch drift between the two states.

Takeaway

When Terraform spans two control planes, AWS and Kubernetes, model the boundary deliberately: separate state, defined outputs, explicit readiness gates between stages, and teardown in reverse. A resource dependency alone will not solve provider initialization or cluster readiness.

Four things to take away:

• A provider cannot wait on a resource. If a provider depends on something Terraform creates, split the state.
• Outputs are an API between layers. Treat renames as breaking changes.
• Gate every stage on what it actually proves: control plane, nodes, then workloads.
• Destroy in the reverse order you create.

Get the Code & Try It Yourself

This setup isn't just theoretical, it is fully implemented and open-source. If you want to explore the multi-root structure, check out the wrapper scripts, or use the modules as a starting point for your own platform, the entire codebase is available on GitHub.

princemaxi / enterprise-k8s-observability

Enterprise Kubernetes Logging Platform

This repository deploys the EKS-based enterprise logging stack, including Elasticsearch, Kibana, Filebeat, Vault, ingress, TLS, DNS, and the Order API Terraform is intentionally split into two state boundaries:

terraform/infrastructure/environments/<env>
        -> EKS and AWS resources
terraform/platform/environments/<env>
        -> Kubernetes and Helm resources

The platform root reads EKS connection details from the infrastructure state with terraform_remote_state. It never declares a Kubernetes provider in the same state that creates EKS, so a clean deployment does not need terraform apply -target.

Deploy dev

Prerequisites are Terraform >= 1.7, AWS CLI credentials, kubectl, Docker, and an S3 bucket plus DynamoDB lock table for Terraform state. The split-root workflow expects the real backend files in the environment directories and the repo-root copies used by the helper scripts:

cp terraform/infrastructure/environments/dev/backend.hcl \
  terraform/backend-dev-infrastructure.hcl
cp terraform/platform/environments/dev/backend.hcl \
  terraform/backend-dev-platform.hcl

The live deployment files are real configuration, not placeholders. Edit the real files in the…

View on GitHub

Feel free to clone the repo, inspect the Terraform configuration, and adapt it to your own environments. If you find it helpful, drop a ⭐ on the repository, or open an issue if you see areas where we can harden this further!

Comments

[Maxwell Ugochukwu AWS Community Builders]

How does your team handle the boundary between cloud infrastructure and Kubernetes platform provisioning? Do you stick with split Terraform states, run a single mammoth root with targeted applies, or have you fully offloaded the platform layer to GitOps tools like Argo CD or Flux?

[Vladimir Mikhalev AWS Community Builders]

Split, with the line at the cluster. In the EKS reference I run, Terraform builds the cluster, and Argo CD installs everything after a one-time bootstrap. The handoff is the Argo CD cluster Secret: Terraform's outputs become its annotations (domain, aws_region, velero_bucket, plus platform_revision, the commit that cluster runs), and labels like enable_karpenter: "true" pick the add-ons.

I ruled out helm_release because it puts cluster and add-on lifecycles in one state and one plan, and nothing reconciles drift between applies. Split roots fix the first half. The second half needs a controller.

Ready then means Applications healthy: the first passing run on AWS brought all 19 to healthy on EKS 1.37 and passed 14 checks. The reasoning is in ADR 1.