Santanu Das

Posted on Oct 22 • Edited on Oct 25

Integrating Amazon EFS with EKS Cluster Using Terraform

#eks #terraform #efs #aws

Overview
Objectives
Prerequisites
Architecture Summary
Operational Flow
Security & Compliance
Performance & Reliability
Implementation Choices
Lessons Learned
Terraform Implementation
Validation
References

This is more of a work-log (and lesson-learned) during setting up a shared EFS volumes for our EKS cluster, at zenler.com.The goal was to configre a shared, persistent storage for EKS workloads using Amazon Elastic File System (EFS), AWS KMS encryption, and the EFS CSI driver — fully automated with Terraform.

🧭 Overview

Amazon EFS (Elastic File System) provides scalable, managed NFS storage that can be shared across multiple pods and nodes in Amazon EKS. This guide walks through how to configure EFS for your EKS cluster using Terraform and the AWS EFS CSI driver.

We’ll use:

Terraform to provision and configure infrastructure
AWS KMS for encryption at rest
IRSA (IAM Roles for Service Accounts) for EFS driver permissions
Kubernetes PV/PVC for workload-level persistence

🎯 Objectives

By the end of this guide, you’ll have:

A KMS-encrypted EFS file system accessible to your EKS cluster
Dedicated access points per application (e.g., /data/podinfo)
Kubernetes StorageClass, PV, and PVC configured to use EFS
Verified pod read/write access to the mounted EFS volume

🧩 Prerequisites

Before deploying the EFS integration to your Amazon EKS cluster, ensure that the following conditions are met:

1️⃣ EKS Cluster & Addons

Requirement	Description
`EKS Cluster`	A running Amazon EKS cluster (v1.28 or newer recommended).
`EFS CSI Driver Addon`	The `aws-efs-csi-driver` addon must be installed — either via Terraform (`aws-ia/eks-blueprints-addons`) or manually using `kubectl` / Helm.
`OIDC Provider`	OIDC provider is enabled for the cluster (required for IRSA). You can check with: `aws eks describe-cluster --name <cluster-name> --query "cluster.identity.oidc.issuer"`

2️⃣ AWS Identity & Access Management (IAM)

Requirement	Description
IRSA Role for EFS CSI Driver	An IAM role must exist with trust to the EKS OIDC provider and the following policy attached: AmazonEFSCSIDriverPolicy
Terraform Addon IRSA Integration	If using the Terraform aws-ia/eks-blueprints-addons module, ensure: `enable_aws_efs_csi_driver = true` and `aws_efs_csi_driver_irsa_policies = ["arn:aws:iam::aws:policy/service-role/AmazonEFSCSIDriverPolicy"]`
Custom Encryption Key Access (Optional)	If using a custom KMS key for EFS encryption, add that key’s ARN to the IAM policy for the CSI driver role.

3️⃣ Networking & Connectivity

Requirement	Description
EFS File System	Must exist in the same VPC as the EKS cluster. Terraform can manage this via `aws_efs_file_system`
Mount Targets	An aws_efs_mount_target must exist in each private subnet used by your EKS nodes
Subnet Access	Each subnet used by EFS mount targets must have: • Route to the VPC internal network • Security group rules allowing NFS (TCP 2049) • (Optional) Access to S3 Gateway Endpoint for AWS API calls during EFS provisioning
Security Groups	The node group’s security group must allow outbound traffic to EFS on TCP port 2049. The EFS mount target SG must allow inbound TCP/2049 from node SGs

4️⃣ Kubernetes Configuration

Requirement	Description
EFS StorageClass	Must define `storage_provisioner = "efs.csi.aws.com"` and include the EFS filesystem ID
PersistentVolume / PersistentVolumeClaim	PVs must reference the access point via: `volume_handle = "<filesystem_id>::<access_point_id>"` and PVCs must bind to those PVs
Access Point Permissions	Each aws_efs_access_point should have a distinct root_directory.path and use consistent POSIX UID/GID permissions

5️⃣ Tooling

Tool	Minimum Version	Notes
Terraform	v1.6+	For proper `for_each` evaluation and module dependency handling
AWS CLI	v2.15+	Required for testing and validation commands
kubectl	v1.28+	For direct Kubernetes interaction.
eksctl (optional)	latest	For quick inspection and debugging (e.g. `eksctl utils describe-stacks`)

✅ Optional but Recommended

KMS Encryption enabled for EFS (encrypted = true and kms_key_id = <key-arn>).
Throughput Mode set based on workload (bursting, provisioned, or elastic).
Private Cluster Access only (no public endpoint) for security-sensitive environments.
Separate Access Points per App (e.g. /data/app1, /data/app2) for isolation.

🧱 Architecture Summary

Component	Description
`EFS File System`	Shared, elastic NFS storage managed by AWS
`EFS Mount Targets`	Network endpoints within each private subnet
`EFS Access Points`	App-specific directory paths and ownership controls
`EFS CSI Driver`	Kubernetes driver that mounts EFS volumes to pods
`StorageClass, PV, PVC`	Kubernetes resources that bind apps to EFS volumes

⚙️ Operational Flow

Terraform provisions an EFS file system, mount targets, and access points.
The EFS CSI driver (installed via aws-ia/eks-blueprints-addons) exposes the EFS file system to Kubernetes.
Kubernetes StorageClass, PersistentVolume, and PersistentVolumeClaim map each EFS access point to an application.
Applications reference their PVC in Deployment manifests to mount /data or another path inside their container.
Data written by a pod is stored persistently in EFS and is accessible from any node or replica.

Data flow:
Pod → PVC → PV → EFS Access Point → EFS File System → AWS Storage Backend

🔒 Security & Compliance

Area	Recommendation
Encryption	Enable KMS encryption on the EFS file system to ensure all data at rest is protected
IAM Permissions	The EFS CSI driver service account must have elasticfilesystem:DescribeAccessPoints and elasticfilesystem:ClientMount
POSIX Ownership	Use aws_efs_access_point.posix_user to enforce correct UID/GID ownership per app directory
Reclaim Policy	Use Retain for PersistentVolumes to prevent accidental data deletion when PVCs are removed
Access Mode	Use ReadWriteMany (RWX) to support multiple pods and nodes sharing the same volume
Network Access	Ensure EKS node subnets can reach EFS mount targets over NFS (TCP/2049). No NAT is required for internal access
Dependency Management	Deploy EFS CSI Driver before creating Kubernetes PV/PVC resources

🔒 Security Group Configuration

Starting with EKS clusters created using the modern Terraform EKS module (v21+) and authentication mode set to API, the EKS control plane assigns the Cluster Primary Security Group to all AWS-managed network interfaces used by system components and add-ons — including the EFS CSI driver, CNI, and CoreDNS pods.

Because of this, NFS mount requests to Amazon EFS now originate from the Cluster Primary Security Group, (not from the Node Group Security Group). If inbound NFS (TCP 2049) is allowed only from the Node SG, the mount will fail with timeout errors.

⚡ Rule-set Summery

Component	Security Group Used	Notes
EKS-managed node ENIs	Cluster Primary SG	Default for all AWS-managed add-ons and CSI drivers
Self-managed node EC2 instances	Node SG	Used only in legacy or hybrid setups
EFS Mount Targets	EFS SG	Receives inbound NFS (TCP 2049) from Cluster Primary SG

🧩 EKS → EFS Network Flow

The diagram below shows how traffic flows between EKS components and EFS:

This configuration ensures your EFS mount targets can accept traffic from all EKS-managed interfaces and prevents DeadlineExceeded or MountVolume.SetUp failed errors.

Diagram Explanation

EKS-managed ENIs (for add-ons such as EFS CSI Driver, CoreDNS, and CNI) inherit the Cluster Primary SG.
The Cluster Primary SG initiates NFS (TCP 2049) connections to the EFS Security Group.
The EFS Security Group allows inbound NFS from the Cluster Primary SG (and optionally from the Node SG).
Each Mount Target represents an EFS endpoint in a private subnet for its Availability Zone.
The EFS File System is a regional resource that mounts transparently through the nearest mount target.

⚡ Performance & Reliability Enhancements

Setting	Description
Mount Options	Use optimized NFS parameters: `nfsvers=4.1`, `rsize=1048576`, `wsize=1048576`, `hard`, `timeo=600`, `retrans=2`
Throughput Mode	For consistent high performance, set EFS `throughput_mode = "provisioned"` and define a suitable provisioned_throughput_in_mibps
Binding Mode	Use `WaitForFirstConsumer` to ensure PVs bind only after a node is available
Reclaim Policy	Keep as `Retain` for stateful workloads; change to `Delete` only for ephemeral storage needs

⚙️ Key Implementation Choices

Static Access Points: Each application gets its own pre-created access point for better auditability and isolation.
POSIX Permissions: All access points are initialized with uid/gid = 1000 and permissions 0770.
Terraform Management: EFS, PV, and PVC lifecycle are fully controlled via Terraform for consistency and versioning.
StorageClass Configuration: The same EFS StorageClass is reused cluster-wide for all applications.

💡 Key Lessons Learned

The csi block in Terraform must be nested under persistent_volume_source (schema change from older Terraform releases).
The annotation volume.beta.kubernetes.io/storage-provisioner is deprecated — rely on storage_class_name instead.
gidRangeStart and gidRangeEnd parameters are unnecessary when using pre-created access points.
EFS CSI driver automatically handles directory creation and permissions when the access point is properly configured.
Setting volume_binding_mode = "WaitForFirstConsumer" improves scheduling reliability in private EKS clusters.
Adding mount_options significantly improves I/O performance and NFS resilience.
Use Retain reclaim policy to safeguard persistent data from unintentional deletion.

🏗️ Terraform Implementation

The code below is only for the reference purpose; not supposed to be copy/paste directly. I used terraform-aws-eks & terraform-aws-eks-blueprints-addons for the EKS cluster creation and EKS addons as a seperate Terraform project and used the output from that module as eks_cluster_name and eks_primary_sg_id variable in this project.

1️⃣ Create EKS Cluster & Addons

# ------------------------------------------------
# EKS cluster
# ------------------------------------------------
module "cluster" {
  source   = "terraform-aws-modules/eks/aws"
  version  = "21.3.2"
  ....
}

# ------------------------------------------------
# Cluster addons
# ------------------------------------------------
module "addons" {
  for_each = var.service_enabled ? toset([var.service_name]) : []
  source   = "aws-ia/eks-blueprints-addons/aws"
  version  = "1.22.0"
  ....
  enable_aws_efs_csi_driver = true
  ....
}

2️⃣ Create EFS File System and Mount Targets

# ------------------------------------------------
# Security Group - allow NFS from primary-sg
# ------------------------------------------------
resource "aws_security_group" "asg_efs" {
  name        = "${var.eks_cluster_name}-efs"
  description = "Security group for EFS mount targets"
  vpc_id      = local.my_vpc_info.id

  tags = {
    Name     = "${var.eks_cluster_name}-efs",
  }
}

// Allow inbound NFS from Cluster Primary SG
resource "aws_vpc_security_group_ingress_rule" "efs_allow_nfs" {
  security_group_id = aws_security_group.asg_efs.id
  from_port         = 2049
  to_port           = 2049
  ip_protocol       = "tcp"
  description       = "Allow NFS from EKS Cluster Primary SG"

  # Allow from cluster primary-security-group
  referenced_security_group_id = var.eks_primary_sg_id

  lifecycle {
    precondition {
      condition     = can(var.eks_primary_sg_id)
      error_message = "Must allow NFS from cluster_primary_security_group!!"
    }
  }
}

// Optional: also allow from Node SG for self-managed nodes
resource "aws_vpc_security_group_ingress_rule" "efs_allow_nfs_node_sg" {
  security_group_id            = aws_security_group.asg_efs.id
  from_port                    = 2049
  to_port                      = 2049
  ip_protocol                  = "tcp"
  referenced_security_group_id = var.eks_node_sg_id
  description                  = "Allow NFS from EKS Node SG (optional)"
}

# ------------------------------------------------
# EFS File System
# ------------------------------------------------
resource "aws_efs_file_system" "this" {
  creation_token = "${var.eks_cluster_name}-efs"
  encrypted      = true
  kms_key_id     = var.eks_cluster_kms_key

  lifecycle_policy {
    transition_to_ia = "AFTER_30_DAYS"
  }

  tags = {
    Name = join("-", [
      local.efs_template_name,
      regex("[^-]+$", var.eks_cluster_name)
    ]),
    EKSCluster = var.eks_cluster_name,
  }
}

# ------------------------------------------------
# EFS Mount Targets
# ------------------------------------------------
resource "aws_efs_mount_target" "this" {
  for_each = toset(var.aws_zones)

  file_system_id  = aws_efs_file_system.this.id
  subnet_id       = module.efs_subnets[var.service_efs.id].subnet_ids[each.value]
  security_groups = [aws_security_group.asg_efs.id]
}

# ------------------------------------------------
# EFS Access Points
# ------------------------------------------------
resource "aws_efs_access_point" "this" {
  file_system_id = aws_efs_file_system.this.id

  posix_user {
    gid = 1000
    uid = 1000
  }

  root_directory {
    path = "/data/${each.key}" : each.value.path
    creation_info {
      owner_gid   = 1000
      owner_uid   = 1000
      permissions = "0755"
    }
  }

  tags = {
    Name     = "${local.efs_template_name}ap-${each.key}",
  }
}

3️⃣ Create StorageClass & PersistentVolumes

I did this section as a seperate module in the same TF project; used the Attribute References form the abode resources as the input variable for efs_pvs local module.

# ------------------------------------------------------
# EFS-PVS local-module reference
# ------------------------------------------------------
module "efs_pvs" {
  source = "./modules//efs_pvs"
  efs_access_points = {
    for K1, V1 in aws_efs_access_point.this : K1 => {
      apid = V1.id,
      nmsp = var.apps_config[K1].name_space
    }
  }
  efs_dns_name       = aws_efs_file_system.this.dns_name
  efs_file_system_id = aws_efs_file_system.this.id
  efs_name_prefix    = local.efs_template_name
}

Then in the module/efs_pvs:

# ------------------------------------------------------
# EFS StorageClass - one per cluster
# ------------------------------------------------------
resource "kubernetes_storage_class_v1" "ksc_efs" {

  metadata {
    annotations = {
      "storageclass.kubernetes.io/is-default-class" = "false"
    }
    name = "${var.efs_name_prefix}-sc"
  }
  storage_provisioner = "efs.csi.aws.com"

  mount_options = [
    "nfsvers=4.1",
    "rsize=1048576",
    "wsize=1048576",
    "hard",
    "timeo=600",
    "retrans=2"
  ]

  parameters = {
    provisioningMode = "efs-ap"
    fileSystemId     = var.efs_file_system_id
    directoryPerms   = "0700"
  }
  reclaim_policy         = "Retain"
  allow_volume_expansion = true

  # Prevent PVs being bound before nodes are up
  volume_binding_mode = "WaitForFirstConsumer"
}

# ------------------------------------------------------
# PersistentVolumes — one per access point
# ------------------------------------------------------
resource "kubernetes_persistent_volume_v1" "kpv_efs" {
  for_each = var.efs_access_points

  metadata {
    name = "${var.efs_name_prefix}pv-${each.key}"
    labels = {
      "app.kubernetes.io/name"      = each.key
      "app.kubernetes.io/component" = "storage"
      "efs.csi.aws.com/filesystem"  = var.efs_file_system_id
    }
  }

  spec {
    capacity = {
      storage = "5Gi" # logical value — ignored for EFS, but required by K8s
    }
    access_modes                     = ["ReadWriteMany"]
    persistent_volume_reclaim_policy = "Retain"

    storage_class_name = kubernetes_storage_class_v1.ksc_efs.metadata[0].name

    persistent_volume_source {
      csi {
        driver        = "efs.csi.aws.com"
        volume_handle = "${var.efs_file_system_id}::${each.value.apid}"
      }
    }
  }
}

# ------------------------------------------------------
# PersistentVolumeClaim — one per app/access point
# ------------------------------------------------------
resource "kubernetes_persistent_volume_claim_v1" "kpvc_efs" {
  for_each = var.efs_access_points

  metadata {
    name      = "${var.efs_name_prefix}pvc-${each.key}"
    namespace = each.value.nmsp
  }

  spec {
    access_modes       = ["ReadWriteMany"]
    storage_class_name = kubernetes_storage_class_v1.ksc_efs.metadata[0].name

    resources {
      requests = {
        storage = "5Gi" # logical — EFS ignores, but required
      }
    }

    # Bind PVC to the corresponding PV explicitly
    volume_name = kubernetes_persistent_volume_v1.kpv_efs[each.key].metadata[0].name
  }
}

✅ Outcome

After successful deployment:

Each workload (e.g. podinfo) mounts a dedicated EFS directory (/data/podinfo) with secure access.
Data persists across pod restarts, node terminations, and cluster upgrades.
EFS-backed storage remains encrypted, durable, and accessible across all availability zones in the VPC.

🧪 Validation & Testing

After deploying the EFS integration, you can validate the configuration and verify read/write operations using the following commands.

1️⃣ Verify EFS CSI Driver Installation

Check that the EFS CSI driver is installed and running in the cluster:

kubectl get pods -n kube-system -l app=efs-csi-controller
kubectl get daemonset -n kube-system efs-csi-node

You should see:

Controller pods running on control plane nodes
DaemonSet pods running on all worker nodes

2️⃣ Confirm StorageClass, PV, and PVC Resources

List all storage-related Kubernetes objects:

kubectl get storageclass
kubectl get pv
kubectl get pvc --all-namespaces

You should see your EFS storage class (e.g. efs-sc) and persistent volumes bound to their claims.

3️⃣ Check Volume Binding and Access Points

To inspect which access point or file system a PV is using:

kubectl describe pv <pv-name> | grep -A5 "Source:"

Expected output:

Source:
    Type:    CSI (a Container Storage Interface (CSI) volume source)
    Driver:  efs.csi.aws.com
    VolumeHandle: fs-0abc1234::fsap-0def5678

This confirms the PV correctly references the EFS access point.

4️⃣ Test Pod Read/Write Access to EFS

Create a simple debug pod that mounts the EFS PVC:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: efs-test-pod
spec:
  containers:
  - name: app
    image: amazonlinux
    command: ["/bin/bash", "-c", "sleep infinity"]
    volumeMounts:
    - name: efs-vol
      mountPath: /data
  volumes:
  - name: efs-vol
    persistentVolumeClaim:
      claimName: <your-efs-pvc-name>
EOF

Wait for it to be ready:

kubectl wait pod/efs-test-pod --for=condition=Ready --timeout=60s

Then test file operations:

kubectl exec -it efs-test-pod -- bash
echo "hello from EFS" > /data/test.txt
cat /data/test.txt
exit

✅ You should see the same file accessible from other pods using the same PVC.

5️⃣ Validate Cross-Pod Persistence

To confirm shared access between pods:

kubectl run efs-reader --image=amazonlinux -it --rm --overrides='
{
  "spec": {
    "containers": [{
      "name": "app",
      "image": "amazonlinux",
      "command": ["cat", "/data/test.txt"],
      "volumeMounts": [{
        "name": "efs-vol",
        "mountPath": "/data"
      }]
    }],
    "volumes": [{
      "name": "efs-vol",
      "persistentVolumeClaim": {
        "claimName": "<your-efs-pvc-name>"
      }
    }]
  }
}'

Expected output: hello from EFS
That confirms the data is shared and persistent across pods and nodes.

6️⃣ Check EFS File System on AWS

Use AWS CLI to confirm the file system and access points:

aws efs describe-file-systems --query "FileSystems[*].{ID:FileSystemId,Name:Name,Encrypted:Encrypted,ThroughputMode:ThroughputMode}"
aws efs describe-access-points --file-system-id <your-fs-id> --query "AccessPoints[*].{ID:AccessPointId,Path:RootDirectory.Path}"