Managing AWS Resources with ACK and helmfile

AWS Controllers for Kubernetes (ACK) is a Kubernetes Custom Controller that allows you to manage AWS resources through Kubernetes manifests. By using ACK, you can easily manage AWS resources in the same lifecycle as your applications. It is suitable for scenarios where you need to create AWS resources alongside your applications for testing purposes and delete them together with the application when no longer needed. This article explains how to manage AWS resources using Kubernetes manifests. While there are various ways to manage Kubernetes manifest files, this article focuses on using helmfile for management.

Prerequisites

Please install the following on your local machine:

• eksctl
• kubectl
• helm
• helmfile

Set up the following in your AWS account. This IAM policy has the necessary permissions for the IAM Controller of ACK to create IAM resources.

• IAM policy
  • https://github.com/aws-controllers-k8s/iam-controller/blob/main/config/iam/recommended-inline-policy
  • name: ack-iam-controller-policy

First, let's create an EKS cluster. In this example, we will deploy it with minimal configuration using eksctl.

eksctl create cluster --name ack-helmfile

Next, create an OIDC provider, IAM Role and service account.

eksctl utils associate-iam-oidc-provider --cluster=ack-helmfile --approve
eksctl create iamserviceaccount --cluster=ack-helmfile --name=ack-dynamodb-controller --namespace=ack-system --attach-policy-arn=arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess --approve
eksctl create iamserviceaccount --cluster=ack-helmfile --name=ack-iam-controller --namespace=ack-system --attach-policy-arn=arn:aws:iam::[your AWS account ID]:policy/ack-iam-controller-policy --approve

Install ACK on the created EKS cluster. This time, we will install the IAM and DynamoDB controllers. Below is a sample configuration for the ACK controllers using helmfile.

helmfile.yaml

repositories:
  - name: aws-controllers-k8s
    url: public.ecr.aws/aws-controllers-k8s
    oci: true

environments:
  {{ .Environment.Name }}:
    values:
    - environments/{{ .Environment.Name }}/values.yaml

---

releases:
  - name: "ack-dynamodb-controller"
    namespace: "{{ .Namespace | default .Values.awsControllersK8s.namespace }}"
    createNamespace: true
    chart: "aws-controllers-k8s/dynamodb-chart"
    version: "{{ .Values.awsControllersK8s.dynamodb.version }}"
    installedTemplate: "{{ .Values.awsControllersK8s.dynamodb.installed }}"
    values:
    - values/dynamodb.yaml.gotmpl

  - name: "ack-iam-controller"
    namespace: "{{ .Namespace | default .Values.awsControllersK8s.namespace }}"
    createNamespace: true
    chart: "aws-controllers-k8s/iam-chart"
    version: "{{ .Values.awsControllersK8s.iam.version }}"
    installedTemplate: "{{ .Values.awsControllersK8s.iam.installed }}"
    values:
    - values/iam.yaml.gotmpl

environments/test/values.yaml

awsControllersK8s:
  namespace: ack-system
  region: ap-northeast-1

  dynamodb:
    installed: true
    version: 1.1.1

  iam:
    installed: true
    version: 1.2.1

values/dynamodb.yaml.gotmpl

aws:
  region: {{ .Values.awsControllersK8s.region }}

serviceAccount:
  create: false
  name: ack-dynamodb-controller

values/iam.yaml.gotmpl

aws:
  region: {{ .Values.awsControllersK8s.region }}

serviceAccount:
  create: false
  name: ack-iam-controller

Applying helmfile will create the IAM and DynamoDB controllers.

helmfile -e test apply .

kubectl get deployment -n ack-system
NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
ack-dynamodb-controller-dynamodb-chart   1/1     1            1           79s
ack-iam-controller-iam-chart             1/1     1            1           79s

Deploying Required AWS Resources in the Application

Now that the DynamoDB controller is ready, let's use it to create a DynamoDB resource. Here, we will deploy a simple table called "ack-dynamodb-table".

helmfile.yaml

environments:
  {{ .Environment.Name }}:
    values:
    - environments/{{ .Environment.Name }}/values.yaml

---

releases:
  - name: "ack-dynamodb-table"
    namespace: default
    installedTemplate: "{{ .Values.dynamodb.installed }}"
    chart: "./manifests"

environments/test/values.yaml

dynamodb:
  installed: true

manifests/table.yaml

apiVersion: dynamodb.services.k8s.aws/v1alpha1
kind: Table
metadata:
  name: ack-dynamodb-table
spec:
  tableName: ack-dynamodb-table
  attributeDefinitions:
    - attributeName: id
      attributeType: "N"
  keySchema:
    - attributeName: id
      keyType: "HASH"
  provisionedThroughput:
    writeCapacityUnits: 1
    readCapacityUnits: 1

helmfile -e test apply .

Applying it will create the DynamoDB table defined in the manifest.

Deploying Role and Service Account for IRSA

To utilize DynamoDB from the application, we need to use IRSA (IAM Roles for Service Accounts). IRSA enables the association of Kubernetes service accounts with IAM roles. Specifically, it involves setting up the IAM role, appropriate trust relationships, and annotating the service account to make it usable. While eksctl can handle this through commands(as we did earlier), we will create them using ACK this time.

helmfile.yaml

environments:
  {{ .Environment.Name }}:
    values:
    - environments/{{ .Environment.Name }}/values.yaml

---

releases:
  - name: "ack-iam-role"
    namespace: default
    installedTemplate: "{{ .Values.iam.installed }}"
    chart: "./manifests"

We will obtain the OIDC URL required for the trust relationship of the role used in service account from the EKS management console and add it to the values.yaml file.

environments/test/values.yaml

iam:
  installed: true
  account_id: [your AWS account ID]
  oidc_url: [your EKS oidc url]

We will define the policy, role, and service account in the manifest that have permissions to access DynamoDB.

manifests/role.yaml

apiVersion: iam.services.k8s.aws/v1alpha1
kind: Policy
metadata:
  name: ack-dynamodb-policy
spec:
  policyDocument: |
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "",
                "Effect": "Allow",
                "Action": [
                    "dynamodb:UpdateTable",
                    "dynamodb:UpdateItem",
                    "dynamodb:Scan",
                    "dynamodb:Query",
                    "dynamodb:PutItem",
                    "dynamodb:ListTables",
                    "dynamodb:GetItem",
                    "dynamodb:DescribeTable",
                    "dynamodb:DeleteItem",
                    "dynamodb:BatchWriteItem",
                    "dynamodb:BatchGetItem"
                ],
                "Resource": [
                    "arn:aws:dynamodb:ap-northeast-1:{{ .Values.iam.account_id }}:table/ack-dynamodb-table"
                ]
            }
        ]
    }
  name: ack-dynamodb-policy
---
apiVersion: iam.services.k8s.aws/v1alpha1
kind: Role
metadata:
  name: ack-app-role
spec:
  assumeRolePolicyDocument: |
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "Federated": "arn:aws:iam::{{ .Values.iam.account_id }}:oidc-provider/{{ .Values.iam.oidc_url }}"
                },
                "Action": "sts:AssumeRoleWithWebIdentity",
                "Condition": {
                    "StringEquals": {
                        "{{ .Values.iam.oidc_url }}:aud": "sts.amazonaws.com",
                        "{{ .Values.iam.oidc_url }}:sub": "system:serviceaccount:default:ack-app-serviceaccount"
                    }
                }
            }
        ]
    }
  name: ack-app-role
  policies:
  - "arn:aws:iam::{{ .Values.iam.account_id }}:policy/ack-dynamodb-policy"
---
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::{{ .Values.iam.account_id }}:role/ack-app-role
    eks.amazonaws.com/sts-regional-endpoints: "true"
    eks.amazonaws.com/token-expiration: "86400"
    argocd.argoproj.io/sync-wave: "1"
  name: ack-app-serviceaccount
  namespace: default

helmfile -e test apply .

Applying it will create the IAM policy, IAM role with the attached policy, and Kubernetes service account defined in the manifest.

kubectl get serviceaccount

NAME                     SECRETS   AGE
ack-app-serviceaccount   0         7m57s
default                  0         49m

Testing Access to DynamoDB

With the created service account, access to DynamoDB is now permitted from the application. Let's verify this by using the AWS CLI to access DynamoDB. First, let's try executing the PutItem operation on DynamoDB without specifying the service account.

kubectl run awscli --image=amazon/aws-cli -- dynamodb put-item --table-name ack-dynamodb-table --item '{ "id": { "N": "1" }, "value": { "S": "test" }}'

This will result in an error:

kubectl logs awscli

An error occurred (AccessDeniedException) when calling the PutItem operation: User: arn:aws:sts::xxxxxxxxxxxx:assumed-role/eksctl-ack-helmfile-nodegroup-ng-NodeInstanceRole-EVFGB3QY0C29/i-097e6311fa2cc1da1 is not authorized to perform: dynamodb:PutItem on resource: arn:aws:dynamodb:ap-northeast-1:xxxxxxxxxxxx:table/ack-dynamodb-table because no identity-based policy allows the dynamodb:PutItem action

Let's now execute PutItem on DynamoDB using the service account.

kubectl run awscli --image=amazon/aws-cli --overrides='{ "spec": { "serviceAccountName": "ack-app-serviceaccount" } }' -- dynamodb put-item --table-name ack-dynamodb-table --item '{ "id": { "N": "1" }, "value": { "S": "test" }}'

The PutItem operation is now successful.

Summary

In this article, we used AWS Controllers for Kubernetes (ACK) to manage AWS resources using Kubernetes manifests. We discussed the configuration to make the application able to utilize AWS resources and explored how to manage Kubernetes manifest files using helmfile.

Using ACK and the described workflow, it becomes easier to manage AWS resources in conjunction with the application's lifecycle, allowing for convenient creation and deletion of AWS resources alongside the application.