DEV Community

Keisuke FURUYA for AWS Community Builders

Posted on

Managing AWS Resources with ACK and helmfile

AWS Controllers for Kubernetes (ACK) is a Kubernetes Custom Controller that allows you to manage AWS resources through Kubernetes manifests. By using ACK, you can easily manage AWS resources in the same lifecycle as your applications. It is suitable for scenarios where you need to create AWS resources alongside your applications for testing purposes and delete them together with the application when no longer needed. This article explains how to manage AWS resources using Kubernetes manifests. While there are various ways to manage Kubernetes manifest files, this article focuses on using helmfile for management.

Prerequisites

Please install the following on your local machine:

  • eksctl
  • kubectl
  • helm
  • helmfile

Set up the following in your AWS account. This IAM policy has the necessary permissions for the IAM Controller of ACK to create IAM resources.

First, let's create an EKS cluster. In this example, we will deploy it with minimal configuration using eksctl.

eksctl create cluster --name ack-helmfile
Enter fullscreen mode Exit fullscreen mode

Next, create an OIDC provider, IAM Role and service account.

eksctl utils associate-iam-oidc-provider --cluster=ack-helmfile --approve
eksctl create iamserviceaccount --cluster=ack-helmfile --name=ack-dynamodb-controller --namespace=ack-system --attach-policy-arn=arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess --approve
eksctl create iamserviceaccount --cluster=ack-helmfile --name=ack-iam-controller --namespace=ack-system --attach-policy-arn=arn:aws:iam::[your AWS account ID]:policy/ack-iam-controller-policy --approve
Enter fullscreen mode Exit fullscreen mode

Install ACK on the created EKS cluster. This time, we will install the IAM and DynamoDB controllers. Below is a sample configuration for the ACK controllers using helmfile.

helmfile.yaml

repositories:
  - name: aws-controllers-k8s
    url: public.ecr.aws/aws-controllers-k8s
    oci: true

environments:
  {{ .Environment.Name }}:
    values:
    - environments/{{ .Environment.Name }}/values.yaml

---

releases:
  - name: "ack-dynamodb-controller"
    namespace: "{{ .Namespace | default .Values.awsControllersK8s.namespace }}"
    createNamespace: true
    chart: "aws-controllers-k8s/dynamodb-chart"
    version: "{{ .Values.awsControllersK8s.dynamodb.version }}"
    installedTemplate: "{{ .Values.awsControllersK8s.dynamodb.installed }}"
    values:
    - values/dynamodb.yaml.gotmpl

  - name: "ack-iam-controller"
    namespace: "{{ .Namespace | default .Values.awsControllersK8s.namespace }}"
    createNamespace: true
    chart: "aws-controllers-k8s/iam-chart"
    version: "{{ .Values.awsControllersK8s.iam.version }}"
    installedTemplate: "{{ .Values.awsControllersK8s.iam.installed }}"
    values:
    - values/iam.yaml.gotmpl
Enter fullscreen mode Exit fullscreen mode

environments/test/values.yaml

awsControllersK8s:
  namespace: ack-system
  region: ap-northeast-1

  dynamodb:
    installed: true
    version: 1.1.1

  iam:
    installed: true
    version: 1.2.1
Enter fullscreen mode Exit fullscreen mode

values/dynamodb.yaml.gotmpl

aws:
  region: {{ .Values.awsControllersK8s.region }}

serviceAccount:
  create: false
  name: ack-dynamodb-controller
Enter fullscreen mode Exit fullscreen mode

values/iam.yaml.gotmpl

aws:
  region: {{ .Values.awsControllersK8s.region }}

serviceAccount:
  create: false
  name: ack-iam-controller
Enter fullscreen mode Exit fullscreen mode

Applying helmfile will create the IAM and DynamoDB controllers.

helmfile -e test apply .
Enter fullscreen mode Exit fullscreen mode
kubectl get deployment -n ack-system
NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
ack-dynamodb-controller-dynamodb-chart   1/1     1            1           79s
ack-iam-controller-iam-chart             1/1     1            1           79s
Enter fullscreen mode Exit fullscreen mode

Deploying Required AWS Resources in the Application

Now that the DynamoDB controller is ready, let's use it to create a DynamoDB resource. Here, we will deploy a simple table called "ack-dynamodb-table".

helmfile.yaml

environments:
  {{ .Environment.Name }}:
    values:
    - environments/{{ .Environment.Name }}/values.yaml

---

releases:
  - name: "ack-dynamodb-table"
    namespace: default
    installedTemplate: "{{ .Values.dynamodb.installed }}"
    chart: "./manifests"
Enter fullscreen mode Exit fullscreen mode

environments/test/values.yaml

dynamodb:
  installed: true
Enter fullscreen mode Exit fullscreen mode

manifests/table.yaml

apiVersion: dynamodb.services.k8s.aws/v1alpha1
kind: Table
metadata:
  name: ack-dynamodb-table
spec:
  tableName: ack-dynamodb-table
  attributeDefinitions:
    - attributeName: id
      attributeType: "N"
  keySchema:
    - attributeName: id
      keyType: "HASH"
  provisionedThroughput:
    writeCapacityUnits: 1
    readCapacityUnits: 1
Enter fullscreen mode Exit fullscreen mode
helmfile -e test apply .
Enter fullscreen mode Exit fullscreen mode

Applying it will create the DynamoDB table defined in the manifest.

Created DynamoDB Table

Deploying Role and Service Account for IRSA

To utilize DynamoDB from the application, we need to use IRSA (IAM Roles for Service Accounts). IRSA enables the association of Kubernetes service accounts with IAM roles. Specifically, it involves setting up the IAM role, appropriate trust relationships, and annotating the service account to make it usable. While eksctl can handle this through commands(as we did earlier), we will create them using ACK this time.

helmfile.yaml

environments:
  {{ .Environment.Name }}:
    values:
    - environments/{{ .Environment.Name }}/values.yaml

---

releases:
  - name: "ack-iam-role"
    namespace: default
    installedTemplate: "{{ .Values.iam.installed }}"
    chart: "./manifests"
Enter fullscreen mode Exit fullscreen mode

We will obtain the OIDC URL required for the trust relationship of the role used in service account from the EKS management console and add it to the values.yaml file.

EKS OIDC info

environments/test/values.yaml

iam:
  installed: true
  account_id: [your AWS account ID]
  oidc_url: [your EKS oidc url]
Enter fullscreen mode Exit fullscreen mode

We will define the policy, role, and service account in the manifest that have permissions to access DynamoDB.

manifests/role.yaml

apiVersion: iam.services.k8s.aws/v1alpha1
kind: Policy
metadata:
  name: ack-dynamodb-policy
spec:
  policyDocument: |
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "",
                "Effect": "Allow",
                "Action": [
                    "dynamodb:UpdateTable",
                    "dynamodb:UpdateItem",
                    "dynamodb:Scan",
                    "dynamodb:Query",
                    "dynamodb:PutItem",
                    "dynamodb:ListTables",
                    "dynamodb:GetItem",
                    "dynamodb:DescribeTable",
                    "dynamodb:DeleteItem",
                    "dynamodb:BatchWriteItem",
                    "dynamodb:BatchGetItem"
                ],
                "Resource": [
                    "arn:aws:dynamodb:ap-northeast-1:{{ .Values.iam.account_id }}:table/ack-dynamodb-table"
                ]
            }
        ]
    }
  name: ack-dynamodb-policy
---
apiVersion: iam.services.k8s.aws/v1alpha1
kind: Role
metadata:
  name: ack-app-role
spec:
  assumeRolePolicyDocument: |
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "Federated": "arn:aws:iam::{{ .Values.iam.account_id }}:oidc-provider/{{ .Values.iam.oidc_url }}"
                },
                "Action": "sts:AssumeRoleWithWebIdentity",
                "Condition": {
                    "StringEquals": {
                        "{{ .Values.iam.oidc_url }}:aud": "sts.amazonaws.com",
                        "{{ .Values.iam.oidc_url }}:sub": "system:serviceaccount:default:ack-app-serviceaccount"
                    }
                }
            }
        ]
    }
  name: ack-app-role
  policies:
  - "arn:aws:iam::{{ .Values.iam.account_id }}:policy/ack-dynamodb-policy"
---
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::{{ .Values.iam.account_id }}:role/ack-app-role
    eks.amazonaws.com/sts-regional-endpoints: "true"
    eks.amazonaws.com/token-expiration: "86400"
    argocd.argoproj.io/sync-wave: "1"
  name: ack-app-serviceaccount
  namespace: default
Enter fullscreen mode Exit fullscreen mode
helmfile -e test apply .
Enter fullscreen mode Exit fullscreen mode

Applying it will create the IAM policy, IAM role with the attached policy, and Kubernetes service account defined in the manifest.

Created IAM Role

kubectl get serviceaccount

NAME                     SECRETS   AGE
ack-app-serviceaccount   0         7m57s
default                  0         49m
Enter fullscreen mode Exit fullscreen mode

Testing Access to DynamoDB

With the created service account, access to DynamoDB is now permitted from the application. Let's verify this by using the AWS CLI to access DynamoDB. First, let's try executing the PutItem operation on DynamoDB without specifying the service account.

kubectl run awscli --image=amazon/aws-cli -- dynamodb put-item --table-name ack-dynamodb-table --item '{ "id": { "N": "1" }, "value": { "S": "test" }}'
Enter fullscreen mode Exit fullscreen mode

This will result in an error:

kubectl logs awscli

An error occurred (AccessDeniedException) when calling the PutItem operation: User: arn:aws:sts::xxxxxxxxxxxx:assumed-role/eksctl-ack-helmfile-nodegroup-ng-NodeInstanceRole-EVFGB3QY0C29/i-097e6311fa2cc1da1 is not authorized to perform: dynamodb:PutItem on resource: arn:aws:dynamodb:ap-northeast-1:xxxxxxxxxxxx:table/ack-dynamodb-table because no identity-based policy allows the dynamodb:PutItem action
Enter fullscreen mode Exit fullscreen mode

Let's now execute PutItem on DynamoDB using the service account.

kubectl run awscli --image=amazon/aws-cli --overrides='{ "spec": { "serviceAccountName": "ack-app-serviceaccount" } }' -- dynamodb put-item --table-name ack-dynamodb-table --item '{ "id": { "N": "1" }, "value": { "S": "test" }}'
Enter fullscreen mode Exit fullscreen mode

The PutItem operation is now successful.

Succeed PutItem

Summary

In this article, we used AWS Controllers for Kubernetes (ACK) to manage AWS resources using Kubernetes manifests. We discussed the configuration to make the application able to utilize AWS resources and explored how to manage Kubernetes manifest files using helmfile.

Using ACK and the described workflow, it becomes easier to manage AWS resources in conjunction with the application's lifecycle, allowing for convenient creation and deletion of AWS resources alongside the application.

Top comments (0)