Javier Sepúlveda for AWS Community Builders

Posted on Jul 25

GitOps beyond Git: immutable platform delivery with OCI and Argo CD using EKS Capabilities.

#aws #terraform #kubernetes #gitops

In the last blog, KIRO and the AMDF MCP was used to generate reusable Kubernetes packages, KCL modules and a KRO ResourceGraphDefinition that groups multiple CRDs into a single self service API CR.

But building the package is only the firs step. To use it in production enviroments, is necessary to ship it and when there are more than one cluster or teams, a new question shows up: how do these packages actually reach the clusters reliably, versioned, and inmutable?

Quick recap: in the last blog was built a basic Keycloak resource, component by component. Remember that a KRO package is just KCL that renders to YAML.

Check last blog

Prerequisites

EKS Capabilities Architecture.

In this architecture, we use Terragrunt to provision a complete stack Network, EKS, and the Capabilities, a bootstrap step that installs Argo CD and the resources it needs.

Check all code in this repository for replicate this blog.

segoja7 / aws_eks_capabilities

Terraform Terragrunt Scaffold

Enterprise-grade AWS infrastructure scaffold using Terraform and Terragrunt with a layered architecture based on Domain-Driven Design (DDD) principles.

Architecture

The project implements four infrastructure layers with strict dependency ordering:

Foundation → Platform → Application → Observability

Stacks

stacks/
├── foundation/
│   ├── network/
│   │   ├── vpc/                  # VPC, subnets, NAT gateways
│   │   └── security-groups/      # Security group definitions
│   └── iam/
│       ├── roles/                # IAM roles
│       └── policies/             # IAM policies
├── platform/
│   ├── containers/
│   │   ├── eks/                  # EKS cluster and node groups
│   │   └── ecr/                  # Container registry
│   └── data/
│       ├── rds/                  # RDS PostgreSQL/MySQL
│       └── elasticache/          # ElastiCache Redis/Memcached
├── application/
│   ├── compute/
│   │   ├── alb/                  # Application Load Balancer
│   │   └── asg/                  # Auto Scaling Groups
│   └── storage/
│       ├── s3/                   # S3 buckets
│       └── efs/                  # Elastic

…

View on GitHub

├── common
│   ├── additional_providers
│   │   └── provider_kubectl.hcl
│   ├── common.hcl
│   ├── common.tfvars
│   └── variables.tf
├── environments
│   └── dev
│       ├── foundations.tfvars
│       └── platform.tfvars
├── modules
│   └── cicd
│       ├── codebuild.tf
│       ├── codecommit.tf
│       ├── codepipeline.tf
│       ├── events.tf
│       ├── iam.tf
│       ├── kms.tf
│       ├── main.tf
│       ├── outputs.tf
│       ├── variables.tf
│       └── versions.tf
├── README.md
├── root.hcl
└── stacks
    ├── foundation
    │   └── network
    │       └── vpc
    │           ├── main.tf
    │           ├── outputs.tf
    │           ├── provider.tf
    │           ├── remotebackend.tf
    │           ├── terragrunt.hcl
    │           └── variables.tf
    └── platform
        ├── cicd
        │   └── pipeline
        │       ├── main.tf
        │       ├── outputs.tf
        │       ├── provider.tf
        │       ├── remotebackend.tf
        │       ├── terragrunt.hcl
        │       └── variables.tf
        ├── containers
        │   ├── ecr
        │   │   ├── main.tf
        │   │   ├── outputs.tf
        │   │   ├── provider.tf
        │   │   ├── remotebackend.tf
        │   │   ├── terragrunt.hcl
        │   │   └── variables.tf
        │   └── eks
        │       ├── capabilities.tf
        │       ├── main.tf
        │       ├── outputs.tf
        │       ├── provider.tf
        │       ├── remotebackend.tf
        │       ├── terragrunt.hcl
        │       └── variables.tf
        └── gitops
            └── bootstrap
                ├── kubectl_provider.tf
                ├── main.tf
                ├── provider.tf
                ├── remotebackend.tf
                ├── root-app.yaml.tftpl
                ├── terragrunt.hcl
                └── variables.tf

The hub is a plain EKS cluster provisioned with Terragrunt but with a capabilities: ACK, KRO, and Argo CD run as managed capabilities. AWS runs them in the control plane.

From KCL to a running cluster: the delivery pipeline

a KRO package is just KCL that renders to YAML. So delivery has two phases: publish the reusable KCL, and publish the rendered result in Argo CD.

Phase1: The reusable KCL modules, pushed with kcl mod push. Think of them as versioned building blocks other packages depend on.
Phase2: the rendered manifests. The final YAML (the RGDs + the app-of-apps), packed as a single OCI artifact with oras push. This is what Argo CD actually pulls and applies.

With managed Argo CD is not possible run KCL there is no config management plugin in the control plane for that reason is necesary rendering to plain YAML, but there is not problem because OCI is just the vehicle that carries it.

The trigger

A push to CodeCommit trigger the pipeline:

version: 0.2

#Managed ArgoCD pulls that artifact directly (repoURL oci://). 

phases:
  install:
    commands:
      # KCL_VERSION (a CodeBuild env var) pins the CLI; the install script reads it.
      - curl -fsSL https://kcl-lang.io/script/install-cli.sh | bash
      - export PATH="$HOME/.kcl/bin:$PATH"
      - kcl version
      # ORAS — pinned (ORAS_VERSION), same flaky-`latest` lesson as KCL. Packages
      # the rendered manifests as a single-layer OCI artifact ArgoCD understands.
      - |
        V="${ORAS_VERSION#v}"
        curl -fsSL "https://github.com/oras-project/oras/releases/download/v${V}/oras_${V}_linux_amd64.tar.gz" -o /tmp/oras.tar.gz
        mkdir -p /tmp/oras && tar -xzf /tmp/oras.tar.gz -C /tmp/oras oras
        mv /tmp/oras/oras /usr/local/bin/oras
        oras version
  pre_build:
    commands:
      - export PATH="$HOME/.kcl/bin:$PATH"
      # ~/.docker/config.json. 
      - |
        mkdir -p "$HOME/.docker"
        TOKEN=$(aws ecr get-login-password --region "$AWS_REGION")
        AUTH=$(printf 'AWS:%s' "$TOKEN" | base64 -w0)
        printf '{"auths":{"%s":{"auth":"%s"}}}' "$ECR_REGISTRY" "$AUTH" > "$HOME/.docker/config.json"
  build:
    commands:
      - export PATH="$HOME/.kcl/bin:$PATH"
      # publish each platform library package to its own OCI repo, tagged
      # with the package's own kcl.mod `version` (single source of truth).
      - |
        for d in platform/libraries/*/; do
          lib=$(basename "$d")
          (cd "$d" && kcl mod push --force "oci://$ECR_REGISTRY/$ECR_NAMESPACE/$lib")
        done
      # Inject the APIs' blueprints dependency
      - |
        BP_VER=$(awk -F'"' '/^version[[:space:]]*=/{print $2}' platform/libraries/blueprints/kcl.mod)
        (cd platform/apis && kcl mod add "oci://$ECR_REGISTRY/$ECR_NAMESPACE/blueprints" --tag "$BP_VER")
      - mkdir -p rendered/layer-1 rendered/apps
      # platform/apis = the RGDs (pulls blueprints from ECR). 
      - (cd platform/apis && kcl run -S items -q) > rendered/layer-1/manifests.yaml
      - |
        for f in platform/gitops/apps/*.yaml; do
          envsubst '$MANIFESTS_REPO $MANIFESTS_TAG $CLAIMS_REPO_URL' < "$f" > "rendered/apps/$(basename "$f")"
        done
      - echo "rendered:" && wc -l rendered/layer-1/manifests.yaml && ls rendered/apps
  post_build:
    commands:
      # Package rendered/ as ONE tar+gzip layer
      - |
        set -e   # fail the build on a bad push (without this, oras errors were silent)
        tar -czf /tmp/manifests.tar.gz rendered
        SHA="sha-$(echo "${CODEBUILD_RESOLVED_SOURCE_VERSION:-manual}" | cut -c1-7)"
        MT="application/vnd.oci.image.layer.v1.tar+gzip"
        # --disable-path-validation: the tarball is an absolute /tmp path (oras
        # rejects those by default).
        oras push --disable-path-validation "$MANIFESTS_REPO:$MANIFESTS_TAG" "/tmp/manifests.tar.gz:$MT"
        oras push --disable-path-validation "$MANIFESTS_REPO:$SHA"           "/tmp/manifests.tar.gz:$MT"
        echo "pushed $MANIFESTS_REPO tags: $MANIFESTS_TAG, $SHA"

How to work this pipeline.

Step 1: Install the tools.

KCL
ORAS.

install:
  commands:
    - curl -fsSL https://kcl-lang.io/script/install-cli.sh | bash   # reads $KCL_VERSION
    - export PATH="$HOME/.kcl/bin:$PATH"
    - |
      V="${ORAS_VERSION#v}"
      curl -fsSL ".../oras_${V}_linux_amd64.tar.gz" -o /tmp/oras.tar.gz
      tar -xzf /tmp/oras.tar.gz -C /tmp/oras oras && mv /tmp/oras/oras /usr/local/bin/oras

Step 2: Authenticate to ECR.

writes the ECR credentials into ~/.docker/config.json.

pre_build:
  commands:
    - |
      mkdir -p "$HOME/.docker"
      TOKEN=$(aws ecr get-login-password --region "$AWS_REGION")
      AUTH=$(printf 'AWS:%s' "$TOKEN" | base64 -w0)
      printf '{"auths":{"%s":{"auth":"%s"}}}' "$ECR_REGISTRY" "$AUTH" > "$HOME/.docker/config.json"

Step 3: publish the KCL libraries.

Pushes every KCL library to its own OCI repo (e.g. kcl-modules/blueprints). Why per-library: each module is versioned independently by its own kcl.mod version.

  build:
    commands:
      ###
      - |
        for d in platform/libraries/*/; do
          lib=$(basename "$d")
          (cd "$d" && kcl mod push --force "oci://$ECR_REGISTRY/$ECR_NAMESPACE/$lib")
        done

Step 4: Connect the dependency by version

tells the APIs package to pull blueprints from OCI, at the exact version declared in the library's own kcl.mod. nothing is hardcoded in git the URL comes from env vars, the tag from the source of truth.

# Inject the APIs' blueprints dependency
      - |
        BP_VER=$(awk -F'"' '/^version[[:space:]]*=/{print $2}' platform/libraries/blueprints/kcl.mod)
        (cd platform/apis && kcl mod add "oci://$ECR_REGISTRY/$ECR_NAMESPACE/blueprints" --tag "$BP_VER")

Step 5: Render the RGDs to YAML

compiles the KCL APIs (ResourceGraphDefinitions) into plain Kubernetes YAML. This is the "just KCL that renders to YAML" the output is exactly what a developer only generated and typed.

mkdir -p rendered/layer-1 rendered/apps
(cd platform/apis && kcl run -S items -q) > rendered/layer-1/manifests.yaml

Step 6: The app-of-apps with real values

Injects account-specific values (the OCI repo/tag, the git URL for user claims) into the Argo CD Applications via envsubst.

|
for f in platform/gitops/apps/*.yaml; do
  envsubst '$MANIFESTS_REPO $MANIFESTS_TAG $CLAIMS_REPO_URL' < "$f" > "rendered/apps/$(basename "$f")"
done

Step 7: publish the rendered artifact

Packs the whole rendered/ directory into one tar+gzip layer and pushes it. Argo CD watches oci://…/manifests:dev. On a new push it resolves the tag to a digest, records that digest as the deployed revision, and applies the manifests. The platform team publishes an artifact, and the cluster converges on it.

post_build:
    commands:
      # Package rendered/ as ONE tar+gzip layer
      - |
        set -e   # fail the build on a bad push (without this, oras errors were silent)
        tar -czf /tmp/manifests.tar.gz rendered
        SHA="sha-$(echo "${CODEBUILD_RESOLVED_SOURCE_VERSION:-manual}" | cut -c1-7)"
        MT="application/vnd.oci.image.layer.v1.tar+gzip"
        # --disable-path-validation: the tarball is an absolute /tmp path (oras
        # rejects those by default).
        oras push --disable-path-validation "$MANIFESTS_REPO:$MANIFESTS_TAG" "/tmp/manifests.tar.gz:$MT"
        oras push --disable-path-validation "$MANIFESTS_REPO:$SHA"           "/tmp/manifests.tar.gz:$MT"
        echo "pushed $MANIFESTS_REPO tags: $MANIFESTS_TAG, $SHA"

The source of truth is an immutable artifact, not a mutable Git branch.

Step 8: push and watch it working.

After of terragrunt run --all apply in the directory of the repo aws_eks_capabilities, the EKS hub is up with the capabilities, the empty ECR repos, the pipeline and the Argo CD Application. But this is pointing at an OCI artifact that does not exist yet.

kubectl get application platform-root -n argocd
NAME            SYNC STATUS   HEALTH STATUS
platform-root   Unknown       Unknown

Step 8.1: Configure CodeCommit

git config --global credential.helper '!aws --profile **segoja7** codecommit credential-helper $@'
git config --global credential.UseHttpPath true

Clone the public repo from github and push the same code in empty repo and add the platform source.

git remote add codecommit https://git-codecommit.us-east-1.amazonaws.com/v1/repos/aws-eks-capabilities-kro-ack-argocd-oci-pipeline
git push codecommit master

the push to master trigger the pipeline and this runs from Steps 1–7 for generate the OCI package with RGD from KRO and ACK.

Step 8.3: The pipeline publishes manifests:dev, and platform-root change of Unknown to Synced.

kubectl get applications -n argocd
NAME            SYNC STATUS   HEALTH STATUS
layer-1-rgds    Synced        Healthy
platform-root   Synced        Healthy
user-claims     Synced        Healthy

kubectl get resourcegraphdefinitions
NAME          APIVERSION   KIND         STATE    READY   AGE
eks-cluster   v1alpha1     EKSCluster   Active   True    92s
vpc-network   v1alpha1     VPCNetwork   Active   True    92s

In this demo, a single CodeCommit repo holds both the platform source (platform/) and the user claims (user/claims/). That keeps the walkthrough simple but they are really two different jobs with two different owners: the platform team publishes platform/ (through the OCI pipeline), and another the claim users, a platform repo and a claims repo eachone with its own permissions.

Step 8.4 Apply User claim.

cat > user/claims/my-spoke.yaml <<'YAML'
apiVersion: kro.run/v1alpha1
kind: VPCNetwork
metadata:
  name: my-spoke
  namespace: default
spec:
  name: my-spoke
  vpcCIDR: '10.0.0.0/16'
  az1: us-east-1a
  az2: us-east-1b
---
apiVersion: kro.run/v1alpha1
kind: EKSCluster
metadata:
  name: my-spoke
  namespace: default
spec:
  name: my-spoke
  version: '1.36'
  networkName: my-spoke
  amiType: BOTTLEROCKET_x86_64
  capacityType: ON_DEMAND
  diskSize: 20
  instanceTypes:
  - t3.medium
  desiredSize: 1
  minSize: 1
  maxSize: 3
YAML
git add . && git commit -m "claim: my-spoke" && git push codecommit master

A few Seconds later is possible see the new CustomResource in progress.

kubectl get vpcnetworks,eksclusters -n default
NAME                          STATE         READY   AGE
vpcnetwork.kro.run/my-spoke   IN_PROGRESS   False   115s
NAME                          STATE         READY   AGE
ekscluster.kro.run/my-spoke   IN_PROGRESS   False   115s

A few minutes later, a new EKS cluster exists and has registered itself into the hub's Argo CD.

kubectl get vpcnetworks,eksclusters -n default
NAME                          STATE    READY   AGE
vpcnetwork.kro.run/my-spoke   ACTIVE   True    9m42s

NAME                          STATE    READY   AGE
ekscluster.kro.run/my-spoke   ACTIVE   True    9m42s

CONCLUSION:

We made delivery immutable. The next step is making it verifiable: signing every artifact, and having the cluster refuse to run anything it can't verify.

DEV Community