Building an S3-Backed App on EKS: Comparing kro and Crossplane Composition

Recently, I gave a talk at the Cloud Native Platform Engineering Japan Meetup, where another speaker presented a session titled "Building elegant platform with KRO."

I had already been interested in kro after attending a related session at KubeCon in July, and seeing this presentation gave me another reason to try it myself. I also wanted to find out whether kro could be useful for our internal developer platform, so I decided to experiment with kro and Crossplane together.

For the demo, I built a simple application that stores files in S3. The idea was that a developer could say, "I want to run an application using this image," and the platform would provision not only the Deployment but also the S3 bucket and IAM resources required by the application.

https://github.com/suzuki0430/kro-crossplane-eks-idp-lab

In the meetup presentation, the Platform Team prepared an RGD as the template for the resources, while developers only needed to write a small YAML file to create both the application and its infrastructure. The goal was to avoid requiring developers to understand the details of Kubernetes.

During the Q&A, the speaker also explained that they had evaluated Crossplane Composition before choosing kro, with the simplicity of the definitions being one of the reasons for that decision.

Do I Need Both kro and Crossplane?

For what I wanted to achieve in this experiment, kro is not strictly required.

Later in this article, I also set up another EKS cluster without kro and ran the same application using Crossplane Composition.

Architecture	Can it do what I want in this experiment?
kro only	Can create resources such as Deployments and Services together. However, kro itself does not have functionality for managing S3 or IAM, so another controller is required to manage AWS resources.
Crossplane v2 + AWS Provider + Function	Can achieve the same result without kro. A custom API and Composition can be defined to create the Deployment and AWS resources together.
kro + Crossplane + AWS Provider	kro handles the custom API and relationships between resources, while the Provider handles AWS operations.

What I Assigned to Each Component

The entry point for this experiment is a custom resource called StorageApp. When a developer creates a single StorageApp, kro creates the required Kubernetes resources. Among those resources, the ones representing AWS resources are reconciled to AWS by the Provider.

Component	Responsibility in this experiment
kro	Define the StorageApp API, the required resources and their dependencies, and the conditions for considering them Ready.
Crossplane + AWS Provider	Create S3, IAM, and the Pod Identity Association in AWS and keep their state synchronized.
EKS	Run the controllers and the application. Provide AWS credentials through Pod Identity.
eksctl / CloudFormation	Provision the EKS cluster itself and resources such as the IAM configuration required by the Provider beforehand.

The EKS cluster itself is prepared before creating a StorageApp. What I made self-service in this experiment was the application running on an existing EKS cluster and the AWS resources required specifically by that application.

How I Deployed the Application

For this experiment, I ran everything from the terminal. I did not build a portal UI or a GitOps-based deployment flow.

After preparing EKS and the controllers, I use the following scripts:

# Build the application image and push it to ECR
bash scripts/04-image.sh

# Create the StorageApp and wait until it becomes Ready
bash scripts/05-demo.sh

The application-side input consists of three values: a storage ID, an image, and the number of replicas. In YAML, it looks like this:

apiVersion: platform.example.com/v1alpha1
kind: StorageApp
metadata:
  name: demo
  namespace: idp-lab
spec:
  storageId: demo
  image: ACCOUNT.dkr.ecr.ap-northeast-1.amazonaws.com/idplab-LAB/storage-api@sha256:DIGEST
  replicas: 1

The command creates the StorageApp. From there, kro creates resources such as the Deployment, and the standard Kubernetes controllers start the Pods. S3 and IAM resources are created by the AWS Provider after it observes the MRs created by kro.

The developer ServiceAccount is allowed to create and update StorageApp resources, but it cannot create IAM MRs or modify the ConfigMap containing platform settings.

Where to Define Creation Order and Ready Conditions

On the kro side, I register a ResourceGraphDefinition (RGD). It defines both what users can specify in StorageApp and what resources should be created from those inputs.

In this experiment, I create the following six types of MRs and combine them with a Deployment, Service, and ServiceAccount.

MR	What it manages
Bucket	S3 bucket
BucketPublicAccessBlock	Settings that block public access to S3
BucketServerSideEncryptionConfiguration	Default S3 encryption
Role / RolePolicy	IAM role for the application and its S3 access permissions
PodIdentityAssociation	Association between the EKS ServiceAccount and IAM role

In the RGD, for example, the Association references the Role ARN, while the Deployment references the Association ID. kro uses these references to build the dependency graph.

To make sure the application policy is also ready before proceeding, I added a reference to the RolePolicy in an annotation on the Association.

The conditions used to determine whether each resource is ready are defined with kro's readyWhen. For the S3 and IAM MRs, I check both Ready=True, which indicates that the Provider considers the resource available, and Synced=True, which indicates that the latest reconciliation completed successfully.

kro considers the Deployment ready only when all three of the following conditions are satisfied:

# Excerpt from platform/storage-app.yaml
# Explanatory comments added
- id: deployment
  readyWhen:
    # The Deployment controller has observed the latest configuration
    - ${deployment.status.observedGeneration == deployment.metadata.generation}
    # The number of Pods using the latest Pod template matches the requested replica count
    - ${deployment.status.?updatedReplicas.orValue(0) == deployment.spec.replicas}
    # The number of available Pods that satisfy readiness conditions matches the requested replica count
    - ${deployment.status.?availableReplicas.orValue(0) == deployment.spec.replicas}

Verifying S3 Reads and Writes Through the Application

The application itself is a simple API written in Go.

API	What it does
GET /healthz	Reports whether the process is alive. It does not access AWS.
GET /readyz	Put / Get / Delete a temporary object in S3 and compare its contents.
PUT /objects/{key}	Store a file of up to 1 MiB under uploads/.
GET /objects/{key}	Retrieve a file. Returns 404 if it does not exist.

The readiness probe runs every 10 seconds and uses a dedicated _health/ key. The liveness probe does not depend on AWS because I do not want the process to restart just because S3 is temporarily unavailable.

For the normal functional test, I PUT a binary file over HTTP, GET it back, and compare the SHA256 hash of the retrieved file with the original.

I use Pod Identity for AWS authentication. Both the application and the Provider use temporary credentials, so neither requires long-lived access keys.

What Happens When I Intentionally Remove a Required Permission?

Next, I attach a dedicated Deny policy to the IAM role used by the running application. The denied action is s3:PutObject for the _health/* objects used by the health check.

The bucket and Pod Identity Association remain unchanged; only the S3 health check is forced to fail.

Here is what happened:

What I checked	Before the Deny	After the Deny takes effect
Ready / Synced for all six types of MRs	All True	Still all True
S3 health check	Success	PutObject returns 403 / AccessDenied
Deployment	1/1	0/1
StorageApp	Ready=True	Ready=False, STATE=IN_PROGRESS

The Deployment becoming 0/1 when readiness fails is normal Kubernetes behavior. What I wanted to verify here was that the Deployment conditions defined in the RGD's readyWhen also propagate to the StorageApp, causing it to become Ready=False.

In other words, even if all of the MRs remain Ready, the StorageApp as a whole does not become Ready unless the Deployment conditions are also satisfied.

After removing the Deny policy, the StorageApp returned to Ready=True.

Does Deleting the Application Also Delete the S3 Data?

For this experiment, I decided to clean up the application, IAM resources, and Pod Identity Association while preserving the data in S3.

For the three S3-related MRs, I do not include Delete in managementPolicies.

spec:
  managementPolicies: [Observe, Create, Update, LateInitialize]
  providerConfigRef:
    name: aws
    kind: ProviderConfig

With this configuration, even if the Kubernetes MRs are deleted, the actual bucket and its configuration remain in AWS. I preserve not only the bucket itself but also its public access block and encryption configuration.

When I deleted the StorageApp, the Deployment and all six types of MRs were removed. The original file in S3, however, remained, and its SHA256 hash still matched. All four public access block settings and the default AES256 encryption configuration also remained.

Next, I recreate the StorageApp using the same storageId: demo.

I do not upload the file again. Instead, I retrieve the original file through the HTTP API of the new Pod.

The SHA256 hashes match for all three of the following:

• The original file that was initially uploaded
• The file retrieved directly from S3 after deleting the application
• The file retrieved through HTTP GET from the recreated application

Trying the Same Thing with Crossplane Composition Without kro

Up to this point, the experiment only showed that the architecture could be implemented with kro. To make the comparison more meaningful, I created another EKS cluster without kro.

This time, I define the StorageApp API using a Crossplane XRD and use a Composition to create the required resources. I refer to this architecture without kro as the "Composition version" and compare it with the previous "kro + Crossplane version."

In a Composition, a program called a Function looks at the StorageApp inputs and the state of existing resources, then returns the list of resources that should be created and maintained. Crossplane manages the resources according to that list.

For this experiment, I use the publicly available function-go-templating and define the Deployment and MRs using Go templates. The Function itself runs as a Pod inside EKS.

The main versions used for the comparison are shown below. I also use the same application image digest and the same input fields for StorageApp.

Item	kro + Crossplane version	Composition version
kro	0.9.4	None
Crossplane	2.4.2	2.4.2
AWS Provider	2.8.1	2.8.1
EKS Kubernetes	1.36.4	1.36.4
Function	None	function-go-templating 0.13.0

In the end, everything I tested in this experiment worked with both approaches.

What I tested	kro + Crossplane version	Composition version
Create the application, S3, and IAM from StorageApp	Success	Success
PUT / GET a file over HTTP and compare hashes	Match	Match
Apply and remove the Deny for _health/*	Ready=False → True	Ready=False → True
Change storageId or set replicas: 4	Rejected	Rejected
Change replicas from 1 → 2 → 1	Applied	Applied
Preserve S3 and its protection settings after deleting the application	Preserved	Preserved
Recreate with the same storageId and GET the original file	Match	Match

Comparing Dependencies and Ready Conditions

In the kro + Crossplane version, dependencies are expressed through references between resources, while readyWhen defines the conditions for considering a resource ready.

For example, when the Deployment references the Association ID, kro waits for the Association to become ready before creating the Deployment.

The following excerpt from the actual RGD shows the conditions for waiting for the Association and the references from the Deployment. The Deployment body and other details are omitted here.

# platform/storage-app.yaml (excerpt)
- id: association
  readyWhen:
    - ${association.status.conditions.exists(c, c.type == 'Ready' && c.status == 'True')}
    - ${association.status.conditions.exists(c, c.type == 'Synced' && c.status == 'True')}
  # template omitted
- id: deployment
  template:
    kind: Deployment
    spec:
      template:
        metadata:
          annotations:
            platform.example.com/association: ${association.status.atProvider.associationId}
            platform.example.com/public-access: ${publicAccess.metadata.name}
            platform.example.com/encryption: ${encryption.metadata.name}

I also include references to the public access block and encryption configuration. Because kro determines the creation order from these references, I do not need to write conditional logic to decide whether the Deployment should be output.

In the Composition version, I instead write logic in the Go template that says, "Add the Deployment to the list if the Association and S3 protection settings are Ready." This conditional controls the initial creation order.

The relevant section is shown below. $seen contains resources that have already been observed, while $ready contains the results of checking whether each MR has both Ready=True and Synced=True.

{{- if or
    (not (empty (index $seen "deployment")))
    (and
        (index $ready "association")
        (index $ready "publicAccess")
        (index $ready "encryption")
        (ne $associationId "")
    )
}}
---
apiVersion: apps/v1
kind: Deployment
# Deployment body omitted
{{- end }}

The and branch defines the condition for creating the Deployment for the first time. The Deployment is output once the Association, public access block, and encryption configuration are ready and the Association ID has been obtained.

The other side of the or handles the case where the Deployment already exists. If the Deployment disappears from the list returned by the Function, Crossplane deletes it. For that reason, once the Deployment has been created, I keep it in the list even if one of its dependencies temporarily becomes not Ready.

I also implement the condition for marking the entire StorageApp as Ready inside the Function. It checks whether all nine required resources exist, whether the MRs have Ready=True and Synced=True, and whether the Deployment is running with the requested number of replicas.

On kind, I modified the Bucket status to simulate a synchronization failure. The Deployment UID did not change, while the StorageApp became Ready=False.

S3 Data Was Preserved in the Composition Version Too

In the Composition version, the file in S3 also remained after deleting the StorageApp. When I recreated it using the same storageId, I was able to retrieve the original file without uploading it again.

As in the kro + Crossplane version, the S3-related MRs do not include Delete in their managementPolicies. Therefore, even when the MRs are deleted, the Provider does not delete the bucket or its protection settings from AWS.

Conclusion

In this experiment, the fields developers specify in StorageApp were the same with both approaches.

One reason to use kro together with Crossplane would be to make the platform-side definitions easier to read and write. In fact, for the definitions I created in this experiment, I found the kro version easier to read.

On the other hand, if the goal is simply to provide something like this StorageApp, and Crossplane is already being operated in the environment, consolidating everything into a Composition also seems like a reasonable option.