DEV Community: Matt Adorjan

Enhancing Cloud Resilience: Unveiling Amazon Application Recovery Controller’s Latest Features for Seamless Application Recovery

Matt Adorjan — Mon, 25 Nov 2024 18:33:23 +0000

Amazon Application Recovery Controller (ARC) is an AWS service designed to help organizations prepare for and execute faster recovery of applications running on AWS’s global cloud infrastructure. ARC provides insights into whether applications and resources are ready for recovery and helps manage and coordinate recovery across AWS Regions and Availability Zones (AZs). This capability reduces the manual steps traditionally required for application recovery, making it simpler and more reliable. It can also help put control of application failovers and fail backs into the hands of your developers, minimizing dependence on your infrastructure teams. In this post, I want to summarize some of the key benefits, recent announcements, and why these are important.

Key Features

Multi-Availability Zone (AZ) Recovery: ARC offers zonal shift and zonal autoshift capabilities to recover from single AZ impairments by redirecting traffic from an impaired AZ to a healthy one.
Multi-Region Recovery: This includes routing control for failover and readiness checks to monitor application readiness, ensuring applications are configured to handle failover traffic.
Routing Control: Allows rerouting of application traffic across different AWS Regions or AZs using simple on/off switches integrated with Amazon Route 53 health checks.

Example Walkthrough: Using ARC

To demonstrate how to use ARC, consider setting up a multi-Region recovery strategy:

Prepare Your Applications: Ensure your applications are set up as siloed replicas in multiple AWS Regions. This setup allows traffic failover from a primary application to a secondary one during an event.
Create Routing Controls:
- Establish routing controls in ARC to manage traffic flow between Regions.
- Use the Amazon Route 53 data plane for DNS-based failover, associating each replica with a routing control.
Implement Safety Rules:
- Define safety rules to prevent unintended outcomes during failover, such as ensuring only one replica is active at any time.
Use Readiness Checks:
- Continuously monitor resource quotas, capacity, and network routing policies to ensure readiness for failover.
Failover Execution:
- During an event, use the ARC API or AWS CLI to update routing control states and reroute traffic, ensuring application availability across Regions.

Expanding Zonal Shift Capabilities: From Load Balancers to Compute Services

Most recently, Amazon Application Recovery Controller (ARC) has significantly expanded its zonal shift and zonal autoshift capabilities, moving beyond load balancers to encompass critical compute services. This expansion marks a crucial evolution in AWS’s approach to application resilience and recovery.

From Load Balancers to Compute Services

Initially, ARC’s zonal shift functionality was limited to Application Load Balancers (ALBs) and Network Load Balancers (NLBs). While this provided valuable traffic management during AZ impairments, it didn’t address the underlying compute resources. Now, ARC has extended its support to include:

EC2 Auto Scaling Groups (ASGs): As of November 18, 2024, EC2 Auto Scaling now supports ARC zonal shift and zonal autoshift. This integration allows for the rapid recovery of applications by shifting EC2 instance launches away from impaired AZs.
Amazon Elastic Kubernetes Service (EKS): Announced on November 8, 2024, ARC now supports zonal shift and zonal autoshift for EKS clusters. This capability helps in managing Kubernetes workloads during AZ impairments.

Importance of Compute Service Integration

The extension of zonal shift to compute services is significant for several reasons:

Comprehensive Recovery: By including EC2 Auto Scaling and EKS, ARC now offers a more holistic approach to application recovery. It addresses not just traffic routing but also the underlying compute resources, ensuring a more robust recovery process.
Automated Instance Management: For EC2 Auto Scaling, zonal shift can prevent new instance launches in impaired AZs and redirect them to healthy ones, reducing the impact of “gray failures” that might not be immediately detected by standard health checks.
Kubernetes-Specific Benefits: In EKS clusters, zonal shift cordons nodes in the impacted AZ and ensures new pods are scheduled only in healthy AZs, maintaining application availability in containerized environments.
Enhanced Multi-AZ Resilience: This expansion allows for more sophisticated multi-AZ resilience strategies, enabling organizations to maintain application performance and availability even during significant AZ impairments.
Reduced Manual Intervention: The integration with compute services, especially with the autoshift feature, reduces the need for manual intervention during AZ failures, leading to faster recovery times and reduced operational overhead.

By extending zonal shift capabilities to these core compute services, AWS has significantly enhanced the ability of organizations to maintain application resilience and quickly recover from AZ-level failures. This evolution represents a more integrated and automated approach to application recovery in cloud environments, aligning closely with the needs of modern, highly available applications.

Building Faster Event-Driven Architectures: Exploring Amazon EventBridge’s New Latency Gains

Matt Adorjan — Sun, 17 Nov 2024 14:53:43 +0000

When milliseconds matter, every improvement counts. Amazon EventBridge just announced a staggering 94% reduction in end-to-end latency for Event Buses, now as low as 129ms at the 99th percentile. For developers and organizations building latency-sensitive, mission-critical applications, this change opens the door to even more responsive and efficient event-driven architectures.

In this post, I'll explore why latency improvements like these are game-changers, demonstrate how you can test and monitor this performance, and highlight the benefits for real-world use cases.

Why Latency Matters

In event-driven systems, delays, however small, can create bottlenecks. Consider applications like:

Fraud detection: A delay in processing transaction data could allow fraudulent activity to slip through.
Industrial automation: Late responses to sensor events might disrupt production or reduce operational efficiency.
Gaming applications: High latency can ruin player experiences in real-time multiplayer environments.

By reducing average latency from 2235ms in January 2023 to just 129ms in August 2024 (P99), Amazon EventBridge empowers developers to react to critical events faster than ever. This improvement translates to better customer experiences, more agile business processes, and the potential for innovation in real-time systems.

Testing the Promise: Real-World Observations from CloudPing.co

To validate Amazon EventBridge’s latency improvements, I turned to a real-world application: CloudPing.co, which uses EventBridge and AWS Lambda functions to handle event-driven workflows. By analyzing the IngestionToInvocationStartLatency metric from one of our backend data-collection Lambda functions, I observed a clear and dramatic improvement in latency trends starting July 31, 2024.

Insights from the Data:

Before July 31, 2024: The latency was more variable and consistently higher, with peaks near the upper end of the range (~141ms).
After July 31, 2024: There’s a noticeable and sustained drop in average latency, aligning with Amazon EventBridge’s announcement of up to a 94% reduction. The trendline shows a much lower and stable average, making event handling significantly faster.

Impact Observed:

This reduction in latency directly translates into faster invocation of Lambda functions, ensuring more responsive event-driven workflows. Applications that rely on timely processing of events benefit from this improvement by delivering results more quickly and reliably to their users.

Visualization:

The graph below illustrates this trend clearly:

Key Benefits of Lower Latency

These performance gains unlock a host of benefits for businesses:

Real-time insights and decisions: Detect and respond to events faster, enabling agility in decision-making.
Seamless scaling: Handle latency-sensitive applications without re-engineering your infrastructure.
Cost efficiency: Latency improvements are applied by default across all AWS regions, at no additional cost.

How to Monitor and Test Latency

EventBridge provides built-in tools to help you measure and optimize event latency:

Metrics: Use the CloudWatch metrics IngestionToInvocationStartLatency and IngestionToInvocationSuccessLatency.
Console Dashboards: Monitor latency trends directly in the EventBridge console.
Real-world Scenarios: Test specific workflows like order processing, data pipelines, or gaming events to see the performance impact.

Take Advantage of EventBridge’s Performance Gains

Whether you’re building a fraud detection system, optimizing industrial automation, or creating immersive gaming experiences, Amazon EventBridge’s latency improvements can help you achieve new levels of performance.

Getting started is simple! Check out the documentation and spin up your first Event Bus in the AWS Console. By taking advantage of EventBridge’s improvements, you can ensure your applications are faster, more reliable, and ready for the demands of real-time processing.

Unlocking Fine-Grained Authorization with Amazon Verified Permissions: An Underrated AWS Service

Matt Adorjan — Mon, 11 Nov 2024 12:48:57 +0000

In today’s application landscape, a comprehensive authorization solution is crucial for maintaining security and compliance. Amazon Verified Permissions (AVP) is an (I feel) often-overlooked service that offers powerful, fine-grained authorization capabilities for custom applications. Let’s explore this AWS service and discover how it can enhance your application’s security posture.

As I have worked across different AWS implementations and application deployments, there is often a desire by developers to implement their own authorization services, write their own policy language, check permissions stored in various database technologies, and otherwise fragment the authorization process. AWS released Amazon Verified Permissions service a while ago, but I don't see it talked about very much, and for organizations which are heavily invested in AWS, I think it is a great drop in solution for permissions and authorization management.

What is Amazon Verified Permissions?

Amazon Verified Permissions is a fully managed, scalable authorization service designed for custom applications. It uses the Cedar policy language to define and enforce fine-grained permissions, allowing developers to externalize authorization logic and centralize policy management.

Key features of AVP include:

Fine-grained authorization using roles and attributes
Centralized policy management
Integration with identity providers like Amazon Cognito
Real-time authorization decisions
Automated policy analysis for compliance and auditing

How Amazon Verified Permissions Works

AVP leverages the Cedar policy language, an open-source language designed for writing and evaluating authorization policies. Here’s how it works:

Define your authorization model using Cedar policies
Store and manage these policies in AVP
When a user attempts an action, your application sends an authorization request to AVP
AVP evaluates the request against relevant policies and returns an ALLOW or DENY decision
Your application enforces the decision

Key Components of AVP

Policy Management and Validation

AVP provides tools for creating, storing, and managing Cedar policies. It also offers policy validation to ensure that your policies are correctly formatted and align with your defined schema.

Policy Querying and Auditing

The service includes features for analyzing and auditing policies, helping you identify potential security issues or overly privileged access.

Integrations and Extensibility

AVP can integrate with identity providers like Amazon Cognito and works alongside other AWS services to provide a comprehensive authorization solution.

Getting Started with AVP

Let’s walk through a basic example of using AVP with Python. First, set up your AWS credentials and install the boto3 library.

import boto3

# Create a Verified Permissions client
avp_client = boto3.client('verifiedpermissions')

# Define a simple policy
policy = {
    "Sid": "AllowViewDocument",
    "Effect": "Allow",
    "Principal": {"Identifier": "User::Alice"},
    "Action": "Document::View",
    "Resource": {"Identifier": "Document::ProjectReport"}
}

# Create a policy store
policy_store = avp_client.create_policy_store(
    name="MyAppPolicyStore"
)

# Create a policy in the policy store
created_policy = avp_client.create_policy(
    policyStoreId=policy_store['policyStoreId'],
    definition=policy
)

# Check authorization
auth_response = avp_client.is_authorized(
    policyStoreId=policy_store['policyStoreId'],
    principal={"EntityType": "User", "EntityId": "Alice"},
    action="Document::View",
    resource={"EntityType": "Document", "EntityId": "ProjectReport"}
)

print(f"Authorization decision: {auth_response['decision']}")

This example demonstrates creating a policy store, defining a simple policy, and checking authorization using AVP.

The policy defined in the policy variable allows a User called Alice to perform the action Document::View only when the specific resource is a Document with an ID of ProjectReport.
You can see in the is_authorized call, you pass in information about the principal, action, and resource, and then get an authorization decision.

Notice how similar this is to AWS' IAM policies! Within AVP, you can define many different policies, and then say you have an API, you can pass the information about the principal and what they are accessing from each API call into an is_authorized call to determine whether to allow the request to be returned as successful or not!

AVP vs. Traditional Authorization Methods

Compared to traditional in-app authorization, AVP offers several advantages:

Externalized authorization logic, simplifying application code
Fine-grained, context-aware access control
Centralized policy management
Real-time authorization decisions
Built-in policy analysis and auditing capabilities

Best Practices and Tips

When using Amazon Verified Permissions, consider the following best practices:

Design your authorization model carefully before implementation
Use policy templates to standardize and simplify policy creation
Regularly audit and analyze your policies
Leverage AVP’s integration capabilities with other AWS services
Use the test bench feature to validate policies before deployment

Performance Considerations

To optimize AVP usage:

Use bulk authorization when possible to reduce API calls
Implement response caching for frequently accessed resources
Design your policies to be as specific as possible to improve evaluation speed

Areas for Additional Consideration

As with any technology solution, there are always bound to be downsides and considerations to make.

Multi-cloud and hybrid-cloud limitations:
- As a single identity provider, AVP has limitations in interoperability across the identity stack and in a multi-cloud world.
- Additional solutions may be needed to integrate AVP with on-premises or multi-cloud environments
While Cedar is open-source completely, it is something released and maintained by AWS. So, if you are worried about that type of lock-in or dependency, it's important to consider this, as AVP is fully reliant on it. I think AWS has done a good job getting it out into the community, where you could decouple this from AVP if needed.

Conclusion

Amazon Verified Permissions is a powerful service that can enhance the security and manageability of your custom applications. By externalizing and centralizing authorization logic, AVP allows developers to focus on core application functionality while ensuring robust, fine-grained access control. As applications become more complex and security requirements become more stringent, services like AVP will have an increasingly crucial role in maintaining secure and compliant applications.

Using Kyverno Policies for Kubernetes Governance

Matt Adorjan — Fri, 31 Dec 2021 23:58:15 +0000

Kyverno is a great tool that can be installed into any Kubernetes cluster, allowing cluster administrators to enforce policies for resources in the cluster, and even modify resources before they are applied. Kyverno can be used to ensure deployments are secure, ensure deployments meet certain organizational criteria (e.g. define a cost center label), or even ensure all deployments mount a common volume.

Kyverno works by deploying a pod and services into your existing cluster. It creates multiple Admission Webhooks in the cluster. These webhooks are responsible for handling API requests coming in to Kubernetes and either validating something from the request (Validating Admission Webhook, or modifying the request before it is applied (Mutating Admission Webhook).

In the diagram below, you can see where in the process of an API call to Kubernetes, each of the Mutating Admission and Validating Admission webhooks will run.

All of the Kubernetes Manifests and Kyverno Policies are available in this GitHub repo.

Deploying Kyverno

Getting started with Kyverno is pretty simple. While there are a lot of knobs that can be turned to configure Kyverno, the initial default goes a long way. I deployed this into my DigitalOcean Kubernetes cluster.

To install, I simply ran:

$ helm repo add kyverno https://kyverno.github.io/kyverno/
$ helm repo update
$ helm install kyverno kyverno/kyverno --namespace kyverno --create-namespace

I now have a single Kyverno pod running in the cluster:

$ kubectl get pods -n kyverno
NAME                       READY   STATUS    RESTARTS   AGE
kyverno-6d94754db4-tdl9s   1/1     Running   0          5s

Policies

Anatomy of a basic policy

Policies can be written with many different options in Kyverno. The most basic policies can be written to check values of specific field(s) within an API request to Kubernetes, and make a decision whether the request should be allowed or not.

Require specific pod labels

This example requires specific labels to be set on a pod prior to creating those resources. One of the great benefits Kyverno provides is that you can specify Pod as the resource kind, but it will also check against the policy whenever creating resources which will end up creating pods (Deployment, StatefulSet, etc.)

The below example policy requires that all pods have the labels.acmecorp.com/costCenter and labels.acmecorp.com/department labels. They can be set to any value.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-business-labels
  annotations:
    policies.kyverno.io/title: Require Business Labels
    policies.kyverno.io/category: Best Practices
    policies.kyverno.io/severity: medium
    policies.kyverno.io/subject: Pod, Label
    policies.kyverno.io/description: >-
      Define required labels used by our internal business processes to understand which applications
      are running in each cluster, and used to handle chargeback activities for resources consumed
      by this specific application.
spec:
  validationFailureAction: enforce
  background: false
  rules:
  - name: check-for-business-labels
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "The labels `labels.acmecorp.com/costCenter` and `labels.acmecorp.com/department` are required."
      pattern:
        metadata:
          labels:
            labels.acmecorp.com/costCenter: "?*"
            labels.acmecorp.com/department: "?*"

Example Policy Tests:

Demo showing policy in cluster:

Require the use of a specific container registry

This policy checks for 2 important items: 1. images specified in a pod definition must be from a specific container registry (in this case, from the DigitalOcean registry); and 2. images cannot have the latest tag.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
  annotations:
    policies.kyverno.io/title: Restrict Image Registries and Latest
    policies.kyverno.io/category: Best Practices
    policies.kyverno.io/severity: medium
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/description: >-
      Requires all images for pods be sourced from the Digital Ocean Container Registry. Any other
      image sources are denied.
spec:
  validationFailureAction: enforce
  background: false
  rules:
  - name: validate-registries
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Unknown image registry."
      pattern:
        spec:
          containers:
          - image: "registry.digitalocean.com/*"
  - name: validate-image-tag
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Must not use the tag `latest` on any images."
      pattern:
        spec:
          containers:
          - image: "!*:latest"

Example Policy Tests:

Demo showing policy in cluster:

Require a runAsUser be specified

This policy requires that every container specified within a pod has a runAsUser defined and that the value is greater than 0, meaning the container cannot run as root. In reality, there are additional items you will probably want to check in a policy like this, but this provides a good place to get started.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-pod-runasuser
  annotations:
    policies.kyverno.io/title: Require the RunAsUser to be Specified
    policies.kyverno.io/category: Best Practices
    policies.kyverno.io/severity: medium
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/description: >-
      Requires Pods to specify as runAsUser value within their containers which are not root.
spec:
  validationFailureAction: enforce
  background: false
  rules:
  - name: check-userid
    match:
      resources:
        kinds:
        - Pod
    validate:
      message:  >-
        The field spec.containers.*.securityContext.runAsUser must specified and greater than zero.
      pattern:
        spec:
          containers:
          - securityContext:
              runAsUser: ">0"

Example Policy Tests:

Demo showing policy in cluster:

Conclusion

Overall, this is meant to be a couple basic examples of using policies with Kyverno in Kubernetes. You can certainly get much more complex, or implement a policy which mutates a resource, to keep resources in your cluster compliant with your rules.

If you end up trying Kyverno and find out you need something more complex, take a look at Gatekeeper. The Gatekeeper project works very similar to Kyverno, except it allows for defining policies in Rego language, which adds complexity but allow for additional customization.

Trying out AWS Controllers for Kubernetes (ACK)

Matt Adorjan — Wed, 25 Aug 2021 00:13:50 +0000

Introduction

AWS Controllers for Kubernetes (ACK) allows creating AWS resources via the same process you use to deploy other resources in Kubernetes. ACK uses the Kubernetes controller model to interact with AWS APIs. Once the ACK service controller for a specific service is deployed, you can leverage Kubernetes Custom Resource Definitions (CRDs) to declare specific AWS resources in YAML manifests. You then send them to the Kubernetes API server and the specified AWS resources are provisioned within your AWS account.

When I initially heard the news about AWS Controllers for Kubernetes, I was immediately struck by a few previous challenges where having AWS resources tied directly to other Kubernetes deployments would be immensely helpful. I think the key for using ACK is understanding when it makes sense to use them. Most organizations have mature processes for provisioning cloud infrastructure using tools like CloudFormation or Terraform. By introducing a new way of provisioning resources, you introduce an additional place where you now need to govern standards in terms of naming/tagging/security config, validate permissions for the controller follow least privilege, etc.

Use Cases

In my opinion, there are only a few reasons to use ACK to deploy resources and a longer list of situations where you probably want to avoid using ACK. Obviously, this is all very dependent on your organization and your use cases.

When to use ACK

Managing AWS resources which are tied to the lifecycle of Kubernetes resources - there are many use cases where you may want to maintain resources alongside your Kubernetes deployments, or specify them in your Helm Chart. For example, a use case that will be helpful on day 1 for me is the ability to create SQS queues alongside deployments in Kubernetes which are set to scale based on the new SQS queue's length.

When not to use ACK

AWS resources are going to be primarily used outside of Kubernetes - this is not a replacement for an infrastructure as code solution like CloudFormation or Terraform.
Data retention is required - this is a grey area, as you can definitely create data stores using ACK and there might be good cases for this. However, it is important to understand how simply deleting the AWS resource object in Kubernetes accidentally would cause both the deletion of the AWS resource and all of its associated data.
ACK controllers don't expose required parameters - if the controller doesn't expose specific parameters you need set on a resource, you should create the resource using another method to avoid double work. For example, if you need S3 Public Access to be turned off, it's not an available parameter, so ACK isn't a good option.
You want a way to expose resource deployments to developers and they already know Kubernetes - just because a developer is familiar with how Kubernetes resources work, if the resources are not directly tied to a Kubernetes deployment, it doesn't make sense to manage those resources with ACK. It makes more sense for developers to learn CloudFormation or Terraform or to expose resource creation to them using something like AWS Service Catalog.

Choosing a deployment model

ACK calls these installScope and the options are either cluster or namespace. There are some pros and cons for each option.

Cluster:
- Pros: Only need to run one copy of each controller in the cluster, therefore reducing effort required to manage these components.
- Cons: Permissions for the IAM credentials used by each controller will need to be broad to accommodate all of the possible ways a resource needs to be created anywhere in the cluster.
Namespace:
- Pros: Can get finer grained with controller configuration and IAM permissions that are used when the controller creates resources.
- Cons: Need to manage each controller in all namespaces, which can add operational overhead (e.g. need to run upgrades across all namespaces instead of in just 1 place).

I'm a big fan of the "shared service" model in multi-tenant clusters, and will likely use the cluster option here. I think as long as you have proper governance in place within the cluster, you can centrally manage all of your components once and make them available to all users in the cluster. See the last section of this post for more information on an approach for governance.

Getting started with ACK's S3 Controller

I initially started following the Install instructions from the ACK docs, specifically using the Helm chart. Looking at the S3 controller Helm Chart in ECR, the latest version is v0.0.2.

export HELM_EXPERIMENTAL_OCI=1
export SERVICE=s3
export RELEASE_VERSION=v0.0.2
export CHART_EXPORT_PATH=/tmp/chart
export CHART_REPO=public.ecr.aws/aws-controllers-k8s/$SERVICE-chart
export CHART_REF=$CHART_REPO:$RELEASE_VERSION
export ACK_K8S_NAMESPACE=ack-system

mkdir -p $CHART_EXPORT_PATH

helm chart pull $CHART_REF
helm chart export $CHART_REF --destination $CHART_EXPORT_PATH

kubectl create namespace $ACK_K8S_NAMESPACE

helm install --namespace $ACK_K8S_NAMESPACE ack-$SERVICE-controller \
    $CHART_EXPORT_PATH/ack-$SERVICE-controller

This worked overall, but as I started deploying a test S3 bucket, a lot of the Specs listed in the docs were generating errors and not working. For example, my bucket manifest is below. v0.0.2 did not have any support for tagging so this would create a bucket without tags and throw a not implemented error.

apiVersion: s3.services.k8s.aws/v1alpha1
kind: Bucket
metadata:
  name: test-s3-matt-bucket
spec:
  tagging:
    tagSet:
      - key: CostCenter
        value: Development
  name: test-s3-matt-bucket

After trying to understand what was going on, I realized that both the Helm Chart and the Docker image available in ECR were out of date. Looking at the s3-controller repository, changes were actively being made but there were no releases indicating what commit the v0.0.2 versions were tied to.

As a next step, I decided to build a newer Docker image and use the latest Helm chart from the repository to deploy the controller. This ended up being a bit more difficult than I anticipated. The ACK solution is designed to be very modular. The pro of this design is that it's very easy to add new services to ACK. The biggest con is that it is a bit confusing to figure out how everything fits together as someone new to the project.

After reading through the documentation a bit more and reviewing all of the different repositories in the aws-controllers-k8s org, I found the code-generator repository which contains the proper scripts to generate new Docker images for services based on the latest commits on a service repository.

To build a new S3 image:

Run the following to clone the code-generator repo and run the build script for s3:

git clone https://github.com/aws-controllers-k8s/code-generator.git
cd code-generator/scripts/
./build-controller-image.sh s3

Tag the newly generated image and push it to my own ECR repo.

docker tag <image-tag> public.ecr.aws/<repository>/s3-ack:<tag>
docker push public.ecr.aws/<repository>/s3-ack:<tag>

Run the following to clone the s3-controller repository:

git clone https://github.com/aws-controllers-k8s/s3-controller.git
cd s3-controller/helm/

Assuming you've already deployed the Helm chart earlier, you will need to upgrade the chart using the same name as before but instead pointing to the locally cloned copy. The example below upgrades a Helm chart called ack-s3-controller installed in namespace ack-system and sources the Helm chart from the current directory (.). It also sets the image.repository and image.tag values to the new location in ECR from step #2.

helm upgrade ack-s3-controller .  --namespace ack-system \
     --set image.repository=public.ecr.aws/<repository>/s3-ack,image.tag=<tag>

From there, you should now be able to create a new S3 bucket leveraging the latest features available in the documentation. In my experience, as soon as I deployed the new version and used the S3 Bucket YAML file listed above, a new S3 bucket was deployed with the expected tags.

Suggestions

The ACK project has a very in depth review of their releases and versioning process here. It is very thorough and follows a lot of standards you would expect for a solution of this nature. One of the items the documentation makes very clear is that each controller is on its own release and maintenance cycle. This also makes sense, given the amount of different service controllers that will be needed.

All of this being said, my suggestion is:

Follow a standardized release process for all of the controllers. Some controllers have releases defined in GitHub which match up with Docker images and Helm charts, some don't (as we saw with S3).
If the service controller documentation is being updated on the Docs website, make sure there is an actual released Docker image and Helm chart available which allows the usage of what the Docs are referencing.
Document the release versions in a centralized location on the Docs website. This page would be a great place to list the latest released version.

I don't think it's a huge ask to attach some simple release processes to each of the controller repositories which handle the above. I'm completely cognizant of the fact that these are all still in alpha and may not be ready for "stable" tags, but I think this simple change significantly lowers the barrier to entry and will allow others in the community to try out these controllers.

Governing resource creation by ACK with Gatekeeper

You may want to govern that AWS resources are created using specified guardrails when using ACK. One way you could do this is to try to customize the IAM policy attached to the controllers to restrict creation of resources unless they meet certain conditions. This works well but different resources offer different conditions meaning you may or may not be able to restrict creation based on the parameters you would like. Also, error messages returned from AWS for Access Denied are not passed through to the end user, leaving them blind to issues caused by IAM restrictions.

In the world of Kubernetes, Gatekeeper can fill this gap for us when using ACK. Because AWS resources to be deployed with ACK are translated into a standard JSON format when submitting to the Kubernetes API, we can write policies in rego which are then enforced using Gatekeeper.

I will leave a full Gatekeeper tutorial for other posts that already exist, but will post a sample Gatekeeper Template and Constraint below.

Example Gatekeeper Template and Constraint - S3 Standards

template.yaml - this defines the Constraint Template. You can see we are checking that the S3 bucket name starts with a specific string and that a specific tag is present.

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: bucketdefaultrequirements
  annotations:
    description: Requires S3 buckets created by ACK match standards.
spec:
  crd:
    spec:
      names:
        kind: bucketDefaultRequirements
      validation:
        # Schema for the `parameters` field
        openAPIV3Schema:
          type: object
          properties:
            bucketStartsWith:
              type: string
            requiredTagKey:
              type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package bucketdefaultrequirements

        violation[{"msg": msg}] {
          namingConventionStartsWith := input.parameters.bucketStartsWith

          value := input.review.object.spec.name

          # Check if the Bucket Name follows our naming convention
          not startswith(value, namingConventionStartsWith)

          # Construct an error message to return to the user.
          msg := sprintf("Bucket name does not follow proper format; found `%v`; needs to start with `%v`.", [value, namingConventionStartsWith])
        }

        violation[{"msg": msg}] {
          requiredTagKey := input.parameters.requiredTagKey
          value := input.review.object.spec.tagging.tagSet
          not contains(value, requiredTagKey)
          msg := sprintf("%v tag is missing.", [requiredTagKey])
        }

        contains(tagKeys, elem) {
          tagKeys[_]["key"] = elem
        }

constraint.yaml - this defines the actual constraint which uses the template created above. We can create a new constraint to be used across the entire cluster (like below) or we can create specific namespace scoped constraints for this rule.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: bucketDefaultRequirements
metadata:
  name: s3-buckets-must-meet-base-requirements
spec:
  match:
    kinds:
      - apiGroups: ["s3.services.k8s.aws"]
        kinds: ["Bucket"]
  parameters:
    bucketStartsWith: "matt-s3-"
    requiredTagKey: "CostCenter"

When the above constraint fails and I try to create an S3 bucket called test-s3-matt-bucket without a CostCenter tag, I get the following friendly message returned by the Kubernetes API:

error when creating "bucket.yaml": admission webhook "validation.gatekeeper.sh" denied the request: 
[s3-buckets-must-meet-base-requirements] CostCenter tag is missing.
[s3-buckets-must-meet-base-requirements] Bucket name does not follow proper format; found `test-s3-matt-bucket`; needs to start with `matt-s3-`.

In Conclusion

I hope that this has been helpful in some way! ACK is a great step forward for integrating AWS resources with the Kubernetes deployment lifecycle which many developers are already very familiar with. It's still early days, and like everything that comes out of AWS, I am positive customer obsession will continue driving the ACK project towards full API parity when deploying AWS resources in Kubernetes.