DEV Community: George Palangattil

From On-Prem to EKS: Shift-Left Validation, Invisible Throttling, and Autonomous Incident Capture with the AWS DevOps Agent

George Palangattil — Sun, 27 Sep 2026 20:39:53 +0000

Moving a fleet of on-premises Kubernetes clusters to Amazon EKS is rarely blocked by the cloud transition itself — it's blocked by the version gap between where those clusters have been sitting and where EKS lives today. This post walks through that gap, how the AWS DevOps Agent closes it before deployment, and what changes operationally once your workloads are running on EKS.

The modernization challenge: the Kubernetes version gap

On-premises clusters tend to stagnate — often frozen at v1.23 or v1.25 to avoid destabilizing aging applications. Amazon EKS, by contrast, follows a forward-leaning lifecycle that typically targets v1.30 and newer. That delta isn't cosmetic; it's a real shift in the Kubernetes API surface and resource schema, and it's where most migrations stall.

The friction shows up hardest at API deprecation boundaries. Manifests that deploy cleanly on-premises fail outright on EKS when they reference removed resource kinds — extensions/v1beta1, apps/v1beta1, and policy/v1beta1 are the classic blockers. policy/v1beta1, for example, was removed in v1.25, so any manifest still using it will fail on a v1.30 target without prior refactoring.

There's a second dimension beyond API versions: storage and networking drift. Moving off local CSI drivers and custom ingress controllers onto the Amazon EBS CSI driver, the AWS Load Balancer Controller, and the VPC CNI is a structural re-architecture, not a lift-and-shift. At scale, manually auditing every YAML manifest against these two axes stops being viable — human review misses the nuanced OpenAPI schema changes between versions, and trial-and-error deployment against production EKS clusters is an expensive way to find them.

Pre-migration validation: the AWS DevOps Agent CLI compatibility engine

This calls for a shift-left validation model. The AWS DevOps Agent acts as a context-aware, pre-flight compatibility engine — it analyzes on-premises manifests against the OpenAPI specification of the target EKS version before a single resource is provisioned.

1. Export the source cluster state

# Capture every namespaced resource from the on-prem cluster
kubectl get all --all-namespaces -o yaml > onprem-cluster-state.yaml

2. Define the migration task

agent-config.json

{
  "migrationTask": {
    "sourceEnvironment": {
      "platform": "on-premises",
      "kubernetesVersion": "1.24"
    },
    "targetEnvironment": {
      "platform": "aws-eks",
      "kubernetesVersion": "1.30",
      "region": "us-east-1"
    },
    "inputManifests": "./manifests/",
    "outputDirectory": "./eks-migration-report/"
  }
}

3. Run the analysis

aws-devops-agent analyze --config agent-config.json

The agent returns a Migration Report with a compatibility score, a breaking-changes log, and remediation manifests that are ready to deploy — no separate refactoring pass required.

Remediation in practice

Take a legacy PodDisruptionBudget written against policy/v1beta1, which is non-functional starting in EKS 1.26:

Legacy Manifest

apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: payment-service-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: payment-service

Agent-remediated Manifest

# =====================================================================
# AWS DEVOPS AGENT REMEDIATION REPORT
# Resource: PodDisruptionBudget/payment-service-pdb
# Issue: API version policy/v1beta1 removed in Kubernetes 1.26+
# Action: Upgraded apiVersion to policy/v1
# =====================================================================
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payment-service-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: payment-service

This removes the manual burden of chasing schema changes resource-by-resource, and guarantees manifests are compliant before they ever reach the EKS control plane.

Operations: the invisible throttling problem

Migrations frequently surface architectural issues that older clusters simply never enforced. Chief among them: API Priority and Fairness (APF), the mechanism that protects the EKS control plane from overload — and a system that's historically hard for SRE teams to reason about

The APF transparency problem. When the API server is under pressure, APF returns HTTP 429 responses. Because client-go retries these transparently, the offending workload often shows no errors in its own logs — only rising latency. Your dashboards can stay green while the control plane is quietly degrading. WATCH calls are especially dangerous here: unlike short-lived GET/LIST calls, they hold a concurrency "seat" for the life of the connection, so a handful of misbehaving watchers can exhaust available seats fast.

In a fault-injection exercise, a misbehaving Python async controller was scaled to 50 replicas, generating 1,600–2,000 requests per second against the API server. The effects were immediate:

Correlating CloudWatch audit logs, EKS metrics, and CloudTrail, the AWS DevOps Agent isolated the root cause to the controller service account, entirely inside the workload-low APF priority level, with traffic shifting from simple LIST calls to a heavy mix of WATCH and CREATE/DELETE mutations — the signature of a client that stopped caching and started polling.

Automation: the Operator and volatile event capture

EKS state is volatile by design — when a pod is rescheduled or deleted, the logs and events needed for root-cause analysis often disappear with it. Where a static tool only gives you a snapshot after the fact, the EKS DevOps Agent Operator captures a "movie" of the failure by collecting evidence at the moment a state transition happens.

Unlike a Bedrock Agent, which needs manual tool wiring and pipeline configuration, the Operator runs as an autonomous detection-and-collection engine. It uses AWS Systems Manager Run Command to reach node-level data that's invisible to kubectl — kernel dmesg output, IPAMD networking internals — though this depth is limited to managed nodes; on Fargate, collection stays at the Kubernetes manifest and log level.

An OOMKilled failure, narrated

Detection — the Operator's informer catches the Running → OOMKilled transition within milliseconds.
Collection — SSM (on managed nodes) is triggered to pull manifests, container logs, and kernel logs immediately.
Storage — evidence lands in S3 and CloudWatch under a 14-day lifecycle policy to keep costs down.
Trigger — an HMAC-SHA256-signed webhook notifies the DevOps Agent to start the investigation.

In one such investigation of a web-python container, the Agent traced the failure to an unbounded processed_records list inside the _cache_worker thread. Modeling the leak at 20Mi/min against a 200Mi container limit let it mathematically explain why the container died at the 10-minute mark, every time — turning on-call from reactive log spelunking into an architectural finding delivered up front.

Production results and operational guidelines

The ROI shows up as measurable MTTR reduction in production environments:

Organization	Scenario	Result
Customer A	38,000 OneAgents across 500+ accounts, integrated with Dynatrace	Single pane of glass; eliminated troubleshooting "black boxes"
Customer B	Lambda configuration disruptions	MTTR: 2 hrs → 28 min (77% reduction)
Customer C	API integration regressions during peak events	MTTR: 75% reduction, down to 20–30 min

Architectural guidelines to carry into your own environment

Access management — configure EKS Access Entries with the Standard type and attach AmazonAIOpsAssistantPolicy so the Agent has the introspection rights it needs.
Secure identity — use EKS Pod Identity for role-to-role communication between the Operator and the Agent, keeping least privilege intact.
Cost governance — apply 14-day lifecycle policies to the CloudWatch log groups and S3 buckets used for diagnostic data; value drops off fast after the point of impact.
Pipeline gating — run the Agent CLI as a mandatory shift-left gate in CI/CD so incompatible manifests never reach production.

Conclusion

Closing the Kubernetes version gap is a one-time migration problem; keeping an EKS control plane healthy afterward is an ongoing one. Pairing pre-migration validation with Operator-driven incident capture turns both into a single, intelligent operational layer — one that catches breaking API changes before they ship and explains production incidents in the time it used to take to notice them.

[Boost]

George Palangattil — Sat, 05 Sep 2026 11:00:38 +0000

George Palangattil

Aug 4

Stop Storing AWS Access Keys on Your Servers: Modern Machine Identity with IAM Roles Anywhere

7 min read

Stop Storing AWS Access Keys on Your Servers: Modern Machine Identity with IAM Roles Anywhere

George Palangattil — Tue, 04 Aug 2026 07:30:20 +0000

Introduction

Cloud adoption has changed how organisations build and operate applications, but one challenge continues to exist across almost every enterprise environment: secure machine identity.
While AWS-native workloads can use IAM roles to obtain temporary credentials, many critical business systems still run outside AWS. These include on-premises applications, manufacturing systems, VMware workloads, third-party hosted applications, edge devices, and workloads running in other cloud providers.
These systems often need access to AWS services such as Amazon S3, AWS Secrets Manager, AWS KMS, Amazon DynamoDB, Amazon ECR, or Amazon CloudWatch. The question is simple:
How can a workload outside AWS securely authenticate to AWS without storing long-term access keys?
For years, the default answer was to create an IAM user, generate an access key and secret access key, place those credentials on a server, and periodically rotate them. While functional, this model introduces operational complexity, governance concerns, and security risks.
AWS IAM Roles Anywhere provides a different approach. It allows workloads running outside AWS to obtain temporary AWS credentials using X.509 certificates from an existing public key infrastructure (PKI), enabling organisations to extend IAM-based authentication beyond AWS boundaries. IAM Roles Anywhere enables workloads outside AWS to access AWS resources using temporary credentials and integrates with existing enterprise PKI and AWS Private CA environments.

The Real Problem Is Not Connectivity—It Is Identity

In most enterprises, connectivity is rarely the biggest challenge.

Applications can connect to AWS using:

AWS Direct Connect
VPN connections
Internet connectivity
Private networking solutions
Hybrid networking architectures

The real challenge begins when those applications need secure access to AWS services.

Consider a common enterprise scenario.

A legacy application runs in a private data centre because of regulatory requirements and technical dependencies. Every night, the application exports data and uploads files to Amazon S3, where downstream analytics platforms process the information.
The development team needs AWS access.

The quickest solution is often:

This works initially.

Six months later, another application requires access.
A year later, dozens of applications have AWS access keys stored on servers, configuration files, automation platforms, and scheduled jobs.

At that point, organisations begin asking difficult questions:

Who owns these credentials?
Are they rotated regularly?
Which application is using which key?
What happens when someone leaves the organisation?
How quickly can compromised credentials be revoked?
Are multiple systems sharing the same credential?

These questions often reveal that access key management has become an operational burden.

IAM Roles Anywhere addresses this challenge by eliminating the need for long-term AWS credentials and replacing them with temporary credentials derived from certificate-based identities. AWS positions IAM Roles Anywhere as a mechanism for reducing reliance on long-term credentials through temporary, automatically rotating credentials while leveraging existing PKI investments.

Understanding IAM Roles Anywhere

At its core, IAM Roles Anywhere extends the AWS IAM role model to workloads running outside AWS.

Instead of proving identity through AWS access keys, a workload presents an X.509 certificate issued by a trusted certificate authority.

AWS validates the certificate and, if all conditions are met, provides temporary AWS credentials through AWS Security Token Service (STS).

AWS documentation describes IAM Roles Anywhere as bridging AWS IAM trust relationships with public key infrastructure by connecting IAM roles, trusted certificate authorities, and certificate identities.

In simplified form:

The workload never needs to store long-term AWS access keys.
Instead, access is granted dynamically using temporary credentials associated with an IAM role.

Why This Matters for Enterprise Architecture

Many organisations are investing heavily in Zero Trust principles and modern identity management strategies.

Human users increasingly authenticate through:

Single Sign-On
Federated Identity
Multi-Factor Authentication
Centralised Identity Providers

Yet machine identities often remain dependent on long-lived secrets.

This creates an imbalance.
While human authentication has evolved significantly, machine authentication often relies on stored credentials that may remain unchanged for months or years.

IAM Roles Anywhere helps close that gap by enabling organisations to leverage certificate-based trust and temporary credentials for workloads outside AWS. AWS states that IAM Roles Anywhere allows organisations to extend the same IAM roles and policies used in AWS to on-premises, hybrid, and multi-cloud workloads

From a security perspective, this creates a more consistent identity model across the enterprise.

Real-World Business Use Cases

Use Case 1: Financial Services Data Processing

A bank operates a settlement platform inside its private data centre.

The application cannot be moved to AWS due to regulatory constraints and integration dependencies.

However, nightly settlement reports must be:

Uploaded to Amazon S3
Encrypted using AWS KMS
Processed by analytics services hosted on AWS

Historically, the application used access keys stored on the server.

With IAM Roles Anywhere:

The architectural benefit is straightforward:

No long-term AWS credentials
Improved auditability
Centralised permission management
Alignment with enterprise PKI standards

Use Case 2: Multi-Cloud Integration

Many enterprises operate across AWS, Azure, and Google Cloud.
Consider an application hosted on Azure Virtual Machines that needs secure access to Amazon S3.

Traditionally, AWS access keys are distributed into Azure environments.

Instead, the workload can use a certificate issued by corporate PKI and obtain temporary AWS credentials through IAM Roles Anywhere.

This creates a cleaner trust model while maintaining separation between cloud providers.

Use Case 3: Manufacturing and Edge Systems

Manufacturing organisations often operate thousands of distributed systems and edge nodes.

These systems may:

Collect telemetry
Store production statistics
Upload files to AWS
Send operational metrics

Managing thousands of AWS access keys becomes difficult at scale.
Certificate-based identities are often already used within manufacturing environments.

IAM Roles Anywhere allows these existing PKI investments to extend into AWS authentication workflows.

Core Architectural Components

IAM Roles Anywhere relies on three primary components.

Trust Anchor

A trust anchor represents the certificate authority trusted by IAM Roles Anywhere.

AWS supports both:

AWS Private Certificate Authority
External certificate authorities

Root and intermediate certificate authorities can be used as trust anchors.

This decision significantly impacts security and governance.

IAM Role

The IAM role defines permissions granted to the external workload.

The role trust policy must allow the IAM Roles Anywhere service principal to assume the role and establish a session. AWS documentation specifies that trust policies for IAM Roles Anywhere roles include permissions such as sts:AssumeRole, sts:SetSourceIdentity,and sts:TagSession.

The role should follow least-privilege principles.

Examples include:

S3 ingestion role
Backup role
Monitoring role
Secrets retrieval role

Avoid creating broad "catch-all" roles.

Profile

Profiles define which IAM roles are available to the requesting workload and can apply additional controls to credential sessions. AWS documentation identifies profiles as the mechanism for defining which roles can be assumed and how permissions are managed for the requesting workload.

Think of profiles as a governance layer between certificate identities and AWS permissions.

Critical Architecture Decisions

Decision 1: Which CA Should Be Your Trust Anchor?

This is one of the most important design decisions.

Many organisations are tempted to select their enterprise root CA.
Technically this works.

Architecturally it may not be ideal.

A higher-level CA can potentially allow a broader set of certificates to participate in authentication workflows. AWS guidance recommends carefully evaluating trust-anchor placement because trust anchor choice affects which end-entity certificates can obtain AWS credentials.

A better approach is often:

This provides:

Clear ownership
Reduced blast radius
Better governance
Easier auditing

Decision 2: Server Identity vs Application Identity

A common question is whether certificates should identify servers or applications.

Option A: Server Identity

Certificate = Server

Benefits:

Easier management
Fewer certificates

Challenges:

Multiple applications on the same server may share permissions

Option B: Application Identity

Certificate = Application

Benefits:

Better isolation
Stronger least-privilege model
Improved auditability

For highly regulated environments, application-centric identities often provide better governance.

Decision 3: Role Design

Roles should align with business functions.

Examples:

InvoiceUploadRole
PaymentProcessingRole
FactoryTelemetryRole
BackupServiceRole

Avoid patterns such as:

AllApplicationsRole

Effective role design reduces operational risk and simplifies audits.

Security Best Practices

Use Temporary Credentials Everywhere

One of the primary benefits of IAM Roles Anywhere is temporary credential usage.

The shorter the credential lifetime, the lower the exposure risk if credentials are compromised.

AWS highlights temporary credentials as a core security benefit of the service.

Protect Private Keys

Certificates are only as secure as their associated private keys.

Consider:

Restricted filesystem permissions
Secure storage mechanisms
TPM-backed keys
Hardware security modules for critical workloads
Endpoint protection controls

The focus should shift from protecting AWS access keys to protecting cryptographic identities.

Monitor Activity Through CloudTrail

IAM Roles Anywhere sessions can include source identity information derived from certificate attributes. AWS documentation explains that certificate subject information can be mapped to source identity and principal tags for policy evaluation and auditing purposes.

This improves:

Traceability
Audit investigations
Security monitoring
Incident response

Plan for Revocation

Certificate revocation should be part of the architecture from day one.

AWS provides support for certificate revocation list (CRL) management through IAM Roles Anywhere operations and APIs.

IAM Roles Anywhere vs Other AWS Identity Models

Different workloads require different identity mechanisms.

Workload Type	Recommended Identity Model
Amazon EC2	Instance Profile
AWS Lambda	Execution Role
Amazon ECS	Task Role
Amazon EKS	EKS Pod Identity / IRSA
On-Premises Server	IAM Roles Anywhere
Azure VM Accessing AWS	IAM Roles Anywhere
GCP Workload Accessing AWS	IAM Roles Anywhere
Edge Device with PKI	IAM Roles Anywhere

IAM Roles Anywhere should be viewed as the preferred solution when the workload runs outside AWS but still requires secure access to AWS services.

Lessons Learned from Enterprise Adoption

In many hybrid cloud programmes, implementing IAM Roles Anywhere is not the difficult part.

The challenges usually involve:

Defining certificate ownership
Aligning PKI governance with cloud governance
Establishing renewal processes
Building revocation procedures
Creating operational monitoring

Interestingly, organisations often spend more time discussing IAM permissions than certificate lifecycle management.

In practice, expired certificates frequently cause more operational disruption than IAM role configurations.

Successful implementations treat IAM Roles Anywhere as part of a broader machine identity strategy rather than an isolated AWS service.

Conclusion

IAM Roles Anywhere represents an important evolution in hybrid-cloud security architecture.

It allows organisations to extend the AWS IAM model beyond AWS boundaries and apply temporary credential-based authentication to workloads running in data centres, edge locations, third-party environments, and other cloud providers. It accomplishes this by leveraging certificate-based authentication, trusted certificate authorities, IAM roles, and temporary AWS credentials.

The most important takeaway is that IAM Roles Anywhere is not really a certificate service.

It is a machine identity solution.

The real value is not simply removing access keys from servers. The real value is extending AWS's proven IAM, STS, and governance capabilities into environments that historically relied on static credentials.

For organisations operating hybrid and multi-cloud environments, that is a significant architectural advantage.

Keep your workloads wherever they need to run—but modernise how they prove their identity to AWS.