K3s on AWS in 2026: 4 IAM auth methods benchmarked

#aws #kubernetes #devops #security

If you self-host K3s on EC2 and your pods need AWS API access, there are at least 4 distinct ways to wire that up - and most blog posts only cover 1 or 2.

I built all four on the same 3-node cluster, ran 10 cold starts each, then deliberately broke things to record failure modes:

EC2 Instance Profile (default fallback)
IRSA via S3 public bucket (the classic 2021 pattern)
IRSA via CloudFront + custom domain (private bucket, OAC)
IAM Roles Anywhere with self-signed CA (X.509 cert auth)

## 3 things I didn't expect

Setup B (IRSA via S3 public bucket) was actually FASTER than baseline - 2.59s vs 3.18s median for plain Instance Profile. I assumed S3 GET for JWKS would add latency. Regional S3 STS validation beats it.
Caveat: image was :latest, so timings include some kubelet registry digest resolution overhead.

aws_signing_helper serve mode is broken with helper 1.8.2 + current aws-cli + K3s 1.35.4. Returns 400 on the IMDSv2 listing endpoint, botocore overflows on _evaluate_expiration. AWS docs still
describe it as supported. credential-process mode works fine.

Setup C (IRSA via CloudFront) showed bimodal cold start - 3.97s median, 13.68s p95. 2 of 10 runs hit ~13.5s. Hypothesis: CloudFront edge cache cold misses for the JWKS fetch from AWS STS. No direct STS
visibility, inferred from distribution shape.

## Plus 6 failure scenarios

I broke things deliberately and recorded what each setup does when:

K3s service-account-issuer changes (most damaging - instant fail for ALL existing IRSA pods, no cached creds save you)
Clock skew exceeds 5min
IAM Role deleted mid-flight
OIDC Provider deleted
Trust policy :sub mismatch
K8s Secret deleted (Setup D specific)

The error codes are different between scenarios, which actually helps with debugging - useful distinction nobody else seems to document.