DEV Community: John Ajera

Configuring AWS Business Support+

John Ajera — Mon, 13 Apr 2026 23:21:19 +0000

Configuring AWS Business Support+

AWS Business Support+ is AWS’s paid support tier that combines 24/7 expert access, AI-assisted troubleshooting, Trusted Advisor and health-oriented guidance, and targeted initial response commitments for business-critical issues (see official plan page for current feature wording and pricing). This guide focuses on how to turn it on and who can do it: console enrollment, organization-wide versus single-account subscription, IAM for the Support Plans console, and Organizations SCP pitfalls when regions are restricted.

1. Overview

Open the AWS Support Plans console and upgrade from Basic to Business Support+
Choose organization-wide (management account, Organizations all features) or account-only enrollment
Grant IAM access to change plans (supportplans:* or AWS managed policies) and fix SCP denies if Support Plans fails in member accounts
Know how to downgrade (console path) and where to compare pricing before you confirm

2. Prerequisites

Root user or an IAM principal allowed to change support plans (see Manage access to AWS Support Plans)
For organization-level Business Support+: the account must be the Organizations management account, with all features enabled for the organization
Billing in good standing; paid support has a minimum one-month commitment (Support FAQs)

3. Subscribe from the Support Plans console

Open the console

Optional but recommended before you subscribe:

Choose Compare all Support plans and features to line up Business Support+ against other tiers
Use Pricing calculator (in the console) to estimate support charges from expected monthly AWS usage

Upgrade from Basic to Business Support+

Steps follow Changing AWS Support plans:

In the AWS Business Support+ section, choose Get started
Choose scope:
- My organization — only if you use AWS Organizations with all-feature mode; only the management account can enroll the whole org. New accounts created in the org are automatically on Business Support+. In the console, My organization is disabled when it does not apply (for example, you are signed in to a member account, Organizations is not in use, or all features are not enabled for the organization)
- My account — subscribe this account only
Review plan details and pricing (AWS Support pricing)
Accept the subscription terms (checkbox)
Choose Confirm upgrade

After enrollment: confirm the tier in AWS Support Center (the console shows your current plan) and verify you can open a support case with the severity options you expect.

4. IAM: who may view or change the plan

Users need supportplans API permissions. Common actions (docs):

supportplans:GetSupportPlan — view current plan
supportplans:StartSupportPlanUpdate — start an upgrade or downgrade request
supportplans:GetSupportPlanUpdateStatus — poll update status

AWS managed policies (names may evolve; confirm in IAM):

AWSSupportPlansFullAccess — change plans
AWSSupportPlansReadOnlyAccess — view only

Example read-only custom policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "supportplans:Get*",
        "supportplans:List*"
      ],
      "Resource": "*"
    }
  ]
}

Example full access (delegate carefully):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "supportplans:*",
      "Resource": "*"
    }
  ]
}

5. Organizations: SCPs and global services

If member accounts see errors even with correct IAM, an SCP that denies by Region can block Support Plans because it is a global console/API.

Per Manage access to AWS Support Plans, add supportplans:* to the NotAction list on any broad Region-deny SCP (exact structure depends on your org’s policy layout). If you use AWS Control Tower Region deny controls, read AWS guidance on drift before hand-editing SCPs, because repairs can revert custom exceptions.

6. Downgrades and higher tiers

Downgrade from Business Support+: on Manage Support Plans, use Review downgrade in the Basic Support section (Changing AWS Support plans)
Enterprise Support or Unified Operations: use Contact sales in the console; Enterprise downgrades go through your TAM
Account leaves the org after org-level Business Support+: AWS documents that the account drops to Basic support

7. Optional follow-on configuration

These are not strictly “subscription” steps but are how teams use paid support day to day:

AWS Support App in Slack — create and update cases from Slack
AWS Support API — automate case operations (separate IAM/service permissions)
Trusted Advisor and AWS Health — visible in console once the plan includes them; use Trusted Advisor and AWS Health docs for check types and alerts

Promotional trial or pricing offers change over time; use the Business Support+ plan page and in-console offer links for current eligibility and refund rules.

8. Summary: Copy-Paste

Console entry points (sign in first):

https://console.aws.amazon.com/support/plans/home
https://console.aws.amazon.com/support/home

IAM: attach AWSSupportPlansFullAccess to a named role used only for billing/support changes, or merge the JSON examples in section 4 into your least-privilege model.

Organizations: if Region-deny SCPs exist, ensure supportplans:* is excluded from the deny per AWS documentation.

9. Troubleshooting

Access Denied on Support Plans: Attach AWSSupportPlansFullAccess (or equivalent supportplans:*) to the role or user; confirm you are not using a member account when only the management account may enroll the org.

Member account cannot open the page: Check SCP Region denies and add supportplans:* to NotAction as documented; verify Control Tower drift if that platform manages SCPs.

Wrong scope after the fact: Org-wide subscription is a management-account decision; single-account subscription does not extend to siblings. Adjust scope only by following AWS change plan / downgrade flows and org enrollment rules.

Charges surprise you: Revisit the console Pricing calculator and Support pricing; support fees are separate from service usage on the bill.

10. References

Greenfield EKS: Choosing Standard EKS vs EKS Auto Mode Without Legacy Baggage

John Ajera — Tue, 07 Apr 2026 23:35:59 +0000

Greenfield EKS: Choosing Standard EKS vs EKS Auto Mode Without Legacy Baggage

If you are standing up your first Amazon EKS cluster and you are not dragging along years of Terraform modules, custom AMIs, or a mandated node strategy from another team, the choice between Standard EKS (you own how nodes are provisioned and wired) and EKS Auto Mode (AWS runs more of that for you) is mostly about defaults: speed and delegated operations versus transparency and fine-grained control. This article is a practical decision guide for that greenfield scenario—what differs, how to think about cost and maintenance, and how to separate the EKS operating model from optional add-ons you install on top.

Source: What is Amazon EKS? (AWS Documentation).

1. Overview

This guide helps you decide, for a new cluster with no legacy constraints:

What Standard EKS and EKS Auto Mode each optimize for in the first weeks and months
How to compare cost at a high level (control plane, compute, Auto Mode management fees, and people time)
Who runs what (nodes, scaling, ingress/LB), including an AWS-documented checklist of what Auto Mode manages as built-in infrastructure
A short rubric for “default to Auto Mode” versus “start explicit with managed node groups”

2. Prerequisites

Familiarity with containers and the idea of a Kubernetes control plane versus worker nodes
Access to current Amazon EKS pricing and EKS Auto Mode documentation for numbers and feature details that change over time
No assumption that you already run Karpenter, managed node groups, or a corporate standard—this article assumes greenfield

3. Name the two starting paths

Same managed control plane; who owns the node story is what splits the paths.

Side-by-side

	Standard EKS	EKS Auto Mode
You choose	How nodes are created and scaled: managed node groups, self-managed nodes, or later Karpenter	Less DIY wiring; AWS runs more of lifecycle + integration (docs)
Mental model	“These are my instances / ASGs, my scaling and upgrade path”	“AWS-shaped automation; fewer knobs, less assembly”
APIs you will see (examples)	EC2, MNG, launch templates, plus whatever provisioner you pick	Platform-oriented CRDs such as `NodePool` / `NodeClaim`, `NodeClass`, `CNINode`, LB-related types—exact set varies by version

Standard EKS — responsibility flow

+-------------------------------+
| AWS: Kubernetes API           |  (managed control plane)
+-------------------------------+
              |
              v
+-------------------------------+
| You: pick node strategy       |
| MNG / self-managed / Karpenter|
+-------------------------------+
              |
              v
+-------------------------------+
| Workloads / Pods              |
+-------------------------------+

EKS Auto Mode — responsibility flow

+-------------------------------+
| AWS: Kubernetes API           |  (managed control plane)
+-------------------------------+
              |
              v
+-------------------------------+
| AWS: node lifecycle +         |
| integrations (opinionated)    |
+-------------------------------+
              |
              v
+-------------------------------+
| Workloads / Pods              |
+-------------------------------+

Same for both: observability, backups, secrets, RBAC, network policy, ingress, and mesh are still yours. Picking a mode ≠ production-ready.

4. Cost at a glance

Pricing moves; always verify against the Amazon EKS pricing page. The figures below use US West (Oregon) On-Demand rates at the time of writing and a small greenfield scenario: one cluster, three worker nodes (m5a.xlarge).

Estimated monthly AWS bill

Line item	Rate	Standard EKS	Auto Mode
Control plane	$0.10 / cluster / hr	~$73	~$73
EC2 instances (3 x `m5a.xlarge`)	$0.172 / instance / hr	~$377	~$377
Auto Mode management	$0.02064 / instance / hr	--	~$45
AWS invoice total		~$450	~$495

Source for *$0.02064 / hr** (Auto Mode) and $0.172 / hr (EC2) for m5a.xlarge: AWS Example 4: EKS Auto Mode on Amazon EKS pricing (US West Oregon, On-Demand). Auto Mode fees are per instance type; use that page or the AWS Pricing Calculator for EKS for other shapes.*

Auto Mode adds roughly 10-12% to the AWS bill in this scenario. The management fee is billed per-second (one-minute minimum) and applies regardless of EC2 purchase option (On-Demand, Reserved, Savings Plan, or Spot).

Watch out for extended support. If you let a Kubernetes version drift past standard support (14 months), the cluster fee jumps to $0.60 / hr (~$438 / month). Plan upgrades regardless of mode.

What the invoice does not show

Hidden cost	Standard EKS	Auto Mode
Engineer time building node automation, LB controller Helm charts, upgrade runbooks	Higher -- you build and maintain the glue	Lower -- AWS ships more of that glue
Incident cost when nodes misbehave, AMIs drift, or scaling stalls	Yours to debug end-to-end	Shared with AWS; fewer levers but also fewer moving parts you wrote
Idle capacity from over-provisioned groups or generous defaults	Risk if you set scaling bounds loosely	Risk if Auto Mode defaults are generous for your workload

Bottom line: the ~$45 / month management fee in this example is roughly one hour of a platform engineer's loaded cost. If Auto Mode saves more than that in reduced toil per month, it pays for itself. If your team already has solid automation and rarely touches nodes, Standard keeps the invoice leaner.

5. Who runs what—and what the first year feels like

The choice shows up in division of labor: who keeps nodes, scaling, and traffic into the cluster healthy. That is what fills your calendar in months six through twelve—upgrade planning and Helm charts on one side, AWS platform changes and support cases on the other—not an abstract “mode” badge.

Where the work lands

Area	Standard EKS (typical greenfield)	EKS Auto Mode
Nodes and scaling	You pick the mechanism—managed node groups, self-managed nodes, or Karpenter—and you own upgrades, behavior, and capacity tuning	AWS delivers a more integrated node and scaling experience; you align with Kubernetes objects and practices AWS documents for Auto Mode instead of assembling the same stack yourself (Auto Mode)
Ingress and load balancers	Teams usually install and operate something like AWS Load Balancer Controller—chart upgrades, compatibility with new cluster versions, incidents when labels or annotations drift	More of that integration is AWS-operated for Auto Mode, so you typically spend less time babysitting that slice—still read AWS release notes when the platform changes

Standard EKS in practice

Clarity and skill-building: Obvious ownership (“we chose MNG / Karpenter / …”), strong learning for engineers who will live in AWS and Kubernetes, and a concrete story when auditors ask what creates instances.
Recurring toil: Upgrade choreography across the control plane version, node AMIs, and add-on compatibility; deliberate choices for scaling bounds and instance families; ongoing ownership of ingress/LB tooling unless you outsource it elsewhere.

EKS Auto Mode in practice

Less assembly, faster baseline: Shorter path to a working data plane, fewer Terraform or CloudFormation resources tied to node plumbing, and less day-to-day ownership of the LB integration layer described above.
Tradeoff: Fewer levers when behavior surprises you; success depends on solid observability (logs, metrics, tracing) and comfort escalating or adapting when AWS-owned behavior changes.

What Auto Mode treats as managed infrastructure (AWS checklist)

AWS describes these as built-in cluster capabilities, not as separate EKS console add-ons you install and version yourself. This is the high-level list from Automate cluster infrastructure with EKS Auto Mode and the Automated components section there—use the docs for the latest detail and limits.

Compute and nodes

Data-plane EC2 instances (Bottlerocket-based variants): AMI selection, locked-down OS (SELinux enforcing, read-only root), no direct SSH or SSM to nodes
Security and lifecycle: patching and upgrades with minimal disruption; maximum node lifetime (default up to 21 days) so nodes are replaced regularly
Accelerated workloads: GPU-related kernel drivers and plugins (for example NVIDIA and AWS Neuron)
Spot: handling of Spot interruption notices and EC2 instance health signals

Scaling

Karpenter-style autoscaling: reacts to unschedulable Pods, adds capacity, and removes or consolidates nodes when demand drops

Load balancing

Elastic Load Balancing tied to Kubernetes Service and Ingress objects: provisions and manages ALB and NLB resources and scales them with cluster demand

Networking

Pod and Service networking, including IPv4/IPv6 and use of secondary CIDRs where needed for Pod IP space
Network policy enforcement for Pods
Cluster DNS (local DNS for the cluster)

Storage

EBS CSI-class block storage as a managed capability
Ephemeral storage defaults on nodes (volume type, size, encryption, and cleanup behavior on termination—per AWS documentation)

Identity

EKS Pod Identity: you do not install the EKS Pod Identity Agent yourself on Auto Mode clusters (AWS documents that it is not required in this model)

Neither mode delivers “production in a box.” Metrics, secret rotation, mesh, policy engines, backups, and business-specific operators remain your software lifecycle unless you adopt separate managed services.

Avoid the “busy cluster” trap. A Standard environment can look heavier because it has more add-ons (monitoring, GitOps, security tools). That says what you installed, not that Standard is inherently more capable than Auto Mode. Judge the choice on who operates nodes and built-in integrations, not on how crowded the UI feels.

6. Decision rubric (greenfield, no baggage)

Lean toward EKS Auto Mode when:

The team is small and the priority is shipping a standard container platform quickly
Workloads are typical Linux containers without exotic kernel, sysctl, or custom AMI requirements today
You are comfortable adopting AWS-shaped APIs for nodes and integrations for the next phase of growth
You prefer fewer moving parts you must patch and tune in the first year

Lean toward Standard EKS (often with managed node groups first) when:

You want maximum transparency into every layer for training, compliance groundwork, or multi-cloud discipline
You already know you need specific instance families, purchase options, or strict cost caps encoded explicitly from day one
You expect to standardize on an in-house or community node provisioning story (for example Karpenter you fully control) and want to learn that model without an additional management layer

Default suggestion for a true greenfield product team: if your main risk is slow delivery and operational overload, EKS Auto Mode is often the better starting default; if your main risk is understanding and owning every dependency for regulatory or career reasons, Standard EKS with managed node groups is a clean teaching path. You can revisit the choice after the team has real production traffic and metrics.

7. Troubleshooting: common misconceptions

“Auto Mode is more Kubernetes.” It is more AWS-managed automation exposed through Kubernetes APIs, not a superset of every optional addon.
“Standard is cheaper.” Invoice lines can be lower; total cost may not be once you count engineering time and incidents for a small team.
“We picked a mode, so we are secure and observable.” RBAC, network policy, secrets, auditing, and monitoring are still your design choices.
“Our Standard cluster has more moving parts, so it must be better.” Often that is more optional software you added—not proof that Standard beats Auto Mode; see section 5.

8. References

What is Amazon EKS?
Amazon EKS pricing (see Example 4: EKS Auto Mode for per-instance-type Auto Mode fees used in section 4)
EKS Auto Mode
Learn how EKS Auto Mode works (deeper component topics)
Amazon EKS best practices
Managed node groups
Karpenter

Argo CD and AWS CodeConnections: The Upside, the Redeploy Pain, and How I Fixed It

John Ajera — Sat, 28 Mar 2026 10:46:16 +0000

Argo CD and AWS CodeConnections: The Upside, the Redeploy Pain, and How I Fixed It

I run Argo CD on Amazon EKS using the managed Argo CD capability and AWS CodeConnections for Git. CodeConnections has been a clear win for day-to-day operations. Then I had to recreate the connection (new resource, new identity in the URL). Every Application went to Sync: Unknown until I updated URLs in two places—Git and the live cluster—and fixed ApplicationSets so they stopped writing the old URL back. This article leads with why I still choose CodeConnections, then what breaks on redeploy, then what I did when it inevitably happened, in that order.

1. Why CodeConnections is worth it for Argo CD on EKS

No SSH keys or personal tokens in the cluster. Argo pulls Git using IAM: the capability role is allowed to use the connection (UseConnection, GetConnection). You are not copying PATs into Secrets or rotating leaked keys because someone printed kubectl get secret.

One connection, many repos. The HTTPS URL includes your account, region, a connection UUID, and then owner/repo. Same connection, different path segment for each repository. Setup details and Terraform patterns are in Argo CD on EKS: Git repo access with AWS CodeConnections and Terraform.

Fits how enterprises already govern access. Connections are AWS resources; approval and auditing live next to the rest of your cloud controls.

That does not mean zero tradeoffs. Managed Argo CD often applies one Git credential broadly (for example every github.com fetch through Kustomize can inherit it). If that bites you, vendoring or URL strategy fixes it—see Why Your Kustomize Remote Bases Break on Managed Argo CD (and How to Fix It). The tradeoff this article focuses on is different: replacing the connection.

2. What actually hurts when you update or redeploy the connection

The Git URL Argo uses is not abstract. It embeds a connection UUID in the path. A new connection is a new UUID. Anything that still points at the old path keeps asking AWS to authorize the wrong resource, so repo fetch fails and sync never reconciles.

Pain point one — Git only is not enough. Your GitOps repo is the source of truth, but Kubernetes already has Application and ApplicationSet objects applied yesterday. Their spec.source.repoURL (and generator URLs on sets) stay on the old string until something updates them. Pushing Git does not retroactively patch those CRs.

Pain point two — ApplicationSets fight you. Many sets declare repoURL twice: on the git generator and again on the template. If you patch child Applications but leave the set on the old URL, the controller reconciles and puts the old URL back on the apps.

Pain point three — Terraform layout. If CodeConnections and the EKS cluster share one tangled module and state, recreating the connection can feel like you are planning half the platform when you only wanted a new Git pipe. I now prefer the connection (and its IAM attachment) in something standalone, with outputs the cluster stack consumes—so the next rotation is a smaller blast radius.

What this is not. If Argo shows forbidden listing some API group (for example heavy CRD surfaces from controllers like ACK), that is usually cluster RBAC / EKS access policies for the Argo identity, not the CodeConnections URL. Fix that on its own; do not confuse it with a UUID swap.

Do not mass-delete Applications to “fix” a bad URL. Workloads may still be fine; you risk prune tearing down real resources. Fix the URLs.

3. Prerequisites

kubectl against the cluster, with permission to read and patch applications and applicationsets in the Argo CD namespace (below I use argocd, the usual default)
Rights to commit and push every Git repo that hardcodes the CodeConnections URL
The new clone URL or at least the new UUID from the AWS Console or Terraform output

4. If it happens to you: what worked for me

These steps assume you already know the old and new connection UUID (search your shell history, Terraform state, or an old Application YAML). The host pattern is documented in the CodeConnections and EKS guides linked at the end.

Step A — Fix Git first, everywhere

Search across all repos that participate in GitOps—not only the “main” infra repo:

rg 'codeconnections\.' --glob '*.yaml' --glob '*.yml' --glob '*.tf' --glob '*.md'

Update bootstrap Application manifests, every ApplicationSet generator and template repoURL, any child Application checked in with a literal URL, and docs or scripts that build repository secrets. Commit and push.

Step B — Patch Applications in the cluster

Still in a broken state, the cluster cannot always sync from Git, so you patch live objects once. Set shell variables to your real values (namespace if not argocd, old UUID, new UUID):

NS=argocd
OLD_UUID=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
NEW_UUID=ffffffff-eeee-dddd-cccc-bbbbbbbbbbbb

List apps whose single spec.source.repoURL still contains the old UUID:

kubectl get applications -n "$NS" -o json | jq -r --arg u "$OLD_UUID" \
  '.items[] | select(.spec.source.repoURL // "" | contains($u)) | .metadata.name'

For each name, the safest approach is to take the existing URL from the object and swap the UUID in the shell, then merge-patch:

app=my-app
oldurl=$(kubectl get application "$app" -n "$NS" -o jsonpath='{.spec.source.repoURL}')
newurl="${oldurl//$OLD_UUID/$NEW_UUID}"
kubectl patch application "$app" -n "$NS" --type merge -p "{\"spec\":{\"source\":{\"repoURL\":\"$newurl\"}}}"

If you use spec.sources (multi-source), repeat the idea per entry that points at CodeConnections.

Step C — Patch ApplicationSets (do not skip this)

Confirm the old UUID still appears on spec (ignore status for a moment):

kubectl get applicationsets -n "$NS" -o yaml | rg "$OLD_UUID"

To replace the UUID everywhere under each set’s JSON (review before you run this in production—I exported one set first and eyeballed it):

for name in $(kubectl get applicationsets -n "$NS" -o json | jq -r --arg u "$OLD_UUID" \
  '.items[] | select(.. | strings? | contains($u)) | .metadata.name' | sort -u); do
  kubectl get applicationset "$name" -n "$NS" -o json | \
    jq --arg o "$OLD_UUID" --arg n "$NEW_UUID" \
    'del(.status) | walk(if type == "string" then gsub($o; $n) else . end)' | \
    kubectl replace -f -
done

The walk replaces every occurrence of the old UUID string in that JSON. Pick a UUID substring unique to the connection so you do not hit unrelated fields.

Step D — Refresh and converge

Hard refresh apps in the UI or CLI, then let GitOps catch up. If Git still cannot be fetched until one app is fixed, patch your bootstrap Application (the one that applies the rest of the tree) to the new URL first, then repeat the rest—classic chicken-and-egg.

5. Troubleshooting

Application stuck deleting

kubectl patch application my-app -n argocd -p '{"metadata":{"finalizers":null}}' --type=merge

I deleted an app and it came back

An ApplicationSet owns it (ownerReferences). Fix the set and Git; deleting the app alone will not stick.

6. References

Host a Static Site on EC2 with Terraform (VPC, Optional ALB)

John Ajera — Sat, 28 Mar 2026 08:54:20 +0000

Host a Static Site on EC2 with Terraform (VPC, Optional ALB, Session Manager)

For static sites, S3 + CloudFront is usually the better default. This post points at a small Terraform demo and pulls a few excerpts from main.tf, variables.tf, iam.tf, and user_data.tftpl. Full layout: tf-aws-ec2-static-demo (local path ~/workspace/jdevto/tf-aws-ec2-static-demo if you keep it beside this blog repo). S3 + CloudFront with Terraform: art0018.

Overview

The demo provisions a VPC, nginx on Amazon Linux 2023, index.html (AZ + private IP from IMDSv2), and robots.txt. use_alb=false (default): one instance in one public subnet; clients hit :80 on the instance public IP (CIDR from allowed_http_cidr). use_alb=true: internet ALB across az_count ≥ 2 public subnets; az_count instances in private subnets (one per AZ), NAT for egress, no instance public IP; instances register to one target group with HTTP / health check expecting 200; instance SG allows :80 from the ALB SG and from the VPC CIDR. enable_ssm=true (default): Session Manager with AmazonSSMManagedInstanceCore—no SSH in the template.

locals {
  subnet_count   = var.use_alb ? var.az_count : 1
  instance_count = var.use_alb ? var.az_count : 1
}

variable "az_count" {
  default = 3
  # ...
  validation {
    condition     = var.az_count >= 1 && var.az_count <= 6 && (!var.use_alb || var.az_count >= 2)
    error_message = "az_count must be between 1 and 6, and at least 2 when use_alb is true (ALB requirement)."
  }
}

Why EC2?

Learning stack (Terraform + VPC + optional ELB), policy that prefers VPC-hosted sites, temporary lift-and-shift, or a box that might grow non-static behavior later. None of that makes EC2 the default for a pure static site.

VPC

Every instance is in a VPC and a subnet (EC2-Classic is gone for new accounts). The demo creates its own VPC—public subnets always; private subnets + NAT when use_alb=true.

Architecture

use_alb = false
  +----------+   :80 (public IP)   +----------------+
  | Clients  | ------------------> | EC2 (nginx)    |
  +----------+                     +----------------+

use_alb = true
  +----------+   :80   +-----+   :80   +----------------+
  | Clients  | ------> | ALB | ------> | EC2 × az_count |
  +----------+         +-----+         | (private)      |
                                      +----------------+

NAT is billed when the ALB path is on. Repeated curl to the ALB can show different AZ / private IP in index.html as backends rotate. user_data fills those fields via IMDSv2 after nginx is up:

IMDS_TOKEN=$(curl -sS -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
PRIVATE_IP=$(curl -sS -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" \
  "http://169.254.169.254/latest/meta-data/local-ipv4")
AZ=$(curl -sS -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" \
  "http://169.254.169.254/latest/meta-data/placement/availability-zone")

main.tf — placement and bootstrap:

resource "aws_instance" "web" {
  count = local.instance_count

  subnet_id              = var.use_alb ? aws_subnet.private[count.index].id : aws_subnet.public[0].id
  vpc_security_group_ids = [aws_security_group.instance.id]
  user_data              = templatefile("${path.module}/user_data.tftpl", { enable_ssm = var.enable_ssm })
  user_data_replace_on_change = true
  iam_instance_profile   = var.enable_ssm ? aws_iam_instance_profile.ssm[0].name : null
  # ...
}

main.tf — instance ingress (ALB on) and target group health check:

dynamic "ingress" {
  for_each = var.use_alb ? [1] : []
  content {
    description     = "HTTP from ALB security group (forwarded client traffic)"
    from_port       = 80
    to_port         = 80
    protocol        = "tcp"
    security_groups = [aws_security_group.alb[0].id]
  }
}

dynamic "ingress" {
  for_each = var.use_alb ? [1] : []
  content {
    description = "HTTP from VPC for ALB health checks and internal probes"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = [aws_vpc.main.cidr_block]
  }
}

resource "aws_lb_target_group" "web" {
  count = var.use_alb ? 1 : 0

  port             = 80
  protocol         = "HTTP"
  protocol_version = "HTTP1"
  # ...

  health_check {
    enabled             = true
    protocol            = "HTTP"
    port                = "traffic-port"
    path                = "/"
    matcher             = "200"
    interval            = 15
    timeout             = 10
    healthy_threshold   = 2
    unhealthy_threshold = 3
  }
}

Session Manager

Agent + AmazonSSMManagedInstanceCore; user_data.tftpl via templatefile so #!/bin/bash is at column 0 (indented heredoc in Terraform often breaks the shebang). With enable_ssm, the template pulls the SSM Agent RPM from S3 and starts the service. Use the AMI’s curl—do not dnf install curl on AL2023 (conflicts with curl-minimal, set -e aborts before nginx). Egress: NAT or SSM VPC endpoints. Docs: agent install, status. Your principal needs ssm:StartSession; Session Manager plugin for CLI. After apply, wait for Online in Fleet Manager; with use_alb=true, use instance_ids or ssm_start_session_command (first instance). %{ if enable_ssm ~} … %{ endif ~} wraps the SSM block in the template.

iam.tf:

resource "aws_iam_role_policy_attachment" "ssm_core" {
  count      = var.enable_ssm ? 1 : 0
  role       = aws_iam_role.ssm[0].name
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}

user_data.tftpl (SSM path):

#!/bin/bash
# Column-0 shebang required: indented Terraform heredocs break #!/bin/bash and cloud-init may skip the script.
set -eux
# Do not `dnf install curl` here: AL2023 ships curl-minimal; installing full curl conflicts and aborts the whole script under set -e.
SSM_RPM=""
case "$(uname -m)" in
  x86_64) SSM_RPM="https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm" ;;
  aarch64) SSM_RPM="https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_arm64/amazon-ssm-agent.rpm" ;;
  *) echo "Unsupported arch for SSM agent RPM"; exit 1 ;;
esac
curl -sS -o /tmp/amazon-ssm-agent.rpm "$SSM_RPM"
dnf install -y /tmp/amazon-ssm-agent.rpm
rm -f /tmp/amazon-ssm-agent.rpm
systemctl enable amazon-ssm-agent
systemctl restart amazon-ssm-agent

robots.txt

Crawler hint file—not security. The demo writes:

User-agent: *
Allow: /

user_data.tftpl:

cat >/usr/share/nginx/html/robots.txt <<'ROBOTS'
User-agent: *
Allow: /
ROBOTS

Google: robots.txt.

Run it

git clone https://github.com/jdevto/tf-aws-ec2-static-demo.git
cd tf-aws-ec2-static-demo
terraform init && terraform apply

ALB (needs ≥2 AZs; default az_count is 3):

terraform apply -var="use_alb=true"
# terraform apply -var="use_alb=true" -var="az_count=2"

Wait 1–2 minutes after first boot for user_data. user_data_replace_on_change = true replaces instances when the template changes. Variables: repo README.

variable "use_alb" {
  type    = bool
  default = false
}

variable "allowed_http_cidr" {
  type    = string
  default = "0.0.0.0/0"
}

variable "enable_ssm" {
  type    = bool
  default = true
}

terraform output verify_commands
# use_alb=false:
curl -sS "http://$(terraform output -raw instance_public_ip)/robots.txt"
# use_alb=true (no instance public IP):
# curl -sS "$(terraform output -raw website_url_alb)/robots.txt"
terraform output -raw ssm_start_session_command

Troubleshooting

Issue	Check
Timeout / connection refused	`use_alb=true`: use ALB URL, not instance IP. `false`: SG, `allowed_http_cidr`, `user_data`, `instance_public_ip`.
ALB unhealthy / 502	SG :80 from ALB + VPC; `/` returns 200. `cloud-init-output.log`: failed `user_data` (e.g. `dnf install curl` vs `curl-minimal`).
Apply limits	Other region/account or limit increase.
SSM offline / denied	`enable_ssm`, agent time, `ssm:StartSession`, outbound to SSM (NAT or endpoints).

References

Using GoAccess to View nginx Logs

John Ajera — Sat, 28 Mar 2026 08:54:19 +0000

Using GoAccess to View nginx Logs

GoAccess is a fast terminal and HTML log analyzer for nginx access logs (requests/sec, status codes, top URLs, referrers, user agents). This guide covers installation, optional S3 download, and typical run modes.

1. Overview

Install GoAccess (package manager, source, or jdevto/cli-tools script)
Optional: sync logs from S3 when they are not on the machine you analyze from
Run TUI, HTML report, live tail, or compressed input; use COMBINED or COMMON to match nginx log_format

Skip §4 if logs are already on disk.

2. Prerequisites

Readable nginx access logs (usually combined or common format)
sudo for distro package installs
S3 (optional): AWS CLI v2 and s3:GetObject / s3:ListBucket for the prefix you use

3. Install GoAccess

Fedora / RHEL / CentOS Stream

sudo dnf install -y goaccess

Debian / Ubuntu

sudo apt update && sudo apt install -y goaccess

goaccess --version

Newer or custom build: official build instructions.

jdevto/cli-tools installer (optional)

Builds from the official tarball; details and env vars in install_goaccess.md.

bash <(curl -s https://raw.githubusercontent.com/jdevto/cli-tools/main/scripts/install_goaccess.sh) install

bash <(curl -s https://raw.githubusercontent.com/jdevto/cli-tools/main/scripts/install_goaccess.sh) uninstall

4. Optional: Download nginx Logs from S3

Single file

mkdir -p ~/nginx-logs
aws s3 cp s3://my-bucket/path/to/access.log ~/nginx-logs/access.log

Key is .gz only: download, decompress, or stream without extracting:

aws s3 cp s3://my-bucket/path/to/access.log.gz ~/nginx-logs/access.log.gz
gzip -d ~/nginx-logs/access.log.gz
# or: zcat ~/nginx-logs/access.log.gz | goaccess - --log-format=COMBINED

Prefix (many files)

mkdir -p ~/nginx-logs
aws s3 sync s3://my-bucket/nginx-logs/ ~/nginx-logs/

Replace bucket and prefix. If you only have *.gz, after sync use zcat ~/nginx-logs/*.gz > ~/nginx-logs/merged-access.log (order across files may not be chronological) or decompress then merge plain *.log.

Merge plain logs (optional)

cat ~/nginx-logs/*.log > ~/nginx-logs/merged-access.log

5. Run GoAccess

Match --log-format to nginx (COMBINED is the usual default).

Terminal UI

goaccess /var/log/nginx/access.log --log-format=COMBINED

HTML report

goaccess /var/log/nginx/access.log --log-format=COMBINED -o report.html --no-parsing-spinner
xdg-open report.html

Live log

tail -f /var/log/nginx/access.log | goaccess - --log-format=COMBINED

Common log format

goaccess /var/log/nginx/access.log --log-format=COMMON

Custom log_format: map fields with GoAccess custom format tokens.

Compressed on disk

zcat /var/log/nginx/access.log.*.gz | goaccess - --log-format=COMBINED

6. Summary: Copy-Paste

sudo dnf install -y goaccess
mkdir -p ~/nginx-logs && aws s3 sync s3://YOUR_BUCKET/YOUR_PREFIX/ ~/nginx-logs/   # optional
goaccess /var/log/nginx/access.log --log-format=COMBINED
goaccess /var/log/nginx/access.log --log-format=COMBINED -o report.html --no-parsing-spinner && xdg-open report.html
tail -f /var/log/nginx/access.log | goaccess - --log-format=COMBINED

7. Troubleshooting

Parsed 0 lines / wrong numbers: Log line shape ≠ COMBINED/COMMON. Compare head -1 to nginx log_format and adjust --log-format.

Permission denied on log: sudo goaccess ..., fix group on log files, or copy the log to your home directory.

S3 Access Denied: IAM on bucket/prefix; check AWS_PROFILE / role.

Live tail idle: Confirm nginx access_log path and read permissions on the active file.

8. References

Using GoAccess to View nginx Logs

John Ajera — Sun, 22 Mar 2026 23:22:42 +0000

Using GoAccess to View nginx Logs

1. Overview

Install GoAccess (package manager, source, or jdevto/cli-tools script)
Optional: sync logs from S3 when they are not on the machine you analyze from
Run TUI, HTML report, live tail, or compressed input; use COMBINED or COMMON to match nginx log_format

Skip §4 if logs are already on disk.

2. Prerequisites

Readable nginx access logs (usually combined or common format)
sudo for distro package installs
S3 (optional): AWS CLI v2 and s3:GetObject / s3:ListBucket for the prefix you use

3. Install GoAccess

Fedora / RHEL / CentOS Stream

sudo dnf install -y goaccess

Debian / Ubuntu

sudo apt update && sudo apt install -y goaccess

goaccess --version

Newer or custom build: official build instructions.

jdevto/cli-tools installer (optional)

Builds from the official tarball; details and env vars in install_goaccess.md.

bash <(curl -s https://raw.githubusercontent.com/jdevto/cli-tools/main/scripts/install_goaccess.sh) install

bash <(curl -s https://raw.githubusercontent.com/jdevto/cli-tools/main/scripts/install_goaccess.sh) uninstall

4. Optional: Download nginx Logs from S3

Single file

mkdir -p ~/nginx-logs
aws s3 cp s3://my-bucket/path/to/access.log ~/nginx-logs/access.log

Key is .gz only: download, decompress, or stream without extracting:

aws s3 cp s3://my-bucket/path/to/access.log.gz ~/nginx-logs/access.log.gz
gzip -d ~/nginx-logs/access.log.gz
# or: zcat ~/nginx-logs/access.log.gz | goaccess - --log-format=COMBINED

Prefix (many files)

mkdir -p ~/nginx-logs
aws s3 sync s3://my-bucket/nginx-logs/ ~/nginx-logs/

Merge plain logs (optional)

cat ~/nginx-logs/*.log > ~/nginx-logs/merged-access.log

5. Run GoAccess

Match --log-format to nginx (COMBINED is the usual default).

Terminal UI

goaccess /var/log/nginx/access.log --log-format=COMBINED

HTML report

goaccess /var/log/nginx/access.log --log-format=COMBINED -o report.html --no-parsing-spinner
xdg-open report.html

Live log

tail -f /var/log/nginx/access.log | goaccess - --log-format=COMBINED

Common log format

goaccess /var/log/nginx/access.log --log-format=COMMON

Custom log_format: map fields with GoAccess custom format tokens.

Compressed on disk

zcat /var/log/nginx/access.log.*.gz | goaccess - --log-format=COMBINED

6. Summary: Copy-Paste

sudo dnf install -y goaccess
mkdir -p ~/nginx-logs && aws s3 sync s3://YOUR_BUCKET/YOUR_PREFIX/ ~/nginx-logs/   # optional
goaccess /var/log/nginx/access.log --log-format=COMBINED
goaccess /var/log/nginx/access.log --log-format=COMBINED -o report.html --no-parsing-spinner && xdg-open report.html
tail -f /var/log/nginx/access.log | goaccess - --log-format=COMBINED

7. Troubleshooting

Parsed 0 lines / wrong numbers: Log line shape ≠ COMBINED/COMMON. Compare head -1 to nginx log_format and adjust --log-format.

Permission denied on log: sudo goaccess ..., fix group on log files, or copy the log to your home directory.

S3 Access Denied: IAM on bucket/prefix; check AWS_PROFILE / role.

Live tail idle: Confirm nginx access_log path and read permissions on the active file.

8. References

Why Your Kustomize Remote Bases Break on Managed Argo CD (and How to Fix It)

John Ajera — Mon, 16 Mar 2026 10:50:40 +0000

Why Your Kustomize Remote Bases Break on Managed Argo CD (and How to Fix It)

You switched to managed Argo CD (e.g. EKS Capabilities) and use AWS CodeConnections for Git. Your Applications that pull from public GitHub remotes in Kustomize suddenly fail with errors like "Password authentication is not supported" or "Authentication failed". The same kustomization worked with self-managed Argo CD. This article explains why CodeConnections causes that and how vendoring those remote bases into your repo fixes it.

1. Overview

What this guide does:

Explains why Kustomize remote resources (e.g. github.com/open-policy-agent/gatekeeper-library/...) break when Argo CD uses CodeConnections for Git
Shows that the credential helper is applied to every github.com URL, including public repos, and GitHub rejects those credentials
Describes when to vendor and a minimal how-to: download remote files at a pinned ref, add them to your repo, and point your kustomization at local files instead of remote URLs
Suggests keeping a README in the vendored directory so future upgrades are a repeatable re-download and commit

Why it breaks:

With managed Argo CD + CodeConnections, one credential is used for all GitHub access. When Kustomize runs git fetch for a remote base, Argo CD passes that same credential. Public repos don't need auth—but they receive it anyway, and GitHub rejects the format (e.g. "Password authentication is not supported").

2. Prerequisites

Managed Argo CD (e.g. EKS Argo CD capability) with AWS CodeConnections used for Git (see Argo CD on EKS: Git repo access with AWS CodeConnections and Terraform for setup)
At least one Argo CD Application whose source uses Kustomize with remote resources or bases pointing at GitHub (e.g. github.com/org/repo/path?ref=master)

3. Why remote bases break with CodeConnections

When Argo CD syncs an Application, the repo-server runs kustomize build over your repo. If your kustomization.yaml lists remote resources like:

resources:
  - github.com/open-policy-agent/gatekeeper-library/library/general/httpsonly?ref=master

Kustomize invokes git fetch for that URL. Argo CD configures a Git credential helper so that any git operation gets credentials. With CodeConnections, that helper returns the same credential (the one for your connected repos) for every github.com request—including requests to public repos such as open-policy-agent/gatekeeper-library.

Git sends those credentials to GitHub. GitHub then responds with something like:

fatal: Authentication failed for 'https://github.com/open-policy-agent/gatekeeper-library/'
remote: Invalid username or token. Password authentication is not supported for Git operations.

So the build fails even though the remote repo is public and would work with no credentials. With self-managed Argo CD you might have used SSH for your app repo; Kustomize’s HTTPS fetches to public GitHub then didn’t use that credential and succeeded. With CodeConnections, one HTTPS credential is used everywhere, and it gets sent to public URLs where it’s invalid.

4. When to vendor

Vendor remote bases when:

You use managed Argo CD with CodeConnections and your Kustomize app references public GitHub remotes (e.g. gatekeeper-library, shared bases from other orgs).
You see authentication or "Password authentication is not supported" errors during manifest generation for those remotes.

You can also vendor remotes you don’t control so that upgrades are explicit and reproducible (pin to a commit, re-download when you want to upgrade).

5. How to vendor

Vendoring means copying the remote manifest files into your repo and pointing Kustomize at local files instead of remote URLs.

Steps:

Choose a ref — Use a commit SHA or tag from the upstream repo (e.g. master or a specific SHA) so you can reproduce the same content later.
Download the remote files — Each remote resource is usually a directory containing one or more YAML files (e.g. template.yaml). Download those files using the raw GitHub URL pattern: https://raw.githubusercontent.com/<org>/<repo>/<ref>/<path>/<file>.yaml Save them under a directory in your repo (e.g. infrastructure/my-app/vendored/) with clear names (e.g. httpsonly.yaml, requiredlabels.yaml).
Update your kustomization — Replace remote resources entries with the local file names:

# Before (remote – fails with CodeConnections)
resources:
  - github.com/open-policy-agent/gatekeeper-library/library/general/httpsonly?ref=master

# After (vendored)
resources:
  - httpsonly.yaml

Document the source and upgrade process — Add a README in the vendored directory that lists:
- The upstream repo and the ref (commit SHA) you used
- The mapping from each local file to its upstream path
- How to upgrade: re-download from a new ref, overwrite the files, run kustomize build . to verify, commit, and update the README with the new ref.

Optionally, add a small script or one-liner (e.g. a shell loop with curl) that downloads all vendored files given a ref, so upgrades are a single command plus commit.

6. Summary

Problem: Managed Argo CD + CodeConnections sends one GitHub credential to every git URL. Kustomize remote bases to public GitHub repos get that credential; GitHub rejects it and the build fails.
Fix: Vendor those remote bases: download the manifest files at a pinned ref, put them in your repo, and point kustomization.yaml at the local files. No remote fetch at build time, so no credential is used for those resources.
Upgrades: Re-download from a new ref, overwrite the vendored files, verify with kustomize build ., commit, and update the README with the new ref.

Checklist when vendoring:

Pick a ref (e.g. latest master or a commit SHA) from the upstream repo.
Download each remote file via https://raw.githubusercontent.com/<org>/<repo>/<ref>/<path>/<file>.yaml into a directory in your repo.
Replace remote resources in kustomization.yaml with the local file names.
Add a README in that directory with source ref, file mapping, and upgrade steps (and optionally a download script).

7. Troubleshooting

Manifest generation still fails after vendoring

Confirm kustomization.yaml lists only local file names (e.g. httpsonly.yaml), not github.com/... URLs.
Run kustomize build . from the directory that contains the kustomization and vendored files; fix any path or resource errors before relying on Argo CD.

Upstream added or removed a file

Re-download from the ref you want (e.g. new commit on master). Add or remove the corresponding local file and update the resources list and README.

8. References

Argo CD – Kustomize (including private remote bases)
Argo CD on EKS: Git repo access with AWS CodeConnections and Terraform (CodeConnections setup)
gatekeeper-library (example upstream that’s often used as a Kustomize remote)

Argo CD on EKS: Git repo access with AWS CodeConnections and Terraform

John Ajera — Sat, 14 Mar 2026 07:31:29 +0000

Argo CD on EKS: Git repo access with AWS CodeConnections and Terraform

Argo CD on EKS Capabilities needs to pull from your Git repos. Instead of storing personal access tokens or SSH keys in the cluster, use AWS CodeConnections: one connection authorizes Argo CD to access GitHub via IAM. This guide gives the minimal Terraform (connection + IAM policy on the Argo CD role) and the one-time Console steps to move the connection from Pending to Available. One connection can serve many repos; you only change the owner/repo in the URL.

1. Overview

What this guide does:

Creates a CodeStar connection (GitHub) and grants the Argo CD capability role codeconnections:UseConnection and codeconnections:GetConnection so Argo CD can pull from Git without credentials in the cluster
Walks through the one-time Console step to complete the connection (Pending → Available), including using the GitHub App (e.g. AWS Connector for GitHub) for org repos
Shows the CodeConnections repo URL format for Argo CD Applications (connection ID is the UUID only, not the ARN)

Why CodeConnections?

No PAT or SSH key in Kubernetes; the Argo CD role gets IAM permission to use the connection
One connection = multiple GitHub repos (same connection ID, different owner/repo in the URL)

2. Prerequisites

EKS cluster with the Argo CD capability enabled (Identity Center configured)
The name of the Argo CD capability IAM role (e.g. from your EKS module: cluster_name-argocd-capability-role, or from AWS Console → IAM → Roles)
AWS Console access to complete the connection (GitHub OAuth)

3. Terraform: connection and IAM

Add the following to your Terraform. Replace argocd_capability_role_name with the actual role name (the role EKS created for the Argo CD capability).

variable "argocd_capability_role_name" {
  description = "Name of the IAM role used by the Argo CD EKS Capability"
  type        = string
}

resource "aws_codestarconnections_connection" "github" {
  name          = "github"
  provider_type = "GitHub"
}

data "aws_iam_policy_document" "argocd_codeconnections" {
  statement {
    effect = "Allow"
    actions = [
      "codeconnections:UseConnection",
      "codeconnections:GetConnection"
    ]
    resources = [aws_codestarconnections_connection.github.arn]
  }
}

resource "aws_iam_role_policy" "argocd_codeconnections" {
  name   = "argocd-codeconnections"
  role   = var.argocd_capability_role_name
  policy = data.aws_iam_policy_document.argocd_codeconnections.json
}

After terraform apply, the connection exists in Pending state. Terraform cannot complete it; you do that once in the Console.

4. After apply: complete the connection (Pending → Available)

Open the AWS Console in the same region as your Terraform.
Go to Developer Tools → Connections (or search Connections).
Find the connection (e.g. github). Status will be Pending.
Click the connection name → Update pending connection (or Connect to GitHub). You’ll land on the Connect to GitHub page.

GitHub connection settings:

Connection name — Confirm or set the name (e.g. github to match your Terraform name). This is how the connection appears in the Console.
App Installation (optional) — Two choices:
- Leave blank to connect as a GitHub user. That user must have (at least read) access to the repos you use in Argo CD. Fine for personal or single-account repos.
- Use a GitHub App (recommended for org repos): e.g. choose AWS Connector for GitHub via “Install a new app”, then select the org where your repos live (e.g. my-org) as the installation target.
Click the orange Connect button to start the GitHub authorization flow (or complete it after installing the app).

If you chose the GitHub App, you’ll land on GitHub’s Install & Authorize AWS Connector for GitHub page for that org. Choose Only select repositories, click Select repositories, and pick the repo(s) Argo CD will use (e.g. my-org/my-repo). Review the permissions (read/write access to code, pull requests, etc.), then click Install & Authorize. You’ll be redirected to https://redirect.codestar.aws/return. Back on the AWS Connect to GitHub page, the App Installation field will show your installation (e.g. an installation ID such as 123456789—yours will differ). Confirm Connection name (e.g. github), then click the orange Connect button. The connection status should become Available.
If you connected as a GitHub user, complete the OAuth prompt, then return to the Console and confirm status is Available.

5. Test the connection

Once the connection is Available:

Register the repo in Argo CD (one-time). Create a repository Secret so the repo appears under Settings → Repositories. Use your real region, account ID, connection UUID, and org/repo—do not leave literal OWNER/REPO in the URL or you’ll get “repository not found”. Example (replace the values if yours differ):

   export REGION=ap-southeast-2
   export ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
   export CONN_ID=a1b2c3d4-e5f6-7890-abcd-ef1234567890   # your connection UUID from Console or Terraform
   export OWNER_REPO=my-org/my-repo                     # your org and repo
   kubectl create secret generic argocd-repo-github -n argocd --from-literal=url="https://codeconnections.${REGION}.amazonaws.com/git-http/${ACCOUNT}/${REGION}/${CONN_ID}/${OWNER_REPO}.git"
   kubectl label secret argocd-repo-github -n argocd argocd.argoproj.io/secret-type=repository

If the secret already exists with the wrong URL, delete it first: kubectl delete secret argocd-repo-github -n argocd, then run the create and label commands above.

In Argo CD → Settings → Repositories, click REFRESH LIST. The repo should show with a green check (Successful). If it shows Failed, see the Troubleshooting section.
Create an Application that uses this repo: set source.repoURL to the same CodeConnections URL and a path/targetRevision. Sync the application and confirm it syncs without errors.

6. Using the repo URL in Argo CD

CodeConnections URL format (replace placeholders):

https://codeconnections.<region>.amazonaws.com/git-http/<account-id>/<region>/<connection-id>/OWNER/REPO.git

connection-id: Must be the connection ID (UUID only), e.g. 9b6c3274-31da-4d1d-8955-e7d5cd9d15e2. Do not put the full ARN in the URL path—that causes HTTP 400. In Terraform the AWS provider’s aws_codestarconnections_connection.github.id is the ARN; use the UUID (the segment after the last / in the ARN) or an output from your module that exposes the UUID.
OWNER/REPO: Your Git org and repo (e.g. my-org/my-repo).

In your Argo CD Application, set source.repoURL to that URL. One connection works for many repos—same connection ID, change only owner/repo.

7. Summary: copy-paste

Terraform: Connection + IAM policy (role name from your EKS setup):

resource "aws_codestarconnections_connection" "github" {
  name          = "github"
  provider_type = "GitHub"
}

resource "aws_iam_role_policy" "argocd_codeconnections" {
  name   = "argocd-codeconnections"
  role   = var.argocd_capability_role_name  # your Argo CD capability role name
  policy = jsonencode({
    Version   = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["codeconnections:UseConnection", "codeconnections:GetConnection"]
      Resource = aws_codestarconnections_connection.github.arn
    }]
  })
}

Console: Developer Tools → Connections → select connection → Update pending connection → on the Connect to GitHub page set/confirm Connection name, then either leave App Installation blank (user connection) or install AWS Connector for GitHub and select the org that owns your repos (e.g. my-org) → click Connect → complete GitHub auth → status Available.

Argo CD: Use the CodeConnections URL with the connection ID and your OWNER/REPO as source.repoURL.

8. Troubleshooting

Issue	What to check
Connection stays Pending	Complete the connection in the AWS Console (authorize in GitHub or install the app). Check region.
HTTP 400 when Argo CD tests repo	The repo URL path must use the connection ID (UUID only), not the full ARN. If your URL contains `arn:aws:codestar-connections:` in the path, fix the URL to use only the UUID (e.g. from Terraform use a value that outputs the UUID, not `connection.id` which is the ARN).
repository not found / ProviderResourceNotFoundException	The URL still has literal `OWNER/REPO` instead of your real org and repo (e.g. `my-org/my-repo`). Delete the repo Secret and recreate it with the correct URL (see §5).
authorization failed: Write access to repository not granted	The GitHub identity (user or app) used by the connection doesn’t have access to that repo. Install the GitHub App (e.g. AWS Connector for GitHub) on the org that owns the repo (e.g. my-org), or ensure the GitHub user that authorized has read access to the repo. Then re-authorize the connection if needed.
Argo CD cannot fetch repo	Argo CD role has `codeconnections:UseConnection` and `codeconnections:GetConnection` on the connection ARN; connection status Available; `source.repoURL` uses the UUID as connection-id and correct owner/repo.
Wrong region	Connection is regional. Console and Terraform region must match.

9. References

Configure repository access (Argo CD, CodeConnections) – AWS EKS User Guide
EKS Capabilities – Argo CD, ACK, KRO

Bitwarden Secrets Manager on EKS – Per-App Integration with Atlantis

John Ajera — Mon, 02 Mar 2026 10:01:15 +0000

Bitwarden Secrets Manager on EKS – Per-App Integration with Atlantis

Sync secrets from Bitwarden Secrets Manager into Kubernetes on EKS using the sm-operator, AWS Secrets Manager for the machine token, and the Secrets Store CSI Driver. This guide expands on the base integration with a per-app, per-namespace pattern and uses Atlantis as a concrete example. It covers Terraform, Kustomize overlays, Argo CD, sync waves, and troubleshooting.

Note: Use placeholder values for org IDs and secret IDs. Never commit real tokens. For production, follow least-privilege IAM and rotation practices.

1. Overview

What this guide does:

Integrates Bitwarden Secrets Manager with EKS via sm-operator, AWS Secrets Manager, and Secrets Store CSI Driver
Uses a per-app namespace pattern: each app (e.g. Atlantis) gets its own SecretProviderClass, bw-auth-token-sync, and BitwardenSecret in its own namespace
Walks through Terraform (EKS + Pod Identity per app), manifests, Argo CD Applications, validation, and force-sync
Uses Atlantis (Terraform PR automation) as a worked example with GitHub App credentials

Why per-app namespaces?

Isolation: each app has its own Bitwarden machine token (scoped in AWS)
Simplicity: the sm-operator creates the K8s Secret in the app’s namespace; no cross-namespace wiring
Consistency: same pattern for every new app

2. Prerequisites

EKS cluster with Secrets Store CSI Driver installed
Argo CD installed
Terraform-managed EKS (e.g. terraform-aws-eks-basic with Secrets Manager support)
Bitwarden Secrets Manager organization with Machine Account
Per-app AWS secret path: bitwarden/sm-operator/<app>/machine-token

3. Architecture Overview

Bitwarden SM (Machine Account + App Secrets)
        │
        │ machine token (per app: bitwarden/sm-operator/<app>/machine-token)
        ▼
AWS Secrets Manager
        │
        │ Pod Identity + CSI (in app namespace, e.g. atlantis-1)
        ▼
SecretProviderClass + bw-auth-token-sync → creates bw-auth-token
        │
        ▼
BitwardenSecret (authToken: bw-auth-token)
        │
        │ sm-operator fetches via Bitwarden API
        ▼
Output K8s Secret (e.g. atlantis-1-vcs) in app namespace

Flow (per app, e.g. atlantis-1):

Machine token stored in AWS Secrets Manager as bitwarden/sm-operator/<app>/machine-token (JSON: {"token":"<value>"}).
SecretProviderClass + bw-auth-token-sync Deployment in the app namespace: CSI Driver mounts the token and creates K8s Secret bw-auth-token.
BitwardenSecret in the same namespace: authToken.secretName: bw-auth-token. The sm-operator reads this token and calls the Bitwarden API.
sm-operator creates the output K8s Secret (e.g. atlantis-1-vcs) in the app’s namespace. Atlantis (or any app) uses it directly.

4. Bitwarden Setup

Machine Account and token: Bitwarden Admin → Machine Accounts → Create Access Token. Copy the token.
Per-app machine token (optional): For isolation, create a separate token per app and store in bitwarden/sm-operator/<app>/machine-token in AWS.
App secrets: In Bitwarden Secrets Manager, create the secrets your apps need. For Atlantis (GitHub App): webhook secret, private key (key.pem).
Copy IDs: From Bitwarden Admin → Settings → Organization, copy organizationId. For each secret, copy its bwSecretId (UUID).

5. Terraform

Requirement: EKS module must enable Secrets Manager with Pod Identity. Add an association per app namespace (e.g. { namespace = "atlantis-1", service_account = "awssm-sync" }) and prefix bitwarden/sm-operator.

Create the AWS Secrets Manager secret per app. Pass the token via -var or TF_VAR_. Never commit it.

variable "bitwarden_sm_machine_token_atlantis" {
  description = "Bitwarden SM machine token for atlantis-1"
  type        = string
  default     = "REPLACE_WITH_REAL_TOKEN"
  sensitive   = true
}

resource "aws_secretsmanager_secret" "bitwarden_atlantis" {
  name        = "bitwarden/sm-operator/atlantis-1/machine-token"
  description = "Bitwarden SM machine token for atlantis-1"
  tags        = var.tags
}

resource "aws_secretsmanager_secret_version" "bitwarden_atlantis" {
  secret_id     = aws_secretsmanager_secret.bitwarden_atlantis.id
  secret_string = jsonencode({ token = var.bitwarden_sm_machine_token_atlantis })
}

6. Per-App Manifests (Atlantis Example)

Each app gets its own overlay with:

SecretProviderClass
ServiceAccount + bw-auth-token-sync Deployment
BitwardenSecret
The app itself (e.g. Atlantis Helm chart)

6.1 SecretProviderClass

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: bitwarden-sm-token
  namespace: atlantis-1
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
spec:
  provider: aws
  parameters:
    region: ap-southeast-2
    usePodIdentity: "true"
    objects: |
      - objectName: "bitwarden/sm-operator/atlantis-1/machine-token"
        objectType: "secretsmanager"
        jmesPath:
          - path: token
            objectAlias: token
  secretObjects:
    - secretName: bw-auth-token
      type: Opaque
      data:
        - objectName: token
          key: token

6.2 ServiceAccount + bw-auth-token-sync

apiVersion: v1
kind: ServiceAccount
metadata:
  name: awssm-sync
  namespace: atlantis-1
  annotations:
    argocd.argoproj.io/sync-wave: "-2"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bw-auth-token-sync
  namespace: atlantis-1
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
  labels:
    app: bw-auth-token-sync
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bw-auth-token-sync
  template:
    metadata:
      labels:
        app: bw-auth-token-sync
    spec:
      serviceAccountName: awssm-sync
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.9
          resources:
            requests:
              cpu: 1m
              memory: 4Mi
          volumeMounts:
            - name: secrets-store
              mountPath: "/mnt/secrets-store"
              readOnly: true
      volumes:
        - name: secrets-store
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: bitwarden-sm-token

6.3 BitwardenSecret

apiVersion: k8s.bitwarden.com/v1
kind: BitwardenSecret
metadata:
  name: atlantis-1-vcs
  namespace: atlantis-1
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  organizationId: "REPLACE_ORG_ID"
  secretName: atlantis-1-vcs
  onlyMappedSecrets: true
  map:
    - secretKeyName: github_secret
      bwSecretId: "REPLACE_WEBHOOK_SECRET_ID"
    - secretKeyName: key.pem
      bwSecretId: "REPLACE_PRIVATE_KEY_ID"
  authToken:
    secretName: bw-auth-token
    secretKey: token

6.4 Atlantis Helm values

# application-patch.yaml
vcsSecretName: atlantis-1-vcs
githubApp:
  id: "YOUR_GH_APP_ID"
  installationId: "YOUR_GH_INSTALLATION_ID"
ingress:
  enabled: false   # Enable when you have ingress + atlantisUrl

Use argocd.argoproj.io/sync-wave: "5" on the Atlantis Application so it is created after the BitwardenSecret and bw-auth-token-sync.

7. Sync Waves (Ordering)

To avoid the app starting before secrets exist:

-2: ServiceAccount awssm-sync
-1: SecretProviderClass, bw-auth-token-sync Deployment
0: BitwardenSecret, ConfigMaps
5: Atlantis Application (Helm)

8. Validation

# Verify secrets in app namespace
kubectl get secret atlantis-1-vcs bw-auth-token -n atlantis-1

# BitwardenSecret status
kubectl get bitwardensecret atlantis-1-vcs -n atlantis-1
kubectl get bitwardensecret atlantis-1-vcs -n atlantis-1 \
  -o jsonpath='{.status.conditions[?(@.type=="SuccessfulSync")].status}'

# Output secret keys (should show github_secret, key.pem)
kubectl get secret atlantis-1-vcs -n atlantis-1 -o jsonpath='{.data}' | jq -r 'keys[]'

# Atlantis pod
kubectl get pods -n atlantis-1
kubectl logs -n atlantis-1 -l app=atlantis --tail=20

9. Local Access (Port-Forward)

The Atlantis Service exposes port 80 (targetPort 4141):

kubectl port-forward svc/atlantis-1 -n atlantis-1 4141:80

Then open http://localhost:4141.

10. Force Sync (Token Refresh)

When the machine token in AWS Secrets Manager changes:

# Per-app namespace
kubectl delete secret bw-auth-token -n atlantis-1
kubectl delete pod -n atlantis-1 -l app=bw-auth-token-sync --force --grace-period=0
kubectl wait --for=condition=ready pod -l app=bw-auth-token-sync -n atlantis-1 --timeout=60s
kubectl rollout restart deployment/sm-operator-controller-manager -n sm-operator-system
kubectl rollout restart statefulset atlantis-1 -n atlantis-1

11. Adding Another App

For each new app:

Create app directory: apps/<app>/overlays/<cluster>/
Add SecretProviderClass (AWS path: bitwarden/sm-operator/<app>/machine-token)
Add bw-auth-token-sync (SA + Deployment)
Add BitwardenSecret with the app’s secret mapping
Add Terraform: { namespace = "<app>", service_account = "awssm-sync" } + AWS secret
Deploy the app (Helm, etc.) with the created secret

12. Summary: Copy-Paste

Validation:

kubectl get secret atlantis-1-vcs bw-auth-token -n atlantis-1
kubectl get bitwardensecret atlantis-1-vcs -n atlantis-1 -o jsonpath='{.status.conditions[?(@.type=="SuccessfulSync")].status}'
kubectl get secret atlantis-1-vcs -n atlantis-1 -o jsonpath='{.data}' | jq -r 'keys[]'

Port-forward:

kubectl port-forward svc/atlantis-1 -n atlantis-1 4141:80

Force sync (token changed):

kubectl delete secret bw-auth-token -n atlantis-1
kubectl delete pod -n atlantis-1 -l app=bw-auth-token-sync --force --grace-period=0
kubectl wait --for=condition=ready pod -l app=bw-auth-token-sync -n atlantis-1 --timeout=60s
kubectl rollout restart deployment/sm-operator-controller-manager -n sm-operator-system
kubectl rollout restart statefulset atlantis-1 -n atlantis-1

13. Troubleshooting

Issue: CreateContainerConfigError – Secret atlantis-1-vcs missing

Solution: Wait for sm-operator to sync BitwardenSecret; or restart pod after secret exists.

Issue: Pod Identity / token association error

Solution: Add { namespace = "atlantis-1", service_account = "awssm-sync" } to secrets_manager_associations in Terraform.

Issue: ResourceNotFoundException – AWS secret missing

Solution: Create bitwarden/sm-operator/atlantis-1/machine-token in Secrets Manager (JSON: {"token":"<value>"}).

Issue: Error: --gh-app-id/--gh-app-key-file... – VCS secret not reaching container

Solution: Ensure vcsSecretName and githubApp.id/installationId in Helm values; verify atlantis-1-vcs exists.

Issue: Argo CD app stuck "Progressing"

Solution: Ingress without controller. Set ingress.enabled: false until ingress is configured.

14. References

Bitwarden Secrets Manager on EKS – End-to-End Integration

John Ajera — Sun, 01 Mar 2026 18:41:10 +0000

Bitwarden Secrets Manager on EKS – End-to-End Integration

Note: Use placeholder values for org IDs and secret IDs. Never commit real tokens. For production, follow least-privilege IAM and rotation practices.

1. Overview

What this guide does:

Integrates Bitwarden Secrets Manager with EKS via sm-operator, AWS Secrets Manager, and Secrets Store CSI Driver
Stores machine token in AWS Secrets Manager; CSI Driver mounts it; sm-operator syncs Bitwarden secrets into K8s Secrets
Walks through Terraform (EKS + secret), flat manifests, Argo CD Application, validation, and force-sync

2. Prerequisites

EKS cluster with Secrets Store CSI Driver installed
Argo CD installed
Terraform-managed EKS (e.g. terraform-aws-eks-basic or equivalent)
Bitwarden Secrets Manager organization with Machine Account

3. Architecture Overview

Bitwarden SM (Machine Account + App Secrets)
        │
        │ machine token
        ▼
AWS Secrets Manager (bitwarden/sm-operator/machine-token)
        │
        │ Pod Identity + CSI
        ▼
Secrets Store CSI Driver → creates bw-auth-token
        │
        ▼
sm-operator (reads BitwardenSecret CR, uses bw-auth-token)
        │
        │ Bitwarden API fetches secrets
        ▼
github-app-secrets (K8s Secret)

Flow:

Machine token stored in AWS Secrets Manager as {"token":"<value>"}
CSI Driver mounts it into a pod; creates K8s Secret bw-auth-token
sm-operator uses bw-auth-token to auth to Bitwarden API and sync secrets into github-app-secrets (or your chosen output secret)

4. Bitwarden Setup

Machine Account and token: Bitwarden Admin → Machine Accounts → Create Access Token. Copy the token.
App secrets: In Bitwarden Secrets Manager, create the secrets your apps need (e.g. github-app-id, github-app-key, github-app-webhook-secret, github-app-installation-id).
Copy IDs: From Bitwarden Admin → Settings → Organization, copy organizationId. For each secret, copy its bwSecretId (UUID).

5. Terraform

Requirement: EKS module must enable Secrets Manager with Pod Identity for sm-operator-system / awssm-sync and secret prefix bitwarden/sm-operator.

Create the AWS Secrets Manager secret and pass the token via -var or TF_VAR_bitwarden_sm_machine_token. Never commit it.

variable "bitwarden_sm_machine_token" {
  description = "Bitwarden SM machine token from Machine Account → Create Access Token"
  type        = string
  default     = "REPLACE_WITH_REAL_TOKEN"
  sensitive   = true
}

resource "aws_secretsmanager_secret" "bitwarden_sm_token" {
  name        = "bitwarden/sm-operator/machine-token"
  description = "Bitwarden Secrets Manager machine token for sm-operator"
  tags        = var.tags
}

resource "aws_secretsmanager_secret_version" "bitwarden_sm_token" {
  secret_id     = aws_secretsmanager_secret.bitwarden_sm_token.id
  secret_string = jsonencode({ token = var.bitwarden_sm_machine_token })
}

6. Deploy sm-operator

Save as sm-operator.yaml and apply. Replace placeholders in BitwardenSecret; adjust region if needed.

# Namespace for sm-operator and CSI resources
apiVersion: v1
kind: Namespace
metadata:
  name: sm-operator-system
  labels:
    name: sm-operator-system
    app.kubernetes.io/name: sm-operator
---
# ServiceAccount with Pod Identity for AWS Secrets Manager access
apiVersion: v1
kind: ServiceAccount
metadata:
  name: awssm-sync
  namespace: sm-operator-system
  annotations:
    argocd.argoproj.io/sync-wave: "-2"
---
# Fetches machine token from AWS SM and creates K8s Secret bw-auth-token
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: bitwarden-sm-token
  namespace: sm-operator-system
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
spec:
  provider: aws
  parameters:
    region: ap-southeast-2
    usePodIdentity: "true"
    objects: |
      - objectName: "bitwarden/sm-operator/machine-token"
        objectType: "secretsmanager"
        jmesPath:
          - path: token
            objectAlias: token
  secretObjects:
    - secretName: bw-auth-token
      type: Opaque
      data:
        - objectName: token
          key: token
---
# Pod that triggers CSI mount; creates bw-auth-token used by sm-operator
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bw-auth-token-sync
  namespace: sm-operator-system
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
  labels:
    app: bw-auth-token-sync
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bw-auth-token-sync
  template:
    metadata:
      labels:
        app: bw-auth-token-sync
    spec:
      serviceAccountName: awssm-sync
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.9
          resources:
            requests:
              cpu: 1m
              memory: 4Mi
          volumeMounts:
            - name: secrets-store
              mountPath: "/mnt/secrets-store"
              readOnly: true
      volumes:
        - name: secrets-store
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: bitwarden-sm-token
---
# Maps Bitwarden secrets to github-app-secrets K8s Secret
apiVersion: k8s.bitwarden.com/v1
kind: BitwardenSecret
metadata:
  name: bitwarden-secret
  namespace: sm-operator-system
spec:
  organizationId: "REPLACE_ORG_ID"
  secretName: github-app-secrets
  onlyMappedSecrets: true
  map:
    - secretKeyName: github-app-id
      bwSecretId: "REPLACE_SECRET_ID_1"
    - secretKeyName: github-app-key
      bwSecretId: "REPLACE_SECRET_ID_2"
    - secretKeyName: github-app-webhook-secret
      bwSecretId: "REPLACE_SECRET_ID_3"
    - secretKeyName: github-app-installation-id
      bwSecretId: "REPLACE_SECRET_ID_4"
  authToken:
    secretName: bw-auth-token
    secretKey: token
---
# Argo CD Application; deploys sm-operator Helm chart
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sm-operator
  namespace: argocd
spec:
  project: platform
  source:
    repoURL: https://charts.bitwarden.com/
    chart: sm-operator
    targetRevision: 2.0.0
    helm:
      values: |
        settings:
          bwSecretsManagerRefreshInterval: 300
          cloudRegion: US
          replicas: 1
  destination:
    server: https://kubernetes.default.svc
    namespace: sm-operator-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
  syncOptions:
    - CreateNamespace=true

7. Validation

After Argo CD syncs the app:

# BitwardenSecret status
kubectl get bitwardensecret -n sm-operator-system
kubectl get bitwardensecret bitwarden-secret -n sm-operator-system \
  -o jsonpath='{.status.conditions[?(@.type=="SuccessfulSync")].status}'

# Output secret keys
kubectl get secret github-app-secrets -n sm-operator-system \
  -o jsonpath='{.data}' | jq -r 'keys[]'

Expected: SuccessfulSync status True and the mapped keys listed.

8. Force Sync (Token Refresh)

When the machine token in AWS Secrets Manager changes:

kubectl delete secret bw-auth-token -n sm-operator-system
kubectl delete pod -n sm-operator-system -l app=bw-auth-token-sync --force --grace-period=0
kubectl wait --for=condition=ready pod -l app=bw-auth-token-sync -n sm-operator-system --timeout=60s
kubectl rollout restart deployment/sm-operator-controller-manager -n sm-operator-system

When only BitwardenSecret mapping changes (e.g. new secrets):

kubectl delete bitwardensecret bitwarden-secret -n sm-operator-system
# Argo CD recreates it; wait for sync
kubectl annotate bitwardensecret bitwarden-secret -n sm-operator-system \
  force-reconcile="$(date +%s)" --overwrite

9. Summary: Copy-Paste

Validation (after sync):

kubectl get bitwardensecret -n sm-operator-system
kubectl get bitwardensecret bitwarden-secret -n sm-operator-system \
  -o jsonpath='{.status.conditions[?(@.type=="SuccessfulSync")].status}'
kubectl get secret github-app-secrets -n sm-operator-system -o jsonpath='{.data}' | jq -r 'keys[]'

Force sync (machine token changed):

kubectl delete secret bw-auth-token -n sm-operator-system
kubectl delete pod -n sm-operator-system -l app=bw-auth-token-sync --force --grace-period=0
kubectl wait --for=condition=ready pod -l app=bw-auth-token-sync -n sm-operator-system --timeout=60s
kubectl rollout restart deployment/sm-operator-controller-manager -n sm-operator-system

Force sync (BitwardenSecret mapping changed):

kubectl delete bitwardensecret bitwarden-secret -n sm-operator-system
kubectl annotate bitwardensecret bitwarden-secret -n sm-operator-system force-reconcile="$(date +%s)" --overwrite

10. Troubleshooting

Issue: Has an invalid identifier

Solution: One or more bwSecretId UUIDs are wrong. Verify each ID in Bitwarden Secrets Manager.

Issue: CSI secret not created

Solution: Check Pod Identity for SA awssm-sync in sm-operator-system; ensure SecretProviderClass and bw-auth-token-sync pod exist. Check CSI driver logs: kubectl logs -n kube-system -l app=csi-secrets-store.

Issue: No changes / Skipping sync

Solution: Delete the BitwardenSecret; Argo recreates it and the operator performs a fresh reconcile.

11. References

FastSchema on EKS – Install and Test

John Ajera — Sun, 01 Mar 2026 04:09:39 +0000

FastSchema on EKS – Install and Test

Deploy FastSchema via Argo CD on EKS and access it locally with port-forward. Uses SQLite with EBS-backed PVC. First-run setup token from pod logs.

FastSchema has no official Helm chart. This guide uses a community chart at k8sforge/fastschema-chart.

Note: This setup is for evaluation, not production-ready. For production, use external database (MySQL/PostgreSQL), configure auth providers, and follow security best practices.

1. Overview

What this guide does:

Deploys FastSchema using the community Helm chart via an Argo CD Application
Persists data with SQLite on EBS-backed PVC
Uses port-forward for local access (no ingress required)
Walks through first-run setup: token from pod logs, create admin account, verify storage

Prerequisites:

kubectl configured for your EKS cluster
Argo CD installed

2. Prerequisites

Before starting, ensure you have:

kubectl configured for your EKS cluster
Argo CD installed

3. Install FastSchema

Save the manifest below as fastschema-application.yaml and apply:

kubectl apply -f fastschema-application.yaml

Wait until the Application shows Synced in the Argo CD UI or CLI.

Manifest:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: devops
  namespace: argocd
spec:
  clusterResourceWhitelist:
    - group: "*"
      kind: "*"
  description: DevOps-managed applications
  destinations:
    - namespace: "*"
      server: "*"
  namespaceResourceWhitelist:
    - group: "*"
      kind: "*"
  sourceRepos:
    - "*"
---
apiVersion: v1
kind: Namespace
metadata:
  name: fastschema
  labels:
    name: fastschema
    owner: devops
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: fastschema
  namespace: argocd
spec:
  project: devops
  source:
    repoURL: https://k8sforge.github.io/fastschema-chart
    chart: fastschema
    targetRevision: 0.1.4
    helm:
      values: |
        service:
          type: ClusterIP
        persistence:
          enabled: true
          size: 10Gi
          storageClassName: ebs-sc
          accessModes:
            - ReadWriteOnce
        appBaseUrl: "http://localhost:8000"
        appDashUrl: "http://localhost:8000/dash"
        resources:
          limits:
            cpu: 500m
            memory: 512Mi
          requests:
            cpu: 100m
            memory: 256Mi
  destination:
    server: https://kubernetes.default.svc
    namespace: fastschema
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Omit AppProject and Namespace if you already have them (e.g. via devops-apps ApplicationSet).

4. Test Access

Port-forward

kubectl port-forward svc/fastschema -n fastschema 8000:8000

Get setup token (first run)

kubectl logs -n fastschema -l app.kubernetes.io/name=fastschema --tail=30

Look for: Visit the following URL to setup the app: http://localhost:8000/dash/setup/?token=...

Setup and login

Open the URL from the logs. Complete setup to create admin account. Log in at http://localhost:8000/dash with the credentials you created.

5. Verify EBS Storage

kubectl exec -it -n fastschema deploy/fastschema -- sh

Inside the pod:

df -h /fastschema/data
ls -la /fastschema/data
echo "storage-test-$(date)" > /fastschema/data/storage-test.txt
cat /fastschema/data/storage-test.txt

Exit, then from your terminal:

kubectl delete pod -n fastschema -l app.kubernetes.io/name=fastschema

Wait for the new pod to be ready, exec in again and run:

cat /fastschema/data/storage-test.txt

If the file and content persist, the PVC is working.

6. Summary: Copy-Paste

# 1. Apply manifest
kubectl apply -f fastschema-application.yaml

# 2. Wait for sync (argocd app get fastschema)

# 3. Port-forward and get token
kubectl port-forward svc/fastschema -n fastschema 8000:8000 &
kubectl logs -n fastschema -l app.kubernetes.io/name=fastschema --tail=30

Open the setup URL from logs, create admin account, then log in at http://localhost:8000/dash.

7. Troubleshooting

Issue: No setup token in logs

Solution: Wait for pod to be ready. Token prints once on startup. If setup already done, use login at /dash with your credentials.

Issue: PVC stuck Pending

Solution: Ensure storageClassName: ebs-sc exists. Check EBS CSI driver is installed.

Issue: Service not found

Solution: kubectl get svc -n fastschema. Ensure app is synced.

8. References

FastSchema: https://fastschema.com
Configuration: https://fastschema.com/docs/configuration.html
Chart: https://github.com/k8sforge/fastschema-chart

Headlamp on EKS – Install and Test

John Ajera — Sat, 28 Feb 2026 22:21:50 +0000

Headlamp on EKS – Install and Test

Need a lightweight Kubernetes UI without the overhead of kubectl or cloud consoles? Headlamp runs in-cluster, gives you real-time visibility into workloads, logs, and events—and it's free and open source. This guide deploys it via Argo CD with Service Account token auth: no OIDC setup, no manual secrets, and access via simple port-forward. Secure, easy to run, and GitOps-friendly.

Note: This setup uses cluster-admin for simplicity—ideal for learning or individual use. For production, configure proper role-based access (RBAC) and restrict the Service Account to least-privilege roles.

1. Overview

What this guide does:

Deploys Headlamp using the official Helm chart via an Argo CD Application
Creates a Service Account headlamp with cluster-admin for token-based login
Persists data with EBS-backed PVC
Uses port-forward for local access (no ingress required)

Prerequisites:

EKS cluster running with kubectl context set
Argo CD installed

2. Prerequisites

Before starting, ensure you have:

kubectl configured with context set to your EKS cluster
Argo CD installed and syncing Applications

3. Install Headlamp

Save the manifest below as headlamp-application.yaml and apply:

kubectl apply -f headlamp-application.yaml

Argo CD will create the Application and sync Headlamp. Wait until the Application shows Synced in the Argo CD UI or CLI.

Manifest:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: platform
  namespace: argocd
spec:
  clusterResourceWhitelist:
    - group: "*"
      kind: "*"
  destinations:
    - namespace: "*"
      server: "*"
  namespaceResourceWhitelist:
    - group: "*"
      kind: "*"
  sourceRepos:
    - "*"
---
apiVersion: v1
kind: Namespace
metadata:
  name: headlamp
  labels:
    name: headlamp
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: headlamp
  namespace: argocd
spec:
  project: platform
  source:
    repoURL: https://kubernetes-sigs.github.io/headlamp/
    chart: headlamp
    targetRevision: 0.40.0
    helm:
      values: |
        service:
          type: ClusterIP
        config:
          inCluster: true
          oidc:
            secret:
              create: false
        serviceAccount:
          create: true
          name: headlamp
        clusterRoleBinding:
          clusterRoleName: cluster-admin
        persistentVolumeClaim:
          enabled: true
          size: 10Gi
          storageClassName: ebs-sc
          accessModes:
            - ReadWriteOnce
        resources:
          limits:
            cpu: 200m
            memory: 256Mi
          requests:
            cpu: 100m
            memory: 128Mi
  destination:
    server: https://kubernetes.default.svc
    namespace: headlamp
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Note: The serviceAccount.create: true block creates the headlamp Service Account—this is what kubectl create token headlamp -n headlamp uses for login. Adjust storageClassName if your EKS cluster uses a different EBS storage class. Omit the AppProject and Namespace if you already have them.

4. Test Access

Port-forward

kubectl port-forward svc/headlamp -n headlamp 8080:80

Create token

kubectl create token headlamp -n headlamp

Login

Open http://localhost:8080 and paste the token when Headlamp prompts for authentication.

5. Summary: Copy-Paste

# 1. Apply manifest
kubectl apply -f headlamp-application.yaml

# 2. Wait for sync (check Argo CD UI or: argocd app get headlamp)

# 3. Port-forward and get token
kubectl port-forward svc/headlamp -n headlamp 8080:80 &
kubectl create token headlamp -n headlamp

Then open http://localhost:8080 and paste the token when prompted.

6. Troubleshooting

Issue: Application stuck in Syncing or OutOfSync

Solution: Check Argo CD logs and the Application status. Ensure the Headlamp Helm repo is reachable and storageClassName: ebs-sc exists in your cluster. If using a different storage class, update the manifest.

Issue: Token invalid or login fails

Solution: Ensure the Service Account exists: kubectl get sa headlamp -n headlamp. Re-run kubectl create token headlamp -n headlamp to generate a fresh token (tokens expire).

Issue: Argo CD ingress stuck or inaccessible

Solution: See your cluster's Argo CD troubleshooting docs.

7. References

Headlamp: https://headlamp.dev
Installation (Service Account token): https://headlamp.dev/docs/latest/installation/#create-a-service-account-token
Argo CD: https://argo-cd.readthedocs.io