DEV Community: hugolesta

KMCP Turns MCP Servers Into Boring Infrastructure

hugolesta — Fri, 31 Jul 2026 08:55:26 +0000

Everyone building with LLMs eventually writes an MCP server. It works great on a laptop: npx some-mcp-server, point your client at stdio, done. Then someone asks the obvious question — "can the rest of the team use it?" — and you discover you've just signed up to write a Dockerfile, a Deployment, a Service, an Ingress, a secret wiring story, and a health check, for a process whose entire job is to expose four tools.

Multiply that by every tool your organization wants an agent to reach — ArgoCD, Jira, the incident tracker, the cost API — and you have a small fleet of bespoke microservices, each maintained by whoever wrote it first.

KMCP is the fix. It's a Kubernetes controller plus a CRD that turns "run this MCP server" into about 20 lines of YAML. This is what deploying it into a management EKS cluster actually looked like, and — more usefully — what changed once the platform's tools were on the other side of it.

The architecture

flowchart TD
    A["Helmfile — kmcp-crds + kmcp charts"] -->|"OCI chart from ghcr.io"| B["KMCP controller — kmcp-system namespace"]
    C["Git repo — mcpserver.yaml"] -->|"ArgoCD + cdk8s plugin"| D["MCPServer CR — team namespace"]
    B -->|"watches and reconciles"| D
    D -->|"controller creates"| E["Deployment + Service"]
    E -->|"Traefik IngressRoute — internal only"| F["mcp.internal.example.com"]
    F -->|"streamable HTTP — /mcp"| G["Agents, IDE clients, chat assistants"]
    E -->|"scoped API token"| H[("Platform tool — ArgoCD API")]

Two moving parts, deliberately separated. The controller is platform infrastructure — installed once, per EKS cluster, by the platform team's Helmfile. The MCPServer resources are application config — owned by whoever owns the tool being exposed, shipped through the normal GitOps path. A team adding an MCP server never touches the platform repo.

Installing the controller

KMCP ships as OCI Helm charts, which means the repository entry needs oci: true — easy to miss, and the failure mode is a confusing "not a valid chart repository" error:

repositories:
  - name: kmcp
    url: ghcr.io/kagent-dev/kmcp/helm
    oci: true

Then two releases, CRDs first:

releases:
  - name: kmcp-crds
    namespace: kmcp-system
    chart: kmcp/kmcp-crds
    version: ~>0.1

  - name: kmcp
    namespace: kmcp-system
    chart: kmcp/kmcp
    version: ~>0.1
    needs:
      - kmcp-crds
    installed: {{ .Environment.Values.kmcp.installed }}

Splitting CRDs into their own release is not ceremony. Helm's CRD handling is famously half-hearted — it installs crds/ directory contents on first install and then never upgrades them. A separate chart with an explicit needs dependency means CRD upgrades are a normal Helm operation, and the controller never starts before the types it watches exist.

The installed: flag is wired per environment. In this rollout, kmcp.installed is true only in the management EKS cluster and explicitly false in dev, acc, and prod:

  mng:
    values:
      - kmcp:
          installed: true

  prod:
    values:
      - kmcp:
          installed: false

Setting installed: false explicitly in every other environment rather than relying on a | default false is worth the extra lines. It makes the rollout state legible in one file — you can answer "where is KMCP running?" by reading, not by reasoning about template defaults.

Why the management cluster and not the workload EKS clusters? Because the tools worth exposing over MCP — ArgoCD, the CI control plane, the cost reporting API — already live there. Putting the MCP layer next to its backends avoids cross-cluster networking for what is otherwise a trivial HTTP hop.

An MCP server as 20 lines of YAML

Here's a real one — the ArgoCD MCP server, which gives an agent read access to application state, sync status, and logs:

apiVersion: kagent.dev/v1alpha1
kind: MCPServer
metadata:
  name: argocd-mcp
  namespace: platform-tools
  labels:
    kagent.dev/discovery: "disabled"
spec:
  transportType: http
  httpTransport:
    targetPort: 8770
    path: /mcp
  deployment:
    image: node:24-bookworm
    cmd: npx
    args:
      - argocd-mcp@latest
      - http
      - --port
      - "8770"
    port: 8770
    env:
      ARGOCD_BASE_URL: "https://argocd.internal.example.com"

That's the whole thing. No Dockerfile, no Deployment, no Service, no readiness probe. The controller reconciles this into a Deployment and a Service, wires the port mapping, and manages the lifecycle from there.

A few details that matter more than they look:

transportType: http. MCP's original transport is stdio — the client spawns the server as a subprocess and talks over pipes. That model does not survive contact with Kubernetes: there's nothing to load balance, nothing to health check, and one client per process. Streamable HTTP is what makes an MCP server a service — many clients, one deployment, standard networking. If you're deploying MCP servers to a cluster and reaching for stdio, stop and switch.

image: node:24-bookworm with cmd: npx. This is a deliberate shortcut, not sloppiness. Most published MCP servers are npm packages with no official container image. Rather than maintaining a build pipeline for a wrapper image around a package you don't own, you run the stock Node image and let npx fetch it. The tradeoff is real and you should name it out loud: argocd-mcp@latest resolves at pod start, so a pod restart six weeks from now can silently pick up a different version. Fine for a management cluster running internal tooling; not fine for anything on the critical path. Pin the version (argocd-mcp@1.4.2) the moment this stops being an experiment.

kagent.dev/discovery: "disabled". This opts the server out of automatic tool discovery by kagent agents in the cluster. Default-on discovery is a lovely demo and a bad default in a shared environment — it means every agent gets every tool the moment someone deploys a new MCPServer. Explicitly disabling it and granting access deliberately is the version you want when more than one team shares a cluster.

Routing: internal, and deliberately so

The service gets exposed through the existing ingress stack rather than anything KMCP-specific:

ingressRoute:
  enabled: true
  enableDefaultRoute: true
  publicAccess: false
  internetFacing: true
  routesList:
    - serviceName: argocd-mcp
      portName: http
      host: argocd-mcp.internal.example.com

publicAccess: false is doing the heavy lifting. An MCP server is, by construction, a remote code execution surface with a friendly schema on top — it exists to let a language model call functions against your infrastructure. It belongs on the internal load balancer, reachable over the corporate network, and nowhere else.

The portName: http took three attempts to get right in practice. The ingress controller matches the service port by name, and a name mismatch produces a route that resolves, returns a 502, and gives you no useful signal about why. If your MCP route 502s, check the port name before you check anything else.

Shipping it through GitOps, not a Makefile

The kmcp CLI has a kmcp deploy command, and it's genuinely nice for iterating locally. It has no place in your deployment path. In this setup the MCPServer manifests live in the owning team's repo and get synced by ArgoCD, with cdk8s doing the templating:

import os
import cdk8s

env = os.environ.get("ENV")

class MCPServer(cdk8s.Chart):
    def __init__(self, scope, id, env):
        super().__init__(scope, id)
        cdk8s.Include(self, "mcpserver", url=f"{env}/mcpserver.yaml")

app = cdk8s.App(yaml_output_type=cdk8s.YamlOutputType.FILE_PER_APP)
MCPServer(app, "mcpserver", env)
app.synth()

cdk8s.Include on a plain YAML file looks like a no-op wrapper, and mostly it is. The point is that the MCPServer sits inside the same synth as everything else the team deploys — ingress routes, config, the app itself — so one ArgoCD Application covers the whole unit. The MCP server is not a special deployment path. It's just another manifest in the tree.

That's the real design decision here: an MCP server should be indistinguishable from any other workload you run. Same repo layout, same GitOps sync, same review process, same rollback. The moment it needs its own deployment tooling, it becomes something only one person knows how to operate.

The part that actually changes delivery time

Standing up a controller is a day of work. The interesting question is what an organization gets once the tools are behind MCP, and the honest answer is that most of the value comes from something unglamorous: collapsing the distance between a question and its answer.

Think about what "check the deploy status" costs today. An engineer opens the ArgoCD UI, finds the right project among forty, filters applications, reads sync status, opens the diff, maybe pulls logs. Two minutes if they know exactly where to look, fifteen if they don't, and a Slack message to the platform team if they don't have access. Now consider a developer who has never used ArgoCD asking their assistant "is the checkout service synced in acc, and if not, what changed?" and getting an answer grounded in real cluster state.

That is not a chatbot novelty. It's the removal of a lookup tax that every engineer in the organization pays several times a day.

Three concrete shifts worth planning for:

Tribal knowledge becomes queryable. The platform team's real product is not the cluster; it's knowing which of forty ArgoCD projects owns a given service, and that a Progressing status stuck for ten minutes usually means a failed readiness probe. Exposed as tools, that knowledge stops being a queue in front of three people. The measurable win is fewer interrupts, and interrupts are the thing that actually destroys a platform team's throughput.

Read access spreads without the safety problem. Historically, giving a developer visibility into production meant giving them a console login and hoping. Handing them a read-scoped API token they present through an MCP server gives them answers without giving them the console. Different blast radius, entirely.

Onboarding compresses. A new engineer's first week is mostly discovering which internal tool answers which question, and where it lives. With those tools behind MCP, the discovery step goes away — they ask in the language they already have, and the tooling routes it.

The credential belongs to the caller, not the server

This is the part teams skip, and it's the part that determines whether the rollout survives its first security review.

The instinct when deploying a tool server is to give it a service account and a token — mount an ArgoCD credential into the Deployment and let it act on everyone's behalf. Don't. That builds a confused deputy: a single identity, holding the union of everything anyone might need, sitting behind an endpoint whose whole purpose is executing instructions from a language model.

Look closely at the MCPServer spec earlier in this post. The only thing in env is ARGOCD_BASE_URL. There is no token, no envFrom, no mounted secret — the server knows where ArgoCD is and nothing about who may talk to it. Send it an unauthenticated request and it says so:

{"jsonrpc":"2.0","id":1,"method":"initialize"}

x-argocd-api-token must be provided in the request header
(or the ARGOCD_API_TOKEN env var), or a token registry must be
configured via ARGOCD_TOKEN_REGISTRY_PATH.

That error is the design, not a misconfiguration. The caller presents its own token on every request, in the x-argocd-api-token header. The MCP server is a stateless protocol translator: it turns MCP tool calls into ArgoCD API calls and forwards the caller's credential. It holds no standing authority of its own.

The consequences are worth being explicit about, because they invert the usual threat model:

The server is not a prize. Compromising the pod yields a URL. There's no token at rest to exfiltrate, no secret in the namespace, no IRSA role on the ServiceAccount.
Authorization is ArgoCD's job, where it belongs. Every call is evaluated against the caller's own RBAC by the system that owns the data. The MCP layer cannot widen anyone's access, because it has none to lend.
The audit trail stays truthful. ArgoCD logs the acting account, not a shared mcp-server identity. "Who asked?" remains answerable — which it would not be if one token fronted every request.
Revocation is per-team and instant. Killing one team's token stops that team, with no redeploy and no effect on anyone else.

Note the env var mentioned in that error message. ARGOCD_API_TOKEN on the Deployment would work, and it is exactly the shortcut to refuse — it collapses every caller into one identity and undoes everything above. The token registry is the middle path if per-request headers are impractical for some client, but per-request is the model to start from.

This is what makes the per-team ArgoCD accounts load-bearing rather than bureaucratic. Each consuming team gets its own account, scoped to exactly what it owns:

configs:
  params:
    accounts.cloudplatform.enabled: "true"
    accounts.cloudplatform: apiKey
  rbac:
    policy.csv: |
      p, role:cloudplatform-role, projects, *, k8s-tools-project, allow
      p, role:cloudplatform-role, projects, *, kafka*, allow
      p, role:cloudplatform-role, projects, *, vault-project, allow
      p, role:cloudplatform-role, applications, *, k8s-tools-*/*, allow
      p, role:cloudplatform-role, applications, *, kafka*/*, allow
      p, role:cloudplatform-role, applications, *, vault-*/*, allow
      p, role:cloudplatform-role, logs, get, k8s-tools-*/*, allow
      p, role:cloudplatform-role, logs, get, kafka*/*, allow
      p, role:cloudplatform-role, logs, get, vault-*/*, allow
      g, cloudplatform, role:cloudplatform-role

Note apiKey and not login. The account can mint a token for programmatic use and cannot be used to log into the UI as a human. If the token leaks, the blast radius is "read some applications in three projects" instead of "an interactive session".

Note also that logs, get is granted separately from application access. Logs frequently contain more than the deploy metadata does — connection strings in stack traces, customer identifiers in error messages. Deciding about log access explicitly, per team, is worth the extra three lines.

Rolling this out one team at a time — a separate account and policy block per consuming team — is slower than a single shared account and it is the right call. It means an over-permissioned policy affects one team, and revoking access for one team doesn't take down everyone else's assistant.

The pattern generalizes past ArgoCD. Any tool you put behind MCP should answer the same question: whose authority does a call carry? If the answer is "the server's", you've centralized risk and blinded your audit log. If it's "the caller's", the MCP layer stays what it should be — plumbing that translates protocols and carries someone else's identity.

Field notes

oci: true is not optional and the error doesn't tell you. Helmfile treats an OCI registry URL as a classic chart repo without it and fails looking for an index.yaml that will never exist.
Split the CRD chart from the controller chart, always. Helm will install CRDs from a chart's crds/ directory on first install and then never upgrade them. When KMCP's MCPServer spec gains a field in 0.2, you want a chart upgrade, not a manual kubectl apply.
@latest in an MCPServer's args is a time bomb with a slow fuse. The pod that has been running for six weeks and the pod that just restarted are running different code. It won't hurt you until it does, and when it does the symptom will be "it worked yesterday". Pin it before it matters.
Service port names, not numbers, are what ingress matches on. A mismatch gives you a 502 with no useful log line on either side. Check this first when a new MCP route doesn't answer.
Turn off automatic tool discovery in shared clusters. kagent.dev/discovery: "disabled" plus deliberate grants beats every-agent-gets-every-tool. The default is optimized for a demo, not for a cluster with four teams in it.
Terraform-managed IAM roles for the MCP workload were removed, not fixed. The first pass provisioned an IRSA role for the MCP server before anyone had established that it needed AWS access. It didn't — it forwards the caller's ArgoCD token and talks to nothing else. Deleting infrastructure that exists "just in case" is a real win; every unused role is a permission someone will eventually attach a policy to.
Check what your MCP server holds at rest before you ship it. kubectl get deploy <name> -o jsonpath='{.spec.template.spec.containers[0].env}' and a look for mounted secrets takes ten seconds. An empty result is the good outcome — it means a compromised pod yields nothing worth stealing.
Read-only first, and mean it. There will be pressure to add write tools — "just let it trigger a sync". Resist until the read path has been in production long enough that you trust the identity boundaries. A hallucinated get is a wrong answer; a hallucinated sync is an incident.
The controller is not where your time goes. Installing KMCP took an afternoon. Getting ArgoCD accounts, RBAC scoping, and the per-team policy format right took a week and several follow-up PRs. Budget accordingly — the platform piece is easy, the identity piece is the project.

Closing

KMCP's contribution is narrow and worth having: it makes an MCP server a first-class Kubernetes object, so exposing a tool costs a YAML file instead of a service. That reduction is what lets you say yes to the fifth MCP server without inheriting a fleet of snowflakes.

The leverage, though, is in what you connect and whose identity the calls carry. Every tool behind MCP is one fewer thing an engineer has to know where to find, and one fewer interrupt in the platform team's day. Keep the credential with the caller and that scales safely: the MCP layer never accumulates authority, and nobody can ask it for more than they were already allowed to have.

Terraform Enterprise on EKS: The Parts Nobody Documents

hugolesta — Fri, 17 Jul 2026 17:00:23 +0000

Terraform Enterprise's own install docs assume a plain Docker host or a generic Kubernetes cluster with an ingress controller that speaks HTTPS by default. Run it on EKS behind ArgoCD, cdk8s, ExternalSecrets, and Traefik, and you hit at least four failure modes that produce no useful error message — just a 404, a silently-ignored UPDATE, or an OAuth token that pretends a repository doesn't exist.

None of these are TFE bugs. They're the gap between "works on a demo cluster" and "works inside a real platform stack."

Architecture

flowchart TD
    A["Terraform — RDS, S3, IAM, Secrets Manager"] --> B["cdk8s — K8s manifests"]
    B --> C["ArgoCD — sync"]
    C --> D["Terraform Enterprise pod — EKS"]
    D --> E[("Aurora PostgreSQL 16")]
    D --> F[("ElastiCache Redis")]
    D --> G[("S3 — state storage, IRSA")]
    D --> H["Traefik IngressRoute — HTTPS backend"]
    H --> I["ALB / Global Accelerator — external TLS"]

Infra (RDS, ElastiCache, S3, IAM) comes from Terraform. Kubernetes manifests are generated by cdk8s. ArgoCD syncs them. Secrets live in AWS Secrets Manager and land in the pod via ExternalSecrets. Standard platform-engineering plumbing — the failure modes below all live in the seams between these pieces, not inside any one of them.

Image: AMD64 only

TFE publishes no ARM64 image. If your GitHub Actions runners default to linux/arm64, or you've standardized on Graviton for cost, the build fails outright unless you explicitly override:

FROM images.releases.hashicorp.com/hashicorp/terraform-enterprise:v202507-1
LABEL maintainer="platform-team@example.com"

platforms: "linux/amd64"   # required in both build and deploy workflows

Cheap to fix once you know it's the reason — expensive to diagnose if you don't, because the build error just looks like a generic manifest mismatch.

The database needs the admin user, briefly

Aurora Postgres 16 is the ceiling — TFE does not support 17, and Aurora can't be downgraded in place, only destroyed and recreated. Budget for that constraint before you provision, not after.

The subtler trap: on first boot, TFE needs to create schemas. If TFE_DATABASE_USER points at a least-privilege application user (which is the instinct, and the right end state), schema creation fails with a permissions error that doesn't obviously say "wrong user." Use the admin/root credential for the first boot, let migrations run, then rotate to the scoped app user afterward.

TLS: the certificate header that isn't `CERTIFICATE`

TFE's internal nginx uses ssl_trusted_certificate, which requires the PEM header TRUSTED CERTIFICATE — not the standard CERTIFICATE header every other tool expects. A normal self-signed cert fails at container startup with:

nginx: [emerg] cannot load certificate: PEM_read_bio_X509_AUX() failed (Expecting: TRUSTED CERTIFICATE)

Generate it in the right format from the start:

openssl req -x509 -newkey rsa:2048 \
  -keyout /tmp/tfe-key.pem \
  -out /tmp/tfe-cert.pem \
  -days 3650 -nodes \
  -subj "/CN=tfe.example.internal"

openssl x509 -in /tmp/tfe-cert.pem -trustout -out /tmp/tfe-cert-trusted.pem

head -1 /tmp/tfe-cert-trusted.pem
# Expected: -----BEGIN TRUSTED CERTIFICATE-----

Two more ways this quietly breaks even after you get the header right:

Line wrapping matters. openssl x509 -trustout produces correctly wrapped 64-character lines. If anything downstream collapses the file onto a single line — cat in a shell substitution is the usual culprit — nginx fails to parse it even though the header is technically correct.
Secret key names must match the Helm chart's subPath exactly. The chart mounts subPath: tls.crt and subPath: tls.key. Name your ExternalSecret keys anything else (cert.pem, key.pem — the "obvious" names) and you get a silent empty mount: the pod starts, nginx just gets a zero-byte file, and the resulting error looks identical to a missing cert.

If you're piping any multi-line secret (cert, key, license) through an encryption or secrets API as part of provisioning, json.dumps() the file content in Python before sending it — never cat or $(cat file), which collapses newlines and silently produces the same "technically present, functionally broken" secret.

Traefik: HTTPS-only backend, no plain HTTP path

TFE's Helm chart only exposes an HTTPS backend — there's no plain-HTTP listener at all. That's unusual enough among services behind a typical ingress controller that it's easy to configure exactly the same way as everything else and get nothing but a 404.

Two things are non-negotiable:

A ServersTransport with insecureSkipVerify: true. Traefik verifies backend TLS by default; TFE's backend cert is self-signed. Without this resource, Traefik doesn't error — it silently drops the route and returns 404, which looks identical to a misconfigured IngressRoute.

apiVersion: traefik.io/v1alpha1
kind: ServersTransport
metadata:
  name: tfe-transport
  namespace: tfe
spec:
  insecureSkipVerify: true

The IngressRoute must reconnect over HTTPS, referencing that transport, even if your Traefik setup terminates external TLS upstream (ALB, Global Accelerator) and only forwards plain HTTP internally to everything else:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: tfe
  namespace: tfe
spec:
  entryPoints:
    - web
  routes:
    - kind: Rule
      match: Host(`tfe.example.internal`)
      priority: 10
      services:
        - name: terraform-enterprise
          port: 8080
          scheme: https
          serversTransport: tfe-transport

Note the service name: the chart hardcodes it to terraform-enterprise regardless of your Helm release name. Reference the hardcoded name, not whatever you called the release.

If you're generating manifests with cdk8s.Include, put ServersTransport and IngressRoute in separate files. cdk8s.Include only processes the first document in a YAML file — a multi-document file separated by --- silently drops everything after the first ---.

A working route returns a 301 to /session:

curl -sk https://tfe.example.internal/ -o /dev/null -w "%{http_code} %{redirect_url}\n"
# Expected: 301 https://tfe.example.internal/session

The SSO trap that skips your first site admin

This is the one that actually costs people time.

TFE bootstraps its first site admin through a one-time signup endpoint, /admin/account/new. It only renders the signup form while no user exists yet. The moment any user record exists, the route redirects to /app and is permanently closed — there's no supported way to reopen it.

Here's the trap: if the first login to a fresh TFE instance happens via SSO/SAML instead of that local signup form, TFE still considers its job done — a user now exists — but that user is created with is_admin = false. Nobody has site-admin rights. The Site Admin menu and New Organization button don't appear anywhere, for anyone, and there's no UI or API path to fix it.

The correct sequence, in order:

Confirm the pod is healthy and the IngressRoute works.
Before anyone signs in via SSO for the first time, visit /admin/account/new directly and create a local admin account. This is the only flow that auto-sets is_admin = true.
If SSO has already beaten you to it — this happens more often than the docs imply, usually because someone tests the login flow before thinking about admin bootstrapping — promote the existing user directly in the database.

-- fetch admin creds from your secrets manager first
SET search_path TO rails, public;
UPDATE rails.users SET is_admin = true WHERE id = <user_id>;
SELECT id, email, username, is_admin FROM rails.users WHERE id = <user_id>;

The SET search_path line isn't optional. Without it, the UPDATE fails inside an internal trigger (user_hcp_user_email_reservation_check) with relation "hcp_user_email_reservations" does not exist — and depending on your client, that failure can look like it silently did nothing rather than raising a clear error. Always verify with a follow-up SELECT that is_admin actually flipped.

SAML team sync doesn't create the team for you

Once SSO works and an admin exists, the next surprise is team provisioning. TFE grants organization membership through team membership — not directly. Your IdP can send a perfectly correct MemberOf attribute on every login, but if no team with that exact name exists yet in the TFE organization, the user authenticates successfully, gets an account, and is simply not part of the organization. No error. Just an orphaned, authenticated user.

Fix: create the team first, with a name that matches the IdP attribute value exactly, then have the user log out and back in.

curl -sk --header "Authorization: Bearer $USER_TOKEN" \
  --header "Content-Type: application/vnd.api+json" \
  --request POST \
  --data '{
    "data": {
      "type": "teams",
      "attributes": {
        "name": "developers",
        "organization-access": { "manage-workspaces": true }
      }
    }
  }' \
  "https://tfe.example.internal/api/v2/organizations/<org>/teams"

Separately, TFE's SAML config also requires a Single Log-Out URL, even though SLO itself is optional in most SAML deployments and your IdP's setup page may not surface a dedicated logout URL. Leaving it blank fails validation outright. Your IdP's own sign-out endpoint is a legitimate value here, not a placeholder.

And when writing the IdP-side attribute expression for site-admin auto-provisioning: make sure it emits a string, not a boolean. isMemberOfGroupName("admins") alone evaluates to true/false, and TFE's Site Admin Role field matches against a specific string (e.g. site-admins) — a boolean silently never matches, so nobody gets auto-promoted and the failure is invisible unless you go looking for it. A ternary that emits the actual string fixes it.

No inbound webhooks, no VCS auto-apply

If TFE sits behind a private, VPN-only ingress — a reasonable default for an internal platform tool — connecting a VCS provider works fine for the OAuth handshake and API calls (TFE calling out to GitHub), but GitHub's webhook delivery (GitHub calling in to TFE on push) simply cannot reach a private endpoint. There's no stable, publishable GitHub IP range you can safely allow-list without reopening broad public ingress.

Practically: connecting a workspace to a repo in the UI will never auto-trigger a plan on push. The fix is to invert the flow — instead of waiting for GitHub to call TFE, have a CI job that already runs inside your private network call TFE's API directly:

Developer pushes to the repo.
A CI workflow runs on a self-hosted runner inside the network.
The runner calls TFE's API (POST /api/v2/runs, or the CLI-driven workflow) to create a run — outbound from inside the network, so no inbound exposure needed.
TFE executes the plan/apply and posts the commit status back via its own outbound call.

Still attach VCS metadata to the workspace for source display and commit-status reporting — just don't depend on the webhook trigger.

Module registry publish needs its own VCS scope — and a second, invisible SSO check

Two separate gotchas stack on top of each other here, and they produce the identical error message, which makes debugging genuinely confusing.

First: the VCS Provider's scope has three independent toggles — Workspaces, Policy Sets, Modules. Enabling the connection for workspaces does not enable it for module publishing. Miss this and publishing fails with:

Error publishing module — VCS Connection Validation failed: The specified repository "<repo>" doesn't exist or isn't accessible.

— even though the same repo works fine for VCS-linked workspaces. Fix: toggle Modules on in the VCS Provider settings.

Second, and much sneakier: if your GitHub organization enforces SAML SSO, and the VCS Provider is a plain OAuth App (not an installed GitHub App), the OAuth token must be separately authorized for SSO per organization — even for a service account that's already a full org member with read access to the repo. Until that authorization is granted, GitHub silently filters that org's repositories out of every API response made with the token. No error. The repo picker just shows "0 repositories," and direct publish fails with the exact same "doesn't exist or isn't accessible" message as the Modules-toggle issue above.

The tell that distinguishes the two: TFE's manual "can't see your repository? enter its ID" field accepts any string and advances you to a confirmation step regardless of the SSO block, because it doesn't call GitHub's API at that point — the real validation call only happens at final publish. Reaching that confirmation screen proves nothing.

Fix: as the service account, go to github.com/settings/connections/applications, find the OAuth App matching the Client ID shown on the VCS Provider's edit page, and grant/authorize it for the organization. One-click, one-time, per org. No token regeneration or TFE-side change needed.

Field notes

Use a service account for the VCS OAuth connection, not a personal one. It survives offboarding and doesn't break when someone's personal token expires or access is revoked.
Aurora Postgres 16 is a hard ceiling, not a soft recommendation — plan the destroy/recreate cost into your migration timeline if you're already running 17 elsewhere.
Test SSO from a session with a way back in. Confirm a working local admin login exists before testing SAML changes — don't lock yourself out mid-configuration.
The org-level "run/apply everywhere" permission has no narrower sibling. manage-workspaces: true at the org level grants admin on every workspace in the org — there's no org-level "plan/apply but not admin." Scoping down to specific workspaces or a lesser permission means configuring per-workspace Team Access individually.
Every one of these failures is silent by design, not by bug: a dropped route, a filtered repo list, a trigger that swallows an error, a boolean that never matches a string. None of them page you. All of them look like something else until you know the specific shape of this one.

Most of the actual engineering effort in a TFE install isn't the Terraform or the Helm values — it's building the mental map of which of these four subsystems (TLS, ingress, SAML, VCS OAuth) is failing silently, since none of them will tell you directly. Once you know the map, every one of these takes minutes to fix instead of hours to find.

Your AI Platform Needs a Gateway, Not a Credit Card

hugolesta — Thu, 09 Jul 2026 12:28:23 +0000

A team in your org hits Anthropic during a production incident. Anthropic is down. The team is blocked for 20 minutes while someone scrambles to find an alternative API key, change a config, and redeploy. There's no fallback. There was never a plan for this.

Somewhere else, another team has quietly started billing their LLM usage to a personal credit card because procurement takes six weeks. A third team has hardcoded gpt-4o everywhere and doesn't know there's a cheaper model that handles their workload fine.

This isn't a story about AI risk. It's a story about missing infrastructure.

The pattern you've seen before

When teams first started running services in the cloud, the ones who moved fast didn't start with a load balancer and a CDN. They spun up an EC2 instance, pointed DNS at it, and shipped. That worked — until it didn't. Until traffic spiked, until the instance went down, until three teams had three different DNS setups and nobody knew which was canonical.

AI adoption is following the same curve, faster. Every team that can call an API is calling one. The platforms don't have a control plane yet. The control plane is somebody's laptop with a hardcoded key.

An AI gateway is what closes that gap. It's not a product pitch — it's the same thing you built for HTTP traffic: a single ingress point with routing, fallback, observability, and policy enforcement. The LLM is the backend. The gateway is the infrastructure around it.

What the gateway actually does

flowchart TD
    A["Developer — MCP / SDK / REST client"] --> G["AI Gateway — single stable endpoint"]
    B["Business user — internal tooling"] --> G
    G -->|"primary"| P1["Anthropic Claude"]
    G -->|"fallback on 5xx / timeout"| P2["AWS Bedrock — Claude / Titan"]
    G -->|"cost-optimised route"| P3["OpenAI GPT-4o mini"]
    G --> OBS["Observability — spend per team — request logs — latency"]
    G --> POL["Policy — rate limits — budget caps — PII filter"]

Failover. When a provider goes down, the gateway retries against the next configured backend — same request, different model, no client code change. This is the feature that looks optional until someone spends 20 minutes blocked mid-incident.

Cost routing. Not every call needs a frontier model. Summarisation, classification, short completions — route those to something cheaper. The gateway makes this a config change, not a refactor.

Spend visibility. Every request is tagged by team, application, and model. You see where the money goes before finance calls you about it. You can set hard budget caps per team so nobody accidentally runs a batch job against claude-opus all night.

Provider abstraction. When a new model ships — or when your organisation wants to experiment with providers you don't use today — teams swap the gateway config, not their application code. The SDK they're using doesn't change.

Audit logging. In any regulated environment, knowing who sent what prompt to which model isn't a nice-to-have. It's table stakes for governance. The gateway is the right place to capture that, not each individual application.

What self-hosting looks like in practice

The two tools worth knowing are LiteLLM and Bifrost. Both expose an OpenAI-compatible API, which means existing applications need zero changes to the SDK call — only the base URL and API key change.

LiteLLM is the more battle-tested option. It runs as a Docker container, connects to a Postgres database for request logs and virtual key management, and supports every major provider out of the box. A minimal deployment on Kubernetes looks like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: litellm
spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: litellm
          image: ghcr.io/berriai/litellm:main-latest
          args:
            - "--config"
            - "/etc/litellm/config.yaml"
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: litellm-secrets
                  key: database-url
          volumeMounts:
            - name: config
              mountPath: /etc/litellm
      volumes:
        - name: config
          configMap:
            name: litellm-config

The config itself is where the routing logic lives:

model_list:
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: claude-sonnet
    litellm_params:
      model: bedrock/us.anthropic.claude-sonnet-4-5-20251001-v2:0
      aws_region_name: eu-west-1

router_settings:
  routing_strategy: least-busy
  num_retries: 2
  fallbacks:
    - claude-sonnet:
        - claude-sonnet

litellm_settings:
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]

Two entries for the same model_name is the whole failover mechanism. LiteLLM tries the first, fails, retries against the second. From the client's perspective, the call either succeeds or returns an error after both backends fail — never a 20-minute block.

The latency argument doesn't hold here

The pushback you'll hear is latency. "We can't add a proxy hop to every LLM call."

That argument makes sense for traditional APIs where you're measuring response times in milliseconds. It doesn't apply to LLMs. Time-to-first-token on a typical Claude call is 300–800ms. A gateway adds 5–15ms. That's noise, not overhead. The conversation about "proxy latency" is a conversation that predates LLMs and doesn't transfer.

The actual latency concern worth having is gateway availability — if the gateway goes down, everything goes down. The answer is the same as any other critical path: multiple replicas, a readiness probe, a PodDisruptionBudget, and a runbook. Nothing exotic.

The enforcement problem

One thing the gateway doesn't solve on its own: casual users who bypass it entirely. If teams can buy a Claude.ai subscription for $20/month and skip the gateway, many will — especially if the gateway requires a non-trivial onboarding step.

This is a governance problem, not a technical one. The gateway has to be the path of least resistance, not just the official path. That means:

API keys that work without friction (virtual keys in LiteLLM's admin UI take 30 seconds to issue)
A clear internal page that says "here is your endpoint and here is your key"
A procurement policy that routes all model spend through the central key, not individual accounts

The technical infrastructure is the easier part. Getting adoption is the harder part, and that's a platform team problem from day one, not something you solve after the gateway is running.

Field notes

Run the gateway in at least two replicas from day one. A single-replica proxy is a single point of failure you introduced yourself. Two replicas with a PDB that disallows simultaneous evictions cost almost nothing and remove the worst failure mode.
LiteLLM's virtual key system is the right unit of cost tracking. One virtual key per team, budget cap attached, spend dashboard out of the box. Don't try to build this yourself — it's already there.
Failover works at the gateway level, not the application level. Stop trying to handle retries in application code when you have a gateway. Put the retry and fallback logic in one place and let every team benefit from it automatically.
Tag every request with metadata. LiteLLM forwards custom metadata to your observability backend. Use it to tag team, environment, and use case from the first day — you'll want that granularity when someone asks why the bill doubled.
SDK compatibility means the migration is low-friction. If you start with LiteLLM and later want to evaluate Bifrost or something else, the migration is a base URL change for application teams. That's the right amount of coupling.

The teams that will struggle with AI costs and reliability in 12 months are the ones treating model access like a SaaS subscription — something each team manages independently. The teams that won't are the ones who put a control plane in front of it now, before the sprawl hardens into something unauditable.

The gateway isn't the exciting part of the AI platform. It's the infrastructure that makes the exciting parts sustainable.

Give Your LLM Hands: Bedrock Agents, Lambda Tools, and the MCP Pattern in Terraform

hugolesta — Fri, 03 Jul 2026 09:14:49 +0000

The Knowledge Base from the previous post gives the LLM memory. But memory is passive — the model can tell you what the runbook says, but it can't go and create the Confluence page, look up the Jira ticket, or ping Slack. For that you need an agent: a model that can reason over a goal, decide which tool to call, call it, inspect the result, and loop until the task is done.

This is what Bedrock Agents does. The model driving it is Anthropic Claude; the "tools" it invokes are Lambda functions you own; the schema that tells the model what each function accepts and returns lives in Terraform as a function_schema block. It's the Model Context Protocol pattern, just deployed as AWS-managed infrastructure instead of a local sidecar.

The agent covered here — an incident reporter — takes a PagerDuty incident ID, fetches details, searches Jira for related tickets, writes a Confluence incident page, and posts a Slack notification. Four action groups, four Lambdas, one Knowledge Base, one Terraform file.

Architecture

flowchart TD
    A["Caller — incident ID"] -->|"InvokeAgent"| B["Bedrock Agent — Claude Sonnet — eu cross-region"]
    B -->|"Retrieve context"| C[("Knowledge Base — Aurora pgvector")]
    B -->|"action group"| D["Lambda — pagerduty-tools — get_incident — list_incidents — get_incident_alerts"]
    B -->|"action group"| E["Lambda — jira-tools — search_issues — get_issue"]
    B -->|"action group"| F["Lambda — confluence-tools — create_page — get_page — update_page"]
    B -->|"action group"| G["Lambda — slack-tools — send_message — find_channel"]
    D -->|"PagerDuty API"| H["PagerDuty"]
    E -->|"Jira API"| I["Jira"]
    F -->|"Confluence API"| J["Confluence"]
    G -->|"Slack API"| K["Slack"]
    D & E & F & G -->|"tool results"| B
    B -->|"final response"| A

Each Lambda reads its API credentials from Secrets Manager at runtime — no credentials in environment variables, no secrets in Terraform state beyond the ARN.

The Lambda tools pattern

Bedrock calls this concept "action groups". I find it cleaner to think of it as MCP: the agent has a set of named functions with typed parameters, it decides which ones to call based on the task, calls them, and incorporates the responses. The Lambda is the implementation; the Terraform function_schema block is the schema the model reads.

Every tool Lambda in this setup follows the same structure:

A module call that provisions the Lambda (runtime, memory, timeout, S3 source, env vars)
An aws_lambda_permission that allows bedrock.amazonaws.com to invoke it
An aws_bedrockagent_agent_action_group that declares the function schema

The permission and the action group are separate resources — the permission is at the Lambda level, the action group is at the agent level. Both are required.

Secrets handling

Each Lambda that calls an external API gets the secret name as an environment variable, not the secret value. The Lambda's IAM role has secretsmanager:GetSecretValue on that specific secret ARN. This means:

Rotating a token is a Secrets Manager operation, not a Terraform apply
The secret value never appears in a plan output or state file
CloudWatch logs show PAGERDUTY_SECRET_NAME=pagerduty-api-token, not a token

locals {
  lambdas = {
    pagerduty  = "bedrock-agent-pagerduty-tools"
    jira       = "bedrock-agent-jira-tools"
    confluence = "bedrock-agent-confluence-tools"
    slack      = "bedrock-agent-slack-tools"
  }
}

Lambda module + permission (PagerDuty as example)

module "lambda_pagerduty_tools" {
  source  = "your-org/lambda/aws"
  version = "~> 2.15.0"

  env     = local.env
  project = local.project
  vcs     = local.vcs
  owner   = local.owner

  function_name      = local.lambdas.pagerduty
  handler            = "lambda_function.lambda_handler"
  runtime            = "python3.13"
  networking_enabled = false
  cloudwatch_enabled = true
  memory_size        = 256
  timeout            = 30

  s3_bucket = "your-artifacts-bucket"
  s3_key    = "agents/${local.lambdas.pagerduty}/${local.lambdas.pagerduty}.zip"

  env_vars = {
    PAGERDUTY_SECRET_NAME = module.pagerduty_secret.secret_name
    LOG_LEVEL             = "INFO"
  }

  depends_on = [module.pagerduty_secret]
}

resource "aws_lambda_permission" "bedrock_pagerduty_tools" {
  statement_id  = "AllowBedrockInvoke"
  action        = "lambda:InvokeFunction"
  function_name = "${local.project}-${local.lambdas.pagerduty}-${local.env}"
  principal     = "bedrock.amazonaws.com"
  source_arn    = "arn:aws:bedrock:${local.aws_region}:${local.aws_account_id}:agent/*"
  depends_on    = [module.lambda_pagerduty_tools]
}

The source_arn uses agent/* — wildcard over agents in this account and region. If you want tighter scoping, replace the wildcard with the specific agent ARN after it's created, but that creates a circular dependency you'll have to break with a two-phase apply.

The Agent resource

resource "aws_bedrockagent_agent" "incident_reporter" {
  agent_name                  = "${local.project}-incident-reporter-agent-${local.env}"
  agent_resource_role_arn     = aws_iam_role.incident_reporter_agent.arn
  foundation_model            = "eu.anthropic.claude-sonnet-4-6"
  description                 = "Orchestrates PagerDuty, Jira, Confluence, and Slack for automated incident documentation"
  idle_session_ttl_in_seconds = 600

  instruction = <<-EOT
    You are an AI Incident Reporter for the Cloud Platform team.

    When given a PagerDuty incident, complete ALL of the following steps in order:

    1. FETCH INCIDENT DETAILS
       - Use the pagerduty-tools action group to get full incident details and alerts
       - Extract: title, description, severity, affected service, timeline, assignees

    2. FIND RELATED JIRA TICKETS
       - Use the jira-tools action group to search for related tickets
       - DO NOT create new Jira tickets — only retrieve existing ones

    3. CREATE CONFLUENCE INCIDENT PAGE
       - Use the confluence-tools action group to create a new incident page
       - Include: title, severity, timeline, PagerDuty link, related Jira tickets

    4. NOTIFY SLACK
       - Use the slack-tools action group to send a message to the incident channel
       - Include: summary, severity, Confluence URL, Jira links

    5. RETURN SUMMARY
       - Provide a final summary with all links created

    Guidelines:
    - Always complete all steps — do not skip any
    - If a step fails, report the error and continue with remaining steps
    - Use the knowledge base to understand infrastructure context when relevant
  EOT

  tags = local.default_tags

  depends_on = [aws_iam_role_policy_attachment.incident_reporter_agent]
}

A few things worth noting about this resource:

foundation_model: the eu. prefix routes through AWS's cross-region inference, which lets the agent use model capacity across EU regions (eu-west-1, eu-central-1, eu-west-3) automatically. Without this prefix you're locked to a single region's available capacity. Use it if you're in a EU region.

instruction: this is the system prompt. It's what tells the model the task structure, step ordering, and guard rails. Keep it procedural and explicit — the model will follow numbered steps reliably. Vague instructions produce vague agents.

idle_session_ttl_in_seconds: how long a conversation session stays alive between calls. 600 seconds (10 minutes) is reasonable for a human-in-the-loop flow; drop it lower if you're doing fully automated invocations where sessions shouldn't accumulate.

Agent IAM

The agent's execution role needs three permissions:

data "aws_iam_policy_document" "incident_reporter_agent" {
  statement {
    sid    = "InvokeModel"
    effect = "Allow"
    actions = [
      "bedrock:InvokeModel",
      "bedrock:InvokeModelWithResponseStream",
    ]
    resources = ["*"]
  }

  statement {
    sid    = "KBRetrieve"
    effect = "Allow"
    actions = ["bedrock:Retrieve"]
    resources = [
      "arn:aws:bedrock:${local.aws_region}:${local.aws_account_id}:knowledge-base/${var.knowledge_base_id}",
    ]
  }

  statement {
    sid    = "InvokeLambdas"
    effect = "Allow"
    actions = ["lambda:InvokeFunction"]
    resources = [
      "arn:aws:lambda:${local.aws_region}:${local.aws_account_id}:function:${local.project}-${local.lambdas.pagerduty}-${local.env}",
      "arn:aws:lambda:${local.aws_region}:${local.aws_account_id}:function:${local.project}-${local.lambdas.jira}-${local.env}",
      "arn:aws:lambda:${local.aws_region}:${local.aws_account_id}:function:${local.project}-${local.lambdas.confluence}-${local.env}",
      "arn:aws:lambda:${local.aws_region}:${local.aws_account_id}:function:${local.project}-${local.lambdas.slack}-${local.env}",
    ]
  }
}

InvokeModel on * is required — Bedrock Agents uses this to invoke the foundation model internally, and the resource ARN for FM invocation isn't predictable at the time the role is written if you're using cross-region inference prefixes. You can scope it to arn:aws:bedrock:*::foundation-model/* if you want some constraint without breaking cross-region.

Action groups with function schemas

This is where the MCP analogy is most visible. The function_schema block is effectively a tool manifest: the model reads it at runtime to understand what each function is named, what parameters it expects, and what they mean.

resource "aws_bedrockagent_agent_action_group" "pagerduty_tools" {
  agent_id          = aws_bedrockagent_agent.incident_reporter.agent_id
  agent_version     = "DRAFT"
  action_group_name = "pagerduty-tools"
  description       = "PagerDuty tools: get_incident, list_incidents, get_incident_alerts"

  action_group_executor {
    lambda = "arn:aws:lambda:${local.aws_region}:${local.aws_account_id}:function:${local.project}-${local.lambdas.pagerduty}-${local.env}"
  }

  function_schema {
    member_functions {
      functions {
        name        = "get_incident"
        description = "Retrieve full details of a single PagerDuty incident by its ID"
        parameters {
          map_block_key = "incident_id"
          type          = "string"
          description   = "The PagerDuty incident ID (e.g. P1234AB)"
          required      = true
        }
      }
      functions {
        name        = "list_incidents"
        description = "List active PagerDuty incidents filtered by status and/or service"
        parameters {
          map_block_key = "statuses"
          type          = "string"
          description   = "Comma-separated statuses to filter by: triggered, acknowledged, resolved"
          required      = false
        }
        parameters {
          map_block_key = "limit"
          type          = "integer"
          description   = "Maximum number of incidents to return (default: 20)"
          required      = false
        }
      }
      functions {
        name        = "get_incident_alerts"
        description = "Get all alerts associated with a PagerDuty incident"
        parameters {
          map_block_key = "incident_id"
          type          = "string"
          description   = "The PagerDuty incident ID"
          required      = true
        }
      }
    }
  }

  depends_on = [
    module.lambda_pagerduty_tools,
    aws_lambda_permission.bedrock_pagerduty_tools,
  ]
}

The Jira action group is read-only by design — search_issues and get_issue only. This isn't enforced at the Lambda level (you could build a write-capable Lambda and only expose read functions here), but the intent is captured in both the description field and the agent's instruction. Two layers of "don't create tickets" beats one.

resource "aws_bedrockagent_agent_action_group" "jira_tools" {
  agent_id          = aws_bedrockagent_agent.incident_reporter.agent_id
  agent_version     = "DRAFT"
  action_group_name = "jira-tools"
  description       = "Jira tools (read-only): search_issues, get_issue — no ticket creation"

  action_group_executor {
    lambda = "arn:aws:lambda:${local.aws_region}:${local.aws_account_id}:function:${local.project}-${local.lambdas.jira}-${local.env}"
  }

  function_schema {
    member_functions {
      functions {
        name        = "search_issues"
        description = "Search Jira issues using JQL"
        parameters {
          map_block_key = "jql"
          type          = "string"
          description   = "JQL query string"
          required      = false
        }
        parameters {
          map_block_key = "max_results"
          type          = "integer"
          description   = "Maximum number of results to return (default: 10)"
          required      = false
        }
        parameters {
          map_block_key = "fields"
          type          = "string"
          description   = "Comma-separated list of fields to include"
          required      = false
        }
      }
      functions {
        name        = "get_issue"
        description = "Retrieve full details of a single Jira issue by its key"
        parameters {
          map_block_key = "issue_key"
          type          = "string"
          description   = "The Jira issue key (e.g. OPS-1234)"
          required      = true
        }
      }
    }
  }

  depends_on = [
    module.lambda_jira_tools,
    aws_lambda_permission.bedrock_jira_tools,
  ]
}

Attaching the Knowledge Base

One resource to connect the Knowledge Base built in the previous post:

resource "aws_bedrockagent_agent_knowledge_base_association" "incident_reporter_kb" {
  agent_id             = aws_bedrockagent_agent.incident_reporter.agent_id
  knowledge_base_id    = var.knowledge_base_id
  description          = "Internal infrastructure and platform documentation"
  knowledge_base_state = "ENABLED"
}

With this association in place, the agent automatically retrieves relevant KB chunks before deciding which tools to call. The instruction's "use the knowledge base to understand infrastructure context when relevant" is the trigger — the model decides when to retrieve, not you.

Agent alias and SSM parameters

resource "aws_bedrockagent_agent_alias" "incident_reporter" {
  agent_id         = aws_bedrockagent_agent.incident_reporter.agent_id
  agent_alias_name = "live"
  description      = "Live alias for the Incident Reporter agent"
  tags             = local.default_tags

  depends_on = [
    aws_bedrockagent_agent_action_group.pagerduty_tools,
    aws_bedrockagent_agent_action_group.jira_tools,
    aws_bedrockagent_agent_action_group.confluence_tools,
    aws_bedrockagent_agent_action_group.slack_tools,
    aws_bedrockagent_agent_knowledge_base_association.incident_reporter_kb,
  ]
}

resource "aws_ssm_parameter" "incident_reporter_agent_id" {
  name  = "/${local.env}/incident-reporter/agent_id"
  type  = "String"
  value = aws_bedrockagent_agent.incident_reporter.agent_id
}

resource "aws_ssm_parameter" "incident_reporter_agent_alias_id" {
  name  = "/${local.env}/incident-reporter/agent_alias_id"
  type  = "String"
  value = aws_bedrockagent_agent_alias.incident_reporter.agent_alias_id
}

The alias is required to invoke the agent from application code — you always invoke via agentId + agentAliasId. Publishing a named alias decouples callers from the internal version counter: when you update action groups and Bedrock creates a new draft version, you prepare a new alias version and flip it without changing any caller configuration.

The depends_on on the alias resource is load-bearing. Without it, Terraform might try to create the alias before all action groups exist, which causes Bedrock to create the alias pointing at an agent that's still missing tools.

Field notes

agent_version = "DRAFT" on all action groups. Action groups always attach to DRAFT. When you publish a new agent version (via alias routing), Bedrock snapshots the DRAFT config. If you try to attach action groups to a numbered version, the API rejects it. This trips up people who try to version-lock action groups for safety.
The function description matters more than you'd think. The model reads description fields to decide which tool to call and when. Vague descriptions ("does stuff with PagerDuty") produce agents that hallucinate which tool to invoke. Write descriptions as if you're writing API docs for a junior engineer — precise verbs, clear scope.
Parameter types are limited: string, integer, boolean, number, array. No nested objects. If your Lambda needs structured input (a list of filter criteria, a nested config), serialize it as JSON in a string parameter and document the shape in the description. Ugly but it works.
Lambda cold starts extend agent response time noticeably. Each tool call is a synchronous Lambda invocation. Four cold starts in a chain = 4× the latency penalty. Provision concurrency on the most-called tools if response time matters.
The depends_on on the alias is not optional. Terraform's parallelism will race the alias creation against action group creation without it. The apply succeeds but the agent alias captures an incomplete version with missing tools. You won't notice until runtime.
Cross-region inference (eu. prefix) changes the IAM picture. With the cross-region prefix, the model ARN the agent resolves to at runtime isn't in your account's region — it's wherever AWS routes capacity. InvokeModel on * is practically required here, or you scope to arn:aws:bedrock:*::foundation-model/* to at least limit to foundation models.
Secrets Manager reads happen at every Lambda invocation. For high-throughput agents this adds latency and cost. Cache the secret in the Lambda's global scope with a TTL if invocations are frequent — just don't cache indefinitely or token rotation won't take effect.

The Knowledge Base gave the LLM a read-only window into your documentation. The agent and its Lambda tools give it hands: it can fetch, create, and notify across your tooling stack in a single invocation. The infrastructure for it is one Terraform file, four Lambda deployments, and an IAM role with three statements.

Don't Let Your LLM Wing It: Building a Knowledge Base That Actually Knows Things

hugolesta — Tue, 30 Jun 2026 21:08:46 +0000

Every team eventually asks the same question: "can we make the LLM answer questions about our docs instead of hallucinating something plausible?" The answer is retrieval-augmented generation (RAG), and the unglamorous truth is that RAG is 10% prompt engineering and 90% plumbing — object storage, a vector index, an embedding model, and a pipeline that keeps all three in sync whenever someone edits a markdown file.

This is the plumbing. A Bedrock Knowledge Base backed by Aurora PostgreSQL with pgvector, provisioned entirely in Terraform, synced automatically from a Git repo via GitHub Actions. No notebooks, no manual "let me re-upload the docs" Tuesdays.

The architecture

flowchart TD
    A["Git repository — knowledge-base/*.md"] -->|"push to main or workflow_dispatch"| B["GitHub Actions — sync-knowledge-base.yml"]
    B -->|uploads docs| C[("S3 bucket — KMS encrypted")]
    C -->|start ingestion| D{"Bedrock ingestion job — hierarchical chunking"}
    D -->|"Titan Embed Text v2"| E[("Aurora PostgreSQL — pgvector — HNSW + GIN indexes")]
    E -->|"Retrieve / RetrieveAndGenerate"| F["App / LLM proxy — scoped IAM role"]

Two environments — acc and prod — run this same pipeline side by side, each with its own bucket, KB, and database, gated by branch and dispatch logic in the workflow.

Provisioning the vector store

S3 bucket for ingestion

Bedrock pulls documents from S3, not the other way around, so the bucket is just a private, versioned, encrypted drop zone:

resource "aws_s3_bucket" "kb_docs" {
  bucket = "${local.cluster_name}-knowledge-base"
  tags   = local.default_tags
}

resource "aws_s3_bucket_versioning" "kb_docs" {
  bucket = aws_s3_bucket.kb_docs.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "kb_docs" {
  bucket = aws_s3_bucket.kb_docs.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "aws:kms"
    }
  }
}

resource "aws_s3_bucket_public_access_block" "kb_docs" {
  bucket                  = aws_s3_bucket.kb_docs.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

Aurora pgvector cluster

This is a separate Aurora cluster from any application database — different lifecycle, different access pattern, and you don't want a runaway ingestion job competing for connections with your app:

module "kb_db" {
  source  = "terraform-aws-modules/rds-aurora/aws"
  version = "~> 9.0"

  name              = "bedrock-kb"
  engine            = "aurora-postgresql"
  engine_version    = "16.6"
  instances         = { 1 = {} }

  serverlessv2_scaling_configuration = {
    min_capacity = 0.5
    max_capacity = 4
  }

  database_name        = "bedrock_kb"
  master_username       = "root"
  manage_master_user_password = true

  enable_http_endpoint  = true

  tags = local.default_tags
}

Bedrock's Knowledge Base service expects credentials in {"username": ..., "password": ...} JSON shape in Secrets Manager. If your Terraform module stores a bare password string (most do), mirror it into a second secret in the right shape rather than fighting the module:

resource "aws_secretsmanager_secret" "kb_db_bedrock" {
  name                    = "bedrock-kb-db-credentials"
  recovery_window_in_days = 0
  tags                    = local.default_tags
}

resource "aws_secretsmanager_secret_version" "kb_db_bedrock" {
  secret_id = aws_secretsmanager_secret.kb_db_bedrock.id
  secret_string = jsonencode({
    username = "root"
    password = module.kb_db.cluster_master_password
  })
}

IAM: the Bedrock service role

Bedrock's Knowledge Base needs to assume a role that can read from S3, describe and execute statements against Aurora via the Data API, read the credentials secret, and invoke the embedding model:

data "aws_iam_policy_document" "bedrock_kb_assume_role" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["bedrock.amazonaws.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "aws:SourceAccount"
      values   = [local.aws_account_id]
    }
  }
}

data "aws_iam_policy_document" "bedrock_kb" {
  statement {
    sid       = "S3Read"
    effect    = "Allow"
    actions   = ["s3:GetObject", "s3:ListBucket"]
    resources = [aws_s3_bucket.kb_docs.arn, "${aws_s3_bucket.kb_docs.arn}/*"]
  }

  statement {
    sid       = "RDSDataApi"
    effect    = "Allow"
    actions   = ["rds-data:BatchExecuteStatement", "rds-data:ExecuteStatement"]
    resources = [module.kb_db.cluster_arn]
  }

  statement {
    sid       = "SecretsManagerRead"
    effect    = "Allow"
    actions   = ["secretsmanager:GetSecretValue"]
    resources = [aws_secretsmanager_secret.kb_db_bedrock.arn]
  }

  statement {
    sid       = "BedrockEmbeddings"
    effect    = "Allow"
    actions   = ["bedrock:InvokeModel"]
    resources = ["arn:aws:bedrock:eu-west-1::foundation-model/amazon.titan-embed-text-v2:0"]
  }
}

resource "aws_iam_role" "bedrock_kb" {
  name               = "${local.cluster_name}-bedrock-kb"
  assume_role_policy = data.aws_iam_policy_document.bedrock_kb_assume_role.json
}

resource "aws_iam_role_policy" "bedrock_kb" {
  role   = aws_iam_role.bedrock_kb.id
  policy = data.aws_iam_policy_document.bedrock_kb.json
}

The Knowledge Base resource itself

resource "aws_bedrockagent_knowledge_base" "main" {
  name     = "${local.cluster_name}-knowledge-base"
  role_arn = aws_iam_role.bedrock_kb.arn

  knowledge_base_configuration {
    type = "VECTOR"
    vector_knowledge_base_configuration {
      embedding_model_arn = "arn:aws:bedrock:eu-west-1::foundation-model/amazon.titan-embed-text-v2:0"
    }
  }

  storage_configuration {
    type = "RDS"
    rds_configuration {
      resource_arn           = module.kb_db.cluster_arn
      credentials_secret_arn = aws_secretsmanager_secret.kb_db_bedrock.arn
      database_name          = "bedrock_kb"
      table_name              = "bedrock_integration.bedrock_kb"
      field_mapping {
        primary_key_field = "id"
        vector_field      = "embedding"
        text_field        = "chunks"
        metadata_field    = "metadata"
      }
    }
  }

  depends_on = [module.kb_db]
}

resource "aws_bedrockagent_data_source" "s3" {
  knowledge_base_id = aws_bedrockagent_knowledge_base.main.id
  name              = "${local.cluster_name}-knowledge-base-s3"

  data_source_configuration {
    type = "S3"
    s3_configuration {
      bucket_arn = aws_s3_bucket.kb_docs.arn
    }
  }

  vector_ingestion_configuration {
    chunking_configuration {
      chunking_strategy = "HIERARCHICAL"
      hierarchical_chunking_configuration {
        level_configuration {
          max_tokens = 1500
        }
        level_configuration {
          max_tokens = 300
        }
        overlap_tokens = 60
      }
    }
  }
}

Hierarchical chunking is worth calling out: it splits documents into large 1500-token "parent" chunks and smaller 300-token "child" chunks with a 60-token overlap. Retrieval matches on the precise child chunk but can return the broader parent context — better recall on long documents than flat fixed-size chunking, at the cost of slightly more complex ingestion.

One-time schema bootstrap

aws_bedrockagent_knowledge_base expects the target table to already exist with the right columns and indexes — Terraform won't create the pgvector extension or table for you. This is a one-time psql job against the Aurora endpoint:

CREATE EXTENSION IF NOT EXISTS vector;
CREATE SCHEMA IF NOT EXISTS bedrock_integration;

CREATE TABLE IF NOT EXISTS bedrock_integration.bedrock_kb (
  id        UUID PRIMARY KEY,
  embedding vector(1024),
  chunks    TEXT,
  metadata  JSON
);

CREATE INDEX IF NOT EXISTS bedrock_kb_embedding_idx
  ON bedrock_integration.bedrock_kb
  USING hnsw (embedding vector_cosine_ops);

CREATE INDEX IF NOT EXISTS bedrock_kb_chunks_idx
  ON bedrock_integration.bedrock_kb
  USING gin (to_tsvector('simple', chunks));

The HNSW index handles approximate nearest-neighbor search on the embedding vector; the GIN index on chunks enables hybrid search (keyword + semantic) if you ever want it. vector(1024) matches Titan Embed Text v2's output dimensionality — if you swap embedding models later, the column width has to match or ingestion fails outright.

Publishing references via SSM

resource "aws_ssm_parameter" "kb_id" {
  name  = "/${local.env}/knowledge-base/kb_id"
  type  = "String"
  value = aws_bedrockagent_knowledge_base.main.id
}

resource "aws_ssm_parameter" "kb_bucket" {
  name  = "/${local.env}/knowledge-base/bucket"
  type  = "String"
  value = aws_s3_bucket.kb_docs.bucket
}

Apps and CI pipelines read these instead of hardcoding ARNs — when you rebuild the KB in a new account or region, nothing downstream needs a code change.

Syncing docs automatically with GitHub Actions

Provisioning is the easy part. The actual day-to-day value comes from never having to think about ingestion again. Drop a markdown file in a knowledge-base/ folder, push, and it's searchable within minutes.

name: Sync Knowledge Base

on:
  workflow_dispatch:
    inputs:
      environment:
        type: choice
        description: Environment to sync knowledge base
        options:
          - acc
          - prod
        required: true
  push:
    branches:
      - main
    paths:
      - 'knowledge-base/**'

jobs:
  sync-kb-acc:
    if: >
      github.event_name == 'workflow_dispatch' && github.event.inputs.environment == 'acc' ||
      github.event_name == 'push'
    secrets: inherit
    uses: your-org/your-pipelines-repo/.github/workflows/sync-bedrock-knowledge-base.yml@v2
    with:
      environment: acc
      knowledge-base-id: ${{ vars.BEDROCK_KB_ID_ACC }}
      data-source-id: ${{ vars.BEDROCK_KB_DATA_SOURCE_ID_ACC }}
      bucket-name: ${{ vars.BEDROCK_KB_BUCKET_ACC }}
      bucket-prefix: knowledge-base
      source-dir: knowledge-base

  sync-kb-prod:
    if: github.event_name == 'workflow_dispatch' && github.event.inputs.environment == 'prod'
    secrets: inherit
    uses: your-org/your-pipelines-repo/.github/workflows/sync-bedrock-knowledge-base.yml@v2
    with:
      environment: prod
      knowledge-base-id: ${{ vars.BEDROCK_KB_ID_PROD }}
      data-source-id: ${{ vars.BEDROCK_KB_DATA_SOURCE_ID_PROD }}
      bucket-name: ${{ vars.BEDROCK_KB_BUCKET_PROD }}
      bucket-prefix: knowledge-base
      source-dir: knowledge-base

The gating logic is the whole trick here:

push to main, path-filtered to `knowledge-base/** — only fires the acc` job. Routine doc edits land in the acc environment automatically, with zero manual steps.
workflow_dispatch with environment: acc — also runs the acc job. Useful for re-triggering a sync without a new commit (e.g. after fixing a broken IAM policy).
workflow_dispatch with environment: prod — the only path that touches prod. Promotion to production is always a deliberate, manual action, never a side effect of a push.

Both jobs delegate to the same reusable workflow (sync-bedrock-knowledge-base.yml@v2), parameterized per environment. The reusable workflow does the actual work: sync the source-dir to the S3 bucket-name under bucket-prefix, then call StartIngestionJob against data-source-id. Centralizing that logic in one reusable workflow means every team adopting this pattern gets the same sync behavior — and a fix to the sync logic ships everywhere at once instead of needing fifteen copy-pasted workflow files updated individually.

Field notes

The Data API requires enable_http_endpoint = true on the Aurora cluster. Without it, Bedrock's rds-data:ExecuteStatement calls fail with a confusing connectivity error that has nothing to do with security groups — you'll waste an hour checking VPC routing before you find this.
Vector dimension mismatches fail silently at the wrong layer. If vector(1024) doesn't match your embedding model's output size, the table creation succeeds, the Knowledge Base resource creates fine, and ingestion just fails per-document. Check the embedding model's dimensionality before writing the DDL, not after.
Hierarchical chunking is the right default for prose-heavy docs, but if your knowledge base is mostly short, structured files (FAQs, glossaries), flat fixed-size chunking is simpler to reason about and debug.
Don't let push-triggered syncs touch prod. It's tempting to make prod sync automatically on a release tag, but a bad doc — wrong numbers, stale instructions — propagating into a production RAG pipeline with no human in the loop is a worse failure mode than a slightly stale prod KB.
The acc/prod split needs two of everything — bucket, KB, data source, Aurora cluster — not just two sets of IAM permissions on shared resources. It costs more, but it means a bad ingestion config or chunking change gets caught in acc before it can corrupt the index your production LLM proxy actually queries.

Closing

None of this is exotic — it's a bucket, a Postgres extension, two indexes, and a YAML file with an if: condition. That's the whole trick to "production RAG": treat the knowledge base like any other piece of infrastructure, version it, gate promotion to prod behind a manual step, and let the boring CI pipeline do the boring sync work. Your LLM stops winging it, and you stop being the person who manually re-uploads PDFs every time someone asks why the bot doesn't know about last week's runbook update.

How to Upgrade EKS 1.32: Making the Switch from bootstrap.sh to nodeadm

hugolesta — Tue, 28 Oct 2025 17:10:23 +0000

Did you know staying on a deprecated version of Kubernetes in EKS can cost you six times more? At $0.60 per hour instead of the standard $0.10 per hour, you could end up paying nearly $500 per month just for running outdated clusters.

However, the EKS upgrade from version 1.31 to 1.32 isn't just about avoiding extended support fees—it introduces one of the biggest under-the-hood changes in recent EKS history. Specifically, the traditional bootstrap.sh script used for years to configure worker nodes is now replaced by a new tool called nodeadm.

This architectural shift coincides with another critical change: after November 26, 2025, Amazon EKS will no longer publish EKS-optimized Amazon Linux 2 (AL2) AMIs. Furthermore, Kubernetes 1.32 will be the final version with AL2 AMI support, while version 1.33 onwards will only support Amazon Linux 2023 (AL2023) and Bottlerocket based AMIs.

If your clusters currently run EKS 1.31 or earlier on Amazon Linux 2 AMIs, upgrading to 1.32 will break your node initialization unless you adapt your user-data scripts and switch to AL2023. Throughout this article, we'll walk through exactly what changes between versions 1.31 and 1.32, why nodeadm is now required, and how to rewrite your user-data and Terraform templates to ensure a smooth transition.

Why EKS 1.32 Requires a New Bootstrap Approach

Amazon's evolution of EKS introduces significant architectural changes with version 1.32. The most critical change affects how worker nodes bootstrap and join your clusters, requiring careful attention during upgrades.

End of Support for bootstrap.sh in AL2023

EKS 1.32 marks a pivotal shift as it's the final version for which Amazon will release Amazon Linux 2 (AL2) AMIs. Starting with Kubernetes 1.32, AL2023 introduces a completely different node initialization process that abandons the traditional /etc/eks/bootstrap.sh script. This script has been the foundation of EKS node bootstrapping since the service launched, but is now replaced entirely in the AL2023 operating system. The familiar bash-based bootstrap approach that many DevOps teams have built automation around is completely absent in the new OS version.

nodeadm as the New Default Bootstrap Tool

AL2023 replaces bootstrap.sh with nodeadm, a tool that uses a YAML configuration schema. Unlike the previous approach where metadata was discovered automatically through the Amazon EKS DescribeCluster API call, nodeadm requires explicit provision of cluster information. This fundamental change means you must now specify three critical parameters that were previously auto-discovered:

apiServerEndpoint
certificateAuthority
service CIDR

Additionally, the format for applying parameters to kubelet has changed. Previously accomplished with --kubelet-extra-args, node customization now requires using NodeConfigSpec. This shift aims to reduce API throttling risks during large-scale node deployments.

Impact on Existing AL2-Based Clusters

The elimination of bootstrap.sh creates immediate backward compatibility issues. When upgrading an EKS cluster to version 1.32 while still using AL2-based node groups, nodes will fail to join the cluster. Any automation depending on bootstrap.sh will break, as files like /etc/eks/bootstrap.sh and /etc/eks/eni-max-pods.txt no longer exist.

For organizations with self-managed node groups or custom AMI configurations, this requires extracting and explicitly providing cluster metadata that was formerly obtained automatically. Consequently, deployment scripts, Terraform modules, and CloudFormation templates must be rewritten to align with the new declarative approach before successfully migrating to EKS 1.32.

Preparing for the Migration to nodeadm

Before migrating to EKS 1.32 with nodeadm, careful preparation is essential to ensure a smooth transition from the traditional bootstrap approach to the new paradigm.

Identifying Affected Clusters Running AL2

After November 26, 2025, AWS will end support for EKS AL2-optimized AMIs. Kubernetes version 1.32 represents the final release where Amazon EKS will provide AL2 AMIs. This deadline necessitates prompt action, especially for organizations with multiple clusters.

To identify affected clusters:

Check the AMI type for each node group using:

aws eks describe-nodegroup --cluster-name <cluster-name> --nodegroup-name <nodegroup-name>

Examine existing user-data scripts that reference /etc/eks/bootstrap.sh, which won't exist in AL2023.

Choosing Between AL2023 and Bottlerocket AMIs

Upon identifying clusters requiring migration, you must decide between AL2023 and Bottlerocket as your future node operating system.

AL2023 offers several advantages over AL2:

Secure-by-default approach with preconfigured security policies
SELinux in permissive mode
IMDSv2-only mode enabled by default
Optimized boot times and improved package management

Alternatively, Bottlerocket provides:

Purpose-built container-optimized design with minimal attack surface
Enhanced security with read-only file systems
Automatic updates
Improved compliance with security standards like CIS benchmarks

Choose AL2023 when you need significant customizations with direct OS-level access. Opt for Bottlerocket if you prefer a container-native approach with minimal node customization.

Scanning for Deprecated APIs with kubent or pluto

Prior to upgrading, scan for deprecated APIs that might break during the transition. Kubernetes frequently removes beta APIs with each new version, potentially disrupting your workloads.

The kube-no-trouble tool (kubent) efficiently identifies resources using deprecated APIs:

kubent

This scans all accessible namespaces and lists APIs that will be deprecated compared to your current Kubernetes version. For clusters with hundreds of applications across multiple namespaces, this tool proves invaluable in detecting potential upgrade issues beforehand.

Backing Up Cluster State and Node Configurations

A comprehensive migration plan should always include thorough backups. Given that you cannot downgrade an EKS cluster after upgrading, backups become crucial.

Recommended backup steps include:

Taking etcd snapshots for core Kubernetes data
Backing up cluster configuration using tools like Velero
Documenting node-specific configurations, especially custom user-data scripts
Preserving IAM role configurations that will need migration to the new nodeadm format

Additionally, establish a documented rollback procedure with well-defined testing protocols before proceeding with the upgrade.

Implementing nodeadm and AL2023

Image Source: AWS Documentation

Debugging and Validating the Upgrade

After implementing nodeadm and upgrading to EKS 1.32, troubleshooting becomes essential as new components may not function perfectly on the first try. The migration introduces different debugging approaches that we need to master.

Common nodeadm Errors and How to Fix Them

The nodeadm debug command serves as our first line of defense when troubleshooting unhealthy or misconfigured nodes. It validates critical requirements including network access to AWS APIs, credentials for the IAM role, connectivity to the EKS Kubernetes API endpoint, and authentication with the EKS cluster.

nodeadm debug -c file://nodeConfig.yaml

For configuration validation before implementation, the nodeadm config check command proves invaluable:

nodeadm config check -c file://nodeConfig.yaml

Most permission-related issues arise from the Hybrid Nodes IAM role missing the necessary eks:DescribeCluster action. Other common errors include network connectivity problems, incorrect node IP configuration, and timeout issues which can be remedied by extending timeouts:

nodeadm install K8S_VERSION --credential-provider CREDS_PROVIDER --timeout 20m0s

Verifying kubelet and containerd Startup Logs

Examining kubelet logs offers visibility into node initialization problems. For AL2023 nodes, we can use:

systemctl status kubelet
journalctl -u kubelet -o cat

Moreover, checking the status helps verify successful restarts after upgrades. For deeper troubleshooting, we can connect to the node using SSH or kubectl debug and inspect the logs. My personal preference is using AWS Systems Manager Session Manager to enhance security, avoiding opening SSH ports:

chroot /host journalctl -u kubelet -o cat

Checking Node Readiness and Taints

To verify node status after migration, we use standard kubectl commands:

kubectl get nodes -o wide
kubectl describe node NODE_NAME

The STATUS column should display "Ready" for all nodes with the updated version number visible. Untainted nodes are essential for workload scheduling. Furthermore, checking events related to nodes can reveal issues with node registration or initialization.

Validating Add-on Compatibility Post-Upgrade

Post-upgrade add-on validation requires checking deployment versions and ensuring pods are running correctly. For example, to verify the CoreDNS version:

kubectl describe deployment coredns -n kube-system | grep Image | cut -d ':' -f 3

To inspect add-on logs for errors:

kubectl logs -n kube-system -l k8s-app=kube-dns

For networking add-ons like Amazon VPC CNI, we should create test pods to validate IP assignment. Additionally, testing CoreDNS functionality using tools like nslookup ensures proper DNS resolution. Finally, checking if the number of replicas equals the number of nodes for daemonset add-ons like vpc-cni and kube-proxy confirms proper deployment.

Conclusion

Migrating to EKS 1.32 represents a significant paradigm shift for Kubernetes operations on AWS. The transition from bootstrap.sh to nodeadm fundamentally changes how worker nodes join your clusters, requiring careful planning and execution. Additionally, the impending deprecation of Amazon Linux 2 AMIs after November 26, 2025, creates urgency for organizations to adapt their infrastructure.

Throughout this upgrade journey, you must remember that nodeadm demands explicit configuration through YAML rather than command-line arguments. This declarative approach actually offers better consistency and reproducibility for node configurations once implemented correctly. Undoubtedly, the initial migration might seem daunting, especially when rewriting existing automation scripts or Terraform modules.

The choice between AL2023 and Bottlerocket AMIs depends largely on your specific operational requirements. AL2023 provides a familiar environment with improved security features, whereas Bottlerocket offers a container-optimized approach with minimal attack surface. Regardless of your choice, both options eliminate the traditional bootstrap files and require adaptation to the new nodeadm paradigm.

Before initiating any upgrade, thorough preparation becomes essential. First, identify affected clusters running AL2, then scan for deprecated APIs, and finally back up your cluster state. After implementing the necessary changes, debugging tools like nodeadm debug and nodeadm config check help troubleshoot any issues that arise during the migration process.

The EKS 1.32 upgrade, therefore, presents both challenges and opportunities. While it requires significant changes to existing workflows, it also aligns your infrastructure with AWS's future direction. Consequently, organizations that proactively adapt their node initialization processes will avoid extended support fees and benefit from improved security and performance features of newer operating systems.

Ultimately, staying current with EKS versions not only saves operational costs but also ensures compatibility with the evolving Kubernetes ecosystem. Though this particular upgrade demands more effort than typical version bumps, the long-term benefits of embracing nodeadm and AL2023 far outweigh the initial investment required for migration.

Key Takeaways

EKS 1.32 introduces the most significant architectural change in recent history, requiring organizations to abandon the traditional bootstrap.sh script in favor of nodeadm for worker node initialization.

• Critical deadline approaching: Amazon ends AL2 AMI support on November 26, 2025, making EKS 1.32 the final version supporting Amazon Linux 2.

• Bootstrap method completely changes: nodeadm replaces bootstrap.sh and requires explicit YAML configuration instead of automatic cluster metadata discovery.

• Migration requires careful preparation: Identify AL2-based clusters, scan for deprecated APIs with kubent, and backup cluster state before upgrading.

• Choose your AMI strategy: Select between AL2023 for familiar environments with enhanced security or Bottlerocket for container-optimized minimal attack surface.

• Debug with new tools: Use nodeadm debug and nodeadm config check commands to troubleshoot configuration issues and validate node readiness.

• Avoid costly extended support: Staying on deprecated Kubernetes versions costs 6x more ($0.60/hour vs $0.10/hour), potentially adding $500+ monthly per cluster.

The shift to nodeadm represents AWS's commitment to more secure, declarative infrastructure management. While the initial migration requires significant effort in rewriting automation scripts and Terraform modules, organizations that proactively adapt will benefit from improved security, performance, and long-term cost savings.

FAQs

Q1. What is the main change in EKS 1.32 that affects node initialization? EKS 1.32 replaces the traditional bootstrap.sh script with a new tool called nodeadm for worker node initialization. This change requires explicit YAML configuration instead of automatic cluster metadata discovery.

Q2. When will Amazon stop supporting Amazon Linux 2 (AL2) AMIs for EKS? Amazon will end support for EKS AL2-optimized AMIs after November 26, 2025. Kubernetes version 1.32 is the final release where Amazon EKS will provide AL2 AMIs.

Q3. What are the alternatives to Amazon Linux 2 for EKS nodes? The main alternatives are Amazon Linux 2023 (AL2023) and Bottlerocket AMIs. AL2023 offers improved security features and a familiar environment, while Bottlerocket provides a container-optimized design with a minimal attack surface.

Q4. How can I identify and fix common nodeadm errors? You can use the 'nodeadm debug' command to validate critical requirements and the 'nodeadm config check' command to validate configurations. Common issues include missing IAM permissions, network connectivity problems, and incorrect node IP configurations.

Q5. What are the cost implications of staying on a deprecated version of EKS? Running a deprecated version of EKS can cost six times more than the standard rate. Instead of $0.10 per hour, you could end up paying $0.60 per hour, potentially adding over $500 per month for each outdated cluster.

The $10,000 Label: How We Used Go, Clean Architecture, and AWS to Build a FinOps-Driven Cloud Tagging Engine 🏷️

hugolesta — Mon, 29 Sep 2025 16:01:54 +0000

Why Consistent Tagging is Your Company’s Most Underrated FinOps Tool:

The Business Problem: Imagine your cloud bill is a massive corporate expense report. Without proper tagging—simple key-value labels like project: crm-migration or owner: finance-team —you're paying thousands every month for line items labeled simply “Server.” This isn't just an accounting headache; it's a direct threat to cost control and security.
Cost Bloat: Orphaned or forgotten AWS resources (Shadow IT) continue to generate costs because no one is accountable for terminating them.
Billing Disputes: Finance teams struggle to attribute costs accurately, leading to friction and delayed chargebacks.
Security Risks: Unmanaged resources often fall outside compliance or patch cycles.

We decided to solve this with sys-tag-manager, a powerful, automated system built in Golang that acts as our centralized "Cloud Label Printer," ensuring every AWS resource is correctly accounted for, compliant, and cost-trackable.

What Is a Tagging Strategy?

A tagging strategy is a structured approach to applying metadata (tags) to cloud resources. Tags are simple key–value pairs like:

owner: finance-team
project: crm-migration
environment: production

On their own, tags look trivial. But when applied consistently across an entire cloud estate, they form the backbone of organization, governance, and cost management.

A tagging strategy defines:

Which tags are required (e.g. owner, project, environment).
How tags should be formatted (naming conventions, lowercase vs camelCase, separators).
When tags should be applied (at creation time vs automated correction).
Who is responsible for maintaining them?

For further guidance, consult the AWS documentation: Best Practices for Tagging AWS Resources

Technical Foundation: Go, Clean Architecture, and AWS Cost Savings

The Go Advantage: Performance Meets FinOps:

While prototyping in Python is quick, a core, mission-critical tool demands performance and reliability. We chose Golang for sys-tag-manager because:

Cost-Efficient Execution: Go's minimal memory footprint and extremely fast startup time are critical when running as AWS Lambda functions or Kubernetes CronJobs. This translates directly into lower AWS compute costs (less time billed for execution) compared to resource-heavier languages.
Reliability: Static typing and robust concurrency ensure the system can efficiently handle a rapidly growing number of AWS API calls without failure, ensuring 100% compliance coverage.

Clean Architecture: Decoupling Logic from the AWS SDK

To ensure the system remains maintainable as our cloud estate scales, we invested in Clean Architecture. This strategic separation is key to our long-term technical debt reduction:

Domain (Core Business Logic): Pure, independent rules (e.g., "A resource is compliant if it has the required tags: owner and project"). This is highly testable and knows nothing about AWS.
Use Cases (Application Logic): Defines the "what" (e.g., "Check compliance and apply tags").
Adapters (AWS Implementation): Isolated logic that interacts directly with specific AWS SDK services (EC2Tagger, RDSTagger). This prevents vendor lock-in and allows us to add new services (S3, Lambda) without touching the core business rules.

The Compliance Engine: Terraform, SSM, and Metadata Management

The Discovery Layer: Leveraging AWS Resource Explorer

Before sys-tag-manager can fix untagged resources, it must efficiently find them across accounts and Regions. We achieved this by using AWS Resource Explorer as our primary discovery and inventory layer.

Instead of writing complex API calls to list every resource type across every Region, sys-tag-manager utilizes Resource Explorer's unified search capabilities.

The Workflow:

Discovery: sys-tag-manager uses the Resource Explorer API to query the entire cloud estate for resources that are missing required tags (e.g., tag:owner is absent).
Validation: For each untagged resource found, sys-tag-manager checks its metadata against the centralized, correct rules stored in AWS SSM Parameter Store.
Correction: The system then applies the right tags, assigning the resource to the correct owner or project, ensuring immediate compliance.

This design significantly streamlined the core tagging loop, ensuring we are not just efficient in applying tags (Golang), but also efficient in finding them (Resource Explorer), saving API call costs and latency.

Harnessing Shared Infrastructure: The Fallback Mechanism

While our primary goal is to enforce resource-specific tagging, we recognized that some resources are "Shared Infrastructure" (e.g., core networking components, centralized security groups) that don't belong to a single owner. Addressing this was a critical design challenge.

Our solution was a smart fallback mechanism:

The tagging engine first checks for the required resource-specific tags.
If the tags are missing, it then checks a predefined list of AWS ARNs (Amazon Resource Names) that are designated as shared infrastructure.
If an ARN matches, the system applies a generic, shared set of tags (e.g., owner: platform-team, charge-code: shared-infra) instead of flagging it as non-compliant. This prevents false positives and ensures accurate cost attribution for common resources.

The Terraform + SSM Parameter Store Synergy

The true power of sys-tag-manager lies in its ability to dynamically enforce tagging rules based on centralized, auditable metadata.

Centralized Rule Source: We leverage AWS Systems Manager (SSM) Parameter Store to store the required tag keys, values, and compliance rules.
Terraform as the Single Source of Truth: The compliance rules in SSM are managed exclusively by Terraform. This means:
1. Immutability: Every rule change is tracked, reviewed, and deployed via a GitOps workflow.
2. Automation: When a new project is created in Terraform, the required tag values for that project are automatically pushed to SSM, immediately making those tags valid for the tag manager checks.

This integration ensures that the FinOps rules are always aligned with the deployed infrastructure definitions, creating a clean, traceable metadata loop.

Business Impact: Quantifiable Results for FinOps

Metric	Before sys-tag-manager	After sys-tag-manager	Value Proposition
Compliance Time	Weeks (Manual Audits)	Minutes (Automated Correction)	Faster cost allocation & reduced risk.
Orphaned Resources	~12% (Estimate)	<1%	Direct savings on wasted AWS spend.
FinOps Accuracy	High Friction/Disputes	High Trust/Automated Showback	Enables accurate, automated chargeback.

By investing in this system, we have fundamentally shifted from reactive tag auditing to proactive, automated compliance enforcement. This not only saves engineering hours but directly enables our finance team to confidently utilize AWS Cost and Usage Reports (CUR) for accurate showback and chargeback, making our entire cloud operation more accountable and financially efficient.

Wrapping Up: Compliance as Code, Savings as the Result

sys-tag-manager is more than just an automation script; it's the enforcement layer for our FinOps and security policies, ensuring that our cloud environment is self-healing and financially accountable.

By embracing Golang for performance, Clean Architecture for maintainability, and the Terraform + SSM synergy for centralized metadata management, we've transformed tagging from a manual burden into an automated, cost-saving asset. This shift has given us the confidence that every dollar spent on AWS is trackable, auditable, and directly attributed to a business owner or project.

The result is a culture of Compliance as Code where engineers can focus on feature delivery, knowing that the foundational governance—the tagging—is handled automatically and efficiently by sys-tag-manager.

Let's Keep the Conversation Going 🗣️
We've focused on the technical core of sys-tag-manager, but the true organizational victory was how we scaled this system across dozens of teams without friction.

Would you be interested in learning more about how we automated the communication of compliance status and tagging fixes to developers, FinOps, and management?

I'd love to hear your thoughts! What's the biggest tagging challenge you face in your organization? Share your experiences and suggestions in the comments below!

Feel free to connect with me on LinkedIn to discuss our approach to automated communication and team onboarding:

LinkedIn: Hugo Lesta

GitHub: Hugo Lesta's GitHub

DEV Community: hugolesta

KMCP Turns MCP Servers Into Boring Infrastructure

The architecture

Installing the controller

An MCP server as 20 lines of YAML

Routing: internal, and deliberately so

Shipping it through GitOps, not a Makefile

The part that actually changes delivery time

The credential belongs to the caller, not the server

Field notes

Closing

Terraform Enterprise on EKS: The Parts Nobody Documents

Architecture

Image: AMD64 only

The database needs the admin user, briefly

TLS: the certificate header that isn't CERTIFICATE

Traefik: HTTPS-only backend, no plain HTTP path

The SSO trap that skips your first site admin

SAML team sync doesn't create the team for you

No inbound webhooks, no VCS auto-apply

Module registry publish needs its own VCS scope — and a second, invisible SSO check

Field notes

Your AI Platform Needs a Gateway, Not a Credit Card

The pattern you've seen before

What the gateway actually does

What self-hosting looks like in practice

The latency argument doesn't hold here

The enforcement problem

Field notes

Give Your LLM Hands: Bedrock Agents, Lambda Tools, and the MCP Pattern in Terraform

Architecture

The Lambda tools pattern

Secrets handling

Lambda module + permission (PagerDuty as example)

The Agent resource

Agent IAM

Action groups with function schemas

Attaching the Knowledge Base

Agent alias and SSM parameters

Field notes

Don't Let Your LLM Wing It: Building a Knowledge Base That Actually Knows Things

The architecture

Provisioning the vector store

S3 bucket for ingestion

Aurora pgvector cluster

IAM: the Bedrock service role

The Knowledge Base resource itself

One-time schema bootstrap

Publishing references via SSM

Syncing docs automatically with GitHub Actions

Field notes

Closing

How to Upgrade EKS 1.32: Making the Switch from bootstrap.sh to nodeadm

Why EKS 1.32 Requires a New Bootstrap Approach

End of Support for bootstrap.sh in AL2023

nodeadm as the New Default Bootstrap Tool

Impact on Existing AL2-Based Clusters

Preparing for the Migration to nodeadm

Identifying Affected Clusters Running AL2

Choosing Between AL2023 and Bottlerocket AMIs

Scanning for Deprecated APIs with kubent or pluto

Backing Up Cluster State and Node Configurations

Implementing nodeadm and AL2023

Debugging and Validating the Upgrade

Common nodeadm Errors and How to Fix Them

Verifying kubelet and containerd Startup Logs

Checking Node Readiness and Taints

Validating Add-on Compatibility Post-Upgrade

Conclusion

Key Takeaways

FAQs

The $10,000 Label: How We Used Go, Clean Architecture, and AWS to Build a FinOps-Driven Cloud Tagging Engine 🏷️

TLS: the certificate header that isn't `CERTIFICATE`