DEV Community: hugolesta

Your AI Platform Needs a Gateway, Not a Credit Card

hugolesta — Thu, 09 Jul 2026 12:28:23 +0000

A team in your org hits Anthropic during a production incident. Anthropic is down. The team is blocked for 20 minutes while someone scrambles to find an alternative API key, change a config, and redeploy. There's no fallback. There was never a plan for this.

Somewhere else, another team has quietly started billing their LLM usage to a personal credit card because procurement takes six weeks. A third team has hardcoded gpt-4o everywhere and doesn't know there's a cheaper model that handles their workload fine.

This isn't a story about AI risk. It's a story about missing infrastructure.

The pattern you've seen before

When teams first started running services in the cloud, the ones who moved fast didn't start with a load balancer and a CDN. They spun up an EC2 instance, pointed DNS at it, and shipped. That worked — until it didn't. Until traffic spiked, until the instance went down, until three teams had three different DNS setups and nobody knew which was canonical.

AI adoption is following the same curve, faster. Every team that can call an API is calling one. The platforms don't have a control plane yet. The control plane is somebody's laptop with a hardcoded key.

An AI gateway is what closes that gap. It's not a product pitch — it's the same thing you built for HTTP traffic: a single ingress point with routing, fallback, observability, and policy enforcement. The LLM is the backend. The gateway is the infrastructure around it.

What the gateway actually does

flowchart TD
    A["Developer — MCP / SDK / REST client"] --> G["AI Gateway — single stable endpoint"]
    B["Business user — internal tooling"] --> G
    G -->|"primary"| P1["Anthropic Claude"]
    G -->|"fallback on 5xx / timeout"| P2["AWS Bedrock — Claude / Titan"]
    G -->|"cost-optimised route"| P3["OpenAI GPT-4o mini"]
    G --> OBS["Observability — spend per team — request logs — latency"]
    G --> POL["Policy — rate limits — budget caps — PII filter"]

Failover. When a provider goes down, the gateway retries against the next configured backend — same request, different model, no client code change. This is the feature that looks optional until someone spends 20 minutes blocked mid-incident.

Cost routing. Not every call needs a frontier model. Summarisation, classification, short completions — route those to something cheaper. The gateway makes this a config change, not a refactor.

Spend visibility. Every request is tagged by team, application, and model. You see where the money goes before finance calls you about it. You can set hard budget caps per team so nobody accidentally runs a batch job against claude-opus all night.

Provider abstraction. When a new model ships — or when your organisation wants to experiment with providers you don't use today — teams swap the gateway config, not their application code. The SDK they're using doesn't change.

Audit logging. In any regulated environment, knowing who sent what prompt to which model isn't a nice-to-have. It's table stakes for governance. The gateway is the right place to capture that, not each individual application.

What self-hosting looks like in practice

The two tools worth knowing are LiteLLM and Bifrost. Both expose an OpenAI-compatible API, which means existing applications need zero changes to the SDK call — only the base URL and API key change.

LiteLLM is the more battle-tested option. It runs as a Docker container, connects to a Postgres database for request logs and virtual key management, and supports every major provider out of the box. A minimal deployment on Kubernetes looks like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: litellm
spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: litellm
          image: ghcr.io/berriai/litellm:main-latest
          args:
            - "--config"
            - "/etc/litellm/config.yaml"
          env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: litellm-secrets
                  key: database-url
          volumeMounts:
            - name: config
              mountPath: /etc/litellm
      volumes:
        - name: config
          configMap:
            name: litellm-config

The config itself is where the routing logic lives:

model_list:
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: claude-sonnet
    litellm_params:
      model: bedrock/us.anthropic.claude-sonnet-4-5-20251001-v2:0
      aws_region_name: eu-west-1

router_settings:
  routing_strategy: least-busy
  num_retries: 2
  fallbacks:
    - claude-sonnet:
        - claude-sonnet

litellm_settings:
  success_callback: ["langfuse"]
  failure_callback: ["langfuse"]

Two entries for the same model_name is the whole failover mechanism. LiteLLM tries the first, fails, retries against the second. From the client's perspective, the call either succeeds or returns an error after both backends fail — never a 20-minute block.

The latency argument doesn't hold here

The pushback you'll hear is latency. "We can't add a proxy hop to every LLM call."

That argument makes sense for traditional APIs where you're measuring response times in milliseconds. It doesn't apply to LLMs. Time-to-first-token on a typical Claude call is 300–800ms. A gateway adds 5–15ms. That's noise, not overhead. The conversation about "proxy latency" is a conversation that predates LLMs and doesn't transfer.

The actual latency concern worth having is gateway availability — if the gateway goes down, everything goes down. The answer is the same as any other critical path: multiple replicas, a readiness probe, a PodDisruptionBudget, and a runbook. Nothing exotic.

The enforcement problem

One thing the gateway doesn't solve on its own: casual users who bypass it entirely. If teams can buy a Claude.ai subscription for $20/month and skip the gateway, many will — especially if the gateway requires a non-trivial onboarding step.

This is a governance problem, not a technical one. The gateway has to be the path of least resistance, not just the official path. That means:

API keys that work without friction (virtual keys in LiteLLM's admin UI take 30 seconds to issue)
A clear internal page that says "here is your endpoint and here is your key"
A procurement policy that routes all model spend through the central key, not individual accounts

The technical infrastructure is the easier part. Getting adoption is the harder part, and that's a platform team problem from day one, not something you solve after the gateway is running.

Field notes

Run the gateway in at least two replicas from day one. A single-replica proxy is a single point of failure you introduced yourself. Two replicas with a PDB that disallows simultaneous evictions cost almost nothing and remove the worst failure mode.
LiteLLM's virtual key system is the right unit of cost tracking. One virtual key per team, budget cap attached, spend dashboard out of the box. Don't try to build this yourself — it's already there.
Failover works at the gateway level, not the application level. Stop trying to handle retries in application code when you have a gateway. Put the retry and fallback logic in one place and let every team benefit from it automatically.
Tag every request with metadata. LiteLLM forwards custom metadata to your observability backend. Use it to tag team, environment, and use case from the first day — you'll want that granularity when someone asks why the bill doubled.
SDK compatibility means the migration is low-friction. If you start with LiteLLM and later want to evaluate Bifrost or something else, the migration is a base URL change for application teams. That's the right amount of coupling.

The teams that will struggle with AI costs and reliability in 12 months are the ones treating model access like a SaaS subscription — something each team manages independently. The teams that won't are the ones who put a control plane in front of it now, before the sprawl hardens into something unauditable.

The gateway isn't the exciting part of the AI platform. It's the infrastructure that makes the exciting parts sustainable.

Give Your LLM Hands: Bedrock Agents, Lambda Tools, and the MCP Pattern in Terraform

hugolesta — Fri, 03 Jul 2026 09:14:49 +0000

The Knowledge Base from the previous post gives the LLM memory. But memory is passive — the model can tell you what the runbook says, but it can't go and create the Confluence page, look up the Jira ticket, or ping Slack. For that you need an agent: a model that can reason over a goal, decide which tool to call, call it, inspect the result, and loop until the task is done.

This is what Bedrock Agents does. The model driving it is Anthropic Claude; the "tools" it invokes are Lambda functions you own; the schema that tells the model what each function accepts and returns lives in Terraform as a function_schema block. It's the Model Context Protocol pattern, just deployed as AWS-managed infrastructure instead of a local sidecar.

The agent covered here — an incident reporter — takes a PagerDuty incident ID, fetches details, searches Jira for related tickets, writes a Confluence incident page, and posts a Slack notification. Four action groups, four Lambdas, one Knowledge Base, one Terraform file.

Architecture

flowchart TD
    A["Caller — incident ID"] -->|"InvokeAgent"| B["Bedrock Agent — Claude Sonnet — eu cross-region"]
    B -->|"Retrieve context"| C[("Knowledge Base — Aurora pgvector")]
    B -->|"action group"| D["Lambda — pagerduty-tools — get_incident — list_incidents — get_incident_alerts"]
    B -->|"action group"| E["Lambda — jira-tools — search_issues — get_issue"]
    B -->|"action group"| F["Lambda — confluence-tools — create_page — get_page — update_page"]
    B -->|"action group"| G["Lambda — slack-tools — send_message — find_channel"]
    D -->|"PagerDuty API"| H["PagerDuty"]
    E -->|"Jira API"| I["Jira"]
    F -->|"Confluence API"| J["Confluence"]
    G -->|"Slack API"| K["Slack"]
    D & E & F & G -->|"tool results"| B
    B -->|"final response"| A

Each Lambda reads its API credentials from Secrets Manager at runtime — no credentials in environment variables, no secrets in Terraform state beyond the ARN.

The Lambda tools pattern

Bedrock calls this concept "action groups". I find it cleaner to think of it as MCP: the agent has a set of named functions with typed parameters, it decides which ones to call based on the task, calls them, and incorporates the responses. The Lambda is the implementation; the Terraform function_schema block is the schema the model reads.

Every tool Lambda in this setup follows the same structure:

A module call that provisions the Lambda (runtime, memory, timeout, S3 source, env vars)
An aws_lambda_permission that allows bedrock.amazonaws.com to invoke it
An aws_bedrockagent_agent_action_group that declares the function schema

The permission and the action group are separate resources — the permission is at the Lambda level, the action group is at the agent level. Both are required.

Secrets handling

Each Lambda that calls an external API gets the secret name as an environment variable, not the secret value. The Lambda's IAM role has secretsmanager:GetSecretValue on that specific secret ARN. This means:

Rotating a token is a Secrets Manager operation, not a Terraform apply
The secret value never appears in a plan output or state file
CloudWatch logs show PAGERDUTY_SECRET_NAME=pagerduty-api-token, not a token

locals {
  lambdas = {
    pagerduty  = "bedrock-agent-pagerduty-tools"
    jira       = "bedrock-agent-jira-tools"
    confluence = "bedrock-agent-confluence-tools"
    slack      = "bedrock-agent-slack-tools"
  }
}

Lambda module + permission (PagerDuty as example)

module "lambda_pagerduty_tools" {
  source  = "your-org/lambda/aws"
  version = "~> 2.15.0"

  env     = local.env
  project = local.project
  vcs     = local.vcs
  owner   = local.owner

  function_name      = local.lambdas.pagerduty
  handler            = "lambda_function.lambda_handler"
  runtime            = "python3.13"
  networking_enabled = false
  cloudwatch_enabled = true
  memory_size        = 256
  timeout            = 30

  s3_bucket = "your-artifacts-bucket"
  s3_key    = "agents/${local.lambdas.pagerduty}/${local.lambdas.pagerduty}.zip"

  env_vars = {
    PAGERDUTY_SECRET_NAME = module.pagerduty_secret.secret_name
    LOG_LEVEL             = "INFO"
  }

  depends_on = [module.pagerduty_secret]
}

resource "aws_lambda_permission" "bedrock_pagerduty_tools" {
  statement_id  = "AllowBedrockInvoke"
  action        = "lambda:InvokeFunction"
  function_name = "${local.project}-${local.lambdas.pagerduty}-${local.env}"
  principal     = "bedrock.amazonaws.com"
  source_arn    = "arn:aws:bedrock:${local.aws_region}:${local.aws_account_id}:agent/*"
  depends_on    = [module.lambda_pagerduty_tools]
}

The source_arn uses agent/* — wildcard over agents in this account and region. If you want tighter scoping, replace the wildcard with the specific agent ARN after it's created, but that creates a circular dependency you'll have to break with a two-phase apply.

The Agent resource

resource "aws_bedrockagent_agent" "incident_reporter" {
  agent_name                  = "${local.project}-incident-reporter-agent-${local.env}"
  agent_resource_role_arn     = aws_iam_role.incident_reporter_agent.arn
  foundation_model            = "eu.anthropic.claude-sonnet-4-6"
  description                 = "Orchestrates PagerDuty, Jira, Confluence, and Slack for automated incident documentation"
  idle_session_ttl_in_seconds = 600

  instruction = <<-EOT
    You are an AI Incident Reporter for the Cloud Platform team.

    When given a PagerDuty incident, complete ALL of the following steps in order:

    1. FETCH INCIDENT DETAILS
       - Use the pagerduty-tools action group to get full incident details and alerts
       - Extract: title, description, severity, affected service, timeline, assignees

    2. FIND RELATED JIRA TICKETS
       - Use the jira-tools action group to search for related tickets
       - DO NOT create new Jira tickets — only retrieve existing ones

    3. CREATE CONFLUENCE INCIDENT PAGE
       - Use the confluence-tools action group to create a new incident page
       - Include: title, severity, timeline, PagerDuty link, related Jira tickets

    4. NOTIFY SLACK
       - Use the slack-tools action group to send a message to the incident channel
       - Include: summary, severity, Confluence URL, Jira links

    5. RETURN SUMMARY
       - Provide a final summary with all links created

    Guidelines:
    - Always complete all steps — do not skip any
    - If a step fails, report the error and continue with remaining steps
    - Use the knowledge base to understand infrastructure context when relevant
  EOT

  tags = local.default_tags

  depends_on = [aws_iam_role_policy_attachment.incident_reporter_agent]
}

A few things worth noting about this resource:

foundation_model: the eu. prefix routes through AWS's cross-region inference, which lets the agent use model capacity across EU regions (eu-west-1, eu-central-1, eu-west-3) automatically. Without this prefix you're locked to a single region's available capacity. Use it if you're in a EU region.

instruction: this is the system prompt. It's what tells the model the task structure, step ordering, and guard rails. Keep it procedural and explicit — the model will follow numbered steps reliably. Vague instructions produce vague agents.

idle_session_ttl_in_seconds: how long a conversation session stays alive between calls. 600 seconds (10 minutes) is reasonable for a human-in-the-loop flow; drop it lower if you're doing fully automated invocations where sessions shouldn't accumulate.

Agent IAM

The agent's execution role needs three permissions:

data "aws_iam_policy_document" "incident_reporter_agent" {
  statement {
    sid    = "InvokeModel"
    effect = "Allow"
    actions = [
      "bedrock:InvokeModel",
      "bedrock:InvokeModelWithResponseStream",
    ]
    resources = ["*"]
  }

  statement {
    sid    = "KBRetrieve"
    effect = "Allow"
    actions = ["bedrock:Retrieve"]
    resources = [
      "arn:aws:bedrock:${local.aws_region}:${local.aws_account_id}:knowledge-base/${var.knowledge_base_id}",
    ]
  }

  statement {
    sid    = "InvokeLambdas"
    effect = "Allow"
    actions = ["lambda:InvokeFunction"]
    resources = [
      "arn:aws:lambda:${local.aws_region}:${local.aws_account_id}:function:${local.project}-${local.lambdas.pagerduty}-${local.env}",
      "arn:aws:lambda:${local.aws_region}:${local.aws_account_id}:function:${local.project}-${local.lambdas.jira}-${local.env}",
      "arn:aws:lambda:${local.aws_region}:${local.aws_account_id}:function:${local.project}-${local.lambdas.confluence}-${local.env}",
      "arn:aws:lambda:${local.aws_region}:${local.aws_account_id}:function:${local.project}-${local.lambdas.slack}-${local.env}",
    ]
  }
}

InvokeModel on * is required — Bedrock Agents uses this to invoke the foundation model internally, and the resource ARN for FM invocation isn't predictable at the time the role is written if you're using cross-region inference prefixes. You can scope it to arn:aws:bedrock:*::foundation-model/* if you want some constraint without breaking cross-region.

Action groups with function schemas

This is where the MCP analogy is most visible. The function_schema block is effectively a tool manifest: the model reads it at runtime to understand what each function is named, what parameters it expects, and what they mean.

resource "aws_bedrockagent_agent_action_group" "pagerduty_tools" {
  agent_id          = aws_bedrockagent_agent.incident_reporter.agent_id
  agent_version     = "DRAFT"
  action_group_name = "pagerduty-tools"
  description       = "PagerDuty tools: get_incident, list_incidents, get_incident_alerts"

  action_group_executor {
    lambda = "arn:aws:lambda:${local.aws_region}:${local.aws_account_id}:function:${local.project}-${local.lambdas.pagerduty}-${local.env}"
  }

  function_schema {
    member_functions {
      functions {
        name        = "get_incident"
        description = "Retrieve full details of a single PagerDuty incident by its ID"
        parameters {
          map_block_key = "incident_id"
          type          = "string"
          description   = "The PagerDuty incident ID (e.g. P1234AB)"
          required      = true
        }
      }
      functions {
        name        = "list_incidents"
        description = "List active PagerDuty incidents filtered by status and/or service"
        parameters {
          map_block_key = "statuses"
          type          = "string"
          description   = "Comma-separated statuses to filter by: triggered, acknowledged, resolved"
          required      = false
        }
        parameters {
          map_block_key = "limit"
          type          = "integer"
          description   = "Maximum number of incidents to return (default: 20)"
          required      = false
        }
      }
      functions {
        name        = "get_incident_alerts"
        description = "Get all alerts associated with a PagerDuty incident"
        parameters {
          map_block_key = "incident_id"
          type          = "string"
          description   = "The PagerDuty incident ID"
          required      = true
        }
      }
    }
  }

  depends_on = [
    module.lambda_pagerduty_tools,
    aws_lambda_permission.bedrock_pagerduty_tools,
  ]
}

The Jira action group is read-only by design — search_issues and get_issue only. This isn't enforced at the Lambda level (you could build a write-capable Lambda and only expose read functions here), but the intent is captured in both the description field and the agent's instruction. Two layers of "don't create tickets" beats one.

resource "aws_bedrockagent_agent_action_group" "jira_tools" {
  agent_id          = aws_bedrockagent_agent.incident_reporter.agent_id
  agent_version     = "DRAFT"
  action_group_name = "jira-tools"
  description       = "Jira tools (read-only): search_issues, get_issue — no ticket creation"

  action_group_executor {
    lambda = "arn:aws:lambda:${local.aws_region}:${local.aws_account_id}:function:${local.project}-${local.lambdas.jira}-${local.env}"
  }

  function_schema {
    member_functions {
      functions {
        name        = "search_issues"
        description = "Search Jira issues using JQL"
        parameters {
          map_block_key = "jql"
          type          = "string"
          description   = "JQL query string"
          required      = false
        }
        parameters {
          map_block_key = "max_results"
          type          = "integer"
          description   = "Maximum number of results to return (default: 10)"
          required      = false
        }
        parameters {
          map_block_key = "fields"
          type          = "string"
          description   = "Comma-separated list of fields to include"
          required      = false
        }
      }
      functions {
        name        = "get_issue"
        description = "Retrieve full details of a single Jira issue by its key"
        parameters {
          map_block_key = "issue_key"
          type          = "string"
          description   = "The Jira issue key (e.g. OPS-1234)"
          required      = true
        }
      }
    }
  }

  depends_on = [
    module.lambda_jira_tools,
    aws_lambda_permission.bedrock_jira_tools,
  ]
}

Attaching the Knowledge Base

One resource to connect the Knowledge Base built in the previous post:

resource "aws_bedrockagent_agent_knowledge_base_association" "incident_reporter_kb" {
  agent_id             = aws_bedrockagent_agent.incident_reporter.agent_id
  knowledge_base_id    = var.knowledge_base_id
  description          = "Internal infrastructure and platform documentation"
  knowledge_base_state = "ENABLED"
}

With this association in place, the agent automatically retrieves relevant KB chunks before deciding which tools to call. The instruction's "use the knowledge base to understand infrastructure context when relevant" is the trigger — the model decides when to retrieve, not you.

Agent alias and SSM parameters

resource "aws_bedrockagent_agent_alias" "incident_reporter" {
  agent_id         = aws_bedrockagent_agent.incident_reporter.agent_id
  agent_alias_name = "live"
  description      = "Live alias for the Incident Reporter agent"
  tags             = local.default_tags

  depends_on = [
    aws_bedrockagent_agent_action_group.pagerduty_tools,
    aws_bedrockagent_agent_action_group.jira_tools,
    aws_bedrockagent_agent_action_group.confluence_tools,
    aws_bedrockagent_agent_action_group.slack_tools,
    aws_bedrockagent_agent_knowledge_base_association.incident_reporter_kb,
  ]
}

resource "aws_ssm_parameter" "incident_reporter_agent_id" {
  name  = "/${local.env}/incident-reporter/agent_id"
  type  = "String"
  value = aws_bedrockagent_agent.incident_reporter.agent_id
}

resource "aws_ssm_parameter" "incident_reporter_agent_alias_id" {
  name  = "/${local.env}/incident-reporter/agent_alias_id"
  type  = "String"
  value = aws_bedrockagent_agent_alias.incident_reporter.agent_alias_id
}

The alias is required to invoke the agent from application code — you always invoke via agentId + agentAliasId. Publishing a named alias decouples callers from the internal version counter: when you update action groups and Bedrock creates a new draft version, you prepare a new alias version and flip it without changing any caller configuration.

The depends_on on the alias resource is load-bearing. Without it, Terraform might try to create the alias before all action groups exist, which causes Bedrock to create the alias pointing at an agent that's still missing tools.

Field notes

agent_version = "DRAFT" on all action groups. Action groups always attach to DRAFT. When you publish a new agent version (via alias routing), Bedrock snapshots the DRAFT config. If you try to attach action groups to a numbered version, the API rejects it. This trips up people who try to version-lock action groups for safety.
The function description matters more than you'd think. The model reads description fields to decide which tool to call and when. Vague descriptions ("does stuff with PagerDuty") produce agents that hallucinate which tool to invoke. Write descriptions as if you're writing API docs for a junior engineer — precise verbs, clear scope.
Parameter types are limited: string, integer, boolean, number, array. No nested objects. If your Lambda needs structured input (a list of filter criteria, a nested config), serialize it as JSON in a string parameter and document the shape in the description. Ugly but it works.
Lambda cold starts extend agent response time noticeably. Each tool call is a synchronous Lambda invocation. Four cold starts in a chain = 4× the latency penalty. Provision concurrency on the most-called tools if response time matters.
The depends_on on the alias is not optional. Terraform's parallelism will race the alias creation against action group creation without it. The apply succeeds but the agent alias captures an incomplete version with missing tools. You won't notice until runtime.
Cross-region inference (eu. prefix) changes the IAM picture. With the cross-region prefix, the model ARN the agent resolves to at runtime isn't in your account's region — it's wherever AWS routes capacity. InvokeModel on * is practically required here, or you scope to arn:aws:bedrock:*::foundation-model/* to at least limit to foundation models.
Secrets Manager reads happen at every Lambda invocation. For high-throughput agents this adds latency and cost. Cache the secret in the Lambda's global scope with a TTL if invocations are frequent — just don't cache indefinitely or token rotation won't take effect.

The Knowledge Base gave the LLM a read-only window into your documentation. The agent and its Lambda tools give it hands: it can fetch, create, and notify across your tooling stack in a single invocation. The infrastructure for it is one Terraform file, four Lambda deployments, and an IAM role with three statements.

Don't Let Your LLM Wing It: Building a Knowledge Base That Actually Knows Things

hugolesta — Tue, 30 Jun 2026 21:08:46 +0000

Every team eventually asks the same question: "can we make the LLM answer questions about our docs instead of hallucinating something plausible?" The answer is retrieval-augmented generation (RAG), and the unglamorous truth is that RAG is 10% prompt engineering and 90% plumbing — object storage, a vector index, an embedding model, and a pipeline that keeps all three in sync whenever someone edits a markdown file.

This is the plumbing. A Bedrock Knowledge Base backed by Aurora PostgreSQL with pgvector, provisioned entirely in Terraform, synced automatically from a Git repo via GitHub Actions. No notebooks, no manual "let me re-upload the docs" Tuesdays.

The architecture

flowchart TD
    A["Git repository — knowledge-base/*.md"] -->|"push to main or workflow_dispatch"| B["GitHub Actions — sync-knowledge-base.yml"]
    B -->|uploads docs| C[("S3 bucket — KMS encrypted")]
    C -->|start ingestion| D{"Bedrock ingestion job — hierarchical chunking"}
    D -->|"Titan Embed Text v2"| E[("Aurora PostgreSQL — pgvector — HNSW + GIN indexes")]
    E -->|"Retrieve / RetrieveAndGenerate"| F["App / LLM proxy — scoped IAM role"]

Two environments — acc and prod — run this same pipeline side by side, each with its own bucket, KB, and database, gated by branch and dispatch logic in the workflow.

Provisioning the vector store

S3 bucket for ingestion

Bedrock pulls documents from S3, not the other way around, so the bucket is just a private, versioned, encrypted drop zone:

resource "aws_s3_bucket" "kb_docs" {
  bucket = "${local.cluster_name}-knowledge-base"
  tags   = local.default_tags
}

resource "aws_s3_bucket_versioning" "kb_docs" {
  bucket = aws_s3_bucket.kb_docs.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "kb_docs" {
  bucket = aws_s3_bucket.kb_docs.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "aws:kms"
    }
  }
}

resource "aws_s3_bucket_public_access_block" "kb_docs" {
  bucket                  = aws_s3_bucket.kb_docs.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

Aurora pgvector cluster

This is a separate Aurora cluster from any application database — different lifecycle, different access pattern, and you don't want a runaway ingestion job competing for connections with your app:

module "kb_db" {
  source  = "terraform-aws-modules/rds-aurora/aws"
  version = "~> 9.0"

  name              = "bedrock-kb"
  engine            = "aurora-postgresql"
  engine_version    = "16.6"
  instances         = { 1 = {} }

  serverlessv2_scaling_configuration = {
    min_capacity = 0.5
    max_capacity = 4
  }

  database_name        = "bedrock_kb"
  master_username       = "root"
  manage_master_user_password = true

  enable_http_endpoint  = true

  tags = local.default_tags
}

Bedrock's Knowledge Base service expects credentials in {"username": ..., "password": ...} JSON shape in Secrets Manager. If your Terraform module stores a bare password string (most do), mirror it into a second secret in the right shape rather than fighting the module:

resource "aws_secretsmanager_secret" "kb_db_bedrock" {
  name                    = "bedrock-kb-db-credentials"
  recovery_window_in_days = 0
  tags                    = local.default_tags
}

resource "aws_secretsmanager_secret_version" "kb_db_bedrock" {
  secret_id = aws_secretsmanager_secret.kb_db_bedrock.id
  secret_string = jsonencode({
    username = "root"
    password = module.kb_db.cluster_master_password
  })
}

IAM: the Bedrock service role

Bedrock's Knowledge Base needs to assume a role that can read from S3, describe and execute statements against Aurora via the Data API, read the credentials secret, and invoke the embedding model:

data "aws_iam_policy_document" "bedrock_kb_assume_role" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["bedrock.amazonaws.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "aws:SourceAccount"
      values   = [local.aws_account_id]
    }
  }
}

data "aws_iam_policy_document" "bedrock_kb" {
  statement {
    sid       = "S3Read"
    effect    = "Allow"
    actions   = ["s3:GetObject", "s3:ListBucket"]
    resources = [aws_s3_bucket.kb_docs.arn, "${aws_s3_bucket.kb_docs.arn}/*"]
  }

  statement {
    sid       = "RDSDataApi"
    effect    = "Allow"
    actions   = ["rds-data:BatchExecuteStatement", "rds-data:ExecuteStatement"]
    resources = [module.kb_db.cluster_arn]
  }

  statement {
    sid       = "SecretsManagerRead"
    effect    = "Allow"
    actions   = ["secretsmanager:GetSecretValue"]
    resources = [aws_secretsmanager_secret.kb_db_bedrock.arn]
  }

  statement {
    sid       = "BedrockEmbeddings"
    effect    = "Allow"
    actions   = ["bedrock:InvokeModel"]
    resources = ["arn:aws:bedrock:eu-west-1::foundation-model/amazon.titan-embed-text-v2:0"]
  }
}

resource "aws_iam_role" "bedrock_kb" {
  name               = "${local.cluster_name}-bedrock-kb"
  assume_role_policy = data.aws_iam_policy_document.bedrock_kb_assume_role.json
}

resource "aws_iam_role_policy" "bedrock_kb" {
  role   = aws_iam_role.bedrock_kb.id
  policy = data.aws_iam_policy_document.bedrock_kb.json
}

The Knowledge Base resource itself

resource "aws_bedrockagent_knowledge_base" "main" {
  name     = "${local.cluster_name}-knowledge-base"
  role_arn = aws_iam_role.bedrock_kb.arn

  knowledge_base_configuration {
    type = "VECTOR"
    vector_knowledge_base_configuration {
      embedding_model_arn = "arn:aws:bedrock:eu-west-1::foundation-model/amazon.titan-embed-text-v2:0"
    }
  }

  storage_configuration {
    type = "RDS"
    rds_configuration {
      resource_arn           = module.kb_db.cluster_arn
      credentials_secret_arn = aws_secretsmanager_secret.kb_db_bedrock.arn
      database_name          = "bedrock_kb"
      table_name              = "bedrock_integration.bedrock_kb"
      field_mapping {
        primary_key_field = "id"
        vector_field      = "embedding"
        text_field        = "chunks"
        metadata_field    = "metadata"
      }
    }
  }

  depends_on = [module.kb_db]
}

resource "aws_bedrockagent_data_source" "s3" {
  knowledge_base_id = aws_bedrockagent_knowledge_base.main.id
  name              = "${local.cluster_name}-knowledge-base-s3"

  data_source_configuration {
    type = "S3"
    s3_configuration {
      bucket_arn = aws_s3_bucket.kb_docs.arn
    }
  }

  vector_ingestion_configuration {
    chunking_configuration {
      chunking_strategy = "HIERARCHICAL"
      hierarchical_chunking_configuration {
        level_configuration {
          max_tokens = 1500
        }
        level_configuration {
          max_tokens = 300
        }
        overlap_tokens = 60
      }
    }
  }
}

Hierarchical chunking is worth calling out: it splits documents into large 1500-token "parent" chunks and smaller 300-token "child" chunks with a 60-token overlap. Retrieval matches on the precise child chunk but can return the broader parent context — better recall on long documents than flat fixed-size chunking, at the cost of slightly more complex ingestion.

One-time schema bootstrap

aws_bedrockagent_knowledge_base expects the target table to already exist with the right columns and indexes — Terraform won't create the pgvector extension or table for you. This is a one-time psql job against the Aurora endpoint:

CREATE EXTENSION IF NOT EXISTS vector;
CREATE SCHEMA IF NOT EXISTS bedrock_integration;

CREATE TABLE IF NOT EXISTS bedrock_integration.bedrock_kb (
  id        UUID PRIMARY KEY,
  embedding vector(1024),
  chunks    TEXT,
  metadata  JSON
);

CREATE INDEX IF NOT EXISTS bedrock_kb_embedding_idx
  ON bedrock_integration.bedrock_kb
  USING hnsw (embedding vector_cosine_ops);

CREATE INDEX IF NOT EXISTS bedrock_kb_chunks_idx
  ON bedrock_integration.bedrock_kb
  USING gin (to_tsvector('simple', chunks));

The HNSW index handles approximate nearest-neighbor search on the embedding vector; the GIN index on chunks enables hybrid search (keyword + semantic) if you ever want it. vector(1024) matches Titan Embed Text v2's output dimensionality — if you swap embedding models later, the column width has to match or ingestion fails outright.

Publishing references via SSM

resource "aws_ssm_parameter" "kb_id" {
  name  = "/${local.env}/knowledge-base/kb_id"
  type  = "String"
  value = aws_bedrockagent_knowledge_base.main.id
}

resource "aws_ssm_parameter" "kb_bucket" {
  name  = "/${local.env}/knowledge-base/bucket"
  type  = "String"
  value = aws_s3_bucket.kb_docs.bucket
}

Apps and CI pipelines read these instead of hardcoding ARNs — when you rebuild the KB in a new account or region, nothing downstream needs a code change.

Syncing docs automatically with GitHub Actions

Provisioning is the easy part. The actual day-to-day value comes from never having to think about ingestion again. Drop a markdown file in a knowledge-base/ folder, push, and it's searchable within minutes.

name: Sync Knowledge Base

on:
  workflow_dispatch:
    inputs:
      environment:
        type: choice
        description: Environment to sync knowledge base
        options:
          - acc
          - prod
        required: true
  push:
    branches:
      - main
    paths:
      - 'knowledge-base/**'

jobs:
  sync-kb-acc:
    if: >
      github.event_name == 'workflow_dispatch' && github.event.inputs.environment == 'acc' ||
      github.event_name == 'push'
    secrets: inherit
    uses: your-org/your-pipelines-repo/.github/workflows/sync-bedrock-knowledge-base.yml@v2
    with:
      environment: acc
      knowledge-base-id: ${{ vars.BEDROCK_KB_ID_ACC }}
      data-source-id: ${{ vars.BEDROCK_KB_DATA_SOURCE_ID_ACC }}
      bucket-name: ${{ vars.BEDROCK_KB_BUCKET_ACC }}
      bucket-prefix: knowledge-base
      source-dir: knowledge-base

  sync-kb-prod:
    if: github.event_name == 'workflow_dispatch' && github.event.inputs.environment == 'prod'
    secrets: inherit
    uses: your-org/your-pipelines-repo/.github/workflows/sync-bedrock-knowledge-base.yml@v2
    with:
      environment: prod
      knowledge-base-id: ${{ vars.BEDROCK_KB_ID_PROD }}
      data-source-id: ${{ vars.BEDROCK_KB_DATA_SOURCE_ID_PROD }}
      bucket-name: ${{ vars.BEDROCK_KB_BUCKET_PROD }}
      bucket-prefix: knowledge-base
      source-dir: knowledge-base

The gating logic is the whole trick here:

push to main, path-filtered to `knowledge-base/** — only fires the acc` job. Routine doc edits land in the acc environment automatically, with zero manual steps.
workflow_dispatch with environment: acc — also runs the acc job. Useful for re-triggering a sync without a new commit (e.g. after fixing a broken IAM policy).
workflow_dispatch with environment: prod — the only path that touches prod. Promotion to production is always a deliberate, manual action, never a side effect of a push.

Both jobs delegate to the same reusable workflow (sync-bedrock-knowledge-base.yml@v2), parameterized per environment. The reusable workflow does the actual work: sync the source-dir to the S3 bucket-name under bucket-prefix, then call StartIngestionJob against data-source-id. Centralizing that logic in one reusable workflow means every team adopting this pattern gets the same sync behavior — and a fix to the sync logic ships everywhere at once instead of needing fifteen copy-pasted workflow files updated individually.

Field notes

The Data API requires enable_http_endpoint = true on the Aurora cluster. Without it, Bedrock's rds-data:ExecuteStatement calls fail with a confusing connectivity error that has nothing to do with security groups — you'll waste an hour checking VPC routing before you find this.
Vector dimension mismatches fail silently at the wrong layer. If vector(1024) doesn't match your embedding model's output size, the table creation succeeds, the Knowledge Base resource creates fine, and ingestion just fails per-document. Check the embedding model's dimensionality before writing the DDL, not after.
Hierarchical chunking is the right default for prose-heavy docs, but if your knowledge base is mostly short, structured files (FAQs, glossaries), flat fixed-size chunking is simpler to reason about and debug.
Don't let push-triggered syncs touch prod. It's tempting to make prod sync automatically on a release tag, but a bad doc — wrong numbers, stale instructions — propagating into a production RAG pipeline with no human in the loop is a worse failure mode than a slightly stale prod KB.
The acc/prod split needs two of everything — bucket, KB, data source, Aurora cluster — not just two sets of IAM permissions on shared resources. It costs more, but it means a bad ingestion config or chunking change gets caught in acc before it can corrupt the index your production LLM proxy actually queries.

Closing

None of this is exotic — it's a bucket, a Postgres extension, two indexes, and a YAML file with an if: condition. That's the whole trick to "production RAG": treat the knowledge base like any other piece of infrastructure, version it, gate promotion to prod behind a manual step, and let the boring CI pipeline do the boring sync work. Your LLM stops winging it, and you stop being the person who manually re-uploads PDFs every time someone asks why the bot doesn't know about last week's runbook update.

How to Upgrade EKS 1.32: Making the Switch from bootstrap.sh to nodeadm

hugolesta — Tue, 28 Oct 2025 17:10:23 +0000

Did you know staying on a deprecated version of Kubernetes in EKS can cost you six times more? At $0.60 per hour instead of the standard $0.10 per hour, you could end up paying nearly $500 per month just for running outdated clusters.

However, the EKS upgrade from version 1.31 to 1.32 isn't just about avoiding extended support fees—it introduces one of the biggest under-the-hood changes in recent EKS history. Specifically, the traditional bootstrap.sh script used for years to configure worker nodes is now replaced by a new tool called nodeadm.

This architectural shift coincides with another critical change: after November 26, 2025, Amazon EKS will no longer publish EKS-optimized Amazon Linux 2 (AL2) AMIs. Furthermore, Kubernetes 1.32 will be the final version with AL2 AMI support, while version 1.33 onwards will only support Amazon Linux 2023 (AL2023) and Bottlerocket based AMIs.

If your clusters currently run EKS 1.31 or earlier on Amazon Linux 2 AMIs, upgrading to 1.32 will break your node initialization unless you adapt your user-data scripts and switch to AL2023. Throughout this article, we'll walk through exactly what changes between versions 1.31 and 1.32, why nodeadm is now required, and how to rewrite your user-data and Terraform templates to ensure a smooth transition.

Why EKS 1.32 Requires a New Bootstrap Approach

Amazon's evolution of EKS introduces significant architectural changes with version 1.32. The most critical change affects how worker nodes bootstrap and join your clusters, requiring careful attention during upgrades.

End of Support for bootstrap.sh in AL2023

EKS 1.32 marks a pivotal shift as it's the final version for which Amazon will release Amazon Linux 2 (AL2) AMIs. Starting with Kubernetes 1.32, AL2023 introduces a completely different node initialization process that abandons the traditional /etc/eks/bootstrap.sh script. This script has been the foundation of EKS node bootstrapping since the service launched, but is now replaced entirely in the AL2023 operating system. The familiar bash-based bootstrap approach that many DevOps teams have built automation around is completely absent in the new OS version.

nodeadm as the New Default Bootstrap Tool

AL2023 replaces bootstrap.sh with nodeadm, a tool that uses a YAML configuration schema. Unlike the previous approach where metadata was discovered automatically through the Amazon EKS DescribeCluster API call, nodeadm requires explicit provision of cluster information. This fundamental change means you must now specify three critical parameters that were previously auto-discovered:

apiServerEndpoint
certificateAuthority
service CIDR

Additionally, the format for applying parameters to kubelet has changed. Previously accomplished with --kubelet-extra-args, node customization now requires using NodeConfigSpec. This shift aims to reduce API throttling risks during large-scale node deployments.

Impact on Existing AL2-Based Clusters

The elimination of bootstrap.sh creates immediate backward compatibility issues. When upgrading an EKS cluster to version 1.32 while still using AL2-based node groups, nodes will fail to join the cluster. Any automation depending on bootstrap.sh will break, as files like /etc/eks/bootstrap.sh and /etc/eks/eni-max-pods.txt no longer exist.

For organizations with self-managed node groups or custom AMI configurations, this requires extracting and explicitly providing cluster metadata that was formerly obtained automatically. Consequently, deployment scripts, Terraform modules, and CloudFormation templates must be rewritten to align with the new declarative approach before successfully migrating to EKS 1.32.

Preparing for the Migration to nodeadm

Before migrating to EKS 1.32 with nodeadm, careful preparation is essential to ensure a smooth transition from the traditional bootstrap approach to the new paradigm.

Identifying Affected Clusters Running AL2

After November 26, 2025, AWS will end support for EKS AL2-optimized AMIs. Kubernetes version 1.32 represents the final release where Amazon EKS will provide AL2 AMIs. This deadline necessitates prompt action, especially for organizations with multiple clusters.

To identify affected clusters:

Check the AMI type for each node group using:

aws eks describe-nodegroup --cluster-name <cluster-name> --nodegroup-name <nodegroup-name>

Examine existing user-data scripts that reference /etc/eks/bootstrap.sh, which won't exist in AL2023.

Choosing Between AL2023 and Bottlerocket AMIs

Upon identifying clusters requiring migration, you must decide between AL2023 and Bottlerocket as your future node operating system.

AL2023 offers several advantages over AL2:

Secure-by-default approach with preconfigured security policies
SELinux in permissive mode
IMDSv2-only mode enabled by default
Optimized boot times and improved package management

Alternatively, Bottlerocket provides:

Purpose-built container-optimized design with minimal attack surface
Enhanced security with read-only file systems
Automatic updates
Improved compliance with security standards like CIS benchmarks

Choose AL2023 when you need significant customizations with direct OS-level access. Opt for Bottlerocket if you prefer a container-native approach with minimal node customization.

Scanning for Deprecated APIs with kubent or pluto

Prior to upgrading, scan for deprecated APIs that might break during the transition. Kubernetes frequently removes beta APIs with each new version, potentially disrupting your workloads.

The kube-no-trouble tool (kubent) efficiently identifies resources using deprecated APIs:

kubent

This scans all accessible namespaces and lists APIs that will be deprecated compared to your current Kubernetes version. For clusters with hundreds of applications across multiple namespaces, this tool proves invaluable in detecting potential upgrade issues beforehand.

Backing Up Cluster State and Node Configurations

A comprehensive migration plan should always include thorough backups. Given that you cannot downgrade an EKS cluster after upgrading, backups become crucial.

Recommended backup steps include:

Taking etcd snapshots for core Kubernetes data
Backing up cluster configuration using tools like Velero
Documenting node-specific configurations, especially custom user-data scripts
Preserving IAM role configurations that will need migration to the new nodeadm format

Additionally, establish a documented rollback procedure with well-defined testing protocols before proceeding with the upgrade.

Implementing nodeadm and AL2023

Image Source: AWS Documentation

Debugging and Validating the Upgrade

After implementing nodeadm and upgrading to EKS 1.32, troubleshooting becomes essential as new components may not function perfectly on the first try. The migration introduces different debugging approaches that we need to master.

Common nodeadm Errors and How to Fix Them

The nodeadm debug command serves as our first line of defense when troubleshooting unhealthy or misconfigured nodes. It validates critical requirements including network access to AWS APIs, credentials for the IAM role, connectivity to the EKS Kubernetes API endpoint, and authentication with the EKS cluster.

nodeadm debug -c file://nodeConfig.yaml

For configuration validation before implementation, the nodeadm config check command proves invaluable:

nodeadm config check -c file://nodeConfig.yaml

Most permission-related issues arise from the Hybrid Nodes IAM role missing the necessary eks:DescribeCluster action. Other common errors include network connectivity problems, incorrect node IP configuration, and timeout issues which can be remedied by extending timeouts:

nodeadm install K8S_VERSION --credential-provider CREDS_PROVIDER --timeout 20m0s

Verifying kubelet and containerd Startup Logs

Examining kubelet logs offers visibility into node initialization problems. For AL2023 nodes, we can use:

systemctl status kubelet
journalctl -u kubelet -o cat

Moreover, checking the status helps verify successful restarts after upgrades. For deeper troubleshooting, we can connect to the node using SSH or kubectl debug and inspect the logs. My personal preference is using AWS Systems Manager Session Manager to enhance security, avoiding opening SSH ports:

chroot /host journalctl -u kubelet -o cat

Checking Node Readiness and Taints

To verify node status after migration, we use standard kubectl commands:

kubectl get nodes -o wide
kubectl describe node NODE_NAME

The STATUS column should display "Ready" for all nodes with the updated version number visible. Untainted nodes are essential for workload scheduling. Furthermore, checking events related to nodes can reveal issues with node registration or initialization.

Validating Add-on Compatibility Post-Upgrade

Post-upgrade add-on validation requires checking deployment versions and ensuring pods are running correctly. For example, to verify the CoreDNS version:

kubectl describe deployment coredns -n kube-system | grep Image | cut -d ':' -f 3

To inspect add-on logs for errors:

kubectl logs -n kube-system -l k8s-app=kube-dns

For networking add-ons like Amazon VPC CNI, we should create test pods to validate IP assignment. Additionally, testing CoreDNS functionality using tools like nslookup ensures proper DNS resolution. Finally, checking if the number of replicas equals the number of nodes for daemonset add-ons like vpc-cni and kube-proxy confirms proper deployment.

Conclusion

Migrating to EKS 1.32 represents a significant paradigm shift for Kubernetes operations on AWS. The transition from bootstrap.sh to nodeadm fundamentally changes how worker nodes join your clusters, requiring careful planning and execution. Additionally, the impending deprecation of Amazon Linux 2 AMIs after November 26, 2025, creates urgency for organizations to adapt their infrastructure.

Throughout this upgrade journey, you must remember that nodeadm demands explicit configuration through YAML rather than command-line arguments. This declarative approach actually offers better consistency and reproducibility for node configurations once implemented correctly. Undoubtedly, the initial migration might seem daunting, especially when rewriting existing automation scripts or Terraform modules.

The choice between AL2023 and Bottlerocket AMIs depends largely on your specific operational requirements. AL2023 provides a familiar environment with improved security features, whereas Bottlerocket offers a container-optimized approach with minimal attack surface. Regardless of your choice, both options eliminate the traditional bootstrap files and require adaptation to the new nodeadm paradigm.

Before initiating any upgrade, thorough preparation becomes essential. First, identify affected clusters running AL2, then scan for deprecated APIs, and finally back up your cluster state. After implementing the necessary changes, debugging tools like nodeadm debug and nodeadm config check help troubleshoot any issues that arise during the migration process.

The EKS 1.32 upgrade, therefore, presents both challenges and opportunities. While it requires significant changes to existing workflows, it also aligns your infrastructure with AWS's future direction. Consequently, organizations that proactively adapt their node initialization processes will avoid extended support fees and benefit from improved security and performance features of newer operating systems.

Ultimately, staying current with EKS versions not only saves operational costs but also ensures compatibility with the evolving Kubernetes ecosystem. Though this particular upgrade demands more effort than typical version bumps, the long-term benefits of embracing nodeadm and AL2023 far outweigh the initial investment required for migration.

Key Takeaways

EKS 1.32 introduces the most significant architectural change in recent history, requiring organizations to abandon the traditional bootstrap.sh script in favor of nodeadm for worker node initialization.

• Critical deadline approaching: Amazon ends AL2 AMI support on November 26, 2025, making EKS 1.32 the final version supporting Amazon Linux 2.

• Bootstrap method completely changes: nodeadm replaces bootstrap.sh and requires explicit YAML configuration instead of automatic cluster metadata discovery.

• Migration requires careful preparation: Identify AL2-based clusters, scan for deprecated APIs with kubent, and backup cluster state before upgrading.

• Choose your AMI strategy: Select between AL2023 for familiar environments with enhanced security or Bottlerocket for container-optimized minimal attack surface.

• Debug with new tools: Use nodeadm debug and nodeadm config check commands to troubleshoot configuration issues and validate node readiness.

• Avoid costly extended support: Staying on deprecated Kubernetes versions costs 6x more ($0.60/hour vs $0.10/hour), potentially adding $500+ monthly per cluster.

The shift to nodeadm represents AWS's commitment to more secure, declarative infrastructure management. While the initial migration requires significant effort in rewriting automation scripts and Terraform modules, organizations that proactively adapt will benefit from improved security, performance, and long-term cost savings.

FAQs

Q1. What is the main change in EKS 1.32 that affects node initialization? EKS 1.32 replaces the traditional bootstrap.sh script with a new tool called nodeadm for worker node initialization. This change requires explicit YAML configuration instead of automatic cluster metadata discovery.

Q2. When will Amazon stop supporting Amazon Linux 2 (AL2) AMIs for EKS? Amazon will end support for EKS AL2-optimized AMIs after November 26, 2025. Kubernetes version 1.32 is the final release where Amazon EKS will provide AL2 AMIs.

Q3. What are the alternatives to Amazon Linux 2 for EKS nodes? The main alternatives are Amazon Linux 2023 (AL2023) and Bottlerocket AMIs. AL2023 offers improved security features and a familiar environment, while Bottlerocket provides a container-optimized design with a minimal attack surface.

Q4. How can I identify and fix common nodeadm errors? You can use the 'nodeadm debug' command to validate critical requirements and the 'nodeadm config check' command to validate configurations. Common issues include missing IAM permissions, network connectivity problems, and incorrect node IP configurations.

Q5. What are the cost implications of staying on a deprecated version of EKS? Running a deprecated version of EKS can cost six times more than the standard rate. Instead of $0.10 per hour, you could end up paying $0.60 per hour, potentially adding over $500 per month for each outdated cluster.

The $10,000 Label: How We Used Go, Clean Architecture, and AWS to Build a FinOps-Driven Cloud Tagging Engine 🏷️

hugolesta — Mon, 29 Sep 2025 16:01:54 +0000

Why Consistent Tagging is Your Company’s Most Underrated FinOps Tool:

The Business Problem: Imagine your cloud bill is a massive corporate expense report. Without proper tagging—simple key-value labels like project: crm-migration or owner: finance-team —you're paying thousands every month for line items labeled simply “Server.” This isn't just an accounting headache; it's a direct threat to cost control and security.
Cost Bloat: Orphaned or forgotten AWS resources (Shadow IT) continue to generate costs because no one is accountable for terminating them.
Billing Disputes: Finance teams struggle to attribute costs accurately, leading to friction and delayed chargebacks.
Security Risks: Unmanaged resources often fall outside compliance or patch cycles.

We decided to solve this with sys-tag-manager, a powerful, automated system built in Golang that acts as our centralized "Cloud Label Printer," ensuring every AWS resource is correctly accounted for, compliant, and cost-trackable.

What Is a Tagging Strategy?

A tagging strategy is a structured approach to applying metadata (tags) to cloud resources. Tags are simple key–value pairs like:

owner: finance-team
project: crm-migration
environment: production

On their own, tags look trivial. But when applied consistently across an entire cloud estate, they form the backbone of organization, governance, and cost management.

A tagging strategy defines:

Which tags are required (e.g. owner, project, environment).
How tags should be formatted (naming conventions, lowercase vs camelCase, separators).
When tags should be applied (at creation time vs automated correction).
Who is responsible for maintaining them?

For further guidance, consult the AWS documentation: Best Practices for Tagging AWS Resources

Technical Foundation: Go, Clean Architecture, and AWS Cost Savings

The Go Advantage: Performance Meets FinOps:

While prototyping in Python is quick, a core, mission-critical tool demands performance and reliability. We chose Golang for sys-tag-manager because:

Cost-Efficient Execution: Go's minimal memory footprint and extremely fast startup time are critical when running as AWS Lambda functions or Kubernetes CronJobs. This translates directly into lower AWS compute costs (less time billed for execution) compared to resource-heavier languages.
Reliability: Static typing and robust concurrency ensure the system can efficiently handle a rapidly growing number of AWS API calls without failure, ensuring 100% compliance coverage.

Clean Architecture: Decoupling Logic from the AWS SDK

To ensure the system remains maintainable as our cloud estate scales, we invested in Clean Architecture. This strategic separation is key to our long-term technical debt reduction:

Domain (Core Business Logic): Pure, independent rules (e.g., "A resource is compliant if it has the required tags: owner and project"). This is highly testable and knows nothing about AWS.
Use Cases (Application Logic): Defines the "what" (e.g., "Check compliance and apply tags").
Adapters (AWS Implementation): Isolated logic that interacts directly with specific AWS SDK services (EC2Tagger, RDSTagger). This prevents vendor lock-in and allows us to add new services (S3, Lambda) without touching the core business rules.

The Compliance Engine: Terraform, SSM, and Metadata Management

The Discovery Layer: Leveraging AWS Resource Explorer

Before sys-tag-manager can fix untagged resources, it must efficiently find them across accounts and Regions. We achieved this by using AWS Resource Explorer as our primary discovery and inventory layer.

Instead of writing complex API calls to list every resource type across every Region, sys-tag-manager utilizes Resource Explorer's unified search capabilities.

The Workflow:

Discovery: sys-tag-manager uses the Resource Explorer API to query the entire cloud estate for resources that are missing required tags (e.g., tag:owner is absent).
Validation: For each untagged resource found, sys-tag-manager checks its metadata against the centralized, correct rules stored in AWS SSM Parameter Store.
Correction: The system then applies the right tags, assigning the resource to the correct owner or project, ensuring immediate compliance.

This design significantly streamlined the core tagging loop, ensuring we are not just efficient in applying tags (Golang), but also efficient in finding them (Resource Explorer), saving API call costs and latency.

Harnessing Shared Infrastructure: The Fallback Mechanism

While our primary goal is to enforce resource-specific tagging, we recognized that some resources are "Shared Infrastructure" (e.g., core networking components, centralized security groups) that don't belong to a single owner. Addressing this was a critical design challenge.

Our solution was a smart fallback mechanism:

The tagging engine first checks for the required resource-specific tags.
If the tags are missing, it then checks a predefined list of AWS ARNs (Amazon Resource Names) that are designated as shared infrastructure.
If an ARN matches, the system applies a generic, shared set of tags (e.g., owner: platform-team, charge-code: shared-infra) instead of flagging it as non-compliant. This prevents false positives and ensures accurate cost attribution for common resources.

The Terraform + SSM Parameter Store Synergy

The true power of sys-tag-manager lies in its ability to dynamically enforce tagging rules based on centralized, auditable metadata.

Centralized Rule Source: We leverage AWS Systems Manager (SSM) Parameter Store to store the required tag keys, values, and compliance rules.
Terraform as the Single Source of Truth: The compliance rules in SSM are managed exclusively by Terraform. This means:
1. Immutability: Every rule change is tracked, reviewed, and deployed via a GitOps workflow.
2. Automation: When a new project is created in Terraform, the required tag values for that project are automatically pushed to SSM, immediately making those tags valid for the tag manager checks.

This integration ensures that the FinOps rules are always aligned with the deployed infrastructure definitions, creating a clean, traceable metadata loop.

Business Impact: Quantifiable Results for FinOps

Metric	Before sys-tag-manager	After sys-tag-manager	Value Proposition
Compliance Time	Weeks (Manual Audits)	Minutes (Automated Correction)	Faster cost allocation & reduced risk.
Orphaned Resources	~12% (Estimate)	<1%	Direct savings on wasted AWS spend.
FinOps Accuracy	High Friction/Disputes	High Trust/Automated Showback	Enables accurate, automated chargeback.

By investing in this system, we have fundamentally shifted from reactive tag auditing to proactive, automated compliance enforcement. This not only saves engineering hours but directly enables our finance team to confidently utilize AWS Cost and Usage Reports (CUR) for accurate showback and chargeback, making our entire cloud operation more accountable and financially efficient.

Wrapping Up: Compliance as Code, Savings as the Result

sys-tag-manager is more than just an automation script; it's the enforcement layer for our FinOps and security policies, ensuring that our cloud environment is self-healing and financially accountable.

By embracing Golang for performance, Clean Architecture for maintainability, and the Terraform + SSM synergy for centralized metadata management, we've transformed tagging from a manual burden into an automated, cost-saving asset. This shift has given us the confidence that every dollar spent on AWS is trackable, auditable, and directly attributed to a business owner or project.

The result is a culture of Compliance as Code where engineers can focus on feature delivery, knowing that the foundational governance—the tagging—is handled automatically and efficiently by sys-tag-manager.

Let's Keep the Conversation Going 🗣️
We've focused on the technical core of sys-tag-manager, but the true organizational victory was how we scaled this system across dozens of teams without friction.

Would you be interested in learning more about how we automated the communication of compliance status and tagging fixes to developers, FinOps, and management?

I'd love to hear your thoughts! What's the biggest tagging challenge you face in your organization? Share your experiences and suggestions in the comments below!

Feel free to connect with me on LinkedIn to discuss our approach to automated communication and team onboarding:

LinkedIn: Hugo Lesta

GitHub: Hugo Lesta's GitHub