DEV Community: Manoj Kumar Kagitha

Terraform in CI/CD: Plan/Apply Gates with GitHub Actions

Manoj Kumar Kagitha — Mon, 26 Oct 2026 14:07:54 +0000

Versioned modules mean nothing if engineers still run apply from their laptops. Laptop applies are unreviewed, unlogged, and unrepeatable. The fix is a pipeline: every change goes through plan on pull request, review, then apply from CI with short-lived credentials. This post builds that pipeline on GitHub Actions from scratch.

Why CI/CD runs apply

Four reasons, in order of how much pain each one prevents.

Review. A plan posted as a PR comment lets reviewers see exactly what will change before it changes. No more "I applied and something broke, let me check what I did."

Audit trail. Every apply is tied to a commit, a PR, and a GitHub Actions run. When something breaks at 2 AM, you know exactly which change did it and who approved it.

No credential sprawl. Engineers do not need cloud credentials on their laptops. The pipeline assumes a role with OIDC. Credentials live for the duration of the run, then vanish.

Repeatability. The same code, the same Terraform version, the same provider versions, every time. "Works on my machine" stops being a sentence anyone says.

Pipeline stages

A solid Terraform pipeline has four stages. Each one gates the next.

fmt. terraform fmt -check fails the run if any file is misformatted. Cheap, fast, non-negotiable. Formatting arguments are not worth human review time.
validate. terraform init -backend=false followed by terraform validate. Catches syntax errors and invalid references without touching state or cloud APIs.
plan. Full init against the real backend, then terraform plan -out=tfplan. The plan file is saved as an artifact. On pull requests, the human-readable plan is posted as a PR comment.
apply. Only on merge to main. Downloads the exact plan artifact from the plan stage and applies it. Never re-plans at apply time. You apply what was reviewed, not something subtly different.

Linting with tflint slots in between validate and plan. Security scanning with tfsec or Checkov goes there too.

The plan artifact pattern

The critical detail: apply consumes the saved plan file, not a fresh plan. Here is why it matters.

Between plan and apply, time passes. A reviewer takes an hour. Someone merges another PR. The cloud state drifts. If apply re-plans, it applies something nobody reviewed. By saving the plan with -out=tfplan and applying that exact file, the reviewed change is the applied change. If the world moved underneath, the apply fails loudly instead of silently doing something unexpected. That failure is correct behavior.

Posting plan output as a PR comment

A plan artifact is a binary file. Reviewers cannot read it on GitHub. So the pipeline also renders the plan as text and posts it as a comment:

- name: Render plan
  run: terraform show -no-color tfplan > tfplan.txt
  working-directory: ${{ env.WORKING_DIR }}

- name: Post plan comment
  if: github.event_name == 'pull_request'
  uses: actions/github-script@v7
  with:
    script: |
      const fs = require('fs');
      const plan = fs.readFileSync(process.env.WORKING_DIR + '/tfplan.txt', 'utf8');
      const body = `## Terraform Plan\n\n\`\`\`\n${plan.substring(0, 60000)}\n\`\`\``;
      github.rest.issues.createComment({
        issue_number: context.issue.number,
        owner: context.repo.owner,
        repo: context.repo.repo,
        body: body
      });

Long plans get truncated at 60,000 characters, which fits GitHub's comment limits. For most changes, the full plan fits. Reviewers see adds, changes, and destroys inline, right where they already review code.

OIDC auth: no stored secrets

The old way: a long-lived AWS access key or Azure service principal secret sitting in GitHub Secrets, rotating never, shared by everyone. The new way: OpenID Connect. GitHub mints a short-lived token per run. The cloud trusts GitHub's identity provider and hands back temporary credentials scoped to one IAM role.

AWS setup. Create an IAM role with a trust policy that allows GitHub's OIDC provider to assume it, scoped to your repo:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
      },
      "StringLike": {
        "token.actions.githubusercontent.com:sub": "repo:myorg/infra:*"
      }
    }
  }]
}

The sub condition restricts assumption to your infrastructure repo. Tighten it further with branch conditions if you want: only main can assume the production role, while any branch can assume the dev role.

Azure setup. Register an app, add a federated credential trusting repo:myorg/infra:ref:refs/heads/main, and log in with azure/login@v2 using client ID, tenant ID, and subscription ID. No secret needed.

The workflow needs one permission block:

permissions:
  id-token: write
  contents: read
  pull-requests: write

id-token: write is what lets the run mint the OIDC token. Without it, the credential step fails with an unhelpful error.

Apply gates

Not every merge should auto-apply to production. GitHub Environments give you gates.

Define an environment per target: dev, staging, production. Attach rules:

dev: auto-apply on merge. Fast feedback, low blast radius.
staging: apply on merge, but the job waits for no one. Actually, make staging auto-apply too. Staging exists to catch what dev missed, and manual gates on staging just slow the signal.
production: required reviewers. The apply job pauses until a designated approver clicks approve. The approver sees the plan diff from the PR. This is the gate that matters.

apply-prod:
  runs-on: ubuntu-latest
  needs: plan-prod
  if: github.ref == 'refs/heads/main'
  environment: production
  concurrency:
    group: terraform-prod
    cancel-in-progress: false

The environment: production line is what triggers the approval gate. Configure the required reviewers in the repo settings under Environments.

Full working workflow

Here is a complete workflow for one environment. It runs fmt and validate on every PR, plans and comments, and applies only on merge to main behind the production gate.

name: Terraform Prod

on:
  push:
    branches: [main]
    paths: ['envs/prod/**']
  pull_request:
    paths: ['envs/prod/**']

permissions:
  id-token: write
  contents: read
  pull-requests: write

env:
  TF_VERSION: "1.9.0"
  WORKING_DIR: envs/prod

jobs:
  fmt:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: ${{ env.TF_VERSION }}
      - run: terraform fmt -check -recursive
        working-directory: ${{ env.WORKING_DIR }}

  validate:
    runs-on: ubuntu-latest
    needs: fmt
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: ${{ env.TF_VERSION }}
      - run: terraform init -backend=false
        working-directory: ${{ env.WORKING_DIR }}
      - run: terraform validate
        working-directory: ${{ env.WORKING_DIR }}

  plan:
    runs-on: ubuntu-latest
    needs: validate
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: ${{ env.TF_VERSION }}
          terraform_wrapper: false
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/terraform-ci-prod
          aws-region: us-east-1
      - run: terraform init -input=false
        working-directory: ${{ env.WORKING_DIR }}
      - run: terraform plan -out=tfplan -input=false
        working-directory: ${{ env.WORKING_DIR }}
      - run: terraform show -no-color tfplan > tfplan.txt
        working-directory: ${{ env.WORKING_DIR }}
      - uses: actions/upload-artifact@v4
        with:
          name: tfplan-prod
          path: ${{ env.WORKING_DIR }}/tfplan
      - name: Post plan comment
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const plan = fs.readFileSync(process.env.WORKING_DIR + '/tfplan.txt', 'utf8');
            const body = `## Terraform Plan (prod)\n\n\`\`\`\n${plan.substring(0, 60000)}\n\`\`\``;
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: body
            });

  apply:
    runs-on: ubuntu-latest
    needs: plan
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    environment: production
    concurrency:
      group: terraform-prod
      cancel-in-progress: false
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: ${{ env.TF_VERSION }}
          terraform_wrapper: false
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/terraform-ci-prod
          aws-region: us-east-1
      - uses: actions/download-artifact@v4
        with:
          name: tfplan-prod
          path: ${{ env.WORKING_DIR }}
      - run: terraform init -input=false
        working-directory: ${{ env.WORKING_DIR }}
      - run: terraform apply -input=false tfplan
        working-directory: ${{ env.WORKING_DIR }}

Two details worth calling out. First, terraform_wrapper: false on the setup-terraform action. The wrapper captures output for PR comments in the action's own way, which conflicts with the manual plan-comment pattern. Disable it when you post comments yourself.

Second, the apply job downloads the plan artifact and applies that file. It does not run plan again. Reviewed plan in, applied plan out.

Multi-environment promotion

One repo, one workflow per environment, or one workflow with a matrix. The promotion flow:

Merge to main. Dev applies automatically. No gate, fast feedback.
Staging applies automatically after dev succeeds. Use needs: [apply-dev] to serialize.
Production waits. The environment: production gate pauses for a human approver.

Sequential, not parallel. Prod never applies before staging succeeds. If staging apply fails, prod never starts, and the team fixes forward instead of debating whether prod is safe.

Keep one workflow file per environment rather than one mega-workflow with conditionals. terraform-prod.yml, terraform-staging.yml, terraform-dev.yml. Each is readable on its own. Shared steps go into composite actions or reusable workflows if duplication bothers you, but duplication across three files is cheaper than one clever file nobody understands.

Concurrency groups: serialize applies per environment

Two merges in quick succession can queue two applies against the same state. State locking makes the second one wait or fail, but it is cleaner to serialize at the workflow level:

concurrency:
  group: terraform-prod
  cancel-in-progress: false

cancel-in-progress: false is the important part. You want the second run to wait, not to cancel the first run mid-apply. Cancelling mid-apply is how you get tainted resources and half-created infrastructure.

One concurrency group per environment. Dev, staging, and prod apply independently. Two environments never block each other.

State locking and CI

State locking still applies in CI. If a run crashes mid-apply, the lock can stick. The next run fails with a lock error. The fix is the same as locally: terraform force-unlock, but run it deliberately, not in the pipeline.

Do not add auto-unlock to the pipeline. An automatic force-unlock can release a lock held by a genuinely running apply, recreating the race condition locking exists to prevent. When CI reports a stuck lock, a human checks that nothing is actually applying, runs force-unlock once, and re-runs the workflow.

When apply fails mid-run

It happens. A network blip, a cloud API quota, a resource that fails to create. Terraform marks the failed resource as tainted. The next apply destroys and recreates it. Usually that is the right behavior.

When it is not, for example a database that took twenty minutes to provision and failed on a tagging step, you have options:

terraform apply -replace="aws_db_instance.main" recreates one specific resource on the next run, without waiting for a failure to taint it.
terraform state rm followed by terraform import detaches a resource from management when you need to stop Terraform from touching it while you fix things by hand. Use sparingly.
As a last resort, -target applies to a subset of resources. It bypasses dependency ordering guarantees, so treat it as emergency surgery, not routine practice. Every -target apply should be followed by a full, untargeted plan to confirm the state is coherent.

After any recovery, run a full plan. If the plan is empty, you are back to a clean state. If it is not, keep going until it is.

Checklist

[ ] fmt, validate, plan, apply as separate gated stages
[ ] Plan saved with -out=tfplan, applied from the artifact. No re-planning at apply time.
[ ] Plan output posted as a PR comment for review
[ ] OIDC auth configured. No long-lived credentials in GitHub Secrets.
[ ] id-token: write permission set on the workflow
[ ] Production apply behind a GitHub Environment approval gate
[ ] Concurrency group per environment, cancel-in-progress: false
[ ] Promotion order enforced: dev, then staging, then prod
[ ] No auto-unlock in the pipeline. Stuck locks get human attention.
[ ] Terraform, provider, and module versions pinned (see article 2)

The pipeline applies cleanly and every change is reviewed. But infrastructure does not stay as Terraform left it. Someone tweaks a setting in the console. A cloud provider changes a default. The next plan shows diffs nobody made.

Next article: drift detection and remediation. Scheduled scans, telling real drift from cosmetic noise, and the workflow for bringing reality back in line with code.

Who Is Paying for This? Token Metering at the LLM Gateway

Manoj Kumar Kagitha — Fri, 23 Oct 2026 14:52:27 +0000

Once you put a large language model behind a company API gateway, someone asks a simple question.

Who is paying for this?

The model bill arrives as one number. Finance sees one number. Every team in the company sees one endpoint. Nobody can tell you which team burned forty percent of the budget, which use case is growing, or which experiment someone forgot to shut off six weeks ago. That gap between one bill and one endpoint is where metering lives.

We built metering into our LLM gateway before we scaled usage, and it changed how teams behaved. Here is what we learned.

Why LLM metering is not like API metering

With a normal REST API, metering is easy. One request costs one request. You count requests per subscription key, and you are done.

LLM calls do not work that way. One request to a chat endpoint can cost five tokens or five thousand. A prompt injection test script someone ran in a loop overnight can spend more than an entire product team's monthly budget in eight hours. If you count requests, you learn nothing. The unit of cost is the token, and the token count lives in the response, not the request.

That detail shapes everything. Your metering code has to read the response body, extract the usage block, and attribute it before the data is gone.

{
  "usage": {
    "prompt_tokens": 1240,
    "completion_tokens": 312,
    "total_tokens": 1552
  }
}

Azure OpenAI returns this on every response. If you do not capture it at the gateway, you have to reconstruct it later from logs, and that is a losing fight.

Capture it at the gateway, not in the app

Every team wanted to do metering in their own application code. We said no. The gateway is the one layer every call passes through, regardless of language, framework, or team. A metering implementation that depends on each team instrumenting their own service is an implementation that covers sixty percent of traffic on a good month.

We put a policy on the gateway's inbound path to stamp each request with metadata: the subscription the key belongs to, the product, the environment. On the outbound path, another policy parses the response, pulls the token counts, and ships an event to our analytics pipeline. The policy does not touch the request or response bodies in ways the caller can see. It observes and reports.

<policies>
    <outbound>
        <choose>
            <when condition="@(context.Response.StatusCode == 200)">
                <set-body>@{
                    // parse the usage block and forward a metering event
                    // key: subscription id, product, environment, model
                    // value: prompt_tokens, completion_tokens, latency
                }</set-body>
            </when>
        </choose>
    </outbound>
</policies>

This is pseudo-code, not a real policy file, because every gateway has its own syntax. The shape matters more than the syntax: stamp metadata on the way in, extract cost on the way out, send it somewhere durable.

Attribute everything, including the failures

The first version of our metering only counted successful calls. That lasted about two weeks. Someone ran a batch job against a deprecated model deployment that returned 400s all day. The calls were cheap individually, but there were hundreds of thousands of them, and the model provider still charged for the input tokens on each failed request.

Failed requests still consume input tokens. Rate-limited requests still consume them. Requests that hit a content filter still consume them. If you only meter the 200s, your bill and your dashboard will disagree, and finance will believe the bill.

Our outbound policy records every response code. Failures get their own bucket so teams can see them separately, but they are not excluded from the total. The first time a team saw how much they were spending on 400s, they fixed their client code that afternoon.

Give teams their own numbers, not the whole company

A company-wide dashboard that says the LLM bill is fifty thousand dollars a month helps nobody. A dashboard that says your team spent nine thousand, up forty percent from last month, on a specific model, changes behavior overnight.

We publish per-team views: tokens and estimated cost by team, by model, by day. The estimated cost part matters. Tokens are abstract. Dollars are not. We maintain a small price table per model and multiply. The estimate is not perfect, and we say that on the dashboard. It is close enough to drive decisions.

The behavioral change was immediate. Teams started batching requests. They cached repeated prompts. One team switched a summarization job from a large model to a smaller one after seeing the per-run cost. None of that happened when the bill was a single line item.

Budgets need teeth, but start with visibility

Once teams can see their spend, the next question is limits. We added budget alerts first, hard stops second. The sequence matters.

Alerts went to the team's channel when they crossed fifty, eighty, and one hundred percent of their monthly budget. Most teams self-corrected at fifty. For the ones that did not, a hard stop at the gateway rate-limits further calls until someone raises the budget.

We deliberately did not lead with hard stops. An engineer running a time-sensitive demo who hits a hard limit learns that the gateway is the enemy. An engineer who gets a warning at fifty percent learns to watch their budget. The hard stop exists for the runaway loops, not for normal work.

What we would do differently

Three things we would change if we started over.

First, we would meter streaming responses from day one. Token counts arrive at the end of a streamed response, and our first implementation missed them. Streaming became popular faster than we expected.

Second, we would standardize the metadata stamp earlier. We let teams add custom attributes organically, and we ended up with three different keys for the same project name. Normalizing that retroactively was tedious.

Third, we would keep the price table in version control with a review step. Model prices change. When they do, old estimates look wrong. We now treat the price table like code: a pull request, a reviewer, a changelog.

The real lesson

Metering is not a billing feature. It is a feedback loop. The moment teams could see what their calls cost, they started engineering like the cost mattered. Architecture improved. Waste fell. Nobody needed a policy document telling them to be careful with the expensive model, because the numbers told the story better than any document could.

If you are standing up an LLM gateway and nobody has asked about cost attribution yet, they will. Build the metering in before the bill becomes a conversation.

Cross-cloud cost math: when serverless beats fixed tiers

Manoj Kumar Kagitha — Fri, 23 Oct 2026 02:33:42 +0000

Cross-cloud cost math: when serverless beats fixed tiers

Every API consumer asks the same question sooner or later: which gateway
tier should this run on? Most teams answer from habit. The right answer is
arithmetic.

I added a cost command to
api-onboarding-kit
because onboarding should include the answer. It reads the consumer spec,
does the multiplication across every gateway tier, and prints the options
cheapest first. The point is not that the numbers are perfect. They are
from public list pricing and they move. The point is that the method is
boring and it beats arguing from gut feeling.

The shape of the math

Every tier is one of two shapes:

Serverless: cost = requests x per-million price, no base.
Fixed: cost = flat monthly base, no per-request term.

Requests per month come straight from the spec:

monthly_requests = expected_rps x 86,400 x 30

So a consumer at 10 rps average does about 25.9 million requests a month.
Run that through the pricing table in the repo and you get:

Azure APIM Consumption: 25.92 x $4.00 = $103.68
AWS API Gateway HTTP API: 25.92 x $1.00 = $25.92
GCP API Gateway: 25.92 x $3.00 = $77.76
Azure APIM Standard: $700.00 flat
Apigee X pay-as-you-go: 25.92 x $40.00 = $1,036.80

At this traffic level, serverless wins everywhere and the fixed tier looks
absurd. That reverses fast.

The crossover point

Fixed beats serverless when:

monthly_millions > base_usd / per_million_usd

Take Azure APIM Standard ($700 flat) against Consumption ($4 per million):

# 700 / 4 = 175 million requests per month
crossover_millions = 700.0 / 4.00
crossover_rps = crossover_millions * 1e6 / (86_400 * 30)
# => ~67 rps sustained, all month

Below ~67 rps sustained, the consumption tier is cheaper. Above it, the
fixed tier wins and every request past the line is effectively free.
That is the whole decision. One division.

What the math misses

Cheapest is not always right. Two things I learned the hard way:

1. Serverless tiers drop features. In APIM, the consumption tier does
not give you VNet support or the production SLA. If your security review
requires private networking, the "cheaper" tier is not an option and the
comparison is over before it starts. The estimator flags these notes on
every row for a reason.

2. Averages lie. expectedRps drives the cost, but burstRps
decides whether the tier survives. A consumer that averages 5 rps and
bursts to 500 rps can blow through a per-request tier's limits during the
spike while costing nothing on paper. Burst is a capacity question, not a
pricing question. Model it separately.

So the working rule I give teams: run the arithmetic to pick the tier,
then check the feature list and the burst behavior to confirm the tier
is even allowed. Cost picks the winner among legal options.

The command

onboard cost --all --spec payments-team.yaml

It reads expectedRps from the spec and prints every gateway, every
tier, cheapest first. Adding a tier is one dictionary entry:

PRICING["azure-apim"]["tiers"]["premium"] = {
    "base_usd": 2600.0, "per_million_usd": 0.0,
    "note": "Multi-region, higher throughput",
}

Then the crossover math just works, because it is arithmetic, not logic.

The takeaway

Gateway cost is not a negotiation or a vendor pitch. It is one division
problem per tier. Do it in CI, on every new consumer, before the traffic
exists. The numbers move and the features change, but the method does not.

Repo: kagithamanoj/api-onboarding-kit

Every API Breaks Eventually. Versioning Decides Who Feels It.

Manoj Kumar Kagitha — Wed, 21 Oct 2026 14:12:37 +0000

If you have run APIs at enterprise scale, you know the call. A partner's app starts throwing 400s at noon on a Tuesday. Someone renamed a field in a response payload. Nobody versioned anything, so every client got the change at once. The fix takes ten minutes. The trust takes months to rebuild.

Breaking changes are normal. Software grows. Fields get renamed, auth schemes get tightened, pagination shows up where it did not exist before. What is not normal is shipping a breaking change without a version. Versioning decides who feels a breaking change: nobody, or everyone.

Here is how we run API versioning across a multi-gateway estate. I work with Apigee X and Azure APIM side by side, plus backing services on Azure. The estate is the hard part. One versioning policy only works if it holds on every gateway at once.

Pick one versioning scheme and mean it

There are three schemes people argue about: path versioning (/v1/orders), header versioning (Accept: application/vnd.orders.v2+json), and query params (?version=2).

Path versioning wins in practice. Here is why. Every client knows how to hit a different path. Every log aggregator groups by path without custom parsing. Every gateway routes on path without policy gymnastics. Headers get stripped by proxies you do not control. Query params get swallowed by caches. Paths survive.

The rule is simple: the version lives in the path, it is mandatory, and it never changes meaning. /v1 stays /v1 forever. If v1 needs a fix, you ship /v2 and deprecate v1. You never mutate v1 under the same URL.

Put the policy in the gateway, not in every backend team

The biggest versioning failure I have seen was not a bad scheme. It was a good scheme enforced nowhere. Teams were told to version their APIs. Half did it their way, half forgot, and one shipped /v1.2-final-2.

The gateway is the enforcement point. It sits in front of every service. That is where the versioning policy lives, written once, applied to every API that crosses it.

A practical gateway policy does three things:

Rejects requests with unknown or missing versions with a clear 400.
Routes each version to the right backend. /v1/orders and /v2/orders can point at the same service or different ones. The client never knows.
Normalizes version metadata for logging, so analytics can answer "how much traffic is still on v1" without a forensic exercise.

In Azure APIM this is an inbound policy on the product or gateway scope. Something like this:

<policies>
  <inbound>
    <base />
    <choose>
      <when condition="@(context.Request.OriginalUrl.Path.StartsWith("/v1")
          || context.Request.OriginalUrl.Path.StartsWith("/v2"))">
        <set-header name="x-api-version" exists-action="override">
          <value>@(context.Request.OriginalUrl.Path.Substring(1, 2))</value>
        </set-header>
      </when>
      <otherwise>
        <return-response>
          <set-status code="400" reason="Bad Request" />
          <set-body>{"error": "unknown API version. Use /v1 or /v2."}</set-body>
        </return-response>
      </otherwise>
    </choose>
  </inbound>
</policies>

The same logic exists in Apigee as a set of conditional flows and a RaiseFault policy. Different syntax, same behavior. That is the point. The behavior is one policy, expressed twice. We keep a short spec doc for "version enforcement behavior" and review both gateway implementations against it whenever either changes.

Versions need a registry, not just a URL pattern

A URL pattern tells clients where to go. It does not tell anyone what is live, what is deprecated, or who still calls the old version. For that you need a registry: every API, every active version, the owner team, the deprecation date, and the known consumers.

This sounds like paperwork. It is the thing that makes deprecation possible. Without a consumer list, deprecating v1 means guessing. With one, it means sending each consumer a date and a migration guide, then watching the traffic shift on the dashboard.

The registry does not need to be fancy. A structured API catalog backed by the same onboarding flow teams already use works. The important part is that publishing a new version and updating the registry are the same action, not two separate chores that drift apart.

Deprecation is a schedule, not an announcement

Announcing a deprecation and executing one are different jobs. Announcements get read and forgotten. Schedules get followed. A deprecation schedule has four parts:

A published end-of-life date, at least 90 days out for internal consumers, longer for partners. Nobody gets surprised.
Sunset headers on responses from the deprecated version, so clients see the date every time they call. Sunset and Deprecation headers exist for exactly this.
Staged traffic reduction: monitor v1 traffic weekly, reach out to the stragglers directly. The last 5 percent of traffic is always the hardest and always belongs to the team you least expect.
A hard cutoff, enforced at the gateway, on the published date. The gateway returns 410 Gone with a link to the migration guide. Soft cutoffs become permanent.

This is where the gateway policy pays for itself a second time. The cutoff is a one-line change in the routing policy, not a coordinated deploy across backend teams.

Validate the contract, not just the shape

Versioning keeps clients on the contract they coded against. Contract validation keeps the contract honest. At the gateway, that means schema validation on requests and responses where it is cheap enough to run.

The practical version: JSON Schema validation on the inbound side for critical endpoints. It catches the "we added a required field without bumping the version" mistake before it ships. Pair it with contract tests in CI on the backend side. The gateway check is the backstop, not the test suite.

Keep validation scoped. Validating every payload on a high-throughput API adds latency you will feel at P99. Validate the versioned endpoints that carry breaking-change risk, and let observability cover the rest.

The short version

Breaking changes are inevitable. Surprises are optional. One versioning scheme, enforced at every gateway, tracked in a registry, deprecated on a schedule, with contract validation at the choke point. That is the whole playbook. It is unglamorous work. It is also the difference between a Tuesday afternoon incident call and a migration that finishes before anyone notices.

Reusable Terraform Modules for API Gateway Provisioning Across Azure and AWS

Manoj Kumar Kagitha — Mon, 19 Oct 2026 14:07:00 +0000

Every team starts by copy-pasting Terraform. One folder becomes two. Two become ten. Then someone fixes a bug in one copy and the other nine keep the bug. Modules exist to end that cycle. A module is a packaged, versioned, reusable unit of infrastructure. Done well, it is the difference between a team that ships gateways in an afternoon and a team that hand-builds each one.

This post builds two real modules: one for Azure API Management, one for AWS API Gateway. Along the way: module anatomy, the common-interface pattern for multi-cloud teams, versioning, composition, testing, and the anti-patterns to avoid.

Why modules

Three reasons.

First, DRY. An API gateway has dozens of settings: SKUs, policies, products, subscriptions, backends. Define them once in a module. Every team consumes the same hardened configuration.

Second, consistency. When security requires a new policy on every gateway, you change it in one place and roll it out by bumping a version pin. Without modules, you are editing ten folders and hoping you caught them all.

Third, onboarding. A new engineer does not need to learn every APIM setting to ship an API. They call the module with five variables and get a production-ready gateway.

Anatomy of a good module

A module is a folder with a contract. The contract has three parts.

Variables with validation. Every input declares its type, a description, and where it matters, a validation block. Validation catches bad input at plan time instead of apply time.

Outputs. A module that creates things but tells you nothing about them is a black box. Output the IDs, URLs, and names callers need.

README. Document the purpose, inputs, outputs, and one full example. If the README does not show a working example, the module is not done.

Standard layout:

modules/apim/
  main.tf        # resources
  variables.tf   # inputs with validation
  outputs.tf     # what callers get back
  versions.tf    # required providers
  README.md      # purpose, example

The versions.tf pins the provider so the module never silently picks up a breaking provider release:

terraform {
  required_version = ">= 1.5"
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">= 3.0, < 4.0"
    }
  }
}

Azure APIM module

Here is a module that provisions an API Management service and registers a list of APIs on it, each with a rate-limit policy and a product assignment.

variables.tf:

variable "name" {
  type        = string
  description = "Name of the API Management service."
}

variable "location" {
  type        = string
  description = "Azure region for the service."
}

variable "resource_group_name" {
  type        = string
  description = "Resource group for the APIM service."
}

variable "sku_name" {
  type        = string
  description = "APIM SKU."
  default     = "Developer_1"

  validation {
    condition     = contains(["Developer_1", "Basic_1", "Standard_1", "Premium_1"], var.sku_name)
    error_message = "SKU must be one of Developer_1, Basic_1, Standard_1, Premium_1."
  }
}

variable "publisher_name" {
  type        = string
  description = "Publisher name shown in the developer portal."
}

variable "publisher_email" {
  type        = string
  description = "Publisher contact email."

  validation {
    condition     = can(regex("^[^@]+@[^@]+\\.[^@]+$", var.publisher_email))
    error_message = "Publisher email must be a valid email address."
  }
}

variable "apis" {
  type = list(object({
    name         = string
    display_name = string
    path         = string
    protocols    = list(string)
    backend_url  = string
  }))
  description = "APIs to register on the gateway."
  default     = []
}

main.tf:

resource "azurerm_api_management" "this" {
  name                = var.name
  location            = var.location
  resource_group_name = var.resource_group_name
  publisher_name      = var.publisher_name
  publisher_email     = var.publisher_email
  sku_name            = var.sku_name

  identity {
    type = "SystemAssigned"
  }
}

resource "azurerm_api_management_api" "this" {
  for_each = { for api in var.apis : api.name => api }

  name                = each.value.name
  resource_group_name = var.resource_group_name
  api_management_name = azurerm_api_management.this.name
  revision            = "1"
  display_name        = each.value.display_name
  path                = each.value.path
  protocols           = each.value.protocols
  service_url         = each.value.backend_url
}

resource "azurerm_api_management_api_policy" "rate_limit" {
  for_each = azurerm_api_management_api.this

  api_name            = each.value.name
  resource_group_name = var.resource_group_name
  api_management_name = azurerm_api_management.this.name

  xml_content = <<XML
<policies>
  <inbound>
    <rate-limit calls="100" renewal-period="60" />
    <base />
  </inbound>
  <backend>
    <base />
  </backend>
  <outbound>
    <base />
  </outbound>
  <on-error>
    <base />
  </on-error>
</policies>
XML
}

resource "azurerm_api_management_product" "this" {
  product_id            = "default-product"
  resource_group_name   = var.resource_group_name
  api_management_name   = azurerm_api_management.this.name
  display_name          = "Default Product"
  subscription_required = true
  approval_required     = false
  published             = true
}

resource "azurerm_api_management_product_api" "this" {
  for_each = azurerm_api_management_api.this

  api_name            = each.value.name
  product_id          = azurerm_api_management_product.this.product_id
  resource_group_name = var.resource_group_name
  api_management_name = azurerm_api_management.this.name
}

outputs.tf:

output "gateway_url" {
  description = "The gateway URL of the APIM service."
  value       = azurerm_api_management.this.gateway_url
}

output "service_id" {
  description = "The resource ID of the APIM service."
  value       = azurerm_api_management.this.id
}

output "api_ids" {
  description = "Map of API name to resource ID."
  value       = { for k, v in azurerm_api_management_api.this : k => v.id }
}

Notice the pattern: for_each over a map keyed by name. This keeps resource addresses stable when the list order changes. If you use count with a list, reordering the list destroys and recreates APIs. With for_each keyed by name, only real changes trigger updates.

AWS API Gateway module

Same idea, AWS REST API flavor. The module takes a list of endpoints and builds resources, methods, proxy integrations, a deployment, and a stage.

variables.tf:

variable "name" {
  type        = string
  description = "Name of the REST API."
}

variable "description" {
  type        = string
  description = "Description of the REST API."
  default     = "Managed by Terraform"
}

variable "stage_name" {
  type        = string
  description = "Deployment stage name."
  default     = "prod"

  validation {
    condition     = can(regex("^[a-z0-9-]+$", var.stage_name))
    error_message = "Stage name must be lowercase alphanumeric with dashes."
  }
}

variable "endpoints" {
  type = list(object({
    path_part   = string
    http_method = string
    backend_uri = string
  }))
  description = "Endpoints to expose through the gateway."
  default     = []

  validation {
    condition = alltrue([
      for e in var.endpoints : contains(["GET", "POST", "PUT", "PATCH", "DELETE"], e.http_method)
    ])
    error_message = "HTTP methods must be one of GET, POST, PUT, PATCH, DELETE."
  }
}

main.tf:

locals {
  endpoints = { for e in var.endpoints : "${e.http_method}-${e.path_part}" => e }
}

resource "aws_api_gateway_rest_api" "this" {
  name        = var.name
  description = var.description

  endpoint_configuration {
    types = ["REGIONAL"]
  }
}

resource "aws_api_gateway_resource" "this" {
  for_each = local.endpoints

  rest_api_id = aws_api_gateway_rest_api.this.id
  parent_id   = aws_api_gateway_rest_api.this.root_resource_id
  path_part   = each.value.path_part
}

resource "aws_api_gateway_method" "this" {
  for_each = local.endpoints

  rest_api_id   = aws_api_gateway_rest_api.this.id
  resource_id   = aws_api_gateway_resource.this[each.key].id
  http_method   = each.value.http_method
  authorization = "NONE"
}

resource "aws_api_gateway_integration" "this" {
  for_each = local.endpoints

  rest_api_id             = aws_api_gateway_rest_api.this.id
  resource_id             = aws_api_gateway_resource.this[each.key].id
  http_method             = aws_api_gateway_method.this[each.key].http_method
  integration_http_method = "POST"
  type                    = "HTTP_PROXY"
  uri                     = each.value.backend_uri
}

resource "aws_api_gateway_deployment" "this" {
  rest_api_id = aws_api_gateway_rest_api.this.id

  triggers = {
    redeploy = sha1(jsonencode([
      aws_api_gateway_resource.this,
      aws_api_gateway_method.this,
      aws_api_gateway_integration.this,
    ]))
  }

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_api_gateway_stage" "this" {
  deployment_id = aws_api_gateway_deployment.this.id
  rest_api_id   = aws_api_gateway_rest_api.this.id
  stage_name    = var.stage_name
}

Two things worth noting. First, the triggers block with a SHA hash: API Gateway deployments are immutable snapshots. Without the trigger, changing a method does not create a new deployment and your change never goes live. The hash forces redeployment whenever the API configuration changes. This is the single most common gotcha with Terraform-managed API Gateway.

Second, create_before_destroy on the deployment. Without it, Terraform destroys the old deployment before creating the new one, and the stage briefly points at nothing. With it, the new deployment exists before the old one is removed.

outputs.tf:

output "invoke_url" {
  description = "The invoke URL of the deployed stage."
  value       = aws_api_gateway_stage.this.invoke_url
}

output "rest_api_id" {
  description = "The ID of the REST API."
  value       = aws_api_gateway_rest_api.this.id
}

output "execution_arn" {
  description = "The execution ARN for IAM permissions."
  value       = aws_api_gateway_rest_api.this.execution_arn
}

The common-interface pattern

Here is the move that pays off for multi-cloud teams: give both modules the same variable names for the same concepts.

Both modules accept name. Both accept a list of backend definitions shaped as objects with a path, a method, and a backend URL. Callers learn one interface. When a team moves a workload from Azure to AWS, the module call changes but the shape of the inputs does not.

This does not mean the modules are identical inside. Cloud services differ, and each module should use its platform well. APIM gets products and XML policies. API Gateway gets deployments and stages. The interface is shared. The implementation is native.

Where the clouds genuinely diverge, let the variables diverge too. Do not force a fake common variable that maps to nothing on one side. Common where it is real, specific where it is not.

Versioned module sources

Never consume a module from a moving target. Pin to a Git tag:

module "apim" {
  source = "git::https://github.com/myorg/terraform-modules.git//modules/apim?ref=v1.2.0"

  name                = "apim-prod"
  location            = "eastus"
  resource_group_name = azurerm_resource_group.main.name
  publisher_name      = "Platform Team"
  publisher_email     = "platform@example.com"
  apis                = local.apis
}

The ?ref=v1.2.0 is the whole game. Without it, every terraform init can pull different code. With it, upgrades are deliberate: bump the ref, run plan, review the diff, apply.

Tag releases with semantic versioning. Breaking input changes bump the major version. New optional variables bump the minor. Fixes bump the patch. Write a short changelog per release so consumers know what changed before they bump.

Private registry option

If Git URLs get unwieldy across dozens of call sites, Terraform Cloud and Terraform Enterprise offer a private registry. Modules get versioned releases with a changelog UI, and the call site becomes:

module "apim" {
  source  = "app.terraform.io/myorg/apim/azurerm"
  version = "1.2.0"
}

Cleaner to read, easier to browse. It costs money and ties you to the platform. For most teams, Git tags are enough until the module count gets large. Start with tags. Move to a registry when discovery becomes the problem.

Composition pattern

Modules compose. A root module for an environment calls child modules and wires outputs to inputs:

module "network" {
  source = "git::https://github.com/myorg/terraform-modules.git//modules/network?ref=v2.0.1"

  name                = "vnet-prod"
  location            = "eastus"
  resource_group_name = azurerm_resource_group.main.name
  address_space       = ["10.0.0.0/16"]
}

module "apim" {
  source = "git::https://github.com/myorg/terraform-modules.git//modules/apim?ref=v1.2.0"

  name                = "apim-prod"
  location            = "eastus"
  resource_group_name = azurerm_resource_group.main.name
  publisher_name      = "Platform Team"
  publisher_email     = "platform@example.com"
  apis                = local.apis
}

module "dns" {
  source = "git::https://github.com/myorg/terraform-modules.git//modules/dns?ref=v1.0.3"

  zone_name   = "api.example.com"
  gateway_url = module.apim.gateway_url
}

Each child owns one concern. The root owns the wiring. Nobody edits child module internals from the root, and no child reaches into another child's state. Data flows through outputs, in one direction.

Version pinning discipline

Pin everything. Module sources get ?ref=. Providers get version constraints in versions.tf. Terraform itself gets required_version in the root.

Then automate the upgrades. Renovate or Dependabot can open pull requests when new module tags land. Each PR shows a plan diff against the new version. You review the diff, merge, and apply. Upgrades become routine instead of scary, and you never discover six months later that half your environments run a different module version than the other half.

One rule for the team: no PR merges with an unpinned module source. Enforce it in CI with a simple grep if you have to. The discipline matters more than the tool.

Module testing basics

Untested modules rot. Someone changes a default, nothing validates it, and three teams discover the breakage on their next apply. Three levels of testing, cheapest first:

terraform validate and tflint in CI on every pull request. Catches syntax errors, deprecated syntax, and style issues. Fast and free.
Example configurations in examples/ folders that get planned in CI. If the example does not plan cleanly, the module is broken. This catches the most common real-world breakage: a variable change that no longer matches how callers use the module.
Terratest (Go) for modules that must actually apply. It provisions real infrastructure, asserts on outputs, then destroys. Slower and costs cloud spend, so reserve it for widely used modules where a bad release hurts many teams.

At minimum, do levels 1 and 2. They catch most breakage for almost no cost.

Anti-patterns

Mega-modules. One module that provisions networking, gateways, databases, and DNS is not reusable. It is a snowflake with inputs. Split by concern. A module should do one thing well enough that a team can adopt it without adopting everything around it.

Unpinned versions. A source with no ?ref= is a time bomb. Today it works. Tomorrow someone pushes a breaking change and your next init breaks every environment at once. Pin always.

Provider config in child modules. Child modules must never declare provider blocks with credentials or aliases they invent. Providers are configured in the root and passed down. A child that configures its own provider cannot be reused across accounts, regions, or environments.

Boolean sprawl. A module with fifteen boolean flags (enable_x, create_y, skip_z) is twelve modules wearing a trench coat. If the flags cluster into real use cases, split the module along those lines. A few well-chosen flags are fine. Fifteen is a design smell.

Outputs nobody uses. Every output is a promise to maintain. If no caller reads it, delete it. Review outputs when you review the module.

Checklist

[ ] One concern per module. No mega-modules.
[ ] Every variable has a type, a description, and validation where it matters.
[ ] Outputs expose what callers need: IDs, URLs, names. Nothing more.
[ ] README with a working example that actually plans.
[ ] All module sources pinned with ?ref= tags. No exceptions.
[ ] Providers declared in versions.tf, configured only in the root module.
[ ] Semantic versioning on releases. Breaking changes bump major.
[ ] validate, tflint, and example plans running in CI.
[ ] Common variable names across cloud-specific modules where the concepts genuinely match.

Modules give you reusable building blocks with versioned releases. But who runs them, and when? If engineers still apply from laptops, you have versioned modules and unversioned process.

Next article: Terraform in CI/CD with GitHub Actions. Plan/apply gates, OIDC auth with no stored secrets, plan output posted as PR comments for review, and multi-environment promotion from dev to prod.

Terraform Remote State at Enterprise Scale: Backends, Locking, Bootstrapping, and Team Workflows

Manoj Kumar Kagitha — Wed, 14 Oct 2026 04:59:01 +0000

When you are the only one running Terraform, local state is fine. The moment a second engineer runs apply, you are one race condition away from a broken environment. And when you are running dozens of environments across teams, the questions get harder: who creates the state bucket, how do you split state so one bad apply cannot take down everything, and how do you give a team of twenty engineers safe access without handing everyone the keys to production?

This post covers the full picture: choosing a backend, the bootstrapping problem nobody warns you about, state layout strategy, locking, surgery when things go wrong, and the team workflow that holds it together.

Why local state breaks

terraform.tfstate on your laptop is a single point of failure. It cannot be shared. It cannot be locked. If your laptop dies, your state dies with it. Worse, if two people apply at the same time, the last write wins and the loser’s changes vanish from state while still existing in the cloud. That gap between state and reality is called drift, and it is painful to fix.

Remote state solves three problems: shared access, locking, and durability. But setting it up raises a question most tutorials skip.

The bootstrapping problem

Here is the chicken-and-egg problem: Terraform needs a backend to store state, but the backend itself (an S3 bucket, a DynamoDB table, a storage account) is infrastructure. Who creates it?

You have three options, and only one of them is right for teams.

Option 1: ClickOps. Someone creates the bucket by hand in the console. This works exactly once. Then nobody remembers the settings, the bucket has no versioning, no encryption, no lifecycle policy, and six months later someone asks why state history only goes back two weeks.

Option 2: A separate bootstrap Terraform project with local state. This is the standard answer. You keep one tiny Terraform project whose only job is to create the backend infrastructure. Its state stays local (or in a second-level backend), because it has to exist before everything else.

Option 3: Terraform managing its own backend. Do not do this. A Terraform project cannot reliably create the bucket that holds its own state. The init sequence makes this circular and fragile.

Here is what a bootstrap project looks like on AWS:

# bootstrap/main.tf - run once per account, state stays local
terraform {
  required_version = ">= 1.5"
}

provider "aws" {
  region = "us-east-1"
}

resource "aws_s3_bucket" "tfstate" {
  bucket = "myorg-terraform-state-prod"
}

resource "aws_s3_bucket_versioning" "tfstate" {
  bucket = aws_s3_bucket.tfstate.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "tfstate" {
  bucket = aws_s3_bucket.tfstate.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "aws:kms"
    }
  }
}

resource "aws_s3_bucket_public_access_block" "tfstate" {
  bucket                  = aws_s3_bucket.tfstate.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_dynamodb_table" "tflocks" {
  name         = "terraform-state-locks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

Versioning matters more than people think. If a state file gets corrupted, versioning lets you roll back to the last good version. Without it, you are reconstructing state from terraform import, resource by resource, which is exactly as fun as it sounds.

Run this once per account. Commit it to git. Then every other project points at the backend it created.

Picking a backend

Every major cloud has a native backend. The pattern is the same everywhere: an object store for the state file plus a locking mechanism.

AWS (S3 + DynamoDB):

terraform {
  backend "s3" {
    bucket         = "myorg-terraform-state-prod"
    key            = "network/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-state-locks"
    encrypt        = true
  }
}

Azure (Blob Storage):

terraform {
  backend "azurerm" {
    resource_group_name  = "tfstate-rg"
    storage_account_name = "tfstatestore"
    container_name       = "tfstate"
    key                  = "network.terraform.tfstate"
  }
}

Azure handles locking natively through blob leases. No separate lock table to manage. One less moving part, and one less thing to misconfigure.

GCP (GCS):

terraform {
  backend "gcs" {
    bucket = "myorg-terraform-state"
    prefix = "network"
  }
}

GCS locking is native as well.

State layout: one file per blast radius

Do not put your entire infrastructure in one state file. Split by environment and by component. A layout that works:

prod/
  network/terraform.tfstate
  compute/terraform.tfstate
  data/terraform.tfstate
  dns/terraform.tfstate
staging/
  network/terraform.tfstate
  compute/terraform.tfstate

Three reasons for this:

Blast radius. If the network state gets corrupted, compute and data are untouched. A bad apply can only destroy what is in its state file.
Speed. A plan against a 5,000-resource state file is slow. A plan against 200 resources is fast. Engineers stop running plan when it takes ten minutes. Then they apply blind.
Access control. Not everyone needs access to every state file. The data team does not need write access to network state. Smaller state files map cleanly to IAM boundaries.

The tradeoff is cross-component references. When compute needs the VPC ID from network state, use terraform_remote_state data sources or, better, pass values through a shared outputs pattern. It adds a little friction. The blast radius protection is worth it.

State locking in practice

Locking is automatic once the backend supports it. When you run apply, Terraform acquires the lock. Anyone else who tries gets an error instead of corrupting your state.

Two rules I follow:

Never use -lock=false in a shared workflow. It exists for emergencies, like a stuck lock after a crashed apply. If you use it, tell the team first.
If a lock gets stuck, terraform force-unlock <lock-id> clears it. Check with the team before you do. Someone might actually be mid-apply, and force-unlocking under them recreates the exact race condition locking was built to prevent.

State surgery: when you need to move resources

Sometimes the layout needs to change. You split a monolith state into components, or you move a resource from one team’s state to another’s. Terraform has tools for this, and they all operate on state without touching real infrastructure.

Move a resource between modules or addresses:

terraform state mv 'aws_instance.web' 'module.compute.aws_instance.web'

This rewrites the state file. The EC2 instance keeps running. Nothing is destroyed. But get the address wrong and Terraform will plan to destroy the old address and create the new one. Always run terraform plan immediately after a state mv. If the plan shows creates or destroys, you got the address wrong. Fix it before anyone applies.

Pull state down, fix it, push it back:

terraform state pull > /tmp/state.json
# edit carefully
terraform state push /tmp/state.json

This is the escape hatch for corrupted state. It is also the fastest way to make things worse if you edit the JSON by hand without understanding the schema. Use it as a last resort, and keep a backup of the original pull.

Remove something from state without destroying it:

terraform state rm 'aws_s3_bucket.legacy'

The bucket keeps existing in AWS. Terraform just stops managing it. Useful when decommissioning Terraform management of a resource, or when handing a resource to another team’s state file (rm here, import there).

Migrating from local to remote state

If you already have local state and need to move it, the process is straightforward:

Add the backend block to your configuration.
Run terraform init. Terraform detects the migration and asks: "Do you want to copy existing state to the new backend?" Say yes.
Verify with terraform plan. It should show no changes.
Delete the local terraform.tfstate and the backup. Commit the backend config.

The init migration is one of the few genuinely smooth operations in Terraform. It just works.

Keep secrets out of hardcoded config

Do not hardcode bucket names or credentials in your backend block. Use partial backend configuration and pass values at init time:

terraform {
  backend "s3" {}
}

terraform init \
  -backend-config="bucket=myorg-terraform-state-prod" \
  -backend-config="key=network/terraform.tfstate" \
  -backend-config="region=us-east-1" \
  -backend-config="dynamodb_table=terraform-state-locks"

This keeps environment-specific values out of version control and lets the same code work across dev, staging, and prod. In CI/CD, these values come from pipeline variables, not from the repo.

State files contain secrets

This one catches people. Terraform state stores every attribute of every resource, including passwords, keys, and connection strings, in plain text. Treat your state bucket like a secrets store:

Enable encryption at rest (the encrypt = true above, or the KMS config in the bootstrap).
Restrict IAM access to the state bucket to the CI/CD role and break-glass admins only. Engineers get read access for planning, not write.
Never commit a state file to git. Add *.tfstate and *.tfstate.backup to .gitignore on day one.
Enable bucket versioning so a corrupted state can be rolled back.

If you are already using Vault or a secrets manager for application secrets, that does not cover Terraform state. State is a separate secrets surface. Treat it as one.

The team workflow

The setup that works for most teams:

Engineers run terraform plan locally against the remote state. Plans are safe. They change nothing. Read access to state is enough.
Pull requests include the plan output. Reviewers check what will change before it changes.
Only CI/CD runs terraform apply, after the plan is approved. The pipeline uses a dedicated IAM role with write access to state.
Applies run serially per state file. The lock enforces this mechanically, but the pipeline should also avoid queuing parallel applies to the same state.

Local plan, reviewed apply, locked state. That is the whole discipline. It scales from two engineers to two hundred without changing shape.

Pre-flight checklist

Before you call your remote state setup done:

[ ] Backend infrastructure created via a bootstrap project, not by hand
[ ] Versioning enabled on the state bucket
[ ] Encryption at rest enabled
[ ] Public access blocked on the state bucket
[ ] Locking configured and tested (run two applies, confirm the second waits or fails cleanly)
[ ] State split by environment and component, not one giant file
[ ] .gitignore covers *.tfstate and *.tfstate.backup
[ ] IAM: engineers read, CI/CD writes, break-glass admins only
[ ] Backend values passed via -backend-config, not hardcoded
[ ] Team knows the force-unlock procedure and when not to use it

Closing

Remote state is not exciting work. Nobody demos a backend block at a conference. But every production Terraform setup stands on it, and every Terraform horror story I have heard starts with someone skipping one of the items on that checklist.

Get the backend right, split your state by blast radius, lock everything, and keep secrets out of git. The rest of your Terraform practice gets easier from there.

Next week: reusable Terraform modules for API gateway provisioning across Azure and AWS, with versioned module registries and composition patterns that survive team growth.

I Gave My AI Agents an Org Chart

Manoj Kumar Kagitha — Thu, 08 Oct 2026 21:31:44 +0000

I Gave My AI Agents an Org Chart

Most agent frameworks hand you a pile of scripts and wish you luck. You wire up three agents, write the orchestration glue yourself, and six weeks later you have a system nobody can explain. I kept running into this, so I built something with a different shape: a company.

It is called Botropolis. Twenty specialist agents across ten departments, one CEO orchestrator, and a training pipeline so each agent can eventually get its own fine-tuned model. It is open source, MIT licensed, and it runs with no API keys.

The idea

Companies figured out coordination centuries ago. When work gets complex, you do not throw twenty generalists into a chat room. You create departments, give people job titles, and let a manager break big requests into pieces. Botropolis is that idea in Python.

You ask Manoj, the CEO, for something. He looks at the request, decides which departments need to be involved, and routes the work. Research goes to Scout, code goes to Coder, the numbers go to Analyst. Each agent has a YAML spec: name, title, specialty, tools it may call, example tasks. Anyone reading the repo can see who does what, the way an org chart tells you who owns what.

What it actually does

The roster covers ten departments: research, health, finance, code, data, legal, marketing, ops, security, and support. Some highlights:

Teaming. One agent alone is fine, a team is better. You name the agents and the rounds, and they draft, critique, and revise in sequence. Each agent sees everything its teammates produced before it. Manoj writes the final synthesis.
Real tools, no vendor lock-in. Agents with toolkits run a think-act-observe loop over a plain-text protocol. No provider function calling involved, so it works with OpenAI, Anthropic, Gemini, Ollama, or a model you trained yourself. Fourteen tools ship in the box: web search, shell, file tools, Gmail, calendar, a shared company notebook.
A web UI with no build step. Plain HTML, CSS, and JS served by the same FastAPI process. Chat with the CEO, a war room for talking to one agent directly with live tool-call streaming, teaming controls, per-agent analytics, and a roster browser. It works on a phone.
Training pipeline. Datasets, configs, an eval harness, and model cards are all in the repo. I trained Scout-tiny from scratch as a proof of concept: a 5.49M-parameter transformer on 300 Q&A pairs, validation perplexity 5.17. Demo scale, honestly labeled as such. The real path is LoRA on a small open model, and the scaffolding for that is documented.

The honest parts

It runs with zero API keys against an offline stub that labels its output as stub output. Clone it, run python examples/demo.py, and watch the whole company work before spending anything.

The safety model is pragmatic, not paranoid. File and shell tools are sandboxed to a workspace directory with timeouts and a denylist for destructive patterns. Agents draft emails; humans send them. Health and legal agents carry explicit disclaimers in their prompts.

Try it

git clone https://github.com/kagithamanoj/botropolis.git
cd botropolis
pip install -e ".[dev]"
python examples/demo.py

Or skip the clone and talk to the CEO directly:

from botropolis.core.orchestrator import CEO

report = CEO().handle("Research vector databases and compare pricing")
print(report.summary)

The org chart grows by pull request. If you have ever wanted an agent with a job title, come add a department.

Botropolis is built and maintained by Manoj Kumar Kagitha, a Senior Cloud Engineer working on production cloud infrastructure. Nine papers, seven books, and now one company of bots.

We Cut API Tenant Onboarding From Two Days to Three Hours With a Self-Service Portal

Manoj Kumar Kagitha — Wed, 07 Oct 2026 14:58:48 +0000

Before the portal existed, onboarding a new team to our API ecosystem looked like this: a service request, a ticket queue, two or three handoffs between teams, a manual runbook that nobody kept updated, and about two business days of waiting. Not because anyone was slow. Because every step needed a human.

We run a multi-tenant API platform on Azure, with Azure APIM as the front door, Key Vault for secrets, and Terraform underneath. The traffic scale forced us to treat onboarding like a product, not a favor. Here is what we built and what broke along the way.

What onboarding actually costs without self-service

The hidden cost was never the two days. It was the shape of those two days:

A developer submits a request form that half the time misses a field, which restarts the clock.
Someone in my team creates the API subscription, the product assignment in APIM, and the service principal in Azure AD, all by hand.
Credentials get emailed or dropped in a chat, which is a security conversation I no longer want to have.
When something breaks at 2 AM, nobody remembers which team owns which subscription.

Every manual step is a queue. Queues are where latency lives. This was true of our request pipeline exactly the same way it is true of API traffic, which is a fun mirror I keep noticing.

The portal, in concrete terms

We built a self-service onboarding portal backed by a Node.js API. The frontend gives requesting teams a single form. The backend does the actual work through automation:

Validates the request (tenant id, contact, required scopes, environment).
Creates the subscription and product assignment in APIM through the management API.
Provisions credentials into Key Vault, never shown in plaintext after creation.
Wires the Terraform modules that own the tenant's networking and access, so nothing is a snowflake.
Registers the tenant in our traffic analytics, so day-one dashboards include max RPS and P99 latency per endpoint.

The important design decision: the portal does not bypass our existing automation. It calls it. Terraform remains the source of truth for infrastructure. The portal is an interface over the same modules we already run in Azure DevOps pipelines. That meant onboarding a tenant and running the deployment pipeline used the same code paths, which removed an entire class of drift between what the portal did and what IaC expected.

What we got wrong first

We automated the happy path and forgot the exceptions. The first version handled the standard request perfectly and failed loudly on anything unusual, like a team that needed a nonstandard scope or an existing tenant adding a second region. Those requests fell back to tickets anyway, and for a month our metrics looked great while real users quietly went around the portal. Fix: every rejection path in the portal now produces a specific, actionable message, and edge cases route to a human with full context attached instead of a blank ticket.

We treated secrets as a delivery problem. Early on the portal returned API keys in the response payload. It worked and it was wrong. Credentials now go straight into Key Vault with an RBAC assignment for the requesting team, and the portal only ever confirms that provisioning succeeded. If a team loses a key, rotation is one click and the old one is revoked automatically. This is the same rotation discipline I wrote about for certificates: never let a secret's lifecycle depend on a human remembering.

We skipped the audit trail. When an incident involved a tenant's credentials, we could not answer "who created this, when, and with what scopes" without digging. Every provisioning action now writes a structured log entry with requester, approver (when required), timestamp, and resulting scope. Boring. Essential.

The multi-tenant part is where it gets interesting

Single-tenant onboarding is a form. Multi-tenant onboarding is policy. The questions that took real design work:

Isolation boundaries. Tenants share APIM products but get separate subscriptions, separate Key Vault secrets, and scoped RBAC. Blast radius is a first-class design input, not an afterthought.
Rate limits per tenant. Global rate limits protect the platform; per-tenant limits protect tenants from each other. We set both, and the portal assigns them at onboarding time based on the tier the team requests.
Environment parity. Dev, staging, and production onboarding go through the same portal flow. The Terraform workspace differs, not the process. A team that can self-serve in dev already knows how prod works.

What changed, measured honestly

Onboarding went from roughly two business days to about three hours, measured from form submission to working credentials. The three hours is mostly pipeline runtime and approvals, not queueing. The number I actually care about is support tickets: onboarding-related tickets dropped to near zero because the failure modes are now either automated or self-explanatory.

The portal also changed my team's job. We stopped being the onboarding bottleneck and started being the platform owners who review policy changes and handle exceptions. That is a much better use of a cloud engineering team.

If you are about to build one

A few things I would tell myself at the start:

Build the portal on top of your existing automation, not beside it. If the portal and your IaC pipelines can disagree, they will.
Ship the audit trail in version one. You will need it before you think you will.
Automate rejection paths, not just success paths. The quality of an onboarding system is measured by what happens to the request that does not fit.
Never return a secret in an API response. Not once, not for convenience.

Self-service is not about removing humans. It is about moving humans from the critical path to the exception path. That is where judgment belongs.

What Max RPS and P99 Latency Actually Tell You About Your API

Manoj Kumar Kagitha — Mon, 05 Oct 2026 14:41:05 +0000

Our global API serves requests across several regions and teams. A few years ago we started treating traffic analytics as a first class part of running it, not something we look at only after an incident. The two numbers that ended up mattering most were max requests per second and P99 latency. Here is what we actually do with them, and what they taught us.

Why these two numbers

Average latency lies. If one in ten requests is slow, the average still looks fine and users complain anyway. P99 tells you how the slowest one percent of your traffic behaves. When P99 is healthy, almost everyone is having a good experience. When it is not, you have a problem that affects a real slice of your users.

Max RPS tells you how close you are to the ceiling. Every backend, every gateway route, every downstream service has a limit. Knowing the peak rate you actually handled in the last week is more useful than the capacity number someone wrote on a slide. When the peak keeps climbing toward the limit, you plan. When it spikes past it, you scramble.

We track both per endpoint and per consumer where we can. A global number hides the fact that one tenant's burst is what keeps tripping your alarms.

Where the data comes from

Our APIs run behind gateways, Apigee X and Azure API Management, and both emit rich telemetry. Every request produces timing and status data. We aggregate it into dashboards that slice by endpoint, consumer, region, and time window. Nothing exotic. The work is in keeping the dashboards honest and the alerts tuned.

One thing that surprised me early on: gateway timing and backend timing are different numbers, and the difference is where the interesting bugs live. If the gateway reports a request took 800ms but the backend says it took 300ms, the other 500ms went somewhere in between. That gap is policies, retries, connection setup, or network path. We watch the gap per route now. It catches misconfigurations that no amount of backend monitoring would find.

What P99 taught us about retries

Our first P99 problem turned out to be our own retries. A downstream service was occasionally slow, so clients retried. The retries doubled the load on exactly the requests that were already struggling. The average looked okay. P99 climbed steadily for weeks before anyone noticed.

The fix was boring and effective. We added backoff with jitter to the retry policy at the gateway, and we set a hard timeout so a slow request fails fast instead of dragging on. P99 dropped immediately. The lesson: retries are the easiest way to turn a small latency problem into a large one, and P99 is where you see it first.

What max RPS taught us about capacity

We once had a batch job from a partner team that ran every night and hammered one endpoint at many times the normal rate. It was within their quota, so no alarms fired. But it was eating the headroom we needed for real traffic spikes during the day.

Max RPS made this visible. The nightly peak was clearly out of proportion to daytime traffic. We moved the job to a dedicated rate limit tier with its own quota, and daytime headroom went back to normal. Quotas protect you, but they do not tell you that one consumer is distorting your capacity picture. The peak numbers do.

Alerting on these numbers

We set alerts on sustained P99 degradation per endpoint, not on single spikes. A single slow request is noise. Thirty minutes of elevated P99 is a signal. We also alert on max RPS approaching a configurable fraction of known capacity, which gives us a heads up before the limit is real.

The alerts only work because we keep thresholds per endpoint. A login endpoint and a heavy reporting endpoint have completely different latency profiles. One global threshold is either always firing or never firing. Per-endpoint thresholds take a week to tune and then they stay quiet until they matter.

What I would do differently

I would have started tracking the gateway-to-backend gap sooner. It is the single most diagnostic number we have now, and we found it late. I would also have labeled traffic by purpose earlier: interactive user traffic versus batch jobs versus health checks. Mixed together, batch traffic makes your P99 and RPS numbers tell a story that is not really about your users.

The unglamorous truth is that traffic analytics is mostly plumbing and patience. You collect the numbers, you tune the alerts, you look at the dashboards when they page you. The payoff is that incidents stop being mysteries. When something is slow, you already know which endpoint, which consumer, and which part of the path. That is worth more than any single fix we ever shipped.

I built a company of bots

Manoj Kumar Kagitha — Wed, 30 Sep 2026 23:49:17 +0000

Most agent demos are one clever prompt. I wanted to see what happens when you organize agents the way a real company is organized: departments, specialists, and someone at the top deciding who does what.

That is Botropolis: 20 specialist agents across 10 departments. Research has Scout. Code has Coder and Reviewer. Finance, legal, health, data, marketing, ops, security, support all have their people. A CEO agent reads your request, routes it to the right departments, runs them, and compiles one report.

Each agent is a YAML spec: name, title, specialty, model, tools, system prompt, example tasks. The registry loads them, the CEO orchestrates, and a FastAPI server exposes it all. There is a browser UI with three views: chat with the CEO, a war room for talking to any single agent directly, and analytics showing per-agent calls, latency, and token usage.

The part I am proudest of is the training pipeline. I trained botropolis-scout-tiny, a 5.5M-parameter transformer, from scratch on CPU on 300 synthetic Q&A pairs. It is a demo-scale model and honest about it: it learned the shape of research answers, not reliable facts. The pipeline is the point. On top of that I built a 2,940-example curated dataset and a Colab script that fine-tunes SmolLM2-135M with LoRA on a free GPU. That is the path to a real model.

What I learned:

Routing is the whole game. Twenty agents are useless if the wrong ones get the work.
YAML specs make agents reviewable. You can read the whole company in an afternoon.
Usage analytics change how you build. Once you see per-agent latency and tokens, you start routing like a manager watching a budget.

Roadmap: streaming responses, tool execution wired into the model clients, and fine-tunes for more agents.

Repo (MIT): https://github.com/kagithamanoj/botropolis

Certificates Expire on the Worst Possible Day

Manoj Kumar Kagitha — Wed, 30 Sep 2026 14:17:24 +0000

Every team I have worked with has a certificate story. The classic version goes like this. A certificate expires on a holiday weekend. The renewal was assigned to someone who left the team months earlier. Production starts serving TLS errors, and the fix takes eleven minutes while the explanation takes three weeks.

Certificates are the most boring part of infrastructure. They are also the part that fails most loudly. A database can degrade slowly. A certificate either works or it does not. One day the handshake succeeds. The next day every client refuses to connect. There is no graceful mode.

This is a walkthrough of how I think about certificate lifecycle management in a large Azure environment: discovery, automation, alerting, and rotation. Nothing here requires exotic tooling. It requires discipline.

Find every certificate first

You cannot manage what you cannot see. Before automation, before policies, before dashboards, you need a list. Every certificate in every environment, with its expiry date, its issuer, and the service that serves it.

In a real enterprise this is never one place. Certificates live in Azure Key Vault, in API Management custom domains, on Application Gateways, on Front Door endpoints, in Kubernetes secrets, and in that one VM nobody documented. In my experience, the first job on certificate work is not renewal. It is inventory.

One Azure CLI query gives you the foundation:

az keyvault certificate list \
  --vault-name $VAULT \
  --query "[].{name:name, expires:attributes.expires}" \
  -o table

Loop that across every vault in every subscription on a schedule and you have expiry visibility. Pipe the output into your monitoring system and you have the start of alerting.

The inventory needs to live somewhere queryable. A spreadsheet dies the day someone forgets to update it. Pull the data into a dashboard or at least a scheduled report. And every certificate needs an owner. Anything marked "unknown" gets one assigned within a week. The unowned certificates are the ones that expire.

Renewal has to be automatic

Manual renewal does not scale, and it does not survive team changes. The person who knows the renewal procedure today will not be on the team in two years. If the process depends on a person, it will fail.

Azure Key Vault can auto-renew certificates issued through integrated CAs. For public CAs, you set up the renewal once and let the vault handle it from there. For Kubernetes workloads, ACME-based renewal with cert-manager removes the human from the loop entirely.

The hard part is not the renewal. It is the deployment. Renewing a certificate in the vault means nothing if the Application Gateway or API Management instance keeps serving the old one. The renewal pipeline has to push the new certificate to every consumer. Key Vault references in App Service and Functions pick up new versions automatically. For gateways and load balancers, you need a pipeline step or an event-driven function that notices the new version and applies it. Test this end to end in a non-production environment first. The renewal ceremony is the whole chain, not the issuance.

Alert long before expiry

Automation fails too. CAs have outages. Permissions drift. Someone changes a firewall rule and the renewal challenge cannot reach the domain. You need alerts that fire early enough to fix the problem by hand.

The schedule I recommend is 60-30-7. Sixty days out: informational, goes to the team channel. Thirty days out: warning, gets a ticket. Seven days out: page someone. Most certificates are fine at sixty days. The point of the early alert is not urgency. It is to catch the broken automation while you still have options.

The alert must go to a team, not a person. Certificates outlive people. Route expiry alerts to a distribution list or an on-call rotation. If the alert goes to an individual inbox, it will eventually go to a departed employee's inbox, which is the same as no alert at all.

Rotate without downtime

Renewing the certificate is step one. Serving it without dropping connections is step two. Most Azure services handle this well now. Application Gateway lets you update the listener certificate without downtime. API Management custom domains can be updated in place. Kubernetes ingress controllers pick up new secrets on reload.

The pattern that keeps you safe: renew early, deploy the new certificate while the old one is still valid, and verify the serving path before the old one expires. Keep the old certificate in the vault for a short overlap period. If something goes wrong with the new one, you roll back in minutes instead of scrambling for a reissue.

Test your rollback. I mean actually test it, in staging, with someone watching the metrics. A rollback plan you have never run is a theory.

Do not forget internal certificates

Public certificates get all the attention because they are customer facing. Internal certificates expire just as hard. Service-to-service mTLS, internal APIs, webhook endpoints between systems: all of them have expiry dates, and they tend to be managed by nobody in particular.

If your organization runs an internal CA, treat its certificates with the same lifecycle as public ones. Same inventory, same alerts, same automation. The internal CA root itself needs a plan. Root rotation is rare and painful, which is exactly why it needs to be written down before it is urgent.

What I would do on day one

If you inherited certificate management tomorrow, here is the order I would work in.

First, build the inventory. Script discovery across every Key Vault and every TLS endpoint you can reach. Assign an owner to each certificate.

Second, set up expiry alerting on the 60-30-7 schedule, routed to a team channel and an on-call rotation.

Third, pick the certificates closest to expiry or most critical, and automate their renewal end to end, including deployment to the serving infrastructure.

Fourth, extend automation to the rest, starting with the public-facing ones.

Fifth, document the rollback procedure and test it once a quarter.

None of this is exciting work. That is the point. Certificates should be boring. The goal is a system where expiry is a non-event: the certificate renews, the pipeline deploys it, the alert never fires, and nobody remembers it happened. Boring is the victory condition.

An Enterprise LLM Gateway: Centralized Access, Metered Usage, and Guardrails for Production AI on Azure

Manoj Kumar Kagitha — Wed, 30 Sep 2026 06:50:43 +0000

Manoj Kumar Kagitha

Eficens Systems Inc., Dallas, TX, USA

mymanoj11@gmail.com

Abstract: Every enterprise AI effort starts the same way. A team gets an API key, wires it into an application, and ships. Then the second team does the same, and the third. Months later nobody can answer basic questions. Who is calling the models? Which teams spend what? Which data crosses the boundary, and under whose authority? This paper describes an enterprise LLM gateway built to answer those questions: a central control plane for model access on Azure, with identity-aware routing, per-tenant metering, rate limiting, and guardrails. We cover the requirements that shaped it, the architecture, how metering and cost attribution work, how policy is enforced at the gateway, how the platform is provisioned with Terraform and delivered through CI/CD, and what operating it taught us. The aim is practical: a reference design for teams facing the same sprawl. We state plainly what this paper does not do. It reports no controlled experiments and no benchmark numbers. It is an experience report from production work, and the lessons are the contribution.

Index Terms: LLM gateway, AI governance, metering, guardrails, Azure, MLOps, cost attribution.

I. Introduction

The pattern is familiar to anyone who has run platform engineering for a large company. A new capability appears. Early teams adopt it directly, each wiring their own credentials and their own plumbing. It works, so it spreads. Then scale arrives with its usual questions, and nobody has answers. With large language models the cycle ran faster than usual, because the capability was easy to adopt and the costs were easy to ignore. Tokens are invisible until the invoice. Data flows are invisible until the audit.

In our environment the sprawl took a specific shape. Product teams built LLM features against Azure OpenAI and Gemini. Each team held its own keys. Some proxied through small in-house services. Most did not. When finance asked which teams spent what, the answer was a spreadsheet of guesses. When security asked which data reached which model, the answer was worse. The models were the easy part. Governing access to them was the missing system.

The gateway idea is old. Enterprises have run API gateways for years: one front door for authentication, rate limiting, and audit. Our earlier work on onboarding APIs into multi-tenant Azure environments at scale covered exactly that plumbing [1]. An LLM gateway applies the same instinct to model traffic, with three additions that API gateways never needed. Tokens must be metered per call. Prompts and responses must be screened against policy. And requests must be routed across models by task, because model choice is now a cost and quality decision on every call.

This paper describes the gateway we built: a central control plane for LLM access on Azure. Every model call in the enterprise flows through it. It authenticates the caller, resolves the tenant, checks policy, routes to a model, meters the tokens, enforces budgets, and logs everything. Around it sit the provisioning and delivery practices that keep it running: Terraform for infrastructure, CI/CD for safe change, and guardrails from Google's Model Armor on the screening side.

We make five contributions. First, the requirements that an enterprise actually imposes on model access, drawn from finance, security, and platform teams. Second, a reference architecture for the gateway itself. Third, a metering and cost attribution design that turns the loudest budget argument into a dashboard. Fourth, a guardrail and policy enforcement design built on check-before-execute discipline. Fifth, operating lessons from running the gateway under real load. Readers who want agent orchestration patterns should look at our companion work. This paper is about the control plane underneath them.

II. Background and Related Work

A. From API Gateways to LLM Gateways

The enterprise already solved a version of this problem. API gateways centralize authentication, rate limiting, transformation, and audit for service traffic. Our SSRN paper documents onboarding APIs into multi-tenant Azure environments at scale, including the identity and policy plumbing such gateways need [1]. Much of that transfers directly: authenticate every call, resolve the tenant, enforce per-tenant limits, log every decision. What does not transfer is the payload. API gateways meter requests. LLM gateways must meter tokens, screen natural language against policy, and route across models with different cost and capability profiles. The gateway shape is familiar. The metering and screening are new.

B. Metering and Observability

You cannot govern what you cannot measure. Our earlier work on AI-driven observability for hyperscale data centers argued for instrumenting first and analyzing second [2]. The GreenOps work that followed built telemetry pipelines for carbon analysis, which transfer cleanly to token analysis: the same collection, aggregation, and attribution machinery applies [3]. Our book on hyperscale telemetry covers the pipeline thinking in full [4]. The gateway applies it to model traffic. Every call emits token counts, latency, model, tenant, and team. The metering pipeline turns that stream into budgets, alerts, and chargeback.

C. Policy Enforcement

Policy that lives in documentation does not hold. Our 2021 work on automated security compliance in CI/CD used NLP to flag infrastructure-as-code vulnerabilities before deployment, on the principle that the check must run before the action executes [5]. Our Cognitive DevOps book traced the same idea through pipelines: safety properties have to survive automation, not sit beside it [6]. The gateway is the enforcement point where this principle meets model traffic. Policy is expressed as versioned artifacts, evaluated on every call, with denials logged. A policy nobody evaluates is a hope, not a control.

D. Guard Models and Screening

Screening prompts and responses is now a commodity capability. Llama Guard showed that a small fine-tuned model can screen both sides of a conversation at low cost [7]. Google's Model Armor provides the same function as a managed service, which is what our deployment uses. Screening text is necessary but not sufficient. The tool call is where damage happens, and Section VI covers the action side. The text side belongs in the gateway because every prompt and every response already flows through it.

E. Traffic Management

Rate limiting model traffic borrows directly from classic traffic management. Our 2019 work on load balancing in distributed cloud environments covered the scheduling instincts: protect the shared resource, isolate noisy neighbors, shed load gracefully [8]. Tokens are the constrained resource here. Per-tenant rate limits, per-model concurrency caps, and queueing with clear backpressure signals all come from the same playbook, applied to a new bottleneck.

F. Deployment Discipline

A gateway in the critical path must change safely. Our book on the engineering realities of deploying enterprise AI makes the case at length: the unglamorous disciplines of versioning, canarying, and rollback decide whether AI systems survive production [9]. The Sentient Cloud book covers autonomous operations from the infrastructure side [10], and our most recent book unifies the AIOps and hyperscale picture [11]. Section VII applies that discipline to the gateway: versioned configuration, staged promotion, and rollback that covers policy as well as code.

III. Requirements

The requirements came from three groups who rarely agree: finance, security, and the product teams shipping features. Finance wanted attribution. Every token had to map to a team and a tenant, with budgets and a way to do chargeback. Security wanted control. Every call had to carry an identity, cross a policy check, and land in an audit log. Product teams wanted speed. One endpoint, one SDK, one key per team, and no ticket queue to start building. The gateway had to satisfy all three without becoming the reason features shipped late.

Identity and tenancy came first. Every call carries who is asking: user, team, tenant, and the agent role when an agent calls. Tenancy is enforced in the data plane. Retrieval filters by tenant at the index, caches are keyed by tenant, and credentials are minted per tenant. A prompt instruction to respect tenancy is not a control. Filtered data access is.

Metering had to be per call and per token, attributed to teams, with budgets that actually stop traffic. Alerts alone do not change behavior. Hard stops do, so budgets come with enforcement, not just notification. Rate limiting sits beside metering: per-tenant limits, per-model concurrency caps, and clear signals when a caller is being throttled.

Guardrails had to cover text and actions. Prompts and responses go through screening. Tool calls, where agents are involved, go through allowlists scoped to the caller's role. High-risk actions wait for human approval. Every denial is logged, because a mitigation you cannot audit is a hope, not a control.

Reliability was non-negotiable. The gateway sits in the critical path of every AI feature, so it cannot become the outage. Stateless design, horizontal scaling, and graceful degradation: if the screening service is slow, the gateway fails closed on policy but stays open on traffic where policy allows cached decisions. Developer experience closed the list. One endpoint, one SDK, one key per team. If using the gateway is harder than going around it, teams will go around it, and the whole exercise fails.

IV. Architecture

The gateway is a set of stateless services behind a load balancer, with state kept in dedicated stores. Fig. 1 in a longer version of this paper would show the request path. In prose it runs like this. A call arrives at ingress with a team key or a managed identity token. Ingress authenticates the caller and resolves the tenant. The policy engine evaluates the request against the tenant's policy bundle: which models are allowed, what the budget state is, whether the caller is within rate limits. The screener checks the prompt text. The router picks a model and an endpoint. The call executes. The screener checks the response. The metering pipeline records tokens, latency, model, tenant, and team. The audit log records the decisions. Every stage can deny, and every denial is logged with its reason.

Four components deserve detail. The policy engine evaluates versioned policy bundles, one per tenant, reviewed like firewall rules. Policies name allowed models, budget caps, rate limits, screening strictness, and data handling rules. The router applies model routing rules: the strongest model for planning and judgment-heavy calls, smaller models for bulk execution and classification. The general lesson is old. Our 2016 comparison of machine learning algorithms found, unsurprisingly, that no single algorithm wins everywhere [12]. The same holds for models, where reasoning strength, context length, and cost pull in different directions. Routing is how the gateway turns that observation into savings.

The metering pipeline is a stream processor, not a batch job. Token counts arrive per call and are aggregated into per-team, per-tenant, and per-model series within seconds. Budgets are evaluated against live aggregates, which is what makes hard stops possible. The pipeline design follows the telemetry thinking from our GreenOps and observability work: collect everything, aggregate fast, attribute precisely [2][3]. The audit log is append-only and tenant-scoped. When security asks who called what with which data, the answer comes from this log, not from anyone's memory.

Identity flows through the whole path. Human callers authenticate with team keys or Entra ID. Service callers use managed identities, which Azure rotates automatically. Agent callers carry the agent role plus the user and tenant they act for, so tool calls made through the gateway inherit the right scope. Long-lived keys are the exception, issued only where managed identities cannot reach, with rotation enforced. Our earlier work on secure transmission in IoT systems covered the same ground at the device layer: credentials should be short-lived, scoped, and rotated on schedule [13].

A. Tenancy in the Data Plane

This deserves its own subsection because it is where designs fail. Tenancy enforced only in prompts is not enforcement. The gateway enforces it in the data plane at four points. Retrieval filters by tenant at the index, so a query can never match another tenant's documents. Caches are keyed by tenant, so a cached result never crosses the boundary. Credentials are minted per tenant, so a tool call cannot borrow another tenant's authority. And the audit log is partitioned by tenant, so investigations stay scoped. Each of these is a deliberate inefficiency. Sharing caches or credentials across tenants would be cheaper. The cost of a cross-tenant leak dwarfs any infrastructure saving, so we do not share.

B. Reliability

The gateway is in the critical path, so it is designed to fail safely. Services are stateless and scale horizontally. The policy engine caches compiled policy bundles locally with a short TTL, so a control-plane outage does not stop the data plane. Screening calls have tight timeouts with a defined fallback: fail closed for tenants under strict policy, allow with a logged exception where policy permits. Rate limiting uses token buckets per tenant, which degrade gracefully under load instead of collapsing. We monitor the gateway the way we monitor any critical service: latency percentiles at each stage, error rates, screening decision rates, and metering pipeline lag. If the metering pipeline lags, budgets evaluate against stale data, which is itself an alert condition.

V. Metering and Cost Attribution

Agents spend tokens the way services spend CPU: silently, then all at once. The gateway makes the spending visible and attributable. Every call records prompt tokens, completion tokens, model, endpoint, tenant, team, and latency. Attribution is exact because identity is resolved before the call executes, not inferred after.

Three controls matter most. Budgets per team and per tenant, with hard stops when the budget is exhausted and warnings before that. Caching of repeated work: identical prompts within a short window, repeated retrieval results, and repeated tool outputs, all cached with tenant-scoped keys and short TTLs. And model routing, which moves bulk work to smaller models and reserves the strongest models for planning and judgment. Track cost per completed task, not just total spend. A workflow that costs twice as much per task needs to succeed enough more to justify it.

Chargeback turned the loudest argument into the quietest dashboard. Before the gateway, AI spend was a single line item that every team disowned. After it, each team sees its own series: spend by model, by endpoint, by day, against its budget. Finance gets the allocation it needs. Teams get the feedback they need to optimize. The interesting effect was behavioral. Once teams could see their own spend, they started asking for the cheaper routing themselves. Visibility did more than any mandate.

Cache design deserves care because caches are where tenancy and correctness meet. Prompt caches key on the exact prompt plus model plus tenant, with a short TTL. Retrieval caches key on query plus corpus version plus tenant. Tool result caches key on arguments plus tenant, with TTLs matched to how fast the underlying data changes. Nothing caches across tenants. Stale caches cause wrong answers, so every cache entry carries its age and the gateway exposes cache hit rates per endpoint. A sudden drop in hit rate is usually a sign that something upstream changed.

VI. Guardrails and Policy Enforcement

The gateway enforces policy at three points in a call. The prompt is screened before it reaches the model. The tool call, where agents are involved, is checked against an allowlist scoped to the caller's role. The response is screened before it returns to the caller. Each point can deny, and every denial lands in the audit log with the policy rule that fired.

Text screening uses Model Armor on both sides of the call. It catches the obvious categories: disallowed content, prompt injection patterns, and sensitive data in either direction. Screening is tuned per tenant, because a strictness setting that suits a customer-facing assistant will strangle an internal engineering tool. False positives are reviewed weekly, and the tuning notes go back into the policy bundle. A guardrail that nobody tunes becomes either decoration or an obstacle.

The action side is where our earlier compliance work applies directly. Our 2021 paper used NLP to flag infrastructure-as-code vulnerabilities before deployment, on the check-before-execute principle [5]. The gateway applies the same principle to tool calls. Tools are tiered by risk. Read-only tools get broad approval. Tools that write to non-production systems need task-scoped approval. Tools that touch production or move money need human approval, every time, no exceptions. The allowlist is versioned with the agent and reviewed like firewall rules. When a new tool arrives, its tier is the first decision, before any prompt is written.

Indirect prompt injection gets special handling because it defeats input screening. The attack hides instructions in tool output: a document, a ticket comment, a search result that the agent reads mid-run. The gateway tags retrieved and tool-returned content with its source and treats it as data, never as instructions. Output screening re-examines content that crossed a trust boundary mid-run. This is the same boundary discipline as the rest of the gateway. Data from outside the tenant's control is untrusted until checked.

Human approval gates cover the small set of actions that truly need a person. Each approval shows the exact action in plain terms, not the agent's plan paraphrased. Approval fatigue is real: if everything needs approval, approvers stop reading. We reserve the gate for production writes, money movement, and irreversible actions, and we measure approval latency as a service metric. A gate that takes an hour is a gate teams will route around.

VII. Provisioning and Delivery

The gateway runs on Azure, and everything it stands on is defined as code. Terraform provisions the Azure OpenAI instances, private networking, Key Vault, managed identities, RBAC assignments, Container Apps environments, and monitoring. Environment configuration is separated from code, so the same definitions produce dev, staging, and production with different parameters. Secrets live in Key Vault and are referenced, never embedded.

Delivery runs through GitHub Actions. Every change builds container images, runs security validation, and promotes through environments with approvals. Gateway configuration, routes, policy bundles, budgets, and model routing rules, moves through the same pipeline as code. A policy change is a pull request with review, a canary deployment to a subset of tenants, and a rollback path. This is the Cognitive DevOps discipline applied to the control plane: automation with safety properties intact [6].

Two rules proved their worth. First, version everything the gateway decides by: prompts, tool bindings, allowlists, policy bundles, routing rules. When a model update changes behavior, the fastest fix is often the previous configuration version, and it should be one command away. Second, canary every rollout. New policies go to an internal tenant first, then to a friendly pilot team, then wide. The canary tenants are chosen for their tolerance, not their importance. Our book on enterprise AI deployment covers why this boring discipline matters more than model choice [9].

VIII. Operating Experience

Running the gateway taught us things the design did not. Each lesson below cost real time. We record them so other teams can pay less.

The shared cache near-miss. Early in the design, a shared cache across tenants looked like an easy efficiency win. Identical prompts from different tenants would hit the same entry. We caught it in review, before it shipped, and the scare shaped the tenancy rules in Section IV-A. The lesson generalizes. Every shared component in a multi-tenant system is a place where tenancy can leak. Audit the seams on a schedule: caches, batch jobs, logs, exports. The efficient design and the safe design are different designs, and the safe one wins.

Rate limiting is a product feature. Tenants do not experience rate limiting as infrastructure. They experience it as the gateway saying no. Clear error messages, per-tenant dashboards showing limit consumption, and a documented path to higher limits turned throttling from a complaint into a planning input. The noisy neighbor problem is real: one tenant's bulk job can starve interactive traffic without per-tenant buckets. Our load balancing background made this familiar territory [8]. Isolate first, optimize second.

Keys leak, so design for it. Long-lived keys end up in code, in chat logs, in screenshots. We moved every caller we could to managed identities and short-lived tokens, kept team keys only where nothing else worked, and built rotation into the admin API instead of treating it as an incident response. The 2018 work on secure transmission at the device layer taught the same lesson at a different scale: credential lifetime should match the trust you actually have, not the trust you wish you had [13].

Budgets need teeth. Warning emails about budget consumption get filtered. Hard stops get noticed. We set budgets with a warning threshold and a stop threshold, and the stop is automatic. When a stop fires on a real workload, the conversation about budgets gets serious in a way no dashboard achieves. Set the stop threshold with the team, not for them, or the gateway becomes the adversary.

The gateway sees everything, so it becomes the debugging tool. When an agent misbehaved, the first question was always what it did and what it cost. The trace answered both. This was the direct payoff of the observability-first approach from our data center work [2]. Totals are for finance. Traces are for engineering. Build the traces first, and the gateway's central position makes them complete by construction.

Prompt changes are deployments. A routing rule change or a screening strictness change can alter behavior as much as a code deploy. We learned to treat them identically: pull request, review, canary, rollback plan. The teams that treated policy as configuration learned this the painful way, debugging behavior changes with no deploy to blame. Version everything the gateway decides by, as Section VII requires.

Guardrail tuning never ends. Strictness that blocks real work gets disabled by frustrated teams, which is worse than no guardrail at all. We review false positives weekly and tune per tenant. The tuning notes are part of the policy bundle, so the reasoning survives turnover. A guardrail is a living configuration, not a one-time setup.

Automate in bounds. Where the gateway automates (throttling abusive callers, rotating exposed credentials, failing over), every action stays bounded and logged, with a human override path that works in seconds. Our work on self-healing AKS clusters with hybrid reinforcement learning followed the same shape: closed loops are fine when every action is bounded and every bound is visible [14]. The day the automation surprises you is the day the bounds were wrong.

IX. Conclusion

The enterprise LLM gateway turns AI sprawl into a managed platform. One front door for model access, with identity, metering, rate limiting, guardrails, and audit built in. The architecture is deliberately unglamorous: stateless services, versioned policy, tenant-scoped everything, and the same deployment discipline as any critical service. The contribution is not novelty. It is a reference design with the sharp edges labeled, drawn from production work.

Two directions look promising from here. First, agent-to-agent protocols such as MCP and A2A will need mediation at the gateway: discovery, identity, and policy for traffic that never touches a human. Second, cost control will get finer. Per-task budgets, semantic caching with correctness guarantees, and routing driven by measured quality per task are all within reach of the metering pipeline described here. The gateway's central position makes it the natural place to build them.

References

[1] M. K. Kagitha, "Scalable onboarding architectures for global API ecosystems in multi-tenant Azure environments," SSRN, 2026. doi: 10.2139/ssrn.6151926.

[2] M. K. Kagitha, "AI-driven observability for hyperscale colocation data centers," J. Computational Analysis and Applications, vol. 27, no. 7, 2019. doi: 10.5281/zenodo.22836377.

[3] M. K. Kagitha, "GreenOps in the cloud: AI-driven telemetry analysis for reducing carbon footprint," J. Computational Analysis and Applications, vol. 28, no. 6, 2020. doi: 10.5281/zenodo.22836379.

[4] M. K. Kagitha, The Telemetry Blueprint: Hyperscale Data Lakes and Machine Learning for Infrastructure Intelligence. Geh Press, 2023. ISBN 978-1-972547-16-8.

[5] M. K. Kagitha, "Automated security compliance in CI/CD: An NLP approach to IaC vulnerabilities," Computer Fraud & Security, vol. 2021, no. 12, 2021. doi: 10.5281/zenodo.22836366.

[6] M. K. Kagitha, Cognitive DevOps: Integrating LLMs and Predictive Analytics into the CI/CD Pipeline. Geh Press, 2021. ISBN 978-1-972547-14-4.

[7] I. Fedorov et al., "Llama Guard 3-1B-INT4: Compact and efficient safeguard for human-AI conversations," Meta technical report, arXiv:2411.17713, 2024.

[8] M. K. Kagitha, "Load balancing techniques in distributed cloud environments," J. Critical Reviews, vol. 6, no. 1, 2019. doi: 10.5281/zenodo.22836373.

[9] M. K. Kagitha, Beyond the Algorithm: The Engineering Realities of Deploying Enterprise AI. Geh Press, 2024. ISBN 978-1-972547-17-5.

[10] M. K. Kagitha, The Sentient Cloud: Architecting Autonomous Data Centers with AIOps and Azure. Geh Press, 2022. ISBN 978-1-972547-15-1.

[11] M. K. Kagitha, The Future-Proof Enterprise: Unifying AIOps, Hyperscale Cloud, and Quantum Readiness. Geh Press, 2026. ISBN 978-1-972547-20-5.

[12] M. K. Kagitha, "Comparative analysis of machine learning algorithms for predictive analytics," Global J. Eng. Sci. Res., vol. 3, no. 12, 2016. doi: 10.5281/zenodo.19041609.

[13] M. K. Kagitha, "Secure data transmission in IoT-based smart home systems," Int. J. Eng. Researches and Management Studies, vol. 5, no. 11, 2018. doi: 10.5281/zenodo.22836375.

[14] M. K. Kagitha, "Self-healing cloud infrastructure: Hybrid RL framework for AKS," Computer Fraud & Security, vol. 2023, no. 12, 2023. doi: 10.5281/zenodo.22836371.

[15] Google Cloud, "Model Armor documentation." [Online]. Available: https://cloud.google.com/model-armor/docs

[16] OpenTelemetry, "Semantic conventions for generative AI systems." [Online]. Available: https://github.com/open-telemetry/semantic-conventions-genai