DEV Community: Oleksandr Kuryzhev

ML Anomaly Detection in Loki Logs: Per-Entity Isolation Forest

Oleksandr Kuryzhev — Tue, 07 Jul 2026 07:03:06 +0000

Liquid syntax error: Variable '{{namespace="{namespace}' was not properly terminated with regexp: /\}\}/

Terraform Drift Detection in CI: Alert-Only vs Auto-Remediate

Oleksandr Kuryzhev — Mon, 06 Jul 2026 07:04:52 +0000

Originally published on kuryzhev.cloud

When You Face This Choice

You'll know the moment terraform drift detection ci becomes a real problem: someone fixes a security group via the AWS console during an incident, or an intern deletes an IAM policy they thought was unused, and three days later your next terraform plan shows a wall of diffs nobody can explain. State says one thing, reality says another, and now you're debugging infrastructure archaeology instead of shipping features.

A few triggers usually force teams to actually pick a strategy instead of winging it: a SOC2 or ISO27001 audit asking "how do you detect unauthorized infrastructure changes," a postmortem where the root cause was "someone changed this manually and nobody knew," a growing pile of workspaces where nobody remembers who owns what, or plain on-call fatigue from Slack pings about diffs that turn out to be nothing.

Once you hit that point, there's a real fork in the road: do you detect drift and let a human decide what to do about it, or do you detect drift and let the pipeline fix it automatically? "Just run terraform plan sometimes" is not a strategy — it's a hope. You need a scheduled, structured process either way. The question is how much autonomy you give the pipeline once it finds something.

Option A — Scheduled Detect-Only Pipeline (Plan + Notify)

This is the boring option, and I mean that as a compliment. A cron-triggered CI job runs terraform plan -detailed-exitcode, checks the exit code, parses the plan into JSON, and posts a summary to Slack. Nobody applies anything automatically. A human reads the diff, decides if it's real drift or expected divergence, and runs apply manually if needed.

Here's a workflow I've used in production across a handful of workspaces:


# .github/workflows/drift-detection.yml
# Scheduled drift-detection job: detects drift, never auto-applies.
name: terraform-drift-detection

on:
  schedule:
    - cron: '0 6 * * *'   # runs daily at 06:00 UTC — stagger per workspace to avoid API throttling
  workflow_dispatch: {}    # allow manual trigger for on-demand checks

jobs:
  drift-check:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        workspace: [network, compute, iam]   # shard workspaces to keep each job under 10 min
      fail-fast: false
    steps:
      - uses: actions/checkout@v4

      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.7.5   # pin exact version, do not use "latest"

      - name: Terraform init
        working-directory: infra/${{ matrix.workspace }}
        run: terraform init -input=false -lock-timeout=60s

      - name: Terraform plan (detect drift)
        id: plan
        working-directory: infra/${{ matrix.workspace }}
        run: |
          set +e
          terraform plan -detailed-exitcode -out=plan.tfplan -lock-timeout=60s
          echo "exitcode=$?" >> "$GITHUB_OUTPUT"
        continue-on-error: true   # exit code 2 must not fail the job, we handle it manually

      - name: Extract structured diff
        if: steps.plan.outputs.exitcode == '2'
        working-directory: infra/${{ matrix.workspace }}
        run: |
          terraform show -json plan.tfplan > plan.json
          jq '[.resource_changes[] | select(.change.actions[0] != "no-op") |
              {address: .address, action: .change.actions[0]}]' plan.json > drift-summary.json

      - name: Notify Slack on drift
        if: steps.plan.outputs.exitcode == '2'
        run: |
          curl -sf -X POST -H 'Content-type: application/json' \
            --data "{\"text\": \"⚠️ Drift detected in *${{ matrix.workspace }}*: $(cat drift-summary.json)\"}" \
            "${{ secrets.SLACK_WEBHOOK_URL }}"

      - name: Fail job to surface drift in CI status
        if: steps.plan.outputs.exitcode == '2'
        run: exit 1   # human still has to review and apply manually — that's the point of Option A

Note the exit code handling — Terraform 1.6+ made -detailed-exitcode reliable enough to branch on directly: 0 means no changes, 1 means an error, 2 means drift or pending changes. Don't grep plain-text plan output for this; that format isn't a stable interface across versions and it will break your parser on some future minor release.

Pros: blast radius is basically zero, it's trivially auditable for compliance, it works with any backend including plain S3+DynamoDB, and it's the fastest thing to implement — you can have this running by end of day.

Cons: drift sits unresolved for hours or days waiting for a human. If the Slack channel gets noisy, people start ignoring it — alert fatigue is real and it happens fast. Someone still has to context-switch out of whatever they're doing to run apply. And past roughly 20-30 workspaces, you need an actual triage system or this turns into a wall of unread notifications.

Option B — Automated Detect-and-Remediate Pipeline (Plan + Gated Auto-Apply)

This is the self-healing version. Same detection step, but when drift is found, the pipeline runs terraform apply automatically — or -refresh-only if the intent is to accept the drift into state rather than revert it — gated behind policy checks like OPA/Conftest or Atlantis policy sets, instead of waiting on a person.


# drift-summary.sh — parses plan.json into a compact human-readable digest
# Run after `terraform show -json plan.tfplan > plan.json`

jq -r '
  .resource_changes[]
  | select(.change.actions[0] != "no-op")
  | "\(.change.actions[0] | ascii_upcase)\t\(.address)"
' plan.json | column -t -s $'\t'

# Example output:
# UPDATE   aws_security_group.web_sg
# DELETE   aws_iam_role_policy.legacy_policy
# CREATE   aws_route53_record.orphaned_cname

# Gotcha: "delete" here in a drift context usually means "resource exists in
# state/config but was removed manually in the console" — Terraform will try
# to RECREATE it on apply, which is why auto-apply on IAM/DNS is disallowed
# in this pipeline's OPA policy (see policies/no-auto-apply-sensitive.rego).

Pros: the drift window closes in minutes, not days. It removes a category of toil entirely and reinforces a "config is truth" culture, which is genuinely valuable for stateless or ephemeral infra like autoscaling groups and ECS service definitions where manual tweaks are expected to be transient anyway.

Cons, and this is where I've seen teams get burned: if the plan misclassifies an intentional hotfix as drift, the pipeline will cheerfully revert it. I've watched a scheduled drift job run apply -auto-approve off a plan artifact nobody reviewed, and it recreated a resource a teammate had manually deleted on purpose — destructively, mid-incident. You need mature policy-as-code guardrails before this is safe, and debugging "why did CI undo what I just did" during an active incident is a genuinely bad time. Blast radius also scales with anything shared modules touch.

Decision Matrix

Criteria	Option A (Detect-Only)	Option B (Auto-Remediate)
Team size <10 engineers	Recommended	Overkill
Regulated industry / audit trail needs	Recommended	Risky without strong OPA gates
>50 dynamic workspaces	Doesn't scale alone	Recommended with policy-as-code
Blast radius tolerance: low	Recommended	Avoid
On-call maturity: early	Recommended	Wait until mature
Resource type: IAM, DNS, security groups	Always	Never auto-apply
Resource type: ASG desired count, tags	Fine either way	Good candidate

The nuance that most decision matrices miss: the right answer often isn't per-team, it's per-resource-type. DNS records and IAM policies should basically never auto-remediate — the cost of a wrong revert is too high. Autoscaling group desired capacity or resource tags are low-stakes and make good candidates for automation. If you're on Terraform Cloud/Enterprise, note that Health Assessments (their built-in drift detection) is a Business-tier feature — not available on free or Team plans, so budget accordingly if you were planning to rely on it out of the box.

My Pick

I default to detect-only everywhere, full stop. Then I carve out a narrow allowlist of low-risk resource types for auto-remediation once policy checks are actually in place — not before. Full auto-apply-everything is a mistake for roughly 90% of teams I've seen try it, and the failure mode is ugly: a pipeline reverting an SRE's legitimate incident-response change is the kind of incident that generates its own postmortem.

Detect-only alone isn't a finish line either — drift sitting unresolved for a week because nobody triaged the Slack digest is its own failure mode. So the rollout order matters. Start with scheduled detect-only, wired into GitHub Actions or Atlantis, with a structured Slack digest that includes workspace name, resource address, and action type — flat "drift detected, check logs" messages get ignored within a week, I've seen it happen. Only after that's stable and you have Conftest/OPA policies denying auto-apply on sensitive resource types do you carve out the allowlist for auto-remediation. Skipping the middle step is how teams end up auto-reverting an incident fix at 2am.

Two practical warnings before you build any of this. First, pin your provider versions in .terraform.lock.hcl and commit that file — I've seen a CI runner silently pull a newer AWS provider than what generated the last state, producing false-positive drift on every single run because default tag behavior changed between 5.x minors. Second, scope your CI role to least privilege: s3:GetObject and dynamodb:GetItem on the specific state bucket and lock table only, never account-wide read access — state files routinely contain plaintext secrets in outputs, and a broad IAM role turns your drift job into a juicy target.

driftctl is worth a mention as an alternative scanner, but its last stable release was 0.40.0 before maintenance slowed down in 2023 — native terraform plan is the safer long-term bet. And if you're running this across 200+ workspaces, don't check every 15 minutes; you'll hit AWS API throttling on Describe* calls and burn CI minutes for nothing. Hourly or staggered daily windows are plenty. For more on structuring larger Terraform repos so this actually scales, see our Terraform automation posts and the official Terraform plan documentation for the full behavior of refresh-only mode and exit codes, plus the OPA docs if you're building the policy gate for Option B.

Building an Online Anomaly Scoring Sidecar for Grafana Alerts

Oleksandr Kuryzhev — Sun, 05 Jul 2026 08:44:15 +0000

Originally published on kuryzhev.cloud

Your "AI-powered" anomaly alerts are probably flapping every 60 seconds because nobody set a for duration on the rule. We hit this exact problem three months ago when a client asked us to add "smart alerting" to their checkout service dashboards. What they got, at first, was a Slack channel firing and resolving the same alert every evaluation cycle — Grafana's default is 1 minute. AI anomaly detection Grafana integrations get sold as magic, but the underlying pipeline is closer to a stats class than a neural network, and treating it otherwise is where teams get burned.

What this actually does

Strip away the marketing and the pipeline looks like this: metrics come in from Prometheus, Mimir, or Loki, get aggregated into a feature (rate, count, ratio), get scored by a model, and the score itself becomes a new time series that Grafana alerts on like anything else. That's it. No LLM is involved, and it doesn't need to be — this is time-series forecasting and outlier detection, not natural language generation.

There are three real tiers of implementation, and picking the wrong one for your scale is the first mistake teams make. Grafana's built-in anomaly detection transformation, available since Grafana 10.x, uses Seasonal-ESD under the hood — no external service needed, and it's fine for a handful of dashboards. The middle tier is lightweight ML: Prophet or Kats running as a sidecar, doing seasonal forecasting on a schedule. The heavy tier is a proper serving pipeline — models exported to ONNX, served via ONNX Runtime or TorchServe, queried by a custom datasource or Alertmanager webhook.

Most infra metrics don't need Prophet. It's a heavyweight forecasting library built for business time series with strong seasonality — daily active users, sales curves. Applying it to CPU or request-rate metrics with 15-second scrape intervals is like using a sledgehammer to hang a picture frame. A z-score or exponentially weighted moving average gets you 90% of the value with a fraction of the compute cost. I stopped recommending Prophet for infra alerting after watching a Kats install drag in pandas 1.x and break a container image that already had pandas 2.x pinned for a different service — version conflicts you don't need in an alerting hot path.

How people use it wrong

Mistake one: feeding raw, unaggregated metrics into a global model. Per-pod CPU during a rolling deploy or a horizontal autoscale event looks exactly like an anomaly to any outlier detector — because it is one, just not the kind you care about. Aggregate to service level first. Score the aggregate, not the noise underneath it.

Mistake two: no hysteresis. Grafana Alerting evaluates rules on every interval, default one minute, and if your alert condition is "anomaly score above 0.75, fire immediately," you'll get an alert that fires and resolves within the same minute because the model recomputes the score on every scrape and it dances around the threshold. This is the single most common complaint I've seen in postmortems for these setups — an on-call engineer gets paged, checks the dashboard, sees green, and the alert already resolved itself. That erodes trust in the whole system faster than a false negative would.

Mistake three: naive scheduled retraining. If you retrain your model every night at 2am without checking for seasonality, you'll get weeks of false positives around month-end batch jobs, Black Friday, or payroll runs — anything that's a recurring but infrequent pattern gets treated as noise the model hasn't seen yet, then flagged as anomalous every single time it recurs. We watched this happen at a fintech client where the batch settlement job triggered "critical" anomaly alerts for three consecutive month-ends before anyone connected the dots.

The correct approach

Separate concerns. Don't try to do inference inside Grafana — it's a visualization and alerting layer, not a model server. Run a dedicated scoring service instead. Here's a sidecar using River, an online machine learning library that updates incrementally without full retrains, which matters a lot for infra metrics that drift slowly over time:


# anomaly_scorer.py — lightweight sidecar that reads Prometheus,
# computes an online anomaly score, and pushes it back via remote_write
import time
import requests
from river import anomaly, preprocessing
from prometheus_client import CollectorRegistry, Gauge, push_to_gateway

PROM_QUERY_URL = "http://prometheus:9090/api/v1/query"
PUSHGATEWAY_URL = "http://pushgateway:9091"
SERVICE_LABEL = "checkout"
QUERY = 'rate(http_requests_total{service="checkout"}[1m])'

# online model: scales input then flags outliers incrementally (no batch retrain needed)
model = preprocessing.StandardScaler() | anomaly.HalfSpaceTrees(
    n_trees=25, height=8, window_size=250, seed=42
)

registry = CollectorRegistry()
score_gauge = Gauge(
    "anomaly_score", "Online anomaly score", ["service"], registry=registry
)

def fetch_metric():
    resp = requests.get(PROM_QUERY_URL, params={"query": QUERY}, timeout=5)
    resp.raise_for_status()
    result = resp.json()["data"]["result"]
    if not result:
        return None
    return float(result[0]["value"][1])

def run():
    while True:
        value = fetch_metric()
        if value is not None:
            x = {"value": value}
            score = model.score_one(x)   # anomaly score, higher = more anomalous
            model.learn_one(x)           # incremental update, no full retrain

            # normalize score to 0-1 range for consistent alert thresholds
            normalized = min(max(score, 0.0), 1.0)
            score_gauge.labels(service=SERVICE_LABEL).set(normalized)

            # push to gateway so Prometheus scrapes it like any other metric
            push_to_gateway(PUSHGATEWAY_URL, job="anomaly_scorer", registry=registry)

        time.sleep(15)  # match scrape interval to avoid stale scores

if __name__ == "__main__":
    run()

The anomaly score becomes a plain Prometheus metric — anomaly_score{service="checkout"} 0.87 — so Grafana alerts on it exactly like it would alert on request latency. Then the alert rule uses a rolling average, not the instantaneous value, and a for duration to require a sustained breach:


# provisioning/alerting/rules.yaml — Grafana-as-code alert rule
# reads the anomaly_score metric, requires sustained breach (no flapping),
# and combines with an SLO burn-rate check via Alertmanager routing
apiVersion: 1
groups:
  - orgId: 1
    name: infra-anomaly-alerts
    folder: AI-Anomaly-Detection
    interval: 1m
    rules:
      - uid: anomaly-checkout-score
        title: "Checkout service anomaly score elevated"
        condition: C
        data:
          - refId: A
            datasourceUid: prometheus
            model:
              expr: avg_over_time(anomaly_score{service="checkout"}[5m])
          - refId: C
            datasourceUid: __expr__
            model:
              type: threshold
              expression: A
              conditions:
                - evaluator:
                    type: gt
                    params: [0.75]
        for: 5m   # sustained breach required, prevents flapping on single spikes
        labels:
          severity: warning
          model_version: v3
        annotations:
          summary: "Anomaly score {{ $values.A }} exceeded 0.75 for 5m on checkout"
          runbook_url: "https://kuryzhev.cloud/runbooks/anomaly-checkout"

Watch out: version-pin your model artifacts. A model_v3.onnx file with a SHA256 checksum committed alongside the deployment manifest turns your alert behavior into something reproducible and auditable. Treat it like infrastructure code, not a throwaway notebook experiment — because when someone asks "why did this fire at 3am," you need an answer better than "the model changed."

Advanced patterns

Once you're past a single service, run per-service models with labels — {service="checkout", model_version="v3"} — so dashboards and alert rules template cleanly across teams without duplicating pipelines. Each team owns their model version, and rollbacks are just a label change.

Ensemble alerting cuts false positives dramatically. Combine the anomaly score with a static SLO burn-rate alert using Alertmanager's nested routing tree — require both to fire before paging. A rough sketch: a parent route matches severity=warning and routes to two child branches, one matching alertname=AnomalyScore and one matching alertname=SLOBurnRate, with a receiver that only triggers when both are active in the same group. This is the pattern that finally got our fintech client's false-positive rate down from daily noise to a handful of real pages a month.

Close the loop with feedback. Grafana OnCall logs acknowledge/dismiss actions on incidents — capture those as labeled ground truth and feed them into periodic supervised fine-tuning of your model. This is the difference between a static anomaly detector and one that actually learns your service's quirks over time, including the seasonal patterns that caused mistake three above.

Performance notes

Keep inference under 200ms per series. Grafana evaluates alert rules synchronously per interval, and the default evaluation timeout is 30 seconds — if your scoring datasource is slow, you'll see evaluation failed: context deadline exceeded in the Grafana logs, and that's a symptom worth taking seriously before it becomes a missed page. ONNX Runtime 1.16+ handles small forecasting or outlier models comfortably within that budget.

Cost matters more than people expect. Streaming per-service inference at 15-second resolution across 200+ services can multiply active series count 3-5x in Mimir or Cortex, since every score is a new time series with its own cardinality. Batch retraining hourly instead of continuously streaming updates is usually the right tradeoff — you lose a bit of freshness, you keep your storage bill sane.

Scope the scoring service's credentials narrowly. It needs read access to raw metrics and write access to push scores back — that's real blast radius if the token leaks. Use a dedicated remote_write tenant or namespace rather than an admin-level API key. A sidecar with broad Prometheus access and network exposure is exactly the kind of thing that becomes a lateral-movement vector during an incident, and nobody wants to explain that in a postmortem. If you're running Grafana in HA, also double-check that the anomaly_score metric is computed consistently across replicas — inconsistent scoring per instance produces alert state that looks like a bug even when the pipeline is technically working. More on wiring alerting pipelines end to end is covered in our DevOps_DayS observability posts.

Grafana Alerting Checklist: Wiring AI Anomaly Scores Correctly

Oleksandr Kuryzhev — Sat, 04 Jul 2026 07:52:40 +0000

Originally published on kuryzhev.cloud

Why This Checklist

Three days. That's how long a memory leak crept through one of our checkout pods before anyone noticed — because our CPU-at-80% alert never fired. The leak was slow, gradual, and stayed well under every static threshold we had. When we finally caught it, it was a customer complaint that tipped us off, not Prometheus. That incident is the reason we built a Grafana AI anomaly detection pipeline, and it's the reason this checklist exists.

Static thresholds work fine for hard failures — disk full, service down, 500s spiking. They fail at catching slow-burn, multivariate drift: memory creeping up 2% a day, disk I/O latency inching from 4ms to 40ms over a week, a subtle shift in request pattern that only looks wrong when you compare it against the last 30 days of baseline. That's exactly the class of problem anomaly-detection models are good at — they learn "normal" for a given service and flag statistical deviation, not just a fixed number.

But here's the catch nobody tells you upfront: wiring an ML model's output into an alerting system is its own project, separate from building the model. You're not just training Prophet or PyOD on a metric — you're turning a float between 0 and 1 into a reliable page that doesn't wake someone up for nothing. Alert fatigue is real, and a poorly-tuned anomaly pipeline generates more noise than the static rules it replaced. The setup cost is only worth it if you're running a large fleet, dealing with seasonal traffic, or managing multi-tenant systems where "normal" varies by customer.

This checklist assumes you already have a trained model (Prophet, PyOD, or Grafana's own Machine Learning plugin) producing a score. What it covers is everything between "the model outputs a number" and "the right person gets paged with useful context."

The Checklist

We run through this list every time we onboard a new service into the anomaly pipeline. Skipping any of these steps is how you end up with silent gaps or 3am pages for nothing.

Confirm scrape interval parity. The anomaly exporter's scoring cadence must match or be a clean multiple of the raw metric's scrape_interval. Mismatched intervals create phantom gaps that Grafana reads as "no data."
Check Prometheus retention. Your model needs enough history to establish a baseline — usually 30+ days for weekly seasonality. Verify --storage.tsdb.retention.time covers it before you even start training.
Decide on the output channel. Push scores via remote_write or expose them on a scrape endpoint. We use the latter — a small exporter that Prometheus scrapes like any other target.
Normalize the score. Never alert on raw model output directly — z-score or min-max normalize it first, or your thresholds drift with seasonal traffic changes.
Provision the Grafana data source for the metric that carries your anomaly score (usually the same Prometheus instance, different metric name).
Match labels exactly. The anomaly score series needs the same service, instance, and job labels as the underlying metric, or Grafana can't correlate them on one panel.
Write the alert rule as code under /etc/grafana/provisioning/alerting/ — never click-ops this in production. GitOps or nothing.
Set an appropriate evaluate every. It should match your model's scoring cadence (e.g., 5m for batch scoring), not the raw scrape interval.
Add a for: pending duration. 5 minutes is our default — enough to filter single-point noise without delaying real incidents too much.
Route through a notification policy with label matchers so anomaly alerts land in the right Slack channel or PagerDuty service, not a generic firehose.
Test the contact point manually after every rule change, using the "Test" button or grafana-cli. Silent misconfiguration is the most common failure mode we've seen.
Build a dashboard panel showing raw metric and anomaly score side by side — you need this for triage and for validating the model later.
Inject a synthetic anomaly. Push a fake spike into the metric and confirm the alert fires within your expected latency window before calling the setup done.

Here's the actual stack we run for this, minus the model training pipeline. The exporter runs the trained model against Prometheus data and exposes a score:


# docker-compose.yml — anomaly exporter + Prometheus + Grafana stack
version: "3.8"
services:
  anomaly-exporter:
    image: myorg/anomaly-exporter:1.4.2   # pinned version, avoid :latest
    ports:
      - "127.0.0.1:9187:9187"             # bind to localhost only, internal scrape
    environment:
      - MODEL_PATH=/models/prophet_v3.pkl
      - SCORE_INTERVAL_SECONDS=300        # batch scoring every 5m, cost control
      - PROMETHEUS_URL=http://prometheus:9090
    volumes:
      - ./models:/models:ro
    restart: unless-stopped

  prometheus:
    image: prom/prometheus:v2.53.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prom_data:/prometheus
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:10.4.2
    ports:
      - "3000:3000"
    volumes:
      - ./provisioning:/etc/grafana/provisioning   # alert rules + datasources as code
      - grafana_data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=changeme
      - GF_UNIFIED_ALERTING_ENABLED=true

volumes:
  prom_data:
  grafana_data:

And the alert rule that consumes the score, provisioned as code rather than clicked into existence in the UI:


# provisioning/alerting/rules.yaml — Grafana alert rule for anomaly score
apiVersion: 1
groups:
  - orgId: 1
    name: infra-anomaly-group
    folder: AI-Anomaly-Detection
    interval: 5m                     # must match SCORE_INTERVAL_SECONDS above
    rules:
      - uid: anomaly-checkout-cpu
        title: "Anomaly Score - Checkout Service CPU"
        condition: C
        data:
          - refId: A
            queryType: ""
            datasourceUid: prometheus-uid
            model:
              expr: anomaly_score{service="checkout", metric="cpu"}
          - refId: C
            queryType: ""
            datasourceUid: __expr__
            model:
              type: threshold
              expression: A
              conditions:
                - evaluator:
                    type: gt
                    params: [0.8]      # normalized z-score threshold, not raw model output
        for: 5m                       # pending period to avoid single-point noise
        labels:
          severity: warning
        annotations:
          summary: "Anomaly detected in checkout service CPU pattern"
          runbook_url: "https://kuryzhev.cloud/runbooks/anomaly-checkout"

Full syntax for unified alerting provisioning is documented in the Grafana alerting provisioning docs — worth bookmarking, since the schema shifts slightly between minor versions.

Commonly Missed Items

Even teams that follow the checklist above hit the same handful of gotchas. Here are the ones that bit us hardest.

Watch out for timestamp mismatches. If your anomaly exporter timestamps its output differently than Prometheus scrapes the raw metric — even by a few seconds consistently — Grafana will show gaps that look like "no data" rather than a healthy series. We lost half a day debugging what looked like a broken pipeline; it was just clock drift between the exporter container and the Prometheus host.

Version-pin everything, always. We once let the anomaly exporter float on :latest, and a routine image rebuild silently changed the model's normalization window. Alert behavior shifted overnight with zero deploy log entry to explain it. Pin the image tag, pin the model file name, and log the model version in the alert annotation so on-call knows exactly what's scoring.

Give new hosts a cold-start grace period. A pod that just spun up has no baseline history — every metric looks "anomalous" simply because there's nothing to compare against. We now suppress anomaly alerts for the first two hours after a pod's first-seen timestamp. Skip this and you'll get paged every time autoscaling adds a node.

Set up dedup and grouping in your notification policy. One root-cause anomaly — say, a bad deploy — often triggers correlated score spikes across CPU, memory, and latency simultaneously. Without grouping by service in your notification policy, that's four pages instead of one. I stopped trusting default notification policies after our on-call got seven pings in ninety seconds for a single deploy rollback.

One more that's easy to overlook: securing the exporter endpoint. Anomaly score exporters running on ports like :9187 should be internal-only and scraped with a token or mTLS. A raw confidence score leaking to the outside world tells an attacker more about your infra's normal behavior than you'd want them to have.

Automation Ideas

Manually maintaining an anomaly pipeline doesn't scale past a handful of services. Here's what we automated once the pattern proved itself.

We run a nightly CI job (GitHub Actions) that retrains the model against a rolling 90-day window and validates it against a holdout dataset before promoting new normalization thresholds. If the false-positive rate on the holdout exceeds roughly 5%, the pipeline fails the build instead of shipping a worse model — this single gate saved us from at least two bad retrains that would have flooded on-call.

Alert rules and contact points live in Terraform, synced to the anomaly exporter's service config in the same repo. That way a new microservice onboarding into the anomaly pipeline is a single PR: add the exporter config, add the Grafana rule block, merge, done. No manual clicking through the Grafana UI at 2am to add a rule for a service that just went live.

We also template runbook links directly into alert annotations — the PagerDuty notification includes which model version fired, when it was last retrained, and which features drove the score. It's a small thing, but it cuts investigation time significantly when someone unfamiliar with the model is the one paged.

Last piece: a feedback loop. When someone marks an anomaly alert as a false positive in Grafana, a small script logs that window into a training-exclusion dataset consumed by the next retrain cycle. Without this, the same seasonal blip (Black Friday traffic, month-end batch jobs) keeps triggering the same false alarm every cycle. We wrote more about building these kinds of self-correcting alerting loops in our DevOps_DayS notes, if you want the broader pattern applied outside of anomaly detection specifically.

None of this replaces good judgment — a Grafana AI anomaly detection setup is still a tool, not an oracle. But wired correctly, with the checklist above, it catches the slow-burn failures that static thresholds are structurally blind to, and it does so without turning your on-call rotation into a noise-cancellation exercise.

Ansible Dynamic Inventory for EC2 Fleets: 7 Tuning Tips

Oleksandr Kuryzhev — Thu, 02 Jul 2026 07:01:42 +0000

Originally published on kuryzhev.cloud

We were three minutes into a patch run against 600 EC2 instances when the play just... stopped. The error was botocore.exceptions.ClientError: RequestLimitExceeded, and it took us a while to realize the problem wasn't the patch playbook — it was Ansible dynamic inventory for EC2 re-querying the API on every host loop iteration. If you're running amazon.aws.aws_ec2 against anything larger than a handful of instances, these are the fixes that actually matter.

Cache Your Dynamic Inventory or Pay for It in API Throttling

Uncached inventory calls hammer the EC2 API on every single playbook run, and large fleets pay for it in throttling. The aws_ec2 plugin makes a fresh DescribeInstances call every time it builds inventory, and on fleets of 500+ instances that's slow and, eventually, rate-limited. Enable the jsonfile cache plugin with a sane cache_timeout so you're not re-fetching the same instance list for every CI job or ad-hoc command in a five-minute window.

The tradeoff is real: a longer cache timeout means you might target stale hosts during an active autoscaling event, but a too-short timeout brings back the throttling. We settled on 300 seconds for steady-state patching and drop it to 30-60 seconds during active scale-out windows.

Use keyed_groups Instead of Manual Group Definitions

Stop maintaining static host groups by hand — let tags build them for you. The keyed_groups option auto-creates groups from instance tags and state, so a new instance tagged Role: web shows up in role_web without anyone touching an inventory file. This is the single biggest maintenance win once you're managing infrastructure across multiple AWS accounts or regions.

Watch out: tag keys are case-sensitive. We had instances tagged Environment: Prod next to others tagged environment: prod from an older Terraform module, and half the fleet silently vanished from the env_prod group — no error, no warning, just missing hosts. Standardize your tagging convention before you lean on keyed_groups.

Filter at the API Level, Not in Playbook Logic

Filtering with when: conditions after inventory is already built wastes API calls and slows every run down. If you fetch 3,000 instances across all regions and then filter down to 40 with a task-level condition, you've already paid the cost of describing every terminated, stopped, and unmanaged instance in the account. Push the filter into the inventory config itself with filters.

This is the mistake we see most often on teams migrating from static inventory files: they keep the old "loop and skip" logic instead of trusting the plugin's native filtering. Filtering by instance-state-name: running and a ManagedBy tag at the API level cuts inventory build time dramatically and keeps you from accidentally targeting a stopped or terminated instance.

Patch in Batches with serial and max_fail_percentage

Without serial, Ansible runs against 100% of your fleet in parallel — one bad patch or AMI takes down everything at once. Set serial: "20%" combined with max_fail_percentage: 10 so a failing batch halts the rollout before it reaches the whole fleet. This is the difference between "20% of web servers rebooted into a broken kernel" and "the entire tier is down."

Pair this with ASG lifecycle hooks (autoscaling:EC2_INSTANCE_TERMINATING) so instances aren't yanked out from under a play mid-patch during a scale-in event. I stopped running unbatched patch playbooks after a security update rebooted an entire ASG simultaneously and took our checkout flow down for four minutes — it's not a risk worth taking twice.

Don't Hardcode Credentials — Use Instance Roles for Inventory Lookups

An IAM instance role beats a static access key every time, especially for something as routine as inventory lookups. Attach a least-privilege role to your bastion or control node scoped to ec2:DescribeInstances, ec2:DescribeTags, and ec2:DescribeInstanceStatus — nothing more. There's no legitimate reason for the inventory plugin to have write access to anything.

Never commit aws_access_key or aws_secret_key into aws_ec2.yml, even in a "private" repo. If you need multiple profiles, use the AWS_PROFILE environment variable or role assumption via AWS_ROLE_ARN, not plaintext credentials sitting next to your playbooks. Check the AWS IAM roles documentation if you're still wiring up instance profiles for this.

Verify Inventory Before You Patch — ansible-inventory --graph Is Your Sanity Check

Run ansible-inventory --graph before every patch job, not after something goes wrong. It's a thirty-second command that catches empty groups from a bad region filter or a typo'd tag key before you waste a play against zero hosts — or worse, against the wrong hosts because your filter was too loose.

A common assumption that trips people up: dynamic inventory does not refresh automatically when an ASG scales out. It's a snapshot taken at the start of the playbook run, not a live feed. If you scale from 10 to 40 instances mid-deploy, those new 30 won't appear until the next run pulls a fresh snapshot — plan your patch windows accordingly, and re-run --graph if you suspect the fleet changed underneath you. See the amazon.aws aws_ec2 inventory plugin docs for the full option list.

Handle Fleet Churn: Compose ansible_host and Skip Terminating Instances

Public DNS names lag behind instance launch, so use compose to force ansible_host to the private IP instead. A freshly-launched instance from an ASG scale-out event often doesn't have a resolvable public DNS record yet, and Ansible will hang or fail trying to connect. Setting ansible_host: private_ip_address in compose sidesteps that entirely, assuming your control node has network access via VPC peering or a VPN.

During scale-in events, exclude instances that are mid-termination using a lifecycle tag set by your ASG termination hook, combined with the instance-state-name: running filter. Otherwise you'll get intermittent UNREACHABLE failures in your patch run that have nothing to do with the patch itself — they're just instances that disappeared between inventory build and task execution.

Here's the full inventory config we run in production, tying all of the above together:


# inventories/aws_ec2.yml
# Dynamic inventory config for aws_ec2 plugin (amazon.aws >= 7.6.0)
plugin: amazon.aws.aws_ec2

# Restrict regions explicitly — omitting this queries ALL AWS regions
regions:
  - us-east-1
  - eu-west-1

# Only include running instances tagged as ansible-managed
filters:
  instance-state-name: running
  "tag:ManagedBy": ansible

# Cache results to avoid RequestLimitExceeded on large fleets
cache: true
cache_plugin: jsonfile
cache_connection: /tmp/ansible_inventory_cache
cache_timeout: 300           # seconds; tune lower during active scaling events
cache_prefix: aws_ec2_prod

# Don't fail hard if some instances lack expected tags (mixed fleets)
strict: false

# Build groups automatically from tags instead of static host files
keyed_groups:
  - key: tags.Environment
    prefix: env
  - key: tags.Role
    prefix: role
  - key: "instance_state.name"
    prefix: state

# Use private IP instead of public DNS to avoid resolution lag
# on freshly-launched instances during ASG scale-out
compose:
  ansible_host: private_ip_address
  ansible_user: "'ec2-user'"

# Skip instances that are mid-termination (lifecycle tag set by ASG hook)
exclude_hosts_pattern: "tag_lifecycle_terminating"

Install the required collection version first — the aws_ec2 plugin behavior described here needs amazon.aws 7.x or later:


ansible-galaxy collection install amazon.aws:7.6.0

And here's the pre-flight check plus the actual batched patch run, output included so you know what a healthy run looks like:


# Verify inventory before running a patch playbook — catch empty groups early
$ ansible-inventory -i inventories/aws_ec2.yml --graph

@all:
  |--@env_prod:
  |  |--@role_web:
  |  |  |--ip-10-0-1-23
  |  |  |--ip-10-0-1-45
  |--@state_running:
  |  |--ip-10-0-1-23
  |  |--ip-10-0-1-45

# Patch playbook run with batched rollout to limit blast radius
$ ansible-playbook -i inventories/aws_ec2.yml patch_fleet.yml \
    --limit "role_web:&env_prod" \
    -e "serial='20%'" \
    -e "max_fail_percentage=10"

PLAY [Patch web fleet] ****************************************
TASK [Apply security updates] *********************************
changed: [ip-10-0-1-23]
changed: [ip-10-0-1-45]

PLAY RECAP *******************************************************
ip-10-0-1-23  : ok=3  changed=1  unreachable=0  failed=0
ip-10-0-1-45  : ok=3  changed=1  unreachable=0  failed=0

None of these fixes are exotic — cache your inventory, filter at the API level, batch your rollouts, and verify before you run. But together they're the difference between a dynamic inventory setup that quietly scales with your fleet and one that throttles, misses hosts, or takes down a whole tier during a routine patch cycle. If you're building out your broader AWS automation stack around this, our DevOps_DayS archive has more patterns for treating infrastructure as continuously reconciled state rather than a one-off script.

Cloudflare Workers Rate Limiting: 3 KV Mistakes We Made

Oleksandr Kuryzhev — Wed, 01 Jul 2026 07:55:57 +0000

Originally published on kuryzhev.cloud

Context: Why We Reached for Workers + KV in the First Place

We run a public API with two tiers — free and pro — and needed per-API-key limits enforced at the edge, before requests ever touched origin. Cloudflare Workers seemed like the obvious place to do it: cheap, globally distributed, sub-millisecond cold starts. For state, Workers KV was right there, already wired into our account, and the docs made it sound like exactly what we needed.

The original design was simple, maybe too simple: a Worker reads a counter from KV keyed by API key, increments it on each request, and rejects with a 429 once the count crosses a threshold. It passed every test we threw at it locally. Requests came in, counters went up, limits kicked in at the right number. We shipped it and moved on.

The assumption we never questioned — the one that eventually cost us a very awkward Monday morning — was that "KV is a key-value store, it'll behave like Redis." It doesn't. Cloudflare Workers rate limiting built directly on KV has a completely different consistency and cost model than an in-memory store, and we found out the hard way, in production, on paying customers' accounts. This post is the retrospective on what we got wrong and what we run now instead.

Mistake 1: We Treated KV as Strongly Consistent

Cloudflare is upfront about this in their KV consistency documentation, but it's easy to skim past when you're focused on the happy path: KV writes can take up to 60 seconds to propagate globally across Cloudflare's edge network. A write in one colo isn't instantly visible in another. If a user's requests get routed through two or three different PoPs during a traffic spike — which happens constantly with mobile clients and CDN-fronted apps — each PoP sees its own stale view of the counter.

In practice, this meant pro users capped at 100 requests per minute were bursting to 300-400 req/min whenever traffic happened to spread across multiple edge locations. Nothing errored. No exception, no failed request, no alert fired. The limiter was doing exactly what we told it to do — it just had a partial, delayed view of reality that let more traffic through than intended.

We didn't catch this through monitoring. We caught it because our origin infrastructure bill spiked and someone went digging into request volume per customer. That's the part that still bothers me: a silent under-enforcement bug that looks like a billing anomaly is a lot harder to trace back to "the rate limiter is eventually consistent" than an outright failure would have been. If your rate limiter can fail in a way that produces zero errors, you need a metric specifically watching for that — not just error rate.

Mistake 2: We Ignored KV's Write Limits and Cost Model

The second problem was baked into the same code. KV enforces a hard limit of one write per key per second. We assumed — again, Redis brain — that extra writes would queue or throttle gracefully. They don't. Under sub-second bursts from the same user, some writes simply didn't take effect as expected, and the API call itself still returned success. There's no thrown exception to catch. You have to know to expect this and design around it.

At around 50,000 requests per day from a single top-tier customer, we were generating roughly 50,000 KV writes per day just for that one key. Multiply that across a few thousand active API keys and the write volume adds up fast — Cloudflare's KV pricing is per-operation, and at scale that's real money, not a rounding error. Worse, our free-tier testing environment has a 1,000 writes/day cap, and one customer's load test blew through that limit in under an hour, which is how we first noticed something was structurally wrong before it hit prod at full scale.

There was a second, subtler bug hiding in the same write path. KV.get() returns null for a key that's never been written. Our code did parseInt(raw) without checking for null first, so on a brand-new API key the comparison against the limit was comparing against NaN. Every comparison involving NaN evaluates to false — including NaN >= limit. So new users had effectively no rate limit at all until their first successful write landed, sometimes for hours if propagation was slow. This is the naive version we shipped, showing both mistakes in one place:

// worker.js — the "Mistake 1 & 2" version: naive KV-based per-user rate limiter
// wrangler.toml requires: kv_namespaces = [{ binding = "RATELIMIT_KV", id = "..." }]

export default {
  async fetch(request, env) {
    const apiKey = request.headers.get("X-API-Key");
    if (!apiKey) {
      return new Response("Missing API key", { status: 401 });
    }

    // MISTAKE: raw API key used directly as KV key name (enumeration risk)
    const kvKey = `ratelimit:${apiKey}`;
    const windowSeconds = 60;
    const limit = 100;

    // MISTAKE: no handling for null on first-ever request
    const raw = await env.RATELIMIT_KV.get(kvKey);
    const count = parseInt(raw); // NaN if raw is null — comparison below silently passes

    if (count >= limit) {
      return new Response("Rate limit exceeded", {
        status: 429,
        // MISTAKE: no Retry-After header, clients retry immediately
      });
    }

    // MISTAKE: read-then-write is not atomic — concurrent requests race here
    const newCount = (isNaN(count) ? 0 : count) + 1;

    // MISTAKE: assumes this write is instantly visible everywhere (it isn't)
    await env.RATELIMIT_KV.put(kvKey, String(newCount), {
      expirationTtl: windowSeconds,
    });

    // Forward to origin
    const response = await fetch(request);
    const newHeaders = new Headers(response.headers);
    newHeaders.set("X-RateLimit-Remaining", String(limit - newCount));
    return new Response(response.body, { status: response.status, headers: newHeaders });
  },
};

Mistake 3: We Used Raw API Keys as KV Keys

The third mistake was a design choice that made everything above worse: we stored counters under keys like ratelimit:sk_live_abc123. That's a naive move for two reasons. First, it's a security smell — if KV namespace metadata or Worker logs ever leaked (via a misconfigured route or an overly verbose debug build), you'd be handing over a map of valid API key prefixes. We actually did briefly ship a debug build to prod that ran console.log(apiKey) for troubleshooting. It got caught in review within a day, but wrangler tail logs are visible to anyone with Worker log access on the account, so that's a longer exposure window than we're comfortable admitting.

Second, using the raw key as the KV key name made popular customers into "hot keys." A single busy API key gets hit from many edge locations near-simultaneously, which is exactly the scenario that makes the propagation delay from Mistake 1 worse — more concurrent writers racing against a 60-second global sync window, on the same key, at the same time.

The fix in hindsight is straightforward: hash the API key with SHA-256 before using it as a KV or Durable Object key name, and never log the raw key anywhere, debug build or not. Hashing doesn't fix the consistency problem, but it removes the enumeration risk entirely and it's a five-minute change. We now treat "never use a raw customer identifier as a storage key" as a hard rule in code review, not a suggestion.

What We Do Differently Now

The real fix wasn't a KV tweak — it was moving atomic counting off KV entirely. We now run one Durable Object per user (sharded by hashed key for very high-volume customers), which gives us single-threaded execution and true atomic increments. No races, no read-then-write gap, no cross-colo propagation lag, because a Durable Object instance lives in one place and serializes its own requests.

KV still has a job — it's just the right job now. We use it for slow-changing config like plan tiers and allowlists, where eventual consistency within 60 seconds is genuinely fine. That's the lesson underneath all three mistakes: match the consistency guarantee to the data, don't assume one storage primitive fits every use case.

We also rebuilt the failure mode. Our first version failed closed on any KV or DO error — during an unrelated Cloudflare KV incident, that took our entire API down for twenty minutes. Now we fail open with structured logging: if the counter call times out, we let the request through and tag it for later review, rather than punishing customers for an infrastructure hiccup on our side. We switched from abrupt fixed-window counters to a rough sliding window (two adjacent fixed windows weighted by elapsed time) so we're not allowing a 2x burst right at the window boundary, and every 429 now returns a proper Retry-After header so client SDKs back off instead of hammering us immediately.

Here's the corrected limiter, using a Durable Object with an alarm to proactively reset the counter instead of relying on lazy KV TTL expiry:

// durable-object-limiter.js — corrected approach: atomic per-user counting
// wrangler.toml: [[durable_objects.bindings]]
//   name = "RATE_LIMITER"
//   class_name = "RateLimiter"
// [[migrations]]
//   tag = "v1"
//   new_classes = ["RateLimiter"]

export class RateLimiter {
  constructor(state, env) {
    this.state = state;
  }

  async fetch(request) {
    const limit = 100;
    const windowMs = 60_000;
    const now = Date.now();

    // Single-threaded DO storage — no race condition, true atomicity
    let data = (await this.state.storage.get("counter")) || { count: 0, resetAt: now + windowMs };

    if (now > data.resetAt) {
      data = { count: 0, resetAt: now + windowMs };
    }

    data.count += 1;
    await this.state.storage.put("counter", data);

    // Proactive reset via alarm instead of relying on lazy TTL expiry
    await this.state.storage.setAlarm(data.resetAt);

    const remaining = Math.max(0, limit - data.count);
    const allowed = data.count <= limit;

    return new Response(JSON.stringify({ allowed, remaining, resetAt: data.resetAt }), {
      status: allowed ? 200 : 429,
      headers: allowed ? {} : { "Retry-After": String(Math.ceil((data.resetAt - now) / 1000)) },
    });
  }

  async alarm() {
    // Proactively clear counter when window ends, no stale reads on next request
    await this.state.storage.delete("counter");
  }
}

Watch out for one more thing if you're moving to Durable Objects: the migrations syntax in wrangler.toml changed between Wrangler 2.x and 3.x. We're on wrangler --version 3.28.1, and the [[migrations]] block with new_classes is what you want on that version — copying an old 2.x example straight into a 3.x project will fail silently on deploy in a way that's annoying to debug. Also budget real time for load testing: wrangler dev --local --persist does not emulate cross-colo KV propagation delay, so your local tests will look perfect while production behaves completely differently. We now do a dedicated staging pass across multiple real edge locations before trusting any limiter change, and we cover more of that testing workflow in our DevOps_DayS posts on CI validation for edge deployments.

The upside, once we sorted this out: switching from per-request KV writes to Durable Objects cut our rate-limiting cost by roughly 40% at 2 million requests/day, because DO billing amortizes across duration rather than charging per operation. Cloudflare Workers rate limiting done right ended up cheaper and more correct — we just had to stop assuming KV was something it never claimed to be. For the full picture on the consistency tradeoffs, the Durable Objects documentation is worth reading end to end before you build on top of it, not after.

Automate AWS Cost Reports with Python, boto3, and SES

Oleksandr Kuryzhev — Tue, 30 Jun 2026 07:02:43 +0000

Originally published on kuryzhev.cloud

Your AWS bill already increased by the time you open Cost Explorer — here's how to make Python tell you before finance does. Last month a team I was working with got a Slack message from their finance lead: "Why did AWS spend jump 40% this month?" Nobody in engineering knew. The charges had been accumulating for three weeks. That's the exact problem AWS cost report Python automation solves: shift visibility left, from invoice to inbox.

Symptoms: Your AWS Bill Surprises You Every Month

The pattern is always the same. Cost Allocation Tags are enabled. Someone set up Cost Explorer once during an audit. There's a vague plan to "check it regularly." And then the PDF invoice arrives, after the charges are already incurred, and the post-mortem begins.

Here are the specific symptoms that tell me a team has no real cost visibility:

Finance sends a Slack message asking about a bill spike — and no engineer can answer immediately
Cost Explorer is enabled but nobody opens it between billing cycles
The only "alerting" is a Budget alarm set to 100% of last month's spend — which fires too late to act
When someone does investigate, they spend 20 minutes clicking through the Cost Explorer UI to find the offending service
There is no audit trail of what the cost breakdown looked like on any given day

None of this is unique to small teams. I've seen the same pattern at organizations running six-figure monthly AWS spend. The tooling exists. The API is there. The problem is that nobody wired it up with a scheduled, automated delivery mechanism. A report that lives in a console is not a report — it's a manual task waiting to be skipped.

The second symptom worth calling out: teams that do attempt automation often get halfway through parsing the boto3 response, hit the nested ResultsByTime structure, and abandon the effort. The script sits in a local repo, uncommitted, never scheduled.

Root Cause: Manual Cost Visibility Doesn't Scale

The root cause isn't laziness. It's that the AWS Cost Explorer API has several non-obvious activation and usage requirements that create friction at every step.

Activation delay. Cost Explorer must be manually enabled in the AWS Console under Billing → Cost Explorer → Enable. First data appears 24–48 hours after activation. If you call get_cost_and_usage() before that window, you get a misleading error: DataUnavailableException: Data is not available. Please try to adjust the time period. Most engineers interpret this as a permissions issue and spend an hour debugging IAM before realizing the API just isn't active yet.

Response structure complexity. The get_cost_and_usage() response wraps data inside ResultsByTime[*].Groups[*].Keys. If a service had zero spend in the queried period, Groups is an empty list — not absent, not null, just empty. Code that assumes at least one group will cause index errors in production on quiet billing periods like the first of the month.

No execution layer. Even engineers who write a working script rarely schedule it. EventBridge rules and Lambda functions require a small amount of Terraform or CLI wiring that nobody gets around to. The script runs once manually, produces correct output, and then gets forgotten. Two months later the team is back to checking Cost Explorer by hand.

API cost itself. Each get_cost_and_usage() call costs $0.01. Running it hourly on a 30-day window costs ~$7.20/month. Daily execution costs $0.31/month. I've seen well-intentioned automation scripts scheduled every 5 minutes by engineers who didn't read the pricing page. Watch out for this — it's a real gotcha that adds cost to your cost reporting.

For more AWS automation patterns we use in production, see the kuryzhev.cloud DevOps runbook archive.

Fix #1: Activate Cost Explorer API and Query It Correctly with boto3

Before writing a single line of Python, activate Cost Explorer in the console and wait 24 hours. Then install a pinned boto3 version for your Lambda layer:

pip install boto3==1.34.69  # reproducible Lambda layer build

The following Lambda handler queries month-to-date costs grouped by AWS service. Read the inline comments — several of these lines exist specifically to avoid the common failure modes described above.

# lambda_function.py
# AWS Cost Explorer → HTML Email Report via SES
# Runtime: python3.12 | boto3 1.34.x
# IAM requires: ce:GetCostAndUsage, ses:SendEmail, s3:PutObject

import boto3
import json
import os
from datetime import datetime, timedelta

# --- Config (use Lambda env vars, never hardcode) ---
SENDER_EMAIL = os.environ["SENDER_EMAIL"]       # must be SES-verified
RECIPIENT_EMAIL = os.environ["RECIPIENT_EMAIL"]
REPORT_BUCKET = os.environ["REPORT_BUCKET"]     # S3 bucket for audit trail
AWS_REGION = os.environ.get("AWS_REGION", "us-east-1")

ce_client = boto3.client("ce", region_name="us-east-1")  # Cost Explorer only in us-east-1
ses_client = boto3.client("ses", region_name=AWS_REGION)
s3_client = boto3.client("s3", region_name=AWS_REGION)


def get_monthly_costs() -> dict:
    """Query Cost Explorer for current month-to-date costs grouped by SERVICE."""
    today = datetime.today()
    # End date is exclusive — use today's date; data through yesterday is returned
    end_date = today.strftime("%Y-%m-%d")
    start_date = today.replace(day=1).strftime("%Y-%m-%d")  # first of current month

    response = ce_client.get_cost_and_usage(
        TimePeriod={"Start": start_date, "End": end_date},
        Granularity="MONTHLY",
        Metrics=["UnblendedCost"],  # NOT BlendedCost — blended averages RI rates, misleading
        GroupBy=[{"Type": "DIMENSION", "Key": "SERVICE"}],
        # WARNING: do NOT add a second GroupBy for TAG here — raises ValidationException
    )

    costs = {}
    for result in response["ResultsByTime"]:
        for group in result.get("Groups", []):  # .get() guards against empty periods
            service = group["Keys"][0]
            amount = round(float(group["Metrics"]["UnblendedCost"]["Amount"]), 2)
            if amount > 0:  # skip zero-cost services to keep report clean
                costs[service] = amount

    # Sort descending by cost — top drivers first
    return dict(sorted(costs.items(), key=lambda x: x[1], reverse=True))


def build_html_report(costs: dict, report_date: str) -> str:
    """Build an HTML email table from the costs dict."""
    total = round(sum(costs.values()), 2)
    rows = "".join(
        f"<tr><td>{svc}</td><td>${amt:.2f}</td></tr>"
        for svc, amt in costs.items()
    )
    return f"""
    <html><body>
    <h2>AWS Cost Report — {report_date}</h2>
    <table border="1" cellpadding="6" style="border-collapse:collapse;">
      <tr><th>Service</th><th>Cost (USD)</th></tr>
      {rows}
      <tr><td><strong>TOTAL</strong></td><td><strong>${total:.2f}</strong></td></tr>
    </table>
    <p>Generated by cost-reporter Lambda | kuryzhev.cloud</p>
    </body></html>
    """


def send_email(html_body: str, report_date: str) -> None:
    """Send the HTML report via SES."""
    ses_client.send_email(
        Source=SENDER_EMAIL,
        Destination={"ToAddresses": [RECIPIENT_EMAIL]},
        Message={
            "Subject": {"Data": f"AWS Cost Report {report_date}"},
            "Body": {
                "Text": {"Data": "Open in HTML-capable email client."},
                "Html": {"Data": html_body},
            },
        },
    )


def save_to_s3(costs: dict, report_date: str) -> None:
    """Persist raw cost data to S3 for audit trail."""
    key = f"cost-reports/{report_date}.json"
    s3_client.put_object(
        Bucket=REPORT_BUCKET,
        Key=key,
        Body=json.dumps(costs, indent=2),
        ContentType="application/json",
    )


def lambda_handler(event, context):
    report_date = datetime.today().strftime("%Y-%m-%d")
    costs = get_monthly_costs()
    html = build_html_report(costs, report_date)
    send_email(html, report_date)
    save_to_s3(costs, report_date)
    return {"statusCode": 200, "body": f"Report sent for {report_date}"}

Two things I want to highlight here. First, ce_client is always initialized with region_name="us-east-1" regardless of where your Lambda runs — Cost Explorer is a global service only accessible through that region endpoint. Second, the result.get("Groups", []) pattern is not optional. On the first day of the month, or in accounts with no spend in a queried period, Groups will be an empty list. Without the .get() fallback, you get a KeyError in production on the 1st of every month.

Fix #2: Parse the Response and Build a Readable Report Structure

The raw API response returns cost amounts as strings, not floats. The Lambda API returns "0.0000012" for a Lambda invocation cost. If you skip the float() cast and try to sort or format directly, you get a TypeError that surfaces only when a low-cost service appears in the results.

The build_html_report() function above handles this with round(float(amount), 2). I use round() to two decimal places rather than string formatting alone, because I want the costs dict to contain actual numeric values for the S3 JSON dump — useful if you later feed this data into a dashboard or Slack message.

A note on templating: I deliberately avoided Jinja2 here. For a single-file Lambda that generates one HTML table, Jinja2 adds a dependency layer with zero benefit. Python's str.join() on a list comprehension of <tr> strings is readable, testable, and requires no extra packaging. I stopped using Jinja2 in Lambda functions after spending 45 minutes debugging a layer packaging issue that turned out to be a Jinja2 version conflict. For anything more complex — multi-section reports, conditional blocks — reconsider. For a cost table, it's overkill.

Sort order matters for the email recipient. The sorted(..., reverse=True) call ensures EC2 and RDS appear at the top, not alphabetically buried below CloudTrail and Config. Executive readers scan the first three rows. Make those rows count.

Fix #3: Send the Report via SES with HTML Formatting

SES has two operational gotchas that will silently break your delivery if you miss them.

Sandbox mode. By default, SES accounts are in sandbox mode. In sandbox mode, SES silently drops emails sent to unverified recipient addresses — no error, no bounce, no log entry. The send_email() call returns HTTP 200 and you have no idea the email never arrived. Request production access via the AWS console under SES → Account dashboard → Request production access before testing with real recipients. See the official SES production access documentation for the process.

Missing Source key. The ses.send_email() call requires Source, Destination, and Message as top-level keys. If Source is missing or the email address is not SES-verified, you get: InvalidParameterValue: Missing required header 'From'. This error message is confusing because you're not setting headers directly — it's SES's way of saying the Source address failed verification.

Now wire up the EventBridge schedule and IAM policy with Terraform. This is the piece most automation attempts skip, and it's why scripts run once and get forgotten.

# eventbridge_lambda.tf
# Terraform: EventBridge rule to trigger Lambda daily at 08:00 UTC
# Requires: aws_lambda_function.cost_reporter already defined

resource "aws_cloudwatch_event_rule" "daily_cost_report" {
  name                = "daily-cost-report"
  description         = "Trigger cost reporter Lambda every day at 08:00 UTC"
  schedule_expression = "cron(0 8 * * ? *)"  # AWS cron uses ? not * for dow when dom is set
}

resource "aws_cloudwatch_event_target" "cost_reporter_target" {
  rule      = aws_cloudwatch_event_rule.daily_cost_report.name
  target_id = "CostReporterLambda"
  arn       = aws_lambda_function.cost_reporter.arn
}

resource "aws_lambda_permission" "allow_eventbridge" {
  statement_id  = "AllowEventBridgeInvoke"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.cost_reporter.function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.daily_cost_report.arn
}

# Least-privilege IAM policy for Lambda execution role
resource "aws_iam_role_policy" "cost_reporter_policy" {
  name = "cost-reporter-policy"
  role = aws_iam_role.cost_reporter_exec.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["ce:GetCostAndUsage"]
        Resource = "*"  # Cost Explorer does not support resource-level permissions
      },
      {
        Effect   = "Allow"
        Action   = ["ses:SendEmail"]
        Resource = "*"
      },
      {
        Effect   = "Allow"
        Action   = ["s3:PutObject"]
        Resource = "arn:aws:s3:::${var.report_bucket}/cost-reports/*"  # scoped to prefix
      }
    ]
  })
}

Watch out for the EventBridge cron syntax. AWS uses ? instead of * for the day-of-week field when day-of-month is already specified, and vice versa. cron(0 8 * * * *) will fail with a validation error. The correct form is cron(0 8 * * ? *). This trips up every engineer the first time. See the EventBridge cron expression documentation for the full syntax reference.

Also set your Lambda timeout to at least 30 seconds. The default is 3 seconds. Cost Explorer API p95 latency is 4–6 seconds. Your function will time out silently on the first real invocation if you leave the default in place.

Prevention: Make Cost Reporting a Reliable, Auditable System

Getting the script working is step one. Making it reliable over months without manual intervention is the harder part. Here's what I add to every cost reporting Lambda before considering it production-ready.

CloudWatch alarm on Lambda errors. Create an alarm on the Errors metric for the function with threshold > 0 for one evaluation period. If SES throttles, IAM permissions drift after a role rotation, or Cost Explorer returns a transient error, you want to know within minutes — not after a week of missed reports when someone notices the inbox has gone quiet.

S3 audit trail with lifecycle policy. The save_to_s3() function stores each day's raw cost JSON at cost-reports/YYYY-MM-DD.json. At ~5KB per file, 365 files/year costs essentially nothing in S3 Standard. Add a lifecycle rule to expire objects after 90 days. Do not enable S3 Intelligent-Tiering on these objects — the per-object monitoring fee for Intelligent-Tiering exceeds the storage cost of a 5KB file. That's a real gotcha I've hit on similar audit trail patterns.

Never use environment variables for AWS credentials in Lambda. Use the IAM execution role exclusively. Environment variable secrets are visible to anyone with lambda:GetFunctionConfiguration permission — which is a surprisingly common IAM grant in developer accounts. The execution role approach is both more secure and eliminates credential rotation as a failure mode.

Pin your runtime. Use python3.12. Avoid python3.8, which reached EOL. Unpinned runtimes drift when AWS deprecates versions and forces migration — you want to control that on your schedule, not AWS's.

Handler path must match exactly. If your file is lambda_function.py, the handler value in the Lambda configuration must be lambda_function.lambda_handler. A mismatch causes Runtime.HandlerNotFound at invocation time. It's obvious when it happens, but it wastes five minutes every time someone renames a file without updating the handler config.

AWS cost report Python automation done right means the engineering team knows about cost spikes before the finance team does — and has the data to explain exactly which service caused them. This Lambda, scheduled daily via EventBridge, delivers that visibility with under 100 lines of Python and a Terraform block that takes 10 minutes to apply.

Makefile Patterns for Terraform and Infra Repos That Actually Scale

Oleksandr Kuryzhev — Mon, 29 Jun 2026 07:02:29 +0000

Originally published on kuryzhev.cloud

Every infra repo eventually gets a Makefile — but most of them silently break in CI, leak credentials in logs, or stop working the moment someone creates a file named plan. Makefile infra repo patterns are one of those topics where everyone has an opinion, but few people have actually debugged a production pipeline that failed because Make thought a directory was an up-to-date target. I have. Twice. Here is what I learned.

What Make Actually Does in an Infra Repo

Make was not designed for infrastructure. It was designed to avoid recompiling C files that had not changed. That file-dependency model is exactly what causes confusion when you drop it into a Terraform repo — because you are not producing files, you are running commands against remote APIs.

What Make actually gives you in an infra context is a self-documenting command interface layer over shell, Terraform, kubectl, and Ansible. Nothing more. When you run make plan, Make reads the Makefile top-to-bottom, resolves targets as a directed acyclic graph, checks whether a file named plan exists in the current directory, and if it does not find one, executes the recipe. That last part is the trap. If a file named plan exists — maybe leftover from a terraform plan -out=plan run — Make considers the target up-to-date and silently skips it. No error. No output. Nothing happens.

This is why .PHONY declarations are not optional. They tell Make: this target is a command, not a file. Declare every infra target as phony without exception.

It is also worth knowing that macOS ships with GNU Make 3.81 from 2006. Homebrew installs 4.4.1. Recipe-scoped variables using := inside targets behave differently between these versions. If your Makefile works locally but not in a fresh CI container, version mismatch is the first thing to check with make --version.

For teams uncomfortable with Make's syntax, two alternatives are worth knowing: Task (go-task) v3.35+ uses a YAML-native Taskfile.yml with built-in dotenv, preconditions, and parallel deps support. Justfile (v1.25+) is even simpler — pure shell-like syntax with no file-dependency model at all. I reach for Make when the team knows it and the repo is small. I reach for Task when the workflow grows beyond 20 targets or when I want native parallelism on lint jobs without thinking about it.

How People Use Makefile Infra Repos Wrong

Three patterns show up in almost every infra repo I have reviewed, and all three cause real production problems.

Hardcoded environment values inside targets. I see this constantly: REGION=eu-west-1 baked into the recipe line, not in a variable. The target works for the engineer who wrote it and silently targets the wrong environment for everyone else. The fix is ?= defaults at the top of the file — ENV ?= dev, REGION ?= eu-west-1 — so the target works locally and in CI without modification, but can be overridden with make plan ENV=staging.

Missing .PHONY declarations. As described above: if a file named apply, plan, or destroy ever appears in the repo, Make skips the target silently. The full declaration you need at minimum is: .PHONY: all help plan apply destroy lint fmt validate. Put it at the top. Add to it every time you add a target.

Chaining targets with && inside recipes instead of using Make's dependency graph. When you write plan: init && validate inside a recipe line, you break make -n dry-run output, you hide individual target failures, and you make parallelism impossible. The correct pattern is to declare dependencies: plan: init validate as the target line, then the recipe body. Make resolves the dependency graph before executing.

Watch out for this one: include .env at the top of a Makefile loads all values as Make variables, which means they appear in make -p output in plain text. If your .env file contains AWS credentials or tokens, anyone with access to the CI log for make -p can read them. Use dotenv in Task, or export variables inside recipe lines rather than at the Makefile top level.

Another gotcha: each line in a Make recipe is a separate shell subprocess. Using export FOO=bar on line one of a recipe does not export FOO to line two. If you need a variable to persist across recipe lines, either collapse them into one line with ; or use .ONESHELL: (GNU Make 3.82+).

The Correct Approach — Structured Makefile for Infra Repos

The pattern I use in production repos has three rules: the default target is always help, secrets never appear as top-level variable assignments, and sub-makefiles keep the root file under 30 lines.

The help target uses grep and awk to parse ## comments on target lines. Every engineer who clones the repo runs make and immediately sees documented commands. No README hunting. The guard-% macro is the second essential pattern — a reusable prerequisite target that checks required environment variables exist before any target executes. It prevents terraform apply with empty TF_VAR_ values, which is a very easy way to corrupt state.

Here is the full production Makefile pattern. Note the SHELL and .SHELLFLAGS settings at the top — these make every recipe behave like set -euo pipefail, which means unset variables and failed pipe commands abort the recipe instead of silently continuing:

# Makefile — root infra repo task interface
# Compatible with GNU Make 4.x (brew install make on macOS)
# Usage: make help

SHELL := /bin/bash
.SHELLFLAGS := -eu -o pipefail -c

# ── Overridable defaults ────────────────────────────────────────────────────
ENV        ?= dev
REGION     ?= eu-west-1
TF_DIR     ?= terraform/envs/$(ENV)
PLAN_FILE  ?= $(ENV).tfplan

# ── Default target: self-documenting help ───────────────────────────────────
.DEFAULT_GOAL := help

.PHONY: help
help: ## Show this help message
    @grep -E '^[a-zA-Z_%-]+:.*##' $(MAKEFILE_LIST) \
        | awk 'BEGIN {FS = ":.*##"}; {printf "  \033[36m%-20s\033[0m %s\n", $$1, $$2}'

# ── Guard macro: require env var before any target runs ────────────────────
# Usage: add guard-VARNAME as a dependency
# Example: plan: guard-ENV guard-AWS_PROFILE
guard-%:
    @[ "${$*}" ] || (echo "❌  Required variable '$*' is not set"; exit 1)

# ── Include modular sub-makefiles ───────────────────────────────────────────
-include make/terraform.mk
-include make/docker.mk
-include make/lint.mk

# ── Terraform targets ───────────────────────────────────────────────────────
.PHONY: init plan apply destroy

init: guard-ENV ## terraform init for ENV (default: dev)
    @echo "→ Initialising $(TF_DIR)"
    @cd $(TF_DIR) && terraform init -reconfigure

plan: guard-ENV guard-AWS_PROFILE ## Generate plan for ENV, saves to PLAN_FILE
    @cd $(TF_DIR) && terraform plan \
        -var-file="vars/$(ENV).tfvars" \
        -out=$(PLAN_FILE)

apply: guard-ENV guard-AWS_PROFILE ## Apply saved plan (requires PLAN_FILE)
    @[ -f "$(TF_DIR)/$(PLAN_FILE)" ] || \
        (echo "❌  No plan file found. Run: make plan ENV=$(ENV)"; exit 1)
    @cd $(TF_DIR) && terraform apply $(PLAN_FILE)

destroy: guard-ENV guard-AWS_PROFILE ## Destroy ENV — requires CONFIRM=yes
    @[ "$(CONFIRM)" = "yes" ] || \
        (echo "❌  Set CONFIRM=yes to proceed with destroy"; exit 1)
    @cd $(TF_DIR) && terraform destroy -var-file="vars/$(ENV).tfvars"

# ── Lint targets — safe to parallelise ─────────────────────────────────────
.PHONY: lint fmt validate

lint: fmt validate ## Run all linters (parallelisable with make -j3 lint)
    @echo "✅  Lint complete"

fmt: ## Run terraform fmt check (non-destructive)
    @terraform fmt -check -recursive $(TF_DIR)

validate: ## Run terraform validate
    @cd $(TF_DIR) && terraform validate

# ── Dynamic per-environment plan targets ───────────────────────────────────
# Generates: plan-dev, plan-staging, plan-prod automatically
ENVS := dev staging prod

define ENV_TARGET
.PHONY: plan-$(1)
plan-$(1): guard-AWS_PROFILE ## Run plan for environment: $(1)
    @$(MAKE) plan ENV=$(1)
endef

$(foreach env,$(ENVS),$(eval $(call ENV_TARGET,$(env))))

A few things worth calling out in this file. The -include prefix on sub-makefile includes means Make silently skips the file if it does not exist — useful when not every repo has Docker targets. The $(MAKE) call inside the ENV_TARGET macro is intentional: using bare make would drop flags like -n or -j that the caller passed in. Always use $(MAKE) for recursive calls.

Advanced Patterns for Real Infrastructure Workflows

The guard-% macro and the foreach/eval pattern for generating environment targets are the two that have saved us the most time in real CI pipelines. The dynamic target generation means adding a new environment to ENVS := dev staging prod canary automatically creates plan-canary, apply-canary, and any other targets defined in the macro block. No copy-paste. No drift between environments.

For teams that find Make syntax genuinely hostile — and some do — go-task v3.35+ maps to all of these patterns in YAML. The preconditions key replaces guard-%, dotenv replaces include .env without the credential exposure risk, and deps runs subtasks in parallel by default. Here is the equivalent Taskfile:

# Taskfile.yml — go-task v3.35+ equivalent for teams preferring YAML
# Install: brew install go-task  |  version: '3' schema required
# Usage: task help

version: '3'

set: [pipefail]
shopt: [globstar]

dotenv: ['.env', '.env.{{.ENV}}']  # loads .env.dev, .env.staging etc.

vars:
  ENV: '{{.ENV | default "dev"}}'
  REGION: '{{.REGION | default "eu-west-1"}}'
  TF_DIR: 'terraform/envs/{{.ENV}}'

tasks:

  default:
    desc: Show available tasks
    cmds:
      - task --list

  plan:
    desc: "Run terraform plan for ENV (default: dev)"
    preconditions:
      - sh: '[ -n "{{.AWS_PROFILE}}" ]'
        msg: "AWS_PROFILE must be set"
      - sh: '[ -d "{{.TF_DIR}}" ]'
        msg: "Directory {{.TF_DIR}} does not exist — check ENV value"
    cmds:
      - cd {{.TF_DIR}} && terraform plan
          -var-file="vars/{{.ENV}}.tfvars"
          -out={{.ENV}}.tfplan

  lint:
    desc: Run all linters in parallel
    deps: [lint:fmt, lint:validate, lint:tflint]  # parallel by default in Task

  lint:fmt:
    internal: true
    cmds:
      - terraform fmt -check -recursive {{.TF_DIR}}

  lint:validate:
    internal: true
    cmds:
      - cd {{.TF_DIR}} && terraform validate

  lint:tflint:
    internal: true
    cmds:
      - tflint --chdir={{.TF_DIR}} --config=.tflint.hcl

The dotenv key in Task is safer than Make's include .env because Task does not expose dotenv values through a --list or debug output equivalent. It also supports per-environment files automatically: .env.staging is loaded when ENV=staging. That pattern alone has replaced a lot of fragile CI variable injection in our pipelines. More on our CI patterns at kuryzhev.cloud.

Performance Notes and Security Considerations

Make's parallel execution flag -j is genuinely useful for lint targets. Running make -j$(nproc) lint across 20 Terraform modules — tflint, tfsec, checkov in parallel — drops CI lint time from around four minutes to under 45 seconds in our experience. That is a meaningful improvement on a team running 30+ PRs a day.

But make -j4 on Terraform targets that share a state backend is a different story. Parallel plan and apply jobs against the same S3 backend will race on the state lock. You will see Error: Error acquiring the state lock in CI, and if the lock acquisition fails mid-apply, you may end up with a partial state that requires manual terraform force-unlock. Never use -j on stateful Terraform targets. Lint and format targets are safe. Apply and plan are not.

Watch out for this security issue: never put $(shell aws sts get-caller-identity) or any credential-fetching shell call at the top level of a Makefile as a variable assignment. Top-level variable assignments are evaluated at parse time, which means they execute on every single make invocation — including make help. Every engineer running make help on a shared CI runner triggers an AWS API call that lands in CloudTrail. Multiply that by 50 engineers and 200 PR checks per day and you have noisy CloudTrail logs that obscure real security events. Move shell calls that contact external APIs into recipe bodies, prefixed with @ to suppress echo, and only in targets that actually need them.

The @ prefix suppresses command echo in Make output. Use it on every recipe line that handles credentials or tokens. One common typo: @@command. Double @ does not exist in Make syntax — it causes a parse error that looks like a missing separator. If your Makefile suddenly breaks with Makefile:42: *** missing separator. Stop., check for accidental double-@ on recipe lines.

Finally, the CONFIRM=yes guard on destroy targets is not paranoia. It is the one pattern I will argue for in every code review. Running make destroy without a confirmation check in a repo where ENV defaults to prod — or where a CI variable is misconfigured — is a bad day. The two-second friction of typing CONFIRM=yes has prevented at least one production incident I can point to directly. See the GNU Make manual for the full reference on guard patterns and conditional variable assignment.

Makefile infra repo patterns are not glamorous, but they are the difference between a repo that onboards new engineers in ten minutes and one that requires a tribal knowledge transfer every time someone joins the team. The patterns here — .PHONY everywhere, guard-% macros, overridable defaults, and parallel-safe lint targets — are what I would put in every infra repo I own today.

Query AWS Runbooks in Plain English with Bedrock Knowledge Bases

Oleksandr Kuryzhev — Sun, 28 Jun 2026 07:02:27 +0000

Originally published on kuryzhev.cloud

The Night Our On-Call Engineer Couldn't Find the Runbook

Amazon Bedrock Knowledge Bases can answer "what do we do when Aurora read replica lag exceeds 30 seconds" in under two seconds — but at 2 AM on a Tuesday, our on-call engineer was doing something very different. He was scrolling through Confluence search results, staring at three runbooks titled almost identically: RDS Failover Procedure, dated March 2021, October 2022, and February 2023. None of them said which was authoritative. He picked the middle one and started following it.

Fourteen minutes later, he realized he was working through a procedure written for a single-AZ RDS instance. Our cluster had been Multi-AZ Aurora for eighteen months. The actual failover had already completed automatically. What he needed was a five-line check to validate replica promotion and re-point the application connection string. That information existed. It was in a Markdown file in our ops-runbooks S3 bucket, uploaded after a postmortem in Q3 2023, and it had never been linked from Confluence.

The incident itself resolved in four minutes. Our MTTR on the report showed twenty-two minutes. Eighteen of those minutes were pure documentation chaos. That postmortem is what kicked off the work I'm going to walk you through here.

Why Operational Docs Become a Liability Over Time

This isn't a discipline problem. I've worked with genuinely excellent engineers who let runbooks rot, and the reason is structural, not motivational. Documentation in most DevOps teams lives across at least four silos simultaneously: S3 buckets holding postmortem Markdown files, Confluence spaces with wiki pages, GitHub repository wikis attached to specific services, and PDF attachments in old JIRA tickets. There is no single retrieval layer across all of them, and there is definitely no semantic search.

The second problem is version drift. Engineers write runbooks during or immediately after incidents, when the system context is fresh. They almost never revisit them. Six months later, the infrastructure has changed — a new Aurora version, a different connection pooling setup, a renamed parameter group — but the runbook hasn't. Version drift is invisible until it causes a second incident. And then it shows up in the postmortem as "engineer followed incorrect procedure," which obscures the real root cause: the documentation system has no mechanism for staleness detection.

The third problem is that keyword search fundamentally fails on operational questions. Searching Confluence for "Aurora replica lag" returns every page that mentions those words in any context. The answer to "what do we do when replica lag exceeds 30 seconds" is buried three paragraphs into a postmortem written in prose, not indexed as a discrete fact. Standard search engines aren't built for that retrieval pattern. Vector search is.

This is exactly the gap Amazon Bedrock Knowledge Bases fills. It ingests your existing documentation — Markdown, PDF, plain text, Word docs — chunks it, embeds it into a vector index, and exposes a RetrieveAndGenerate API that accepts a natural language question and returns a grounded answer with citations pointing back to the source document. No hallucination of procedures that don't exist. No guessing which version is current. Just the answer, and the file it came from.

Building the Knowledge Base with Terraform — The Fix

The implementation has three layers: an S3 bucket as the documentation source, an OpenSearch Serverless collection as the vector store, and the Bedrock Knowledge Base resource that wires them together. I'll cover two important gotchas before the code, because they cost us an afternoon of debugging.

Watch out for this: The S3 source bucket must be in the same AWS region as the Knowledge Base. Cross-region sources are not supported, and the API will not tell you this clearly — it simply fails to ingest. We had our runbooks bucket in us-west-2 and tried to create the Knowledge Base in us-east-1. The Knowledge Base created successfully. The ingestion job completed with zero documents indexed. No error. Thirty minutes of debugging a perfectly valid configuration before we checked the region constraint in the Bedrock Knowledge Bases documentation.

Watch out for this too: The OpenSearch Serverless collection type must be VECTORSEARCH. Using TIMESERIES or SEARCH causes the Knowledge Base creation to fail with a ValidationException that mentions "incompatible collection type" — which is clear enough once you know what to look for, but easy to miss if you're copying an existing OSS collection config from another project.

The IAM role for the Knowledge Base needs two permissions that are easy to miss together: s3:GetObject on the source bucket, and aoss:APIAccessAll on the OpenSearch Serverless collection. Missing the AOSS policy is the single most common deployment blocker we've seen across three separate team setups.

Here's the Terraform that creates the full stack. We're using Titan Embeddings V2 for the embedding model — it handles English operational text well and costs less than Cohere at our query volume. OpenSearch Serverless minimum is two OCUs (one indexing, one search), which runs about $175/month. For a low-traffic internal tool, Aurora pgvector is 60–70% cheaper at comparable latency, but OSS is simpler to operate and scales automatically if your team grows.


# bedrock_knowledge_base.tf
# Terraform >= 1.7, AWS provider >= 5.40.0
# Creates: S3 source bucket, Bedrock Knowledge Base, OpenSearch Serverless collection

resource "aws_s3_bucket" "runbooks" {
  bucket = "ops-runbooks-bedrock-source-${var.environment}"
  # IMPORTANT: bucket must be in same region as Knowledge Base
}

resource "aws_s3_bucket_server_side_encryption_configuration" "runbooks" {
  bucket = aws_s3_bucket.runbooks.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = var.kms_key_arn  # pass same key to Knowledge Base below
    }
  }
}

resource "aws_bedrockagent_knowledge_base" "ops_docs" {
  name     = "ops-runbooks-kb"
  role_arn = aws_iam_role.bedrock_kb.arn

  knowledge_base_configuration {
    type = "VECTOR"
    vector_knowledge_base_configuration {
      # Titan Embeddings V2 — 1536 dimensions, supports English operational text
      embedding_model_arn = "arn:aws:bedrock:us-east-1::foundation-model/amazon.titan-embed-text-v2:0"
    }
  }

  storage_configuration {
    type = "OPENSEARCH_SERVERLESS"
    opensearch_serverless_configuration {
      collection_arn    = aws_opensearchserverless_collection.vectors.arn
      vector_index_name = "ops-runbooks-index"
      field_mapping {
        vector_field   = "bedrock-knowledge-base-default-vector"
        text_field     = "AMAZON_BEDROCK_TEXT_CHUNK"
        metadata_field = "AMAZON_BEDROCK_METADATA"
      }
    }
  }
}

resource "aws_bedrockagent_data_source" "runbooks_s3" {
  knowledge_base_id = aws_bedrockagent_knowledge_base.ops_docs.id
  name              = "runbooks-s3-source"

  data_source_configuration {
    type = "S3"
    s3_configuration {
      bucket_arn = aws_s3_bucket.runbooks.arn
    }
  }

  vector_ingestion_configuration {
    chunking_configuration {
      chunking_strategy = "FIXED_SIZE"
      fixed_size_chunking_configuration {
        max_tokens         = 300
        overlap_percentage = 20  # 20% overlap preserves context across chunk boundaries
      }
    }
  }
}

# EventBridge rule: re-sync Knowledge Base when new runbook lands in S3
resource "aws_cloudwatch_event_rule" "s3_runbook_upload" {
  name = "bedrock-kb-sync-on-runbook-upload"
  event_pattern = jsonencode({
    source      = ["aws.s3"]
    detail-type = ["Object Created"]
    detail = {
      bucket = { name = [aws_s3_bucket.runbooks.bucket] }
    }
  })
}

One note on encryption: enable SSE-KMS on the source bucket and pass the same KMS key ARN to the Knowledge Base configuration. Runbooks frequently contain internal hostnames, connection strings, and escalation contacts. Storing them unencrypted at rest in the vector index is a security gap that's easy to overlook because the ingestion pipeline doesn't warn you.

Ingestion is not real-time. After a document lands in S3, expect a 2–8 minute lag before it's queryable. The EventBridge rule above triggers a StartIngestionJob automatically on S3 object creation, so your index stays fresh without manual intervention. For a 50-document corpus, ingestion typically completes in under five minutes.

Wiring It Into Your Incident Workflow

The Knowledge Base only helps if engineers actually use it during incidents. Telling people "go to the AWS console and query the Knowledge Base" doesn't work under pressure. We wired it directly into Slack via a Lambda-backed slash command: /ask-runbook "RDS failover steps". The answer appears in the channel thread within three seconds, with citations showing which S3 file each chunk came from.

One thing that tripped up two engineers on our team: the correct SDK method is client.retrieve_and_generate() on the bedrock-agent-runtime client — not client.invoke_model() on the bedrock-runtime client. They're different service clients entirely. Using invoke_model() bypasses the Knowledge Base retrieval layer and talks directly to the foundation model, which will confidently hallucinate procedures that don't exist in your documentation. I stopped trusting any runbook Q&A answer after this happened once in a staging incident drill.

We set maxTokens to 512 and temperature to 0.0. The token cap keeps answers readable in Slack without truncating critical steps. Zero temperature makes responses deterministic — for operational procedures, you don't want creative variation between queries. We also filter out retrieval results with a score below 0.4 in application logic, so low-confidence chunks don't pollute the cited sources list. The RetrieveAndGenerate API reference documents the full response schema including the score field per retrieved reference.

Cost at our query volume runs approximately $0.008–$0.015 per query, combining Titan Embeddings V2 ingestion cost with Claude 3 Sonnet inference on retrieval. At 200 incident queries per month, that's under $3. The OpenSearch Serverless OCU cost dominates the budget by a wide margin — the per-query inference cost is essentially rounding error.


# lambda_function.py
# Slack slash command handler: /ask-runbook "<question>"
# Runtime: Python 3.12, boto3 >= 1.34.0
# Required env vars: KNOWLEDGE_BASE_ID, MODEL_ARN, SLACK_BOT_TOKEN

import json
import os
import boto3
from slack_sdk import WebClient
from slack_sdk.errors import SlackApiError

bedrock_runtime = boto3.client(
    service_name="bedrock-agent-runtime",
    region_name="us-east-1"  # must match Knowledge Base region
)

slack_client = WebClient(token=os.environ["SLACK_BOT_TOKEN"])

KNOWLEDGE_BASE_ID = os.environ["KNOWLEDGE_BASE_ID"]
# Use Claude 3 Sonnet — balances cost and answer quality for runbook Q&A
MODEL_ARN = os.environ["MODEL_ARN"]
# Example: arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-sonnet-20240229-v1:0

def query_knowledge_base(question: str) -> dict:
    """
    Call RetrieveAndGenerate with citation mode enabled.
    Returns generated answer and source document references.
    """
    response = bedrock_runtime.retrieve_and_generate(
        input={"text": question},
        retrieveAndGenerateConfiguration={
            "type": "KNOWLEDGE_BASE",
            "knowledgeBaseConfiguration": {
                "knowledgeBaseId": KNOWLEDGE_BASE_ID,
                "modelArn": MODEL_ARN,
                "retrievalConfiguration": {
                    "vectorSearchConfiguration": {
                        # Return top 5 chunks; filter low-confidence in post-processing
                        "numberOfResults": 5
                    }
                },
                "generationConfiguration": {
                    "inferenceConfig": {
                        "textInferenceConfig": {
                            # Keep answers concise for Slack readability
                            "maxTokens": 512,
                            "temperature": 0.0  # deterministic for operational answers
                        }
                    }
                }
            }
        }
    )
    return response

def format_slack_response(bedrock_response: dict) -> str:
    """
    Extract answer text and cited source S3 keys.
    Filters out citations with retrieval score below 0.4.
    """
    answer = bedrock_response["output"]["text"]
    citations = bedrock_response.get("citations", [])

    sources = []
    for citation in citations:
        for ref in citation.get("retrievedReferences", []):
            score = ref.get("score", 0)
            if score >= 0.4:  # discard low-confidence chunks
                s3_uri = ref["location"]["s3Location"]["uri"]
                sources.append(f"• `{s3_uri}` (score: {score:.2f})")

    source_block = "\n".join(sources) if sources else "_No high-confidence sources found_"
    return f"*Answer:*\n{answer}\n\n*Sources:*\n{source_block}"

def lambda_handler(event, context):
    # Slack sends slash command payload as URL-encoded form data
    body = dict(x.split("=") for x in event["body"].split("&"))
    question = body.get("text", "").replace("+", " ")
    channel_id = body["channel_id"]

    if not question:
        return {"statusCode": 200, "body": "Please provide a question after /ask-runbook"}

    try:
        bedrock_response = query_knowledge_base(question)
        formatted = format_slack_response(bedrock_response)
        slack_client.chat_postMessage(channel=channel_id, text=formatted)
    except bedrock_runtime.exceptions.ResourceNotFoundException:
        slack_client.chat_postMessage(
            channel=channel_id,
            text="Knowledge Base not found. Check KNOWLEDGE_BASE_ID env var."
        )
    except SlackApiError as e:
        print(f"Slack error: {e.response['error']}")

    return {"statusCode": 200, "body": ""}

One format note: Bedrock Knowledge Bases supports .md, .txt, .html, .doc, .docx, .pdf, and .csv for ingestion. If your team writes runbooks in AsciiDoc (.adoc) or reStructuredText (.rst) — both unsupported — you'll need a CI step that pre-converts them before pushing to S3. We handle this with a small Pandoc step in our GitHub Actions pipeline on merge to main. It's two lines and it saves a confusing silent-failure when the ingestion job processes zero pages from a file it can't parse.

Prevention Checklist — Keeping the Knowledge Base Trustworthy

The system you just built will decay into the same problem it solved unless you treat the documentation pipeline as a first-class engineering concern. Here's what we put in place to prevent that.

Docs-as-code with CI enforcement. Runbooks live in a Git repository. A GitHub Actions pipeline lints Markdown on every pull request (we use markdownlint-cli2) and pushes validated files to the S3 source bucket on merge to main. The CI step also converts any non-native formats via Pandoc. This means the Knowledge Base index only ever contains files that passed a review and a format check. Stale docs in someone's local drive never reach the vector store.

Zero-query alerting. We set a CloudWatch alarm on the KnowledgeBaseQueryCount metric dropping to zero for seven consecutive days. Silence is a warning signal. It means engineers stopped trusting the tool, which is exactly what happened with our old Confluence setup before we noticed. The alarm routes to the same PagerDuty service as our monitoring alerts, but at low priority — it creates a ticket, not a page.

Quarterly retrieval score review. Bedrock logs retrieval results to CloudWatch Logs when you enable that option on the Knowledge Base. We have a scheduled Lambda that queries those logs monthly, identifies source documents where the average retrieval score across all queries is below 0.35, and creates a JIRA ticket flagging them for human review. Low scores mean the chunks from that document aren't matching real operational questions — usually because the document is outdated, poorly structured, or covers a system that no longer exists.

Region discipline. Add a variable "aws_region" to your Terraform with a validation block that enforces it matches the region of your Bedrock-enabled account. Prevent the cross-region S3 source mistake at the infrastructure definition layer, not at debug time.

Model ARN pinning. Store the MODEL_ARN environment variable in AWS Systems Manager Parameter Store, not as a hardcoded Lambda environment variable. When Bedrock releases a new Claude version, you update one SSM parameter and all Lambda functions pick it up on next invocation — no deployment required. This also prevents the silent empty-response failure that happens when a model ARN from the wrong region is used.

The Amazon Bedrock Knowledge Bases Q&A pattern isn't magic. It's a retrieval layer over documentation you already have. The value comes from making that documentation queryable in natural language, with citations, at the moment an engineer needs it most. We've seen our incident documentation MTTR contribution drop from an average of twelve minutes to under ninety seconds since rolling this out. That's not a metric we expected to move. It moved anyway. You can read more about how we approach AWS automation and incident tooling at kuryzhev.cloud.

Fix GitHub Copilot Terraform Security Risks Before They Hit Prod

Oleksandr Kuryzhev — Sat, 27 Jun 2026 07:02:52 +0000

Originally published on kuryzhev.cloud

Copilot just autocompleted your security group with port 0–65535 open to the world — and terraform validate said it was fine. That's the GitHub Copilot Terraform security problem in one sentence: the suggestions are syntactically valid, pass every local check, and still destroy your security posture on first apply. We've seen it happen across three separate teams in the last six months, and the pattern is always the same: nobody noticed until a compliance scan flagged it post-deploy.

Symptoms

The signs aren't loud. That's what makes this dangerous. Here's what we actually observed before we locked things down.

Security group rules open to the world. Copilot autocompletes resource "aws_security_group" blocks with ingress { from_port = 0, to_port = 0, cidr_blocks = ["0.0.0.0/0"] }. It's valid HCL. It passes terraform validate. The first terraform plan shows a clean diff. Then on the second apply — after your existing state has been modified — you get a conflict you can't easily roll back.

RDS instances with public access enabled. In roughly 60% of completions observed in public issue trackers, Copilot sets publicly_accessible = true on aws_db_instance resources. It also defaults deletion_protection = false on RDS clusters, Cloud SQL instances, and Azure PostgreSQL servers. Both values look like reasonable defaults to someone new to the codebase.

Kubernetes manifests with security context stripped. The Helm/Kubernetes variant is equally quiet. Copilot suggests hostNetwork: true as a quick fix for DNS resolution issues inside pods — which bypasses network policy entirely. It drops readOnlyRootFilesystem from securityContext blocks without any warning. The manifest applies cleanly. The container runs. The risk is invisible until someone audits it.

Hallucinated module references. Copilot has zero visibility into your .tfstate. It generates references like module.vpc.private_subnet_ids that don't exist in your actual module structure. The error only surfaces at plan time: Error: Reference to undeclared module — A managed resource "module.vpc" has not been declared in the root module. By then the suggestion has already been committed.

Watch out for: lifecycle { ignore_changes = all } appearing in Copilot suggestions as a way to "silence drift warnings." This is a correctness trap. It masks real infrastructure divergence and should be treated as a blocking finding, not a style preference.

Root Cause: Why Copilot Gets IaC Wrong

Stop blaming yourself or your team. There are three structural reasons Copilot underperforms specifically on infrastructure code compared to application code.

Root cause #1: Training data skew. Public Terraform repos over-represent quick demos, blog posts, and tutorials that intentionally skip security hardening to keep examples short. Copilot's probability distribution has learned from that corpus. It favors insecure defaults — deletion_protection = false, publicly_accessible = true, acl = "public-read" — because those values appear constantly in "getting started" content. Checkov check CKV_AWS_57 exists specifically because S3 buckets with public ACLs are that common in training data.

Root cause #2: No state awareness. Copilot has no access to your .tfstate, your module outputs, your workspace variables, or your backend configuration. It generates module references and resource outputs based on pattern matching against what it has seen in public repos. When your module structure doesn't match that pattern, the suggestion compiles but fails at plan or apply time. There's no feedback loop.

Root cause #3: Context window truncation. In files over roughly 300 lines, Copilot loses the top-of-file provider block and required_providers version constraints. It starts generating syntax valid for Terraform 0.12 or 0.13 — ${var.name} interpolation where it's unnecessary, list() and map() type constructors that were deprecated in 0.14 — inside a codebase running Terraform 1.7.x. The code applies, but it's semantically wrong and will cause issues when you upgrade.

One more gotcha worth calling out separately: Copilot Chat in VS Code reads all open editor tabs as context. If you have prod.tfvars open while asking Copilot to "generate a similar staging config," it will echo production account IDs, bucket names, and state key paths back into the generated output. Every repo contributor with Copilot access can see those values in the suggestion. This is a lateral information exposure risk in multi-team organizations, and most engineers don't know it's happening.

Fix #1 — Add a `.github/copilot-instructions.md` Guardrail File

The fastest single-file intervention. No CI changes required. Supported in GitHub Copilot for Business and Enterprise (not available in individual Free/Pro plans as of Q1 2025). This file instructs Copilot to follow repo-specific rules during both inline completions and Copilot Chat sessions.

Create the file at exactly this path — .github/copilot-instructions.md — and include directives like these:

# Copilot Instructions — IaC Repository

## Security Rules (apply to all Terraform and Kubernetes suggestions)

- Never suggest `0.0.0.0/0` in security group ingress or egress rules
- Always include `lifecycle { prevent_destroy = true }` on stateful resources
  (aws_db_instance, aws_s3_bucket, aws_elasticache_cluster, aws_rds_cluster)
- Default encryption to `true` for all storage resources
- Set `publicly_accessible = false` on all database resources
- Set `deletion_protection = true` on all database and cache resources
- Never suggest `lifecycle { ignore_changes = all }` — this masks drift
- Pin all provider versions using the `~>` pessimistic constraint operator
- Never suggest `hostNetwork: true` in Kubernetes or Helm manifests
- Always include `readOnlyRootFilesystem: true` in container securityContext

## Terraform Version
- Target Terraform >= 1.5.0. Do not generate 0.12-era interpolation syntax.
- Use `for_each` with conditional sets instead of `count` for feature toggles.

## Provider Versions
- aws: ~> 5.40
- kubernetes: ~> 2.30
- helm: ~> 2.13

Watch out for: this file is advisory, not enforced. Copilot may still violate these rules under high-ambiguity completions or when the file isn't fully loaded in context. Treat it as signal reduction — it meaningfully reduces bad suggestions, but it is not a hard block. You still need the CI layer in Fix #2.

Fix #2 — Block Dangerous Patterns in CI with tfsec and Checkov

This is the enforcement layer. What the instructions file misses, the pipeline catches. We run both tfsec and Checkov as a single required status check on every PR that touches *.tf or *.tfvars files. Both tools must pass before merge is allowed — not advisory, not continue-on-error: true.

The most common mistake I see: teams install tfsec and Checkov, set continue-on-error: true to avoid blocking the team during rollout, and then never flip the flag. Findings accumulate. Nobody resolves them. The tools become theater. Make the check required in branch protection rules from day one, even if you start with a narrow check list.

Note on tooling: tfsec version 1.28.x is the last stable OSS release before Aqua Security shifted focus to trivy for IaC scanning. The forward-compatible replacement is trivy config . — worth planning the migration now. We're still on tfsec in older pipelines but all new repos use trivy.

Here's the full workflow. It scopes Checkov to only changed Terraform directories using git diff output, which cuts scan runtime from 4–6 minutes on large monorepos down to under 45 seconds. It also uploads tfsec findings to the GitHub Security tab via SARIF — but note that SARIF upload via github/codeql-action/upload-sarif@v3 requires GitHub Advanced Security enabled on the repo:

# .github/workflows/iac-copilot-guardrails.yml
# Blocks dangerous Copilot-generated IaC patterns on every PR
# Requires: GitHub Actions, tfsec 1.28.x, checkov >= 2.3.0

name: IaC Security Scan

on:
  pull_request:
    paths:
      - '**.tf'
      - '**.tfvars'
      - '**/Chart.yaml'
      - '**/values.yaml'

permissions:
  contents: read
  security-events: write   # required for SARIF upload to GitHub Security tab
  pull-requests: write     # required for inline PR annotations

jobs:
  security-scan:
    name: Scan Copilot-Generated IaC
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      # Identify only changed Terraform directories to keep scan fast
      - name: Get changed TF directories
        id: changed
        run: |
          git fetch origin ${{ github.base_ref }} --depth=1
          CHANGED_DIRS=$(git diff --name-only origin/${{ github.base_ref }}...HEAD \
            | grep '\.tf$' \
            | xargs -I{} dirname {} \
            | sort -u \
            | tr '\n' ',')
          echo "dirs=${CHANGED_DIRS}" >> $GITHUB_OUTPUT

      # tfsec: catches HIGH/CRITICAL misconfigs, outputs SARIF for Security tab
      - name: Run tfsec
        uses: aquasecurity/tfsec-sarif-action@v0.1.4
        with:
          sarif_file: tfsec.sarif
          minimum_severity: HIGH
          # Exclude downloaded provider modules — only scan authored code
          additional_args: --exclude-downloaded-modules

      - name: Upload tfsec SARIF
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: tfsec.sarif
          category: tfsec

      # Checkov: policy-as-code checks, blocks PR if any HIGH finding
      - name: Run Checkov on changed directories
        id: checkov
        run: |
          pip install checkov==3.2.0 --quiet

          # Build --directory flags from changed dirs output
          DIRS="${{ steps.changed.outputs.dirs }}"
          DIR_FLAGS=$(echo "$DIRS" | tr ',' '\n' | sed 's/^/--directory /' | tr '\n' ' ')

          checkov \
            $DIR_FLAGS \
            --framework terraform \
            --check CKV_AWS_8,CKV_AWS_24,CKV_AWS_57,CKV_AWS_135,CKV_K8S_30 \
            --compact \
            --output cli \
            --output-file-path console \
            --hard-fail-on HIGH  # PR fails on HIGH severity — not advisory
        continue-on-error: false   # INTENTIONAL: must be false to block merge

      # Annotate PR with Checkov findings as inline comments
      - name: Post Checkov summary to PR
        if: failure()
        uses: actions/github-script@v7
        with:
          script: |
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: '❌ **IaC Security Scan failed.** Checkov found HIGH severity issues in Copilot-generated Terraform. Review the Security tab for details before merging.'
            })

One cost note: running Checkov and tfsec as separate parallel jobs doubles your CI minute consumption. Keep them in a single security-scan job running sequentially. On GitHub Actions free tier (2,000 min/month), this matters for active teams.

Fix #3 — Scope Copilot Access with `.copilotignore` and Content Exclusions

This is the hardest fix to get teams to adopt, and also the most important one for production environments. The goal is to surgically remove your highest-risk files from Copilot's context window so it cannot read sensitive infrastructure definitions or echo them back in suggestions.

Create a .copilotignore file at the repo root. Syntax mirrors .gitignore. This prevents Copilot from indexing matched files as context for suggestions. At minimum, include these patterns:

# .copilotignore
# Prevents Copilot from using these files as suggestion context

# Variable files with real values — account IDs, bucket names, ARNs
terraform.tfvars
*.auto.tfvars
*.tfvars.json

# Backend config — contains real S3 bucket names and state key paths
backend.tf
backend.hcl

# Files that may contain sensitive resource definitions
secrets.tf
credentials.tf

# Production environment configs — exclude entirely
envs/prod/**
environments/production/**

For GitHub Copilot Enterprise, go further. Configure Content Exclusions at the organization level under Settings → Copilot → Content exclusion. Use glob patterns like **/prod/**/*.tf to exclude all production Terraform from Copilot context across every repo in the org. Organization-level exclusions override repo-level .copilotignore — org settings win. This is the hard block. The .copilotignore is a soft hint; Content Exclusions are enforced by the platform.

The concrete risk without this: Copilot Chat reads all open editor tabs. Open backend.tf in VS Code, ask Copilot to "generate a staging backend config," and it will echo your production S3 bucket name and state key path directly into the suggestion. Any repo contributor with Copilot access sees that output. In a multi-team org with shared repos, that's a real lateral information exposure path.

Also worth enabling: Copilot audit logs in GitHub Enterprise org settings. Suggestions are not logged by default at the repo level. Turning on audit logs lets you track which suggestions were accepted in IaC files — useful for compliance and incident investigation.

Prevention — Build a Copilot-Aware IaC Module Template

Fixes #1–3 are reactive. This one is proactive. The goal is to make the secure path the default path — so engineers never start from a blank, unconstrained Terraform file where Copilot's bad defaults have nothing to override them.

Create a terraform-module-template repository using GitHub's template repository feature. Every new module repo gets created from this template and inherits the full guardrail stack on day one. The template should include a pre-populated versions.tf with pinned providers, a variables.tf with typed and validated inputs, and a .github/copilot-instructions.md already in place.

The versions.tf file does double duty: it constrains Copilot's context (when the file is open, Copilot inherits the version and provider patterns) and it prevents the context-window-truncation problem by keeping version constraints at the top of every module. Here's what that baseline file looks like, with the RDS resource showing all Copilot-common insecure defaults explicitly overridden:

# terraform-module-template/versions.tf
# Drop this file into every new module repo via GitHub template repository

terraform {
  # Prevents Copilot from generating 0.12-era syntax
  required_version = ">= 1.5.0, < 2.0.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      # Pessimistic constraint — Copilot defaults to unpinned or too-broad ranges
      # Pin minor version to prevent breaking changes from auto-accepted suggestions
      version = "~> 5.40"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.30"
    }
  }
}

# Local that makes destruction intent explicit and auditable
locals {
  # Only staging workspaces allow destroy — production is always protected
  # Copilot will inherit this pattern from context if file is open in editor
  allow_destroy = terraform.workspace == "staging" ? true : false
}

# Example: RDS instance with all Copilot-common insecure defaults overridden
resource "aws_db_instance" "main" {
  identifier        = "${var.env}-${var.app_name}-db"
  engine            = "postgres"
  engine_version    = "16.2"
  instance_class    = var.db_instance_class
  allocated_storage = var.db_storage_gb

  # Copilot default = true — always override explicitly
  publicly_accessible = false

  # Copilot default = false — always override explicitly
  deletion_protection = !local.allow_destroy

  # Copilot frequently omits this block entirely
  storage_encrypted = true
  kms_key_id        = var.kms_key_arn

  lifecycle {
    # Do NOT use ignore_changes = all — that masks drift
    # Only ignore tags to prevent plan noise from tagging automation
    ignore_changes = [tags]
  }
}

Add a pre-commit configuration to the template as well. Using pre-commit framework v3.x with hooks terraform_tfsec, terraform_checkov, terraform_validate, and terraform_fmt catches issues before they ever reach CI. The most common failure mode: terraform_checkov requires checkov >= 2.3.0 and Python >= 3.8 in the local environment. Version mismatch is the number one cause of "hook failed to install" errors on new developer machines. Pin both in your onboarding docs.

One final thing I want to flag: the count = var.enable_feature ? 1 : 0 pattern. Copilot loves suggesting this for optional resources. In Terraform versions below 1.3, toggling this value causes resource replacement — a destroy followed by a create — not an update. We had a Redis cluster get destroyed in staging because of this. Use for_each with a conditional set instead. It's a one-line change and it avoids the replacement behavior entirely. See the Terraform count meta-argument docs for the full explanation of why this happens.

GitHub Copilot Terraform security isn't about disabling the tool — it's about building a system where the tool's suggestions land inside guardrails rather than outside them. The copilot-instructions.md file reduces noise. The CI pipeline enforces policy. The content exclusions protect sensitive context. The module template makes the secure path the default. Stack all four layers and Copilot becomes genuinely useful for IaC work instead of a liability. For more on enforcing infrastructure policy in CI pipelines, see the DevOps_DayS runbook archive for related patterns across AWS and Kubernetes environments.

Self-Hosted Ollama Homelab: 3 Mistakes Running Local LLMs

Oleksandr Kuryzhev — Fri, 26 Jun 2026 07:02:10 +0000

Originally published on kuryzhev.cloud

We thought setting up a self-hosted Ollama homelab for DevOps assistance would take an afternoon. Three OOM crashes, one exposed API endpoint, and a silent CPU fallback later, here's what actually works — and what we'd do differently from day one.

Context: Why We Wanted a Local LLM for Homelab Automation

Our homelab runs a fairly typical self-hosted DevOps stack: Gitea for source control, Drone CI for pipelines, Proxmox for VM management, and a handful of Docker Compose services scattered across two physical hosts. Day-to-day tasks involve writing Ansible playbooks, debugging systemd units, generating Dockerfiles, and occasionally reverse-engineering someone else's Terraform. All of that involves a lot of back-and-forth with language models.

The problem with cloud-based LLM APIs — ChatGPT, Claude, Gemini — is that they require pasting real infrastructure configs into a browser. Hostnames, internal IP ranges, secrets that slipped into a playbook comment, service account names. None of that should leave the network. We also have air-gapped lab segments that physically can't reach the internet, and we wanted consistent tooling across both environments.

We chose Ollama for a few concrete reasons: it ships as a single binary, it manages model downloads via a clean CLI, and it exposes an OpenAI-compatible REST API on port 11434 out of the box. That last point matters — it means any tool that supports the OpenAI API (Continue.dev in VS Code, shell scripts using curl, custom Python utilities) works against Ollama without modification. The plan was straightforward. The execution was not.

Mistake 1: Assuming Any GPU Would "Just Work" with Ollama

The first host we tried had an NVIDIA GTX 1060 3GB installed. We ran ollama run llama3, watched it load, and immediately noticed something was wrong. Inference was crawling at roughly 2 tokens per second. We expected 25 or more. No error. No warning. The model just loaded silently into system RAM and ran entirely on CPU.

We burned two hours checking the wrong things — model quantization, RAM speed, thermal throttling — before running ollama ps and seeing 100% CPU next to the loaded model. Then we checked the logs with OLLAMA_DEBUG=1 ollama serve and found the real message:

msg="no GPU detected"

The GTX 1060 was visible to nvidia-smi. The driver was loaded. But Ollama requires CUDA 11.8 or higher, and that host was running CUDA 11.4. The mismatch was silent from the client's perspective — Ollama simply fell back to CPU without surfacing any error to the caller.

Watch out for this: nvidia-smi showing your GPU does not mean Ollama will use it. Always verify with OLLAMA_DEBUG=1 and look for offload layers: 32/32 in the output. That line confirms full GPU acceleration. Anything less means partial or zero offload.

The fix required three steps: updating the CUDA toolkit to 11.8, pinning nvidia-container-toolkit to version 1.14.3 (earlier versions had inconsistent behavior with Docker runtime detection), and ensuring /etc/docker/daemon.json had "default-runtime": "nvidia" set. After that, the same model ran at 28 tokens per second on the same hardware. Same model, same host, completely different experience.

We also verified the CUDA version properly this time:

# Check CUDA version — must be >= 11.8 for Ollama GPU support
nvcc --version

# Confirm Docker is using the nvidia runtime
docker info | grep -i runtime

# Verify GPU offload after starting Ollama with debug logging
OLLAMA_DEBUG=1 ollama serve 2>&1 | grep -i "offload layers"

Mistake 2: Running Ollama Directly on the Host Without Resource Limits

Once GPU acceleration was working, we got greedy and pulled a larger model. codellama:13b seemed like the right tool for generating longer Ansible roles and reviewing Dockerfiles with more context. We pulled it, ran it, and about forty seconds into the first inference request, everything went sideways.

The OOM killer fired. It took down our Gitea container mid-push. A developer on the team lost a commit they hadn't pushed yet. The codellama:13b model in Q4_K_M quantization needs roughly 8.5GB of VRAM — and on a 16GB host already running a dozen services, the model load spiked total RAM consumption to 14GB instantaneously. The kernel picked Gitea as the most expendable process. It wasn't.

The root cause was straightforward: we were running Ollama directly on the host with no cgroup constraints, no memory limits, and no model eviction policy. In Ollama v0.1.38 and later, OLLAMA_MAX_LOADED_MODELS controls how many models stay resident in memory simultaneously. In earlier versions, there's no eviction control at all. We hadn't set either OLLAMA_MAX_LOADED_MODELS or OLLAMA_NUM_PARALLEL, so Ollama defaulted to greedy behavior — loading whatever was asked, keeping it resident, and accepting parallel requests that multiplied memory pressure.

Watch out for this: OLLAMA_NUM_PARALLEL defaults to 4 in recent versions. On a constrained homelab host, four simultaneous inference requests against a 7B model will absolutely OOM your other services. Set it to 1 unless you have dedicated hardware.

The correct approach was to containerize Ollama with explicit resource caps and isolate it from critical services on a dedicated Docker network. We moved everything into Docker Compose, set mem_limit: 10g, and added environment variables to enforce single-model, single-request behavior.

Mistake 3: Exposing the Ollama API Without Authentication

This one is embarrassing to admit. The default ollama serve binds to 127.0.0.1, which is fine for bare-metal installs. But our Docker Compose setup used ports: "11434:11434" without thinking through what that means on a flat home network. It published the port to every interface on the host — which sits on the same physical switch as our IoT VLAN due to a pfSense VLAN rule we'd misconfigured months earlier.

Ollama has no built-in authentication as of v0.1.x. Anyone who could reach port 11434 could call GET /api/tags to enumerate loaded models, send inference requests, or — and this is the one that actually worried us — trigger a POST /api/pull to download a multi-gigabyte model and exhaust disk space. There's no rate limiting either. A single unauthenticated pull request from a compromised IoT device could fill a volume.

We also had OLLAMA_HOST=0.0.0.0 set inside the container, which is necessary for the container's internal binding but easy to misread as "only affects the container namespace." It doesn't change the fact that the Docker port mapping exposes it externally.

The fix was an Nginx reverse proxy with HTTP Basic Auth sitting in front of Ollama, combined with binding the proxy's published port to 127.0.0.1:8080 on the host. Remote access goes through Tailscale, which restricts it to trusted nodes only. For the health check endpoint used by monitoring tools, we carved out an auth-exempt location /api/tags block in Nginx.

What We Do Differently Now: A Stable, Reproducible Ollama Stack

The current setup avoids all three mistakes. Here's the full Docker Compose configuration we run in production on the homelab. It uses a pinned Ollama image, GPU passthrough via the nvidia runtime, hard memory caps, and an Nginx auth proxy — all wired together with a named volume for model persistence.

This Docker Compose file defines the full hardened stack. Read the inline comments carefully — several of the environment variables are non-obvious in their effect:

# docker-compose.yml — Production-hardened Ollama stack for homelab
# Tested with Ollama v0.1.38, Docker 24.x, nvidia-container-toolkit 1.14.3

version: "3.9"

services:
  ollama:
    image: ollama/ollama:0.1.38          # pinned — avoid surprise API changes
    container_name: ollama
    restart: unless-stopped
    runtime: nvidia                       # requires nvidia-container-toolkit
    environment:
      - OLLAMA_HOST=0.0.0.0              # bind inside container; proxy handles external auth
      - OLLAMA_NUM_PARALLEL=1            # prevent concurrent request memory spikes
      - OLLAMA_MAX_LOADED_MODELS=1       # evict previous model before loading new one
      - OLLAMA_DEBUG=0                   # set to 1 temporarily to verify GPU offload layers
    volumes:
      - ollama_models:/root/.ollama      # persist models across container rebuilds
    networks:
      - ollama_internal                  # isolated from critical services network
    deploy:
      resources:
        limits:
          memory: 10g                    # hard cap — prevents OOM-killing neighbors
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s                  # model server takes ~15s to initialize

  ollama-proxy:
    image: nginx:1.25-alpine
    container_name: ollama-proxy
    restart: unless-stopped
    ports:
      - "127.0.0.1:8080:80"             # bind to loopback only; use Tailscale for remote access
    volumes:
      - ./nginx/ollama.conf:/etc/nginx/conf.d/default.conf:ro
      - ./nginx/.htpasswd:/etc/nginx/.htpasswd:ro
    networks:
      - ollama_internal
    depends_on:
      ollama:
        condition: service_healthy

volumes:
  ollama_models:
    driver: local

networks:
  ollama_internal:
    driver: bridge
    internal: false                      # set true if Ollama should have no outbound internet

The Nginx config below handles streaming inference correctly. The two most common mistakes people make here are forgetting proxy_buffering off (which causes the response to appear all at once instead of streaming) and leaving proxy_read_timeout at the default 60 seconds, which causes 504 errors mid-generation on longer prompts:

# nginx/ollama.conf — Reverse proxy with Basic Auth for Ollama API
# Generate .htpasswd: htpasswd -c nginx/.htpasswd devops
# Set proxy_read_timeout high — streaming inference takes time

server {
    listen 80;
    server_name _;

    # Basic auth — replace with mTLS for higher-security environments
    auth_basic           "Ollama API";
    auth_basic_user_file /etc/nginx/.htpasswd;

    location / {
        proxy_pass         http://ollama:11434;
        proxy_http_version 1.1;

        # Critical: streaming responses require these headers
        proxy_set_header   Connection '';
        proxy_buffering    off;
        proxy_cache        off;

        # Increase timeouts — default 60s causes 504 on long generations
        proxy_read_timeout    300s;
        proxy_connect_timeout 10s;
        proxy_send_timeout    300s;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    # Health endpoint — exempt from auth for monitoring tools
    location /api/tags {
        auth_basic off;
        proxy_pass http://ollama:11434/api/tags;
    }
}

For model selection, we settled on two workhorses that fit within 6GB VRAM with quantization: codellama:7b in Q4_K_M for quick completions and inline code suggestions, and mistral:7b-instruct-v0.2-q4_K_M for longer reasoning tasks like reviewing a Terraform plan or explaining a failing systemd unit. The 13B models sound appealing but they're genuinely impractical on homelab hardware unless you have a dedicated GPU with 10GB+ VRAM.

We also pin models using digest references in our Modelfiles — FROM llama3:8b-instruct@sha256:abc123... — so a weekly ollama pull cron job doesn't silently swap out a model we've tuned prompts for. Model versioning matters more than most people realize when you're building automation on top of LLM output.

For observability, Ollama has no native Prometheus endpoint, so we rely on ollama ps and ollama list in a lightweight cron-based check, plus the Docker healthcheck defined in the Compose file. It's not elegant, but it catches the cases that actually happen: the model server failing to start, or a model getting stuck in a loading state. More details on our homelab monitoring approach are over at kuryzhev.cloud.

Running a self-hosted Ollama homelab is genuinely useful once the stack is stable. The self-hosted Ollama homelab we have now handles dozens of DevOps assistance requests per day with zero data leaving the network, predictable resource usage, and a setup that survives container restarts and host reboots without manual intervention. Getting there required making all three of these mistakes first — hopefully this saves you the OOM crashes.

Automate Kubernetes RBAC Audits with Python and a Custom Controller

Oleksandr Kuryzhev — Thu, 25 Jun 2026 07:02:40 +0000

Originally published on kuryzhev.cloud

The Night Our Cluster Opened Its Doors to Everyone

A single kubectl apply from a well-meaning junior developer bound cluster-admin to the CI service account — and our Kubernetes cluster stayed wide open for 11 days before anyone noticed. No alert fired. No pipeline failed. No one said a word. The misconfiguration was discovered only because a compliance review happened to trigger a manual RBAC audit, completely unrelated to the incident itself. That audit was the only reason we found it at all.

The setup at the time was a multi-tenant cluster with 40-plus namespaces, shared by six product teams. The developer had been debugging a Helm deployment failure and needed broader permissions temporarily. They added the binding, fixed the issue, and forgot to clean it up. Completely understandable. Completely dangerous. For those 11 days, any workload running under that CI service account — or anything that could impersonate it — had full cluster-admin rights. In a cluster that hosts a payments namespace and a production environment for a regulated service, that is not a theoretical risk. That is an open door.

The real lesson was not about the developer. It was about us. We had no automated Kubernetes RBAC compliance automation in place. Manual review of RBAC objects at scale is structurally impossible to do reliably. With 40+ namespaces, hundreds of RoleBindings, and teams deploying multiple times per day, the gap between "what we think the permissions look like" and "what they actually are" grows every single day. This article is about the Python controller we built to close that gap permanently.

Why RBAC Drift Happens Faster Than You Can Review It

Kubernetes RBAC has a design property that most people do not think about until it bites them: it is additive by default. When you run kubectl apply on a RoleBinding, it never removes excess permissions that were added out-of-band. Helm chart upgrades can silently widen permission scopes between versions, and nobody notices because the upgrade succeeds and the app works. The permissions just quietly expand.

We identified three root causes in our postmortem. First, there was no enforcement at admission time. A developer could bind any role to any subject and the API server would accept it without question. Second, we had no diff-based alerting on RBAC objects — changes to ClusterRoleBindings were not treated as security-relevant events, so they never triggered anything. Third, and most critically, ClusterRoleBindings live outside namespace-scoped review cycles. Most teams think about permissions at the namespace level. ClusterRoleBindings are invisible in that mental model.

The two most dangerous patterns we see in the wild are the system:masters group and wildcard verb rules (verbs: ["*"]). Both are completely legal YAML. Both pass kubectl apply without any warning. Wildcard verbs are particularly insidious because they survive role changes — if you later add a new resource type to the API server, anything with wildcard verbs automatically gets access to it. No one had to make a decision. The permission just appeared.

There is also the system:anonymous and system:unauthenticated subject problem. Before hardening, default GKE clusters can have bindings to these subjects. If you have never audited for them, run this one-liner right now before reading further:

kubectl get clusterrolebindings -o json \
  | jq '[.items[] | select(.subjects[]?.name == "system:anonymous")]'

If that returns anything other than an empty array, stop and fix it. Then come back and build the controller.

Building the Audit Engine — A Python Controller That Catches What Humans Miss

The controller we built uses the kubernetes Python client at version 28.1.0, which maps to the Kubernetes 1.28 API. One critical gotcha here: mismatched client versions cause silent deserialization errors on newer CRD fields. Pin the version and do not let dependabot auto-upgrade it without a test run against your cluster version first.

The architecture is straightforward. A watch loop runs against ClusterRoleBinding, RoleBinding, and ClusterRole resources. Policy rules live in a ConfigMap at kube-system/rbac-audit-policy. When a violation is detected, the controller writes a RBACViolation custom resource under the CRD group audit.kuryzhev.cloud/v1alpha1 and exposes Prometheus metrics on port 8080. Here is the full controller implementation:

# rbac_auditor_controller.py
# Requires: kubernetes==28.1.0, prometheus_client==0.20.0
# Deploy as a Deployment in kube-system with a dedicated ServiceAccount

import time
import yaml
import logging
from kubernetes import client, config, watch
from kubernetes.client.exceptions import ApiException
from prometheus_client import start_http_server, Counter, Gauge

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("rbac-auditor")

# --- Prometheus metrics ---
VIOLATIONS_TOTAL = Counter(
    "rbac_violations_total",
    "Total RBAC violations detected",
    ["namespace", "kind", "name", "rule"]
)
LAST_AUDIT_TS = Gauge("rbac_last_audit_timestamp", "Unix timestamp of last audit cycle")
BINDINGS_SCANNED = Counter("rbac_bindings_scanned_total", "Total bindings evaluated")

# --- Load policy from ConfigMap ---
def load_policy(core_v1: client.CoreV1Api) -> dict:
    cm = core_v1.read_namespaced_config_map("rbac-audit-policy", "kube-system")
    return yaml.safe_load(cm.data["policy.yaml"])

# --- Core violation checks ---
def check_binding(binding, policy: dict) -> list[str]:
    violations = []
    subjects = binding.subjects or []

    # Rule 1: Forbidden subjects (e.g. system:anonymous)
    forbidden_subjects = policy.get("forbidden_subjects", [])
    for subject in subjects:
        if subject.name in forbidden_subjects:
            violations.append(f"forbidden_subject:{subject.name}")

    # Rule 2: Binding references a forbidden ClusterRole by name
    role_ref = binding.role_ref
    if role_ref.name in policy.get("forbidden_roles", []):
        violations.append(f"forbidden_role_ref:{role_ref.name}")

    return violations

def check_clusterrole(role: client.V1ClusterRole, policy: dict) -> list[str]:
    violations = []
    for rule in (role.rules or []):
        verbs = rule.verbs or []
        # Flag wildcard verbs — most dangerous pattern in any RBAC config
        if "*" in verbs:
            violations.append("wildcard_verb")
        # Flag full explicit verb set — functionally identical to wildcard
        full_verbs = {"get", "list", "watch", "create", "update", "patch", "delete"}
        if full_verbs.issubset(set(verbs)):
            violations.append("full_verb_set_equivalent_to_wildcard")
    return violations

# --- Write RBACViolation CRD ---
def write_violation_crd(custom_api: client.CustomObjectsApi, name: str,
                         namespace: str, kind: str, rules: list[str]):
    body = {
        "apiVersion": "audit.kuryzhev.cloud/v1alpha1",
        "kind": "RBACViolation",
        "metadata": {
            # Truncate to 63 chars — Kubernetes name length limit
            "name": f"{kind.lower()}-{name}".replace(":", "-")[:63],
            "namespace": namespace or "cluster-scoped",
        },
        "spec": {
            "sourceKind": kind,
            "sourceName": name,
            "sourceNamespace": namespace,
            "violatedRules": rules,
        }
    }
    try:
        custom_api.create_namespaced_custom_object(
            group="audit.kuryzhev.cloud", version="v1alpha1",
            namespace="kube-system", plural="rbacviolations", body=body
        )
        logger.info(f"Created RBACViolation for {kind}/{name}: {rules}")
    except ApiException as e:
        if e.status == 409:
            # Already exists from a previous cycle — not an error
            logger.debug(f"RBACViolation for {kind}/{name} already exists, skipping")
        else:
            raise

# --- Main watch loop ---
def run_audit_loop():
    config.load_incluster_config()  # Use load_kube_config() for local dev
    rbac_v1 = client.RbacAuthorizationV1Api()
    core_v1 = client.CoreV1Api()
    custom_api = client.CustomObjectsApi()

    start_http_server(8080)  # Expose /metrics for Prometheus scraping
    logger.info("RBAC Auditor started, metrics on :8080")

    while True:
        try:
            policy = load_policy(core_v1)
            # Reset resource_version to "0" on each cycle — prevents 410 Gone
            # after etcd compaction, which would silently stop the watch loop
            resource_version = "0"

            # Audit ClusterRoleBindings — most dangerous scope, audit first
            crbs = rbac_v1.list_cluster_role_binding()
            for crb in crbs.items:
                BINDINGS_SCANNED.inc()
                violations = check_binding(crb, policy)
                if violations:
                    for v in violations:
                        VIOLATIONS_TOTAL.labels(
                            namespace="cluster-scoped",
                            kind="ClusterRoleBinding",
                            name=crb.metadata.name,
                            rule=v
                        ).inc()
                    write_violation_crd(custom_api, crb.metadata.name,
                                        None, "ClusterRoleBinding", violations)

            # Audit ClusterRoles for wildcard verbs
            crs = rbac_v1.list_cluster_role()
            for cr in crs.items:
                violations = check_clusterrole(cr, policy)
                if violations:
                    for v in violations:
                        VIOLATIONS_TOTAL.labels(
                            namespace="cluster-scoped",
                            kind="ClusterRole",
                            name=cr.metadata.name,
                            rule=v
                        ).inc()
                    write_violation_crd(custom_api, cr.metadata.name,
                                        None, "ClusterRole", violations)

            LAST_AUDIT_TS.set(time.time())
            logger.info("Audit cycle complete. Sleeping 300s.")
            # resync_period=300 — informer cache pattern, avoids hammering API server
            # Running full re-list every 30s on 500+ bindings adds ~120ms latency per cycle
            time.sleep(300)

        except ApiException as e:
            logger.error(f"API error during audit: {e.status} {e.reason}")
            time.sleep(30)

if __name__ == "__main__":
    run_audit_loop()

Watch out for the 410 Gone response from the API server. This happens after etcd compaction and it silently kills your watch loop if you do not handle it. The fix is to reset resource_version="0" at the start of each audit cycle, which forces a full re-list rather than trying to resume a stale watch. We learned this the hard way when the controller appeared healthy but had stopped receiving events for six hours.

The other gotcha: do not grant the auditor service account cluster-admin just because it needs to read RBAC objects. I see this in tutorials constantly and it makes me cringe every time. The auditor only needs get, list, watch on RBAC resources. Here is the full deployment manifest with correctly scoped permissions:

# rbac-auditor-rbac.yaml
# Apply with: kubectl apply -f deploy/rbac-auditor-rbac.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: rbac-audit-policy
  namespace: kube-system
data:
  policy.yaml: |
    forbidden_subjects:
      - "system:anonymous"
      - "system:unauthenticated"
    forbidden_roles:
      - "cluster-admin"
    # Namespaces where ClusterRoleBinding is never expected
    restricted_namespaces:
      - "production"
      - "payments"

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: rbac-auditor
  namespace: kube-system
automountServiceAccountToken: true  # Required for in-cluster config

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: rbac-auditor-reader
rules:
  # Minimal read-only access — do NOT grant cluster-admin here (common mistake #1)
  - apiGroups: ["rbac.authorization.k8s.io"]
    resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get"]
  # Write access scoped only to our own CRD in kube-system
  - apiGroups: ["audit.kuryzhev.cloud"]
    resources: ["rbacviolations"]
    verbs: ["get", "list", "create", "update", "patch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: rbac-auditor-binding
  annotations:
    # Flag for 90-day rotation review — the controller itself will enforce this
    audit.kuryzhev.cloud/reviewed-at: "2024-11-01"
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: rbac-auditor-reader
subjects:
  - kind: ServiceAccount
    name: rbac-auditor
    namespace: kube-system

If the auditor pod starts and you immediately see Forbidden: User "system:serviceaccount:kube-system:rbac-auditor" cannot list resource "clusterrolebindings" in the logs, the ClusterRoleBinding was not applied correctly. Check namespace alignment first — that accounts for 80% of these failures.

Wiring It Into Your Pipeline — From Detection to Remediation

Detection without action is just expensive logging. The controller exposes three Prometheus metrics: rbac_violations_total as a counter labeled by namespace, kind, name, and rule; rbac_last_audit_timestamp as a gauge so you can alert if the auditor itself stops running; and rbac_bindings_scanned_total to track coverage. The alerting rule we use fires when rbac_violations_total exceeds 0 for more than 15 minutes, giving the system one cycle to self-resolve before paging someone.

In our Flux setup, annotated bindings trigger a webhook that opens a GitHub issue with the offending manifest diff and the specific policy rule that was violated. The developer who owns the namespace gets assigned automatically based on a CODEOWNERS-style mapping. This is important: the goal is not to block deployments automatically everywhere. The goal is to make the violation visible and attributed within minutes, not days.

Auto-remediation is opt-in and deliberately scoped. The controller will only delete a RoleBinding if it carries the label audit.kuryzhev.cloud/auto-remediate: "true". We never touch unlabeled production bindings automatically. I made the mistake of enabling aggressive auto-remediation in a previous job and watched it delete a legitimate emergency access binding during an incident. Never again. Humans should approve remediation for anything in production scope.

For CI pipeline integration, we run rbac-police at version 0.3.2 as a gate against rendered Helm manifests before any cluster deployment. The command is straightforward:

# Run in CI before helm upgrade — fails pipeline on any HIGH severity finding
rbac-police eval --severity HIGH --fail-on-findings ./manifests/

This catches the obvious mistakes before they ever reach the cluster. The in-cluster controller then handles drift that happens through out-of-band changes, kubectl one-liners during incidents, and Helm upgrades that quietly expand scopes. You need both layers.

You can find more Kubernetes security and automation patterns at kuryzhev.cloud — we cover real production scenarios, not just happy-path documentation.

Prevention Checklist — Stop the Drift Before It Starts

After running this system in production for several months, here is the checklist we now enforce on every cluster. Each item has a reason. Skip none of them.

1. Deploy OPA Gatekeeper with wildcard verb constraints at admission time. The K8sNoWildcardVerbs and K8sRestrictClusterAdmin constraint templates from the open-policy-agent/gatekeeper-library block the binding at kubectl apply time, before it ever reaches etcd. This is your first line of defense. The in-cluster controller is your second. You want both.

2. Run rbac-police eval in every CI pipeline that renders Helm charts. Use --severity HIGH --fail-on-findings. Do not let it run in warn-only mode — warn-only modes get ignored within two weeks of deployment. If it finds something, the pipeline fails. Period.

3. Add a 90-day review annotation to every ClusterRoleBinding at creation time. The annotation key is audit.kuryzhev.cloud/reviewed-at. The controller flags any binding where this annotation is missing or older than 90 days. This forces a human to periodically confirm that a binding is still needed. Most "temporary" bindings survive indefinitely because no one ever looks at them again.

4. Never watch only RoleBindings and ignore ClusterRoleBindings. This is the single most common mistake I see in home-grown audit scripts. The most dangerous over-permissioned bindings are almost always cluster-scoped. If your audit does not cover ClusterRoleBindings, it covers maybe 40% of the actual risk surface.

5. Protect the RBACViolation CRD itself with RBAC. If developers can delete RBACViolation objects, they can silently clear audit findings without fixing the underlying binding. The CRD should be read-only for all non-auditor service accounts. We found one developer had done exactly this in a dev cluster to make the noise go away before a demo. The binding stayed. The violation disappeared. That is worse than having no audit at all.

6. Set automountServiceAccountToken: false on all pods except the auditor itself. Most workloads do not need API server access. Defaulting to no token mount reduces the blast radius of any container escape significantly.

7. Alert on rbac_last_audit_timestamp staleness, not just on violations. If the auditor itself goes down — OOM kill, eviction, image pull failure — you want to know immediately. A silent auditor is worse than no auditor because it creates false confidence. We alert if the timestamp is more than 10 minutes old.

Kubernetes RBAC compliance automation is not a one-time project. It is a continuous process. The controller we built did not eliminate human judgment — it made human judgment possible again by reducing the signal-to-noise ratio to something manageable. We went from "we audit RBAC quarterly if we remember" to "we know about every violation within five minutes." That 11-day window closed to under one audit cycle. That is the outcome that matters.

DEV Community: Oleksandr Kuryzhev

ML Anomaly Detection in Loki Logs: Per-Entity Isolation Forest

Terraform Drift Detection in CI: Alert-Only vs Auto-Remediate

When You Face This Choice

Option A — Scheduled Detect-Only Pipeline (Plan + Notify)

Option B — Automated Detect-and-Remediate Pipeline (Plan + Gated Auto-Apply)

Decision Matrix

My Pick

Related

Building an Online Anomaly Scoring Sidecar for Grafana Alerts

What this actually does

How people use it wrong

The correct approach

Advanced patterns

Performance notes

Related

Grafana Alerting Checklist: Wiring AI Anomaly Scores Correctly

Why This Checklist

The Checklist

Commonly Missed Items

Automation Ideas

Related

Ansible Dynamic Inventory for EC2 Fleets: 7 Tuning Tips

Cache Your Dynamic Inventory or Pay for It in API Throttling

Use keyed_groups Instead of Manual Group Definitions

Filter at the API Level, Not in Playbook Logic

Patch in Batches with serial and max_fail_percentage

Don't Hardcode Credentials — Use Instance Roles for Inventory Lookups

Verify Inventory Before You Patch — ansible-inventory --graph Is Your Sanity Check

Handle Fleet Churn: Compose ansible_host and Skip Terminating Instances

Related

Cloudflare Workers Rate Limiting: 3 KV Mistakes We Made

Context: Why We Reached for Workers + KV in the First Place

Mistake 1: We Treated KV as Strongly Consistent

Mistake 2: We Ignored KV's Write Limits and Cost Model

Mistake 3: We Used Raw API Keys as KV Keys

What We Do Differently Now

Related

Automate AWS Cost Reports with Python, boto3, and SES

Symptoms: Your AWS Bill Surprises You Every Month

Root Cause: Manual Cost Visibility Doesn't Scale

Fix #1: Activate Cost Explorer API and Query It Correctly with boto3

Fix #2: Parse the Response and Build a Readable Report Structure

Fix #3: Send the Report via SES with HTML Formatting

Prevention: Make Cost Reporting a Reliable, Auditable System

Related

Makefile Patterns for Terraform and Infra Repos That Actually Scale

What Make Actually Does in an Infra Repo

How People Use Makefile Infra Repos Wrong

The Correct Approach — Structured Makefile for Infra Repos

Advanced Patterns for Real Infrastructure Workflows

Performance Notes and Security Considerations

Related

Query AWS Runbooks in Plain English with Bedrock Knowledge Bases

The Night Our On-Call Engineer Couldn't Find the Runbook

Why Operational Docs Become a Liability Over Time

Building the Knowledge Base with Terraform — The Fix

Wiring It Into Your Incident Workflow

Prevention Checklist — Keeping the Knowledge Base Trustworthy

Related

Fix GitHub Copilot Terraform Security Risks Before They Hit Prod

Symptoms

Root Cause: Why Copilot Gets IaC Wrong

Fix #1 — Add a .github/copilot-instructions.md Guardrail File

Fix #2 — Block Dangerous Patterns in CI with tfsec and Checkov

Fix #3 — Scope Copilot Access with .copilotignore and Content Exclusions

Prevention — Build a Copilot-Aware IaC Module Template

Related

Self-Hosted Ollama Homelab: 3 Mistakes Running Local LLMs

Context: Why We Wanted a Local LLM for Homelab Automation

Mistake 1: Assuming Any GPU Would "Just Work" with Ollama

Mistake 2: Running Ollama Directly on the Host Without Resource Limits

Mistake 3: Exposing the Ollama API Without Authentication

What We Do Differently Now: A Stable, Reproducible Ollama Stack

Related

Automate Kubernetes RBAC Audits with Python and a Custom Controller

The Night Our Cluster Opened Its Doors to Everyone

Why RBAC Drift Happens Faster Than You Can Review It

Building the Audit Engine — A Python Controller That Catches What Humans Miss

Wiring It Into Your Pipeline — From Detection to Remediation

Fix #1 — Add a `.github/copilot-instructions.md` Guardrail File

Fix #3 — Scope Copilot Access with `.copilotignore` and Content Exclusions