DEV Community: Muskan _zop

AWS Budget Alerts vs Enforcement: Why You Still Can't Cap a Cloud Bill, and What to Do Instead

Muskan _zop — Thu, 03 Sep 2026 08:50:28 +0000

Quick Answer (TL;DR)

There is no native way to hard-cap an AWS bill at a dollar amount, and after a decade of loudly upvoted requests there probably won't be: a true cap means AWS choosing which of your resources to kill mid-month, which converts a billing problem into an outage generator. What exists is alerting (AWS Budgets: actual and forecasted thresholds) and coarse restriction (budget actions: apply a deny policy or stop tagged EC2/RDS when a threshold trips). The working strategy is an alert ladder wired into Slack or Teams where people actually look, budget actions on sandbox accounts only, per-team budget ownership with daily projections, and true fail-closed ceilings only where the architecture allows a gate in front of the spend, which today means AI usage behind budgeted gateway keys, not general cloud usage.

Why this happens

Cloud billing is post-hoc metering: resources run, meters tick, the bill arrives. A hard cap would require the provider to act on your infrastructure the moment a number is crossed: kill the database mid-transaction? Drop the load balancer during the traffic spike that is probably the reason spend rose? Every answer breaks something for someone, so providers ship alerts instead and leave enforcement to you. The result is the trap most teams live in: alerts configured once, delivered to an inbox nobody reads, discovered to have fired three weeks ago during the invoice postmortem. The fix is not wishing for the cap; it's treating alerting as a delivery problem and enforcement as an architecture problem, separately.

Fix #1: The alert ladder, delivered where people look

A budget alert that lands in email is a log line; one that lands in the team's Slack channel is an interruption. Build the ladder per account or per team, not one org-wide number:

aws budgets create-budget --account-id 111122223333 \
  --budget '{"BudgetName":"team-platform-monthly","BudgetLimit":{"Amount":"20000","Unit":"USD"},"TimeUnit":"MONTHLY","BudgetType":"COST"}' \
  --notifications-with-subscribers '[{"Notification":{"NotificationType":"FORECASTED","ComparisonOperator":"GREATER_THAN","Threshold":100},"Subscribers":[{"SubscriptionType":"SNS","Address":"arn:aws:sns:us-east-1:111122223333:budget-alerts"}]}]'

Route the SNS topic to Slack (AWS Chatbot is the no-code path; a small Lambda webhook if you want formatting). The ladder that works: a pacing threshold on actual spend (80%), an act-now threshold on forecasted spend (100%, the one that fires weeks before the money is gone), and a breach record at 100% actual. Forecast quality matters more than people expect: naive projections misfire on weekly rhythms, so a forecast that understands day-of-week seasonality pages you for real trajectory changes instead of every Monday.

This alerting layer is exactly where ZopNight's budgets sit: budgets for whole cloud accounts, teams, and resource groups with month-to-date spend and a daily projection, thresholds at 80, 95, and 100 percent, forecasts that account for day-of-week seasonality, weekly summary emails, and delivery through its Slack app and Microsoft Teams Adaptive Cards with per-alert selection (docs). Worth noting because it's the honest version of this whole post: its own documentation is explicit that budgets track spend rather than cap it; the only true hard ceiling in the product is on AI spend, where budgeted virtual keys can fail closed. A vendor that promises to "cap your AWS bill" is describing something the platform doesn't offer anyone.

Fix #2: Budget actions, the closest native thing to enforcement

AWS Budgets can attach actions to a threshold: apply a restrictive IAM or SCP policy (deny new resource creation), or stop tagged EC2 and RDS instances. This is real enforcement, and it's deliberately blunt: policies don't un-run what's running, stops are limited to two service families, and evaluation runs a few times a day, so a fast leak outruns it. The operating rule: actions belong on sandbox and dev accounts, where "everything stopped at 100%" is a shrug, and never on production, where the same event is an outage you scheduled for yourself. On sandboxes they're excellent: a hard boundary that teaches budget awareness with a blast radius you chose.

Fix #3: The true-cap edge case: gate the spend before it happens

A fail-closed dollar ceiling is only possible where a gate can sit in front of the spend and reject requests. General cloud usage has no such gate (the "gate" would be your production traffic). But some spend categories do: AI usage is the clean case, because every model call already flows through an API key, so a gateway that issues per-team keys with hard USD budgets can genuinely stop spend at a number, failing the request instead of billing it. Rate-bounding quotas (TPM/RPM, service limits) are the blunter cousin: they cap the worst-case burn rate, which bounds a leak's damage per day even though they never speak dollars. And account isolation is the structural version: one team's runaway spend confined to an account you can, in the worst case, suspend without touching anyone else.

How to prevent this

One budget per team or account with an owner, not one org number nobody owns; the 80% pacing alert should land in the owning team's channel.
Forecast alerts armed everywhere (they need weeks of history, so set them before you need them).
Deliver to chat, not email: Slack or Teams via SNS/Chatbot or your tooling; alert fatigue is a routing problem before it's a threshold problem.
Rehearse the breach: when 100% forecasted fires, who looks, within what SLA, with what authority to act? An alert without a runbook is a notification, not a control.
Review breaches monthly: repeated 80% pacing alerts on the same team is a budget-sizing conversation, not an alerting success.

FAQ

Can I set a hard spending limit on an AWS account?

No native one exists. Budgets alert (actual and forecasted), budget actions can restrict IAM or stop tagged EC2/RDS at a threshold, and quotas cap request rates, but nothing stops the meter at a dollar figure. The famous decade-old feature request stays open because a true cap means AWS breaking your workloads for you mid-month.

What can AWS budget actions actually do?

Three things when a threshold trips: apply an IAM policy, apply an SCP (both typically deny-new-creation), or stop EC2 and RDS instances carrying a target tag. Evaluation lags spend by hours and coverage is narrow, so treat actions as sandbox guardrails rather than production enforcement.

How do I send AWS budget alerts to Slack?

Point the budget's notification at an SNS topic, then connect the topic to Slack through AWS Chatbot (console setup, no code) or a small Lambda posting to a webhook. Route per-team budgets to per-team channels; a shared #billing channel everyone mutes recreates the inbox problem with extra steps.

Is AWS Budgets free?

Budgets with alerts are effectively free at typical scale; budgets with attached actions bill a small daily fee per action-enabled budget after a free allowance (as of early 2026; confirm on the pricing page). The real cost is configuration debt: budgets copied from last year with thresholds nobody revisits.

What's the difference between a billing alarm and a budget?

CloudWatch billing alarms watch one total-estimated-charges metric with static thresholds, the 2012-era mechanism. Budgets add forecasting, per-service and per-tag scoping, multiple thresholds, and actions. New setups should use Budgets (plus anomaly detection for shape changes, which neither alarms nor budgets catch).

Related guides

Recommendations Only Save Money When Someone Acts: Turning Cloud Waste Findings into Jira Tickets

Muskan _zop — Thu, 03 Sep 2026 08:49:47 +0000

Quick Answer (TL;DR)

Cloud cost recommendations don't save money; acted-on recommendations do, and in most engineering organizations "acting" means a ticket in the backlog with an owner and a sprint. The bridge from findings to tickets has three load-bearing requirements: each ticket carries the resource, the evidence, the monthly dollar figure, and the suggested fix (so it's actionable without opening another tool), creation is deduplicated by a stable fingerprint (so tomorrow's re-scan doesn't file the same idle database again), and status syncs both ways (closing the ticket resolves the finding, resolving the finding closes the ticket). Skip any of the three and the integration gets disabled within a month, which is the real reason most findings still die in dashboards.

Why this happens

FinOps tooling optimizes for visibility: dashboards, scores, monthly totals. Engineering work happens somewhere else entirely: the backlog, the sprint board, the definition of done. A finding that never crosses that gap is a suggestion, and suggestions lose to roadmap work every single time, not because teams don't care but because unowned work doesn't exist in an engineering org. The naive fix (auto-create a ticket per finding) fails in the opposite direction: the first automation run files three hundred tickets, the second run files three hundred duplicates, the team lead turns the integration off, and the organization learns "we tried that". The craft is entirely in the middle: fewer, richer, deduplicated tickets that behave like work.

Fix #1: The manual bridge, done properly

Before any automation, a weekly 30-minute triage beats most tooling: sort open findings by monthly savings, take the top handful, and file each as a ticket that can be executed without opening the cost tool:

Title: [waste] Idle RDS db-analytics-legacy: $212/month, 0 connections in 21 days
Body:  resource ARN and account
       evidence: DatabaseConnections max = 0, 21-day window
       monthly cost and annual equivalent
       suggested action: snapshot, stop, delete after 30 quiet days
       rollback: restore from final snapshot
Assignee: owning team (from tags)   Label: cloud-waste   Due: within 2 sprints

The dollar figure in the title is not decoration: it's what lets a team lead rank the ticket against feature work, and what lets you total "closed savings" at the end of the quarter, which is the only FinOps metric leadership actually feels.

Fix #2: Policy automation with dedup, the version that survives

Manual triage decays when the finding volume grows, so the durable version is rule-driven: every idle finding over $50 a month files a ticket to the owning team, automatically. The requirements that decide survival are boring and absolute: fingerprint-based dedup on the stable identity of the problem (resource + rule), so re-scans and re-openings never double-file; two-way status sync, so the ticket board and the findings list can't drift apart; and a loop guard, so sync events don't ping-pong.

This is precisely the shape ZopNight's Jira integration ships: file a ticket by hand from any recommendation drawer or set a policy (for example, a ticket for every idle resource over $50 a month), each ticket carrying the resource, the savings, and the suggested fix with a link back; re-running never files a duplicate, and status stays in lock-step both ways: resolve the recommendation and the Jira ticket moves to Done, close or reassign the ticket in Jira and the recommendation updates to match, with both classic and scoped Atlassian tokens supported (Jira integration docs). One honest limitation worth knowing before you standardize on it: ServiceNow is not supported, so ITSM-first shops need Fix #3.

Fix #3: Teams that don't live in Jira

The pattern ports; the plumbing changes. Slack-first teams: route findings to the owning team's channel with an acknowledge action, and treat the ack as assignment; it's weaker than a ticket (no sprint pressure) but infinitely better than a dashboard. Other trackers (ServiceNow, Linear, Asana): a CSV or API export of findings plus a small scheduled job that applies the same three rules (rich payload, fingerprint dedup, status sync) gets you the same outcome; the rules matter, not the vendor. The anti-pattern to avoid everywhere: filing everything. Set a dollar floor per ticket and batch the long tail into one monthly "small cleanups" ticket, because forty $6 findings as forty tickets is how integrations die.

How to prevent this

Prevent the decay back into dashboard purgatory:

Measure action rate and time-to-action, not findings count: findings closed per month and median days from detection to resolution are the adoption metrics; a growing findings count with a flat action rate means the pipeline is broken at the ticket gap.
Route by ownership, which means tags and attribution have to work first; a ticket assigned to nobody is a dashboard entry with a Jira id.
Give waste a standing lane: a small fixed slice of each sprint (one ticket per team per sprint is enough) beats quarterly cleanup heroics.
Age findings loudly: anything open past 60 days gets escalated or explicitly accepted as a documented exception; silent aging is how the backlog becomes a graveyard.
Report closed dollars: "we actioned $9,400/month of waste this quarter" is the sentence that keeps the program funded.

FAQ

How do I get cloud cost recommendations into Jira automatically?

Either through your cost tool's native integration (look for policy-based creation, fingerprint dedup, and two-way status sync; those three decide whether it survives) or a scheduled job reading the tool's export or API and filing through Jira's REST API with your own dedup key (resource id + rule type). The payload matters as much as the plumbing: resource, evidence, dollars, suggested fix.

Why do finding-to-ticket automations get turned off?

Duplicates, almost always: the automation keys tickets to scan runs instead of to the stable identity of the problem, so every re-scan re-files. Second cause: volume without a floor, flooding boards with $5 findings. Both are design choices, which is why "does re-running create duplicates?" is the first question to ask of any integration.

What should a cloud waste ticket contain?

Enough to act without opening the cost tool: the resource and account, the evidence with its window (zero connections, 21 days), the monthly cost, the suggested action with a rollback note, and a link back to the live finding. Put the dollar figure in the title so the ticket ranks honestly against feature work in planning.

How do I measure whether FinOps recommendations are actually being adopted?

Action rate (findings resolved as a share of findings raised), median time from detection to action, and closed dollars per quarter. Dashboards report found waste; programs are judged on removed waste, and the ticket pipeline is what converts one into the other.

Related guides

Scheduling EC2 and RDS Start/Stop at Scale: Why Your Shutdown Script Breaks at 300 Instances

Muskan _zop — Tue, 01 Sep 2026 07:14:16 +0000

Every cloud team writes the same Lambda eventually. Cron fires at 8pm, StopInstances on everything tagged env=dev, StartInstances at 8am. It works on the first ten machines, the bill dips, everyone moves on. Eighteen months later there are three hundred instances, four teams, three timezones, and the savings have quietly gone to zero while the script still reports success.

Nothing dramatic broke. Seven small things did. This post is the taxonomy of how shutdown automation fails at scale, an honest look at AWS Instance Scheduler, and the requirements checklist for scheduling that survives its own success.

The seven ways the script breaks

1. Dependency order. The 8am start brings the app servers up before the database accepts connections. Monday's first deploy of the day fails, someone blames the schedule, and the fix is the worst possible one: that team's instances get quietly removed from scheduling forever. Real infrastructures need ordered startup (storage, then compute, then applications) with delays between steps, because a database that is "running" is not yet a database that is accepting connections.

2. Timezones. The cron expression is in UTC. The Bangalore team's "8pm stop" is the Virginia team's mid-afternoon outage. You end up with either one compromise schedule that saves less for everyone, or per-team cron math that breaks twice a year when daylight saving shifts and the comment above the cron line goes stale.

3. Overrides that never expire. An incident happens on a Thursday night, someone needs the environment up, and the honest quick fix is removing the tag or disabling the rule. The incident ends; the exception doesn't. Indefinite overrides are the single most common way scheduling savings quietly disappear, because every exception is invisible the moment the incident that justified it is forgotten. Overrides need to be time-bounded by policy, with a maximum duration someone senior chose, and an audit trail of who held what open and why.

4. No proof of firing. The Lambda's IAM role got an explicit deny in a security sweep in March. It has thrown AccessDenied every night since. Nobody noticed, because nobody measures whether schedules actually fired; the dashboard measures projected savings, which are a fiction the moment execution stops. (This failure mode deserves its own post, and has one.)

5. Drift. New instances launch without the tag. An autoscaling group replaces its members and the instance IDs your config pinned are gone. Someone resizes an instance and the script's hardcoded assumptions break. At three hundred instances, the population changes daily, and a script keyed to a point-in-time inventory decays at the same rate.

6. Partial failures at API limits. Three hundred StopInstances calls in one burst meets RequestLimitExceeded. Two-thirds of the fleet stops; a third doesn't; the script exits zero because the exception handler was written for the ten-instance era. You now have the worst of both worlds: inconsistent state and a green checkmark.

7. The things that aren't EC2. RDS has its own stop API with its own rules (an RDS instance stopped for seven days starts itself again, which surprises everyone once). Kubernetes namespaces can't be stopped with either. Aurora, Databricks, and autoscaling groups each have their own semantics. The "shutdown script" becomes five scripts with five failure modes.

The honest take on AWS Instance Scheduler

AWS's answer is Instance Scheduler, a CloudFormation solution: DynamoDB holds periods and schedules, a Lambda evaluates them, tags opt instances in. It's a real step up from a hand-rolled script: cross-account, EC2 plus RDS, maintained.

Know what it is and isn't. The solution itself is free; you pay for the Lambda and DynamoDB it runs on (typically a few dollars a month) and for your time operating CloudFormation. Configuration lives in DynamoDB entries, which is workable at ten schedules and painful at a hundred. And the failure modes above mostly remain yours: it has no dependency ordering between resources, no bounded-override workflow, no proof-of-firing reconciliation, and drift management is still tag hygiene. It schedules instances; it doesn't manage a scheduling practice.

The checklist for scheduling that survives scale

Whether you build or buy, this is the list the seven failures imply:

Ordered sequencing with per-resource delays: storage, then compute, then applications, and a way to express "wait two minutes after the database before the app tier".
Timezone-native schedules, so an India schedule and a US-East schedule coexist without UTC arithmetic or DST surprises.
Time-bounded overrides with an admin-set maximum duration, notifications where the team lives (Slack or Teams), and a full audit trail. An override that can't expire is a deletion of the schedule with better manners.
Reconciliation: every window, compare expected state against actual state and alert on mismatch. Success is measured, not assumed.
Group-level operations: bundle an app's resources, start and stop them as a unit, in order, with a budget per group.
Eligibility detection: something that continuously finds resources that could be scheduled but aren't (non-production instances running through weekends), because coverage decays as the fleet grows.
Resilience to change: schedules must survive instance-type changes, autoscaling replacements, and re-tags, or drift eats the program in a quarter.

That checklist is also a fair evaluation rubric for any scheduling product. For what it's worth, this is the shape ZopNight's scheduling implements: dependency-aware sequencing (storage, compute, applications, with per-resource delays), first-class timezones so an Asia/Kolkata schedule runs beside an America/New_York one, overrides that are time-bounded with an admin-configurable maximum precisely so exceptions can't become permanent, and a 15-minute cycle that checks resources are in the state their schedule expects. The listed capabilities above aren't hypothetical; they're what the failure modes forced.

FAQ

How do I automatically start and stop EC2 instances on a schedule?

Four common paths: a cron-triggered Lambda calling StopInstances/StartInstances (fine at small scale), AWS Instance Scheduler (tag-driven, cross-account, DynamoDB-configured), EventBridge Scheduler calling the EC2 APIs directly, or a scheduling product. The mechanism is the easy part; dependency order, overrides, proof of firing, and drift are where the choice actually matters.

Is AWS Instance Scheduler free?

The solution is free to deploy; you pay for the underlying Lambda, DynamoDB, and CloudWatch usage (typically single-digit dollars monthly) plus the operational cost of managing it via CloudFormation and DynamoDB entries. The bigger cost is what it doesn't do: no sequencing, no bounded overrides, no firing reconciliation.

How do I schedule RDS start and stop?

RDS has native stop/start APIs with a catch: a stopped RDS instance automatically restarts after seven days, so weekly-cycle schedules must account for it. Multi-AZ and read-replica configurations have additional restrictions. Schedule databases to stop after their dependent applications, and start before them, with a delay for connection readiness.

What tags should drive instance scheduling?

One schedule tag (like schedule=office-hours-ist) mapping to a named schedule definition, not per-instance cron strings. Enforce it at provision time via IaC modules, and run a weekly report of non-production resources missing the tag, because untagged drift is how coverage decays.

How do I stop a shutdown script from breaking my application startup?

Sequence by tier with explicit delays: storage and databases first, application compute last on start, and the reverse on stop. Then verify: a post-start health check per group beats assuming the order worked. If your tooling can't express ordering, the workaround is separate schedules offset by conservative gaps, which wastes some savings but protects mornings.

Your AI Bill Has a Receipt But No Ceiling: Hard USD Limits on Amazon Bedrock and Azure OpenAI Spend

Muskan _zop — Tue, 01 Sep 2026 07:13:59 +0000

Quick Answer (TL;DR)

Amazon Bedrock and Azure OpenAI provide attribution (detailed receipts per model, per user, per token) but no native enforcement: there is no setting that hard-stops spend at a dollar amount. AWS Budgets and Azure budgets alert after the fact; service quotas cap tokens per minute, not dollars per month. A real ceiling requires a gateway in front of the model APIs that issues per-team virtual keys with hard USD budgets and fails closed when a budget is exhausted. That is the only architecture in which "a team can't overspend" is a property of the system rather than a hope.

Why this happens

Cloud billing was built for infrastructure that spends gradually; agentic AI workloads spend in bursts. One retry loop, one over-eager agent, one leaked key, and a five-figure weekend happens at token speed while your budget alert waits for billing data that lags hours behind. The providers have responded on the visibility side (Bedrock's cost attribution is genuinely granular now), but visibility answers "what happened", and the question teams actually ask is "how do I make it impossible". Meanwhile quotas look like the answer and aren't: tokens-per-minute limits bound the rate of spend, not the amount, and a workload can burn any monthly figure you name without ever hitting a TPM ceiling.

Fix #1: A gateway with budgeted virtual keys

Put a proxy between your applications and the model providers, and stop handing teams raw provider keys. Each team or app gets a virtual key with three properties: a hard budget in dollars (fail closed at the limit), an allow-list of models (nobody needs the frontier model for classification), and full per-key spend accounting. The open-source path is LiteLLM's proxy:

curl http://localhost:4000/key/generate \
  -H "Authorization: Bearer sk-admin" \
  -d '{
    "team_id": "ml-platform",
    "max_budget": 500,
    "budget_duration": "30d",
    "models": ["claude-sonnet-5", "gpt-5-mini"]
  }'

When the $500 is gone, requests fail with a budget error instead of billing more. That failure is the feature: it converts an invoice surprise into an application error somebody sees the same hour.

This is also where ZopNight's AI Gateway sits if you'd rather run it governed inside your cost platform: per-team virtual keys with hard USD budgets and per-key model allow-lists across OpenAI, Anthropic, OpenRouter, and Amazon Bedrock, spend reported by key, provider, model, and team, key rotation and revocation, and a complexity router that classifies each prompt to a cheap or strong model (dry-runnable before you trust it), with hosted LiteLLM as the store of record so the gateway never adds a proprietary layer to your request path (AI Gateway docs). Either way, the architecture is the point: keys with dollars attached.

Fix #2: Quotas and provisioned capacity as blunt backstops

If a gateway isn't landable this quarter, the native controls still bound the blast radius, bluntly. Request lower service quotas (Bedrock TPM/RPM per model, Azure OpenAI deployment capacity) so the worst-case burn rate is capped; a workload that can only spend $40 an hour can only lose $960 a day, which is bad but survivable. Prefer separate AWS accounts or Azure subscriptions per AI team so quotas and budget alerts have team-level blast radius. And set the alerts you do have aggressively (daily granularity, low thresholds, paging a human), understanding they're smoke detectors, not sprinklers.

Fix #3: The runaway agent edge case

Agents fail differently from applications: a planning loop that never converges will happily spend all night inside a single session. Two controls that live in your code, not the platform: per-session budget counters (kill the session at N dollars of estimated spend) and hard iteration caps on any loop that calls a model. And rehearse the kill switch: the only true emergency stop for a leaked or runaway key is revocation, so revoking a virtual key must be a one-command runbook every on-call knows, which is itself an argument for virtual keys, because revoking a shared provider key takes down everyone.

How to prevent this

No shared provider keys. One virtual key per team or app, so spend has an owner and revocation has a scope.
Budgets that fail closed on every key, sized from a month of observed spend plus headroom, reviewed monthly.
Model allow-lists per key: default teams to the mid-tier; the expensive model is an explicit grant.
Route by task complexity where you can; most tokens in most products don't need the strongest model.
Keep the receipts too: per-key spend reports catch the slow creep that never trips a ceiling.

FAQ

Can I set a hard spending limit on Amazon Bedrock natively?

No. AWS Budgets can alert (and its actions can restrict IAM, coarsely and with lag), and service quotas cap request and token rates, but there is no native "stop at $X" for Bedrock spend. A hard dollar ceiling requires the key-and-budget layer to live in front of the API, in a gateway.

Can I cap Azure OpenAI spend at a dollar amount?

Same answer: Azure budgets alert but don't block, and deployment capacity caps throughput, not dollars. Some teams approximate a cap by isolating AI workloads in a subscription and cutting it off on breach, which is an outage disguised as governance. A budgeted gateway key is the clean version.

What is a virtual key for LLMs?

A key your gateway issues to a team instead of the provider's real key. The gateway holds the provider credentials, enforces the key's budget and model allow-list on every request, and accounts spend per key. Compromise or overspend is contained to one key, revocable in seconds without touching other teams.

Does routing through a gateway add latency or risk?

Milliseconds of proxy overhead against token-generation times measured in seconds, so it's rarely observable. The risk profile actually improves: provider keys stop living in dozens of app configs and live in one governed place with rotation and audit.

Will provider-side cost attribution solve this eventually?

Attribution keeps improving (Bedrock's per-user and per-model breakdowns are good now), but attribution and enforcement are different products: one is a receipt, the other is a ceiling. Until providers ship native hard caps, the ceiling has to be architectural.

Related guides

Idle AI Infrastructure: The GPU Cost Category Nobody Names Yet (SageMaker, Bedrock, Vertex AI)

Muskan _zop — Tue, 01 Sep 2026 07:13:49 +0000

Quick Answer (TL;DR)

Idle AI infrastructure is GPU-backed capacity that bills while serving no inference and training nothing: real-time endpoints with zero invocations, notebooks running overnight, provisioned throughput nobody calls, batch clusters that outlived their jobs. It's the same idle-resource problem cloud teams already audit for EC2, at 5-30x the hourly price, hidden behind CPU metrics that say nothing (a GPU box can be 100% idle on the GPU at 5% CPU, and the meter runs the same). The fix pattern is identical to classic idle hunting: a GPU-native threshold (invocations and GPU utilization over 7-14 days), a price per finding, and schedules or scale-to-zero for everything with a human usage pattern.

Why this happens

AI infrastructure gets provisioned in experiment mode: an endpoint for the demo, a notebook for the sprint, provisioned throughput "so latency is safe", a HyperPod cluster for the training push. Experiments end; the infrastructure doesn't, because the team moved to the next model and nobody owns the cleanup. Meanwhile the observability defaults are wrong for the job: CloudWatch gives you CPU for free, but GPU utilization and GPU memory need explicit collection, so the dashboards everyone watches literally cannot show the waste. And because the category has no name, it appears in no cost review: "idle EC2" is a standard audit line; "idle AI infrastructure" mostly isn't, yet, even though a single forgotten p4d.24xlarge burns about $23,900 a month, roughly a rack of forgotten m5.larges.

Fix #1: Find the zero-invocation endpoints first

Real-time inference endpoints are the classic offender: created per experiment, priced per hour, forgotten per quarter. The test is invocations over a meaningful window:

for ep in $(aws sagemaker list-endpoints --query 'Endpoints[].EndpointName' --output text); do
  inv=$(aws cloudwatch get-metric-statistics --namespace AWS/SageMaker \
    --metric-name Invocations --dimensions Name=EndpointName,Value="$ep" \
    --start-time "$(date -u -d '7 days ago' +%FT%TZ)" --end-time "$(date -u +%FT%TZ)" \
    --period 604800 --statistics Sum --query 'Datapoints[0].Sum' --output text)
  case "$inv" in None|0|0.0) echo "$ep: 0 invocations in 7 days";; esac
done

Every hit is an endpoint billing its full instance rate to answer nobody. Price each finding (a single ml.g5.xlarge endpoint idles at roughly $1,000 a month) and the list funds its own cleanup meeting. This is also a category tooling now covers: ZopNight flags GPU-idle SageMaker endpoints and idle or over-provisioned endpoints, clusters, and batch jobs, shows GPU utilization and video memory beside CPU and RAM so the idle verdict is made on the right meter, and extends GPU rightsizing across the G5, G6, P4, and P5 families (recommendation docs).

Fix #2: Schedules and scale-to-zero for anything with human rhythms

Notebooks and dev endpoints have office hours; give them a calendar. Auto-stop idle notebook instances (lifecycle configs make this a one-time setup), stop dev endpoints outside working hours, and move spiky or occasional inference to serverless or asynchronous inference, which scales to zero between requests and converts an always-on instance bill into a per-request one. Training-class clusters deserve the same discipline: schedulable on and off around actual usage instead of holding GPUs on standby between pushes.

Fix #3: The provisioned-capacity edge case

The most expensive idleness doesn't look like a resource at all: it's committed throughput. Bedrock provisioned throughput and Azure OpenAI PTUs bill for the reservation whether or not tokens flow, and a team that provisioned for launch-week traffic is quietly paying launch-week rates in month four. The check is utilization of the commitment against its cost as on-demand tokens; below the crossover, walk it back to on-demand. Same logic as reserved instances, faster decay, because model traffic patterns change monthly.

How to prevent this

Collect GPU metrics by default (DCGM or the platform's GPU utilization metrics) on every GPU workload, or every idle verdict will be made on the wrong meter.
TTL the experiments: endpoints and notebooks created outside IaC get an expiry tag and a weekly reaper report.
Name the category in your cost taxonomy so "idle AI infrastructure" is a standing line in the monthly review, with an owner.
Prefer scale-to-zero shapes (serverless or async inference, job clusters) as the default, and standing endpoints as the justified exception.
Review provisioned throughput monthly against actual token flow, like commitment coverage anywhere else.

FAQ

How do I know if a SageMaker endpoint is idle?

Sum its Invocations metric over 7-14 days; zero or near-zero over that window on a standing endpoint is idle by any defensible bar. Pair with GPU utilization (via DCGM or SageMaker's hardware metrics) to catch the subtler case: traffic exists but a fraction of the GPU serves it, which is a rightsizing finding.

Why doesn't CPU utilization catch idle GPU instances?

Because the expensive silicon is the GPU, and the CPU on a GPU instance does auxiliary work. An inference box can show 30% CPU with the GPU at zero. If your dashboards only show CPU and memory, your GPU fleet is unmonitored where it matters.

Is serverless inference cheaper than a real-time endpoint?

For spiky or low-volume traffic, dramatically: you pay per request and scale to zero between them, trading a cold-start latency penalty. High-steady-volume workloads still favor provisioned endpoints. The decision is the endpoint's invocation histogram, which is the same data the idle check already pulled.

Does provisioned throughput count as idle infrastructure?

When utilization is low, it's the worst kind: pre-paid idleness. Compare the commitment's monthly cost against the same traffic priced as on-demand tokens; sustained utilization below the break-even means the reservation is burning money for latency insurance nobody measured.

Related guides

Reserved Instances vs Savings Plans: Break-Even Math and What a Commitment Costs You If You're Wrong

Muskan _zop — Mon, 24 Aug 2026 06:18:47 +0000

Every Savings Plan pitch, from AWS or anyone else, shows you the same number: the discount. Commit for a year, save 28%. Commit for three, save half. What the pitch never shows is the other curve: what the commitment costs you if your usage drops, your architecture changes, or the product line whose baseline you committed to gets cancelled in month seven.

Finance always asks the second question ("what happens if we're wrong?"), engineering usually can't answer it with a number, and the commitment decision stalls or, worse, gets made on vibes. So here's the downside math, worked all the way through.

The instruments, in one paragraph each

Standard Reserved Instances commit you to an instance family in a region (with size flexibility within the family for Linux). Deepest discounts, least flexibility. The one escape hatch: Standard RIs can be listed on the RI Marketplace, usually at a haircut.

Convertible RIs trade a few points of discount for the right to exchange into other instance families. No marketplace resale, but the exchange right is a real hedge against architecture change.

Compute Savings Plans commit you to a dollar-per-hour spend across almost all compute: any instance family, any region, Fargate, Lambda. Most flexible, slightly shallower discount, and the important fine print: no resale, no exchange, no early exit. A Savings Plan is a promise to spend $X per hour, every hour, for the whole term, whether or not anything runs.

EC2 Instance Savings Plans sit between: deeper discount, locked to a family in a region, same no-exit rule.

Typical discount magnitudes (us-east-1 Linux, on-demand baseline, as of early 2026): a 1-year no-upfront Compute Savings Plan lands around the high-20s percent; 3-year around 50%; EC2 Instance Plans and Standard RIs a few points deeper at each term. Exact numbers vary by family; the shapes below don't.

The break-even formula

A commitment with discount d breaks even when your utilization of it stays above (1 - d). That's the whole formula.

Worked: your workload costs $1.00/hour on demand. A 1-year plan covers it at $0.72/hour (28% off). Break-even utilization is 72%: as long as the covered capacity is actually needed at least 72% of the term's hours, you win.

Now run the downside:

Usage drops to 60% of what you committed. You pay $0.72/hour for every hour anyway. Effective rate on the hours you used: $0.72 divided by 0.60, which is $1.20/hour, 20% above on-demand. On one instance-equivalent that's about $1,050 lost over the year, and it scales linearly with the size of the miss.
The workload dies at month 6 of a 3-year term. Remaining liability: your committed rate times every remaining clock-hour of thirty months. On a $5/hour Compute Savings Plan, that's around $109,000 of spend with nothing behind it, non-cancellable, non-resellable. A Standard RI in the same situation could be listed on the marketplace at a loss; a Convertible could be exchanged toward whatever replaced the workload; the Savings Plan has no lever at all.
The subtle one: you committed to the average, not the trough. Usage that oscillates between 40 and 140 instances with a mean of 90 does not support a 90-instance commitment; every hour below 90 is paying the committed rate for idle coverage. Commitments should be sized on the floor of the usage curve, not its middle.

That last point is why the standard eligibility test is uptime-based: a workload is commitment-grade when it has run at high utilization (70% and up) for a sustained window (30 days and more), because the break-even math for a high-20s discount sits right at that threshold.

Decision rules that survive the downside

Sequence matters: optimize first, commit last. Rightsize, kill idle, schedule non-production, then commit to what remains. Committing before shrinking locks in the waste at a discount.
Commit to the trough. Size against a low percentile (p10) of the last 90 days of usage, not the average.
Ladder the terms. Several smaller commitments purchased quarterly instead of one big annual buy: your coverage tracks reality, and no single bet is fatal.
1-year until proven, 3-year for bedrock. Three-year terms only for load that has already survived a year of architecture churn.
Target 60-80% coverage, not 100%. The last slice of usage is the volatile slice; leave it on demand deliberately.
Prefer flexibility at equal math. Compute Savings Plans over instance-scoped commitments unless the family is genuinely settled; Convertible over Standard when in doubt. The few points of discount you give up are the insurance premium.
Review utilization monthly. A commitment at 93% utilization is a decision working; the same commitment at 70% is a leak with a contract.

One more honesty check worth demanding from any tool (or spreadsheet) that recommends commitments: it should refuse to print a savings number it can't defend. ZopNight's commitment recommendations are a working example of that bar: they fire only on sustained measured uptime, price each 1-year and 3-year option from live rate feeds rather than a flat assumed discount, suppress any recommendation whose projected saving clears break-even by less than 10%, and when uptime can't be measured they say "savings unknown" instead of assuming 100%, with the term, rates, and break-even evidence attached to every figure (rule docs). Whatever you use, reject any recommendation that credits discount on hours you never ran.

FAQ

What's the difference between Reserved Instances and Savings Plans?

RIs reserve a specific instance family in a region (Standard: resellable on the RI Marketplace; Convertible: exchangeable). Savings Plans commit a dollar-per-hour spend (Compute: any family, region, Fargate, Lambda; EC2 Instance: one family, deeper discount) with no resale or exchange. Roughly: RIs are an asset you can partially unwind; Savings Plans are a pure spend promise.

What utilization makes a commitment break even?

One minus the discount. A 28% discount breaks even at 72% utilization of the committed amount; a 50% discount at 50%. Below break-even you're paying more than on-demand for the hours you actually used, which is why sustained high uptime over a meaningful window is the eligibility test, and why sizing on average rather than trough usage quietly guarantees waste.

Can I exit a Savings Plan or RI early?

Savings Plans: no. No cancellation, no resale, no exchange; the hourly commitment bills to the end of the term. Standard RIs can be sold on the RI Marketplace (usually below face value). Convertible RIs can be exchanged for different configurations of equal or greater value. This asymmetry is the strongest argument for flexible instruments when your architecture is still moving.

How much of my usage should be covered by commitments?

Most teams land at 60-80% of steady-state compute. Cover the floor of the usage curve with commitments, leave the volatile top slice on demand, and grow coverage in quarterly tranches as baselines prove themselves. 100% coverage means you've committed to your peaks, and peaks are exactly what usage stops doing.

Should I fix utilization or buy more coverage first?

Fix utilization first, always. Low utilization on an existing commitment means you're already paying for idle coverage; adding more commitment on top compounds it. And before any new commitment: rightsize, delete idle, schedule non-production. Commit to the fleet you should have, not the one you currently do.

Amazon Bedrock, Vertex AI, Azure OpenAI and SageMaker in One Bill: Cross-Cloud AI Cost Visibility

Muskan _zop — Mon, 24 Aug 2026 06:18:32 +0000

Quick Answer (TL;DR)

Most AI-using companies now spend on Amazon Bedrock, Vertex AI, Azure OpenAI, and SageMaker at the same time, because model choice follows tasks, not cloud loyalty. Each provider shows its own slice in its own units (tokens, node-hours, instance-hours, PTUs), so nobody sees the total. One AI bill takes three steps: pull all three clouds' billing exports into one place, map the AI services into a single cost category with a maintained mapping table, and treat standing resources (endpoints, provisioned throughput) differently from jobs (which should be costed by actual runtime, not calendar months). The blocker is never math; it's that the mapping has no owner.

Why this happens

Teams pick models per task: Claude for one workload, Gemini for another, GPT deployments in the Azure tenant the enterprise agreement lives in, and SageMaker for everything self-hosted. Each platform bills in its own vocabulary: Bedrock in tokens and provisioned throughput units, Vertex in node-hours and job runs, Azure OpenAI in tokens and PTU-hours inside a subscription, SageMaker in instance-hours across a dozen resource types. Finance sees four fragments in three invoices, engineering sees four consoles, and the question "what does AI cost us" gets answered with a shrug or a two-week spreadsheet that's stale on arrival. Meanwhile the AI line is usually the fastest-growing item on the bill, which is exactly the wrong place for a shrug.

Fix #1: One category, mapped from the exports

You already have the raw material: every cloud ships a complete billing export (CUR or Data Exports on AWS, BigQuery billing export on GCP, Cost Management exports on Azure). Land all three in one warehouse (the mechanics are a solved problem) and maintain one mapping view that tags AI spend:

CASE
  WHEN service_name IN ('Amazon Bedrock', 'Amazon SageMaker') THEN 'ai'
  WHEN service_name LIKE '%Vertex AI%' THEN 'ai'
  WHEN service_name LIKE '%Cognitive Services%'
       OR service_name LIKE '%Azure OpenAI%' THEN 'ai'
  ELSE category
END AS spend_category

Add the adjacent lines people forget (GPU instances serving self-hosted models, vector databases, AI-dedicated storage) and you have the number: AI spend, per cloud, per month, from the same reconciled data as the rest of the bill.

Fix #2: Attribute inside the category

One total invites the next question: which team, which product, which model. Reuse the attribution machinery the rest of your bill uses: tags and labels on SageMaker and Vertex resources, Azure OpenAI deployments per team, and Bedrock's cost allocation surfaces, plus account or subscription boundaries where tags are thin. The per-model split matters more here than anywhere else in cloud cost, because switching a workload one model tier down is frequently a 5x price change with negligible quality loss, and that decision needs per-model numbers to exist.

Fix #3: The standing-versus-job edge case that skews everything

AI resources split into two billing shapes, and mixing them corrupts reports. Standing resources bill while they exist: real-time endpoints, provisioned throughput, notebook instances, feature stores. Jobs bill for their runtime: training, tuning, batch inference, evaluations. A report that treats a 6-hour training job as a monthly line shows phantom cost; one that ignores an idle 24/7 endpoint hides real cost. Cost jobs by actual runtime and audit standing resources for idleness separately.

This split is also where tooling earns its keep, because doing it by hand across four platforms is the part that decays first. ZopNight, for instance, discovers and prices the full estate as one inventory: Bedrock end to end (agents, knowledge bases, guardrails, custom models, provisioned throughput, and customization, batch-inference, and evaluation jobs), GCP Vertex AI (endpoints, models, feature stores, Workbench notebooks, and training, tuning, and batch-prediction jobs), Azure OpenAI, AI Foundry, AI Search, and Azure ML with cost and start-stop scheduling, and SageMaker down to its job types, with jobs costed by how long they actually ran so a finished job stops adding cost (docs). One inventory, per-resource cost, all four surfaces in the same pane as the rest of the cloud bill.

How to prevent this

Give the mapping an owner. New AI services appear quarterly; an unowned mapping view is stale in one.
Adopt-a-service checklist: before a team uses a new AI platform, name how it bills, which export line it lands on, and who owns its cost.
Tag at creation for AI resources exactly like everything else; endpoints and jobs inherit team attribution from IaC.
Report AI as a first-class category monthly (total, per cloud, per team, per model tier), next to compute and storage, not buried in "other".
Watch the two growth curves separately: standing-resource cost (a capacity decision) and job cost (a usage signal), because they call for different responses.

FAQ

How do I see Amazon Bedrock costs per model or per team?

Bedrock's billing lines break down by model and its cost attribution has grown genuinely granular (per-user and per-model views). For team-level rollups, combine the billing export with your tagging or account boundaries; for enforcement rather than reporting, you need budgeted keys in front of the API, which is a different problem from visibility.

Is provisioned throughput billed when idle?

Yes. Bedrock provisioned throughput and Azure OpenAI PTUs bill for the commitment while it exists, used or not, exactly like a reserved instance. Idle provisioned capacity is the single most expensive AI waste pattern; walk it back to on-demand tokens when sustained utilization doesn't justify it.

What's the FOCUS spec and does it help here?

FOCUS is the FinOps Foundation's common billing schema that AWS and Azure ship natively and GCP maps to. It normalizes column vocabulary so your cross-cloud AI view is one table instead of three schemas; the AI category mapping still has to be yours, since FOCUS normalizes columns, not opinions.

Should self-hosted GPU inference count as AI spend?

Yes, tagged as such. The g5/g6/p4/p5 instances serving your own models are AI cost that never says "AI" on the bill line, and leaving them in generic compute understates the category by whatever your self-hosted footprint is, which for many teams is the majority of it.

Related guides

A FinOps MCP Server: What Your AI Assistant Can Ask About Your Cloud Bill, and What It Can Never Change

Muskan _zop — Mon, 24 Aug 2026 06:18:19 +0000

Quick Answer (TL;DR)

An MCP (Model Context Protocol) server for FinOps gives your AI assistant (Claude, Cursor, and other MCP clients) tools to query real cloud cost data, so "what's costing us money this month" gets answered from the bill instead of from the model's imagination. The design contract that makes this safe has three clauses: the assistant gets read-only tools over costs, resources, and recommendations; writes are rejected at the protocol layer, not by prompt instructions or token scope; and every call is audited. An assistant that can read everything and change nothing is a superpower; one that can change anything is a new incident category.

Why this happens

Two curves crossed. Engineers now live inside AI assistants, and cloud bills are exactly the kind of tedious, structured question those assistants are good at when grounded and dangerous at when not: an ungrounded model asked about your AWS bill will produce confident, plausible, wrong numbers. MCP standardized the fix (a server exposes typed tools; the assistant calls them; answers carry real data). But the same standard makes it trivially easy to hand an assistant write access, and a stochastic system with mutation rights over production infrastructure is a blast radius nobody has finished thinking about. The interesting design question is therefore not "can my assistant see the bill" (yes, easily) but "what is it structurally prevented from doing".

Fix #1: Connect the assistant to cost data, read-only by construction

The bar to hold any FinOps MCP server to (vendor-provided or your own):

Read-only toolset: costs by service, account, team, and period; resource inventories; idle and orphan findings; recommendation lists with their evidence. Enough to answer every "what, where, why" question.
Writes rejected at the protocol layer: mutation attempts fail in the server regardless of what token the assistant holds or what the prompt claims. "The model promised to be careful" is not an enforcement mechanism; a server that has no mutate handlers is.
Per-call audit: every tool invocation logged with its caller and arguments, so "what did the assistant look at" is a query, not a mystery.
Scoped data, same as any client: the server sees what its credential sees, so your existing RBAC keeps meaning something.

With that in place, the workflows are immediately useful: "which team's spend grew fastest this month and why", "list idle databases over $100 a month with their evidence", "draft the cost section of the platform review", all answered from live data, in the tool where the engineer already is.

Fix #2: Build a thin one over your own billing warehouse

If you already land billing exports in a warehouse, a useful internal MCP server is a weekend project: a handful of read-only tools wrapping parameterized queries.

tools:
  get_cost_summary(period, group_by)      -> spend by service/account/team
  get_cost_trend(scope, days)             -> daily series for a scope
  find_idle_resources(type, min_monthly)  -> findings with evidence
  get_untagged_spend(period)              -> unattributed cost by account

Keep the warehouse credential read-only (a database role with SELECT on the billing schema and nothing else) so the safety property is inherited from the database, not promised by the code. Resist the temptation to add "just one" write tool; the moment one exists, every prompt-injection scenario in every document the assistant reads becomes a potential caller of it.

Fix #3: The write question, answered without writes

Teams eventually want the next step: "fine, it found the idle database, let it stop the thing." The safe shape for that is proposal, not action: the assistant files a pending approval record (a ticket, an approval-queue item) that a human reviews and executes through the normal, audited path. The assistant's output is a request; the human holds the trigger. That preserves the entire value (the finding, the context, the prepared action) while keeping mutation authority in systems designed for it, with approvals, preconditions, and rollback, none of which a chat session has.

How to prevent this

Prevention here means preventing the failure modes of assistant-to-infrastructure access, before they're incidents:

Treat assistant sessions as untrusted input. Assistants read web pages, tickets, and docs; any of those can carry injected instructions. Read-only tools make injection a privacy question instead of an outage question.
Separate credentials: the MCP server's own least-privilege identity, revocable in one step, never a human's personal token.
Audit review: skim the tool-call log weekly like any other access log; volume and shape anomalies show up fast.
Version the contract: when the server adds tools, review them like API changes, because that's what they are.
Write the "what it can never do" list down and publish it to the team; unstated guarantees don't calm anyone.

FAQ

What is MCP, in one paragraph?

Model Context Protocol is an open standard that lets AI assistants call tools exposed by external servers: the server declares typed tools, the assistant invokes them, results flow back as structured data. It's the plumbing that turns "ask the model about your systems" from hallucination into database queries, and it's supported by Claude, Cursor, and a growing set of clients.

Is it safe to give an AI assistant access to my cloud bill?

Read access to cost data through an audited, read-only MCP server is one of the lower-risk, higher-value assistant integrations: billing data is sensitive but not secret-bearing, and the failure mode of a bad query is a wrong chart, not an outage. The risk arrives exclusively with write access, which is why the read/write boundary belongs in the server, not in the prompt.

Can the assistant change my infrastructure through MCP?

Only if someone gives it tools that do so. A well-designed FinOps server rejects writes at the protocol layer regardless of token scope, and the mature pattern for actions is proposal-plus-human-approval: the assistant prepares, a person executes through the normal audited path.

Which assistants can use a FinOps MCP server?

Any MCP client: Claude Desktop and Claude Code, Cursor, and the expanding ecosystem of IDEs and agents that speak the protocol. One server, every client, which is precisely why the server's own guarantees (read-only surface, auditing) matter more than any single client's behavior.

Related guides

Cost Allocation Tags When Production Is Already Untagged: Virtual Tags, Inheritance and How to Stop the Bleeding

Muskan _zop — Mon, 24 Aug 2026 06:18:06 +0000

Quick Answer (TL;DR)

You do not need to retro-tag three years of production to get cost allocation, and you probably shouldn't try. Attribution is a reporting problem before it is a tagging problem: derive virtual tags from metadata you already have (account, name patterns, resource type, parents) inside your cost tooling, let untagged children inherit their parent's attribution, and enforce required tags at provision time so the untagged population stops growing. Real tags on real resources become a slow cleanup, not a blocker. Also know the hard rule: AWS cost allocation tags are not retroactive; they only apply to usage after activation, which is one more reason attribution can't wait for tagging.

Why this happens

Every tagging best-practices guide assumes you're starting on day zero. Real accounts have years of history: resources created before the tag policy existed, some by people who left, some by services that can't be tagged at all, and a large class that could be tagged but sits in production behind change approvals nobody wants to spend on metadata. So "tag your resources" becomes a permanently deferred project, 10-30% of the bill attributes to nobody, and every showback report ships with an asterisk. The trap is treating cloud-side tags as the only source of attribution. They're the best source for new resources and the most expensive one for old production.

Fix #1: Virtual tags, derived instead of written

The fastest path to attribution is deriving it from signals that already exist: the account or project a resource lives in, its type, its region, and above all its name, because naming conventions survive even when tagging didn't. A resource called payments-prod-db-01 is telling you its team and environment; a derivation rule can listen.

You can do this directly in SQL over your billing export:

SELECT line_item_resource_id,
  CASE
    WHEN resource_tags_user_env <> '' THEN resource_tags_user_env
    WHEN line_item_usage_account_id IN ('111122223333') THEN 'prod'
    WHEN line_item_resource_id LIKE '%-prod-%' THEN 'prod'
    WHEN line_item_resource_id LIKE '%-stg-%'  THEN 'staging'
    ELSE 'unattributed'
  END AS env_derived,
  SUM(line_item_unblended_cost) AS cost
FROM cur
GROUP BY 1, 2

Real tags win when present, rules fill the gap, and the leftover lands in a visible unattributed bucket that shrinks as rules improve. Nothing in the cloud account was modified, so there's no change approval, no risk, and no waiting.

This is exactly the productized shape of ZopNight's Smart Tags: policy-driven virtual tags derived from provider, region, type, instance type, and name, with a pending-and-accept review workflow so a human confirms each derivation, re-evaluation on every discovery refresh, and the tags feeding showback and tag-coverage reporting while being never written back to your cloud (Smart Tags docs). Whether you use a tool or the SQL above, the principle is the same: attribution first, cloud-side tags second.

Fix #2: Inheritance, so children stop being orphans

A large share of "untagged" resources are children of things that are attributable: the volume attached to a tagged instance, the snapshot of that volume, the network interface on a tagged load balancer. Attribute children to their parents and untagged counts drop dramatically without touching anything. The same logic scales up a level: when a whole account, project, or resource group belongs to one team, map it once and everything inside inherits, giving you instant 100% coverage at coarse granularity that you refine over time. Coarse-but-complete beats precise-but-partial for every decision that matters monthly.

Fix #3: The genuinely untaggable edge cases

Some spend can't carry your tags no matter how disciplined you are: certain shared platform services, data transfer lines, support fees, and legacy resource types with tagging quirks. Don't force it; policy it. Route these to explicit allocation rules (split by the consuming teams' proportions, or hold them in a named platform bucket) and document the rule next to the number. An honest labeled bucket keeps trust; invisible leakage into "other" destroys it.

How to prevent this

Stopping the bleeding is a provision-time problem, not an audit problem:

Require tags at creation: AWS Organizations tag policies and SCPs, Azure Policy deny rules, GCP org policies, so an untagged resource fails to launch in governed accounts.
Bake tags into IaC modules so developers inherit correct tags by default instead of remembering them.
Measure tag coverage weekly (percentage of spend carrying required tags) and treat regressions like failed builds.
Keep a non-compliance report with an owner, because a report nobody owns is a screenshot.
Remember activation: in AWS, cost allocation tags must be activated in the billing console and apply only from that day forward, so activate the keys early even while coverage is still climbing.

FAQ

Do AWS cost allocation tags apply retroactively?

No. A tag only appears in billing data from the moment the key is activated in the billing console, and only on usage after the resource was tagged. Historical spend stays untagged forever, which is precisely why derived or virtual attribution is the only way to allocate the past.

How do I find all untagged resources in AWS?

Tag Editor and resourcegroupstaggingapi get-resources list resources missing given keys, and your billing export shows untagged spend (empty resource_tags columns) ranked by cost, which is the better starting list: fix the expensive untagged resources first, let derivation rules cover the tail.

What's the difference between virtual tags and real tags?

Real tags live on the cloud resource and flow into every native tool; virtual tags live in your cost layer, derived from rules, and touch nothing in the account. Real tags are better where you can have them (new resources, via IaC); virtual tags are how you attribute the past and the untaggable without change windows.

What tags should be mandatory?

Small and enforceable beats comprehensive and ignored: team (or cost center), env, and service cover most allocation questions. Every additional required key lowers compliance, and optional keys can grow later once the required three hold above 90% coverage.

Does Azure support tag inheritance?

Azure Cost Management can inherit subscription and resource-group tags into cost data (a setting, off by default), which is the same attribution-without-modification idea: the resources stay untouched while the billing data gets the tags.

Related guides

Publicly Accessible RDS and Unrestricted Security Groups: The Cost Side of Risky Cloud Config

Muskan _zop — Mon, 24 Aug 2026 05:13:27 +0000

Security findings and cost findings are usually filed by different teams into different tools, and treated as different kinds of problem. But a specific set of configurations sits squarely in both piles: a database open to the internet, a security group accepting traffic from anywhere, an IAM role with a wildcard where a scope should be. The security report calls them risk. The invoice, eventually, calls them money. Treating them as one category gets both teams moving on the same afternoon.

How risky config becomes spend

Publicly accessible RDS. A database with PubliclyAccessible: true and a permissive security group doesn't just widen your attack surface; it works harder. Internet-facing databases absorb continuous scan and connection-attempt traffic, which burns connections, CPU, and sometimes forces the "we need a bigger instance" conversation that a private subnet would have made unnecessary. And every byte an exposed endpoint serves outward is data transfer billed at internet egress rates. The audit is one query: aws rds describe-db-instances --query 'DBInstances[?PubliclyAccessible].DBInstanceIdentifier'. The right answer for almost every production database is an empty list, with access via VPC peering, private endpoints, or a bastion pattern.

Unrestricted security groups (0.0.0.0/0). Open inbound on 80/443 behind a load balancer is the job. Open inbound on SSH, RDP, database ports, or "all traffic" is the classic pre-incident finding, and the incident it precedes is very often a cost incident: the most common monetization of a compromised cloud resource is cryptomining, which means the attacker's first move is launching or saturating compute. Companies discover these breaches on the bill, not in the SIEM: a $40,000 anomaly weekend on GPU instances nobody in the org launched. The audit: aws ec2 describe-security-groups --filters Name=ip-permission.cidr,Values='0.0.0.0/0' and then filtering out the intentional web-facing rules.

Wildcard IAM. Roles with AdministratorAccess or "Principal": "*" don't cost anything while nothing goes wrong. They define the blast radius for when something does: one leaked key with admin scope is the difference between an incident report that says "attacker enumerated one bucket" and one that says "attacker launched 400 instances in three regions we don't use." Cost-wise, wildcard IAM is unpriced tail risk, and the mitigation (scoping roles, requiring MFA, alerting on use of broad roles) is nearly free.

The bastion tax, as a bonus. Teams that fix public exposure the old way (a bastion host per VPC) buy a small standing cost: an instance, its EIP, its patching. Session Manager and its equivalents removed the need for most of these years ago; retiring bastions is a rare win that removes cost and attack surface in the same change.

Why this pairing is worth institutionalizing

Cost has something security usually lacks: a monthly forcing function that executives already read. Security has something cost usually lacks: veto power. Findings that carry both labels ("this database is exposed AND its egress pattern is costing us") clear prioritization queues that pure-risk findings sit in for quarters. In practice that means one shared audit list, run on a cadence, where each item carries both a risk note and a dollar note, and either team can champion it.

The honest scope note: none of this is a security program. Closing 0.0.0.0/0 on port 5432 doesn't make you secure; it makes you not-negligent, cheaper, and harder to monetize. The full security job (identity, detection, response) belongs to security tooling and people. This list is the overlap zone where the cheapest security wins and the cheapest cost wins are the same three commands.

The one-afternoon audit

Public databases: the PubliclyAccessible query above, cross-checked against each one's security group. Expected output: empty, or a short list with written justifications.
Open security groups: the 0.0.0.0/0 filter, minus intentional web listeners. Every remaining rule gets closed or documented today, not sprint-after-next.
Wildcard IAM: list roles carrying AdministratorAccess and policies with * actions on * resources; each either gets scoped or gets an owner and an expiry.
Anomaly tripwire: whatever cost alerting you run, confirm it would actually page on a sudden compute spike in an unused region, because that's what "we got mined" looks like on day one.

FAQ

Why does a publicly accessible RDS instance cost more?

Beyond the risk: internet-facing endpoints absorb continuous scanning and connection-attempt load (consuming connections and CPU that get "solved" by upsizing), and everything they serve outward bills at internet egress rates. Moving the database private removes attack surface and those cost pressures in one change.

Is an open security group ever acceptable?

For web-facing listeners (80/443) behind a load balancer, yes, that's what they're for. For SSH, RDP, database ports, or all-traffic rules, effectively never; those are the standard entry points for compromises whose most common outcome is cryptomining billed to your account. Scope them to known CIDRs or replace interactive access with Session Manager.

How do compromised cloud accounts usually show up?

On the bill. The dominant monetization is compute abuse (mining on large or GPU instances, often in regions the victim doesn't use), which means a sudden spend anomaly is frequently the first detection signal. That's also why cost anomaly alerting with per-region granularity doubles as a security tripwire.

Do wildcard IAM roles cost anything?

Not until they do. Their cost is the blast radius they grant to any leaked credential: the difference between a contained incident and a three-region instance-launch spree. Scoping roles and alerting on broad-role use is one of the cheapest risk reductions in cloud, which is exactly why it belongs on the shared cost-and-security list.

What Config Drift Costs You: CloudWatch Log Retention, S3 Lifecycle Policies and RDS Multi-AZ, Priced

Muskan _zop — Fri, 21 Aug 2026 09:17:08 +0000

Config drift usually gets discussed as a correctness problem: the running state no longer matches the intended one. But three specific drifts are money problems first, and they share a shape: a default nobody chose, set at provision time, growing silently ever since. None of them shows up as an incident. All of them show up on the bill, unlabeled.

Here they are, priced, with the fix for each.

CloudWatch log groups that never expire

The default retention for a CloudWatch log group is never expire. Not 90 days, not a year: forever. Every log group created without an explicit retention setting accumulates at $0.03 per GB-month of storage, on top of the $0.50 per GB you already paid at ingestion.

The math compounds quietly. A service logging 5 GB a day into a no-retention group holds about 3.6 TB after two years: roughly $110 a month in pure storage, growing about $4.50 more every month, for logs nobody has ever queried past week one. Multiply by the forty log groups a typical account accumulates, and drift becomes a line item.

Two commands end it. Find the offenders:

aws logs describe-log-groups \
  --query 'logGroups[?retentionInDays==null].[logGroupName, storedBytes]' --output table

Then set retention (30 to 90 days covers most operational needs; compliance logs belong in S3 with lifecycle rules, not in CloudWatch at 6x the storage price): aws logs put-retention-policy --log-group-name <name> --retention-in-days 90. Existing data past the new retention ages out on its own. While you're in there, the sibling drift: debug-level logging left on after an incident multiplies the ingest bill, and ingest, unlike storage, can't be cleaned up retroactively.

S3 buckets with no lifecycle policy

Standard storage costs $0.023 per GB-month. Infrequent Access costs $0.0125, Glacier Instant Retrieval $0.004, and Deep Archive $0.00099, 23 times cheaper than Standard. A bucket without a lifecycle policy keeps everything in Standard forever, which for logs, backups, exports, and build artifacts means paying the hot price for data whose access probability dropped to near zero after thirty days.

Priced: 10 TB of aging logs in Standard is $230 a month. The same bytes in Deep Archive: about $10. Per bucket, per year, that's a $2,600 difference for one policy document.

The honest fine print, because lifecycle rules have teeth: transitions bill per 1,000 objects (millions of tiny objects can make a transition cost more than it saves; aggregate small files first or filter by size), each colder tier has a minimum storage duration (30 days in IA, 90 in Glacier tiers, 180 in Deep Archive) so churning data doesn't belong there, and retrieval from deep tiers costs time and money, which is fine for backups and audit logs and wrong for anything an application reads. When the access pattern is genuinely unknown, Intelligent-Tiering automates the decision for $0.0025 per 1,000 objects monitored, and its main failure mode is being forgotten on buckets full of tiny objects.

The audit is one loop: list buckets, check get-bucket-lifecycle-configuration, and every bucket that errors with "no lifecycle configuration" and holds logs or backups is drift with a dollar sign.

Multi-AZ in the wrong places, both directions

RDS Multi-AZ doubles the instance and storage cost, exactly. That fact cuts both ways, and both directions are drift.

Production missing Multi-AZ is risk drift: a single-AZ prod database is one AZ event away from an outage that costs more than a decade of the standby's price. The check, run against your list of production identifiers:

aws rds describe-db-instances \
  --query 'DBInstances[?MultiAZ==`false`].DBInstanceIdentifier'

Non-production running Multi-AZ is cost drift, and it's usually a copy-paste artifact: someone cloned the prod Terraform module for staging and the standby came along. A db.m5.large at about $125 a month becomes $250; a dev environment with four such databases is quietly paying $6,000 a year for high availability that protects test data.

The same copy-paste family includes backup retention set to 35 days on throwaway databases and snapshot schedules that never met a deletion policy. The principle across all of it: availability settings should be a deliberate per-environment decision, and any environment where you can't name who chose the setting is running on drift.

The meta-point: drift is a flow

Fix all three today and they return, because the source of drift is provisioning behavior, not the current inventory: new log groups still default to never-expire, new buckets still ship without lifecycle rules, and the next cloned module still carries prod settings into dev. The durable fixes live at provision time (IaC modules with retention and lifecycle baked in, and a linter that rejects a prod-tier setting in a non-prod path) plus a periodic re-audit, because the fleet you audited is not the fleet you'll have in six months.

FAQ

What is the default CloudWatch log retention?

Never expire. Any log group created without an explicit retention policy keeps its data forever at $0.03 per GB-month, on top of the one-time $0.50 per GB ingestion charge. Setting retention (30 to 90 days for operational logs) is one command per group and existing over-age data ages out automatically.

Do S3 lifecycle policies save money on any bucket?

Not automatically. They pay off on data with a decaying access pattern (logs, backups, artifacts) and can backfire on buckets with millions of tiny objects (per-object transition fees) or short-lived data (minimum storage durations of 30 to 180 days in colder tiers). Check object count and churn before writing the rule; use size filters or aggregate small files first.

Is Intelligent-Tiering better than a lifecycle policy?

It's the right default when access patterns are unknown or mixed: it moves objects automatically for a $0.0025 per 1,000 objects monitoring fee with no retrieval charges between the frequent and infrequent tiers. A hand-written lifecycle rule beats it when the pattern is predictable, and neither replaces deleting data with no retention requirement at all.

How much extra does RDS Multi-AZ cost?

Double, exactly: the standby replica bills the same instance and storage rate as the primary. That's cheap insurance for production and pure waste for dev and staging, which is why the audit runs both directions: single-AZ production databases (risk) and Multi-AZ non-production ones (cost).

How do I keep these settings from drifting again?

Move the defaults into provisioning: IaC modules that set log retention, bucket lifecycle, and per-environment availability explicitly, plus a policy check that fails a plan carrying prod-tier settings into non-prod paths. Then re-audit quarterly, because drift is generated by ongoing provisioning, not by the resources you already fixed.

123 Things Wrong With Your Kubernetes Cluster Right Now: Single Replicas, Pinned HPAs and Missing Probes

Muskan _zop — Fri, 21 Aug 2026 09:16:43 +0000

Here's an uncomfortable exercise: pick any production Kubernetes cluster older than a year and audit every workload against a full checklist of known misconfigurations. The typical result isn't five findings or ten. Across reliability, autoscaling, security, storage, and hygiene rules, multiplied across EKS, GKE, and AKS conventions, a serious rulebook runs to well over a hundred distinct checks, and mature clusters fail dozens of them simultaneously.

None of these failures pages anyone, which is the whole problem. A single-replica deployment is fine until the node drains. A missing readiness probe is fine until the next rollout routes traffic to a pod that isn't ready. An HPA pinned at max is fine until the day it needed to scale and couldn't. Kubernetes misconfigurations don't fail; they wait.

The taxonomy: what a complete audit actually checks

Reliability. Missing resource requests and limits (the scheduler places pods blind, and the OOM killer chooses for you). Single-replica Deployments in production with no HPA (one voluntary node drain is an outage). Degraded workloads running below desired replicas long after rollout completed. HPAs sitting at max replicas (that's not autoscaling anymore; it's a fixed fleet at its ceiling that you believe is elastic). Missing liveness and readiness probes (deadlocked containers never restart; rollouts route to unready pods). Containers running image:latest (unrepeatable deploys, unrewindable rollbacks). Failed Jobs nobody cleaned up or noticed.

Security basics. Privileged containers, containers running as root, host-network pods, ingress without TLS, services exposed wider than intended. Each is one line of YAML away from a finding that ends up in a pentest report.

Idle and orphan. Deployments scaled to zero months ago and never deleted, CronJobs suspended and forgotten (or never successfully scheduled at all), unbound PVCs holding claims against nothing, released PersistentVolumes lingering for weeks, Services whose endpoints are empty because everything behind them is gone. This category is where reliability audit meets cost audit: orphans bill.

Rightsizing. Requests set to double what the workload uses (paying for reserved capacity the scheduler can't give anyone else), and the subtler inverse: autoscalers whose own status reports they're constrained (a scaling-limited condition) while everyone assumes capacity is fine.

Run the multiplication (a rule per workload kind, per failure mode, adapted per provider) and you land in the low hundreds. The specific number in this post's title isn't rhetorical: 123 is a real rulebook's count, and it's worth knowing where it comes from. ZopNight ships 123 Kubernetes workload rules, 41 unique checks run across EKS, GKE, and AKS, covering exactly the categories above; a detail worth copying from how it's built is that rules it can't yet evaluate reliably (like OOM detection needing real time-series) are shipped disabled rather than fired on weak evidence, and its rightsizing rule only flags what the HPA's own status condition reports rather than inventing a utilization window. That's the audit-quality bar whether you buy one or build one: every finding names its evidence, and no rule fires on a guess.

The eight worst offenders, with one-liners

Single-replica production Deployments: kubectl get deploy -A -o json | jq -r '.items[] | select(.spec.replicas==1) | .metadata.namespace + "/" + .metadata.name'
HPAs pinned at max: compare status.currentReplicas to spec.maxReplicas across all HPAs; anything equal for days has hit its ceiling silently. (Also flag minReplicas == maxReplicas, which is an HPA cosplaying as a constant.)
Missing probes: select containers without readinessProbe or livenessProbe; the readiness gap breaks rollouts, the liveness gap immortalizes deadlocks.
Missing requests/limits: select containers with no resources.requests; these pods are invisible to capacity planning and first against the wall in contention.
image:latest: grep pod templates for the tag; every hit is a deploy you can't reproduce and a rollback you can't trust.
CrashLoop and restart counts: sort pods by restartCount; triple digits in a namespace nobody watches is an incident on layaway.
Unbound PVCs and released PVs: kubectl get pvc -A for Pending, kubectl get pv for Released older than a week; both are storage bills with no consumer.
Empty-endpoint Services: Services selecting zero pods, which means something upstream still resolves a name that leads nowhere.

Two habits turn the list from trivia into practice. Gate findings on stability: don't flag a deployment as degraded mid-rollout; check that the rollout completed and the degradation persisted, or the audit cries wolf weekly. And rank by blast radius times likelihood: a single-replica payment service outranks forty missing labels, and an audit that can't rank is a report nobody reads twice.

Why nobody catches these

Each item is too small to page on, too boring for sprint planning, and invisible in the dashboards teams actually watch (which show traffic and latency, not configuration posture). The failure is structural: clusters have continuous deployment and point-in-time review. New workloads land daily; the audit, if it ever happened, happened once. Whatever tooling you use, the fix is the same shape: the checklist has to run on a cadence, against every namespace, with findings that carry evidence, or the number in this post's title just grows back.

FAQ

What are the most dangerous Kubernetes misconfigurations?

By incident frequency: single-replica production workloads (any node drain becomes an outage), missing readiness probes (rollouts route traffic to unready pods), missing resource requests (scheduler places blind, OOM killer decides), and HPAs pinned at their max (elasticity that silently ended). All four are invisible until an ordinary event triggers them.

Why is an HPA at max replicas a problem?

Because it means the autoscaler wanted to scale further and couldn't. You're at a fixed ceiling while believing you're elastic: the next traffic increase has nowhere to go, and the condition doesn't alert by default. Persistent max-pinning means the ceiling is wrong, the sizing is wrong, or the demand changed and nobody was told.

Do missing liveness and readiness probes really matter?

Differently, and yes. Without readiness, Kubernetes routes traffic to pods that aren't ready, which turns every deploy into a brief outage. Without liveness, a deadlocked container simply stays deadlocked forever. The reverse failure exists too: aggressive liveness probes that restart slow-starting containers in a loop, so probes need tuning, not just presence.

How often should a cluster configuration audit run?

Continuously, or at worst weekly. Clusters change daily with every deploy, so a quarterly audit describes a cluster that no longer exists. The practical bar: every new workload gets checked within a day of landing, findings persist with evidence until fixed, and the trend (findings per namespace over time) is visible to the teams that own them.

Are these checks different on EKS vs GKE vs AKS?

The workload-level rules (replicas, probes, requests, image tags) are identical Kubernetes-native checks everywhere. Providers differ at the edges: ingress and TLS conventions, storage classes and volume behavior, and each platform's autoscaling integration, which is why serious rulebooks maintain per-provider variants of the same underlying checklist.