DEV Community: Muskan

Chargeback vs Showback: Building Team-Level Cloud Cost Accountability

Muskan — Tue, 05 May 2026 05:17:34 +0000

Most engineering organizations have dashboards. They have tagging policies. They have monthly cost reports that go out to team leads. And spending keeps climbing.

The problem is not visibility. The problem is that visibility without financial consequence produces awareness, not action. During showback-only programs, teams act on 10-20% of cost recommendations. After chargeback goes live, that number jumps to 40-60%. The difference is not better data. It is whether the number hits the team's budget.

This is the governance layer that sits between "we can see our costs" and "teams actually change how they spend." Chargeback and showback are the two models that bridge that gap. Getting the choice and implementation right determines whether your FinOps program produces reports or produces results.

Why Visibility Alone Doesn't Change Spending Behavior

Every FinOps journey starts with tagging. You enforce cost-center, team, and environment tags. You build dashboards in AWS Cost Explorer or Azure Cost Management. You send weekly digests to engineering leads.

Then nothing changes.

The reason is straightforward. A dashboard that shows "your team spent $47,000 last month" creates awareness. It does not create accountability. No one's budget shrinks. No one's quarterly planning adjusts. The number is informational, not operational.

A financial services firm measured this directly. With showback dashboards alone, they cut AWS spend by 18% in one quarter. That sounds productive until you realize the remaining 82% of waste stayed untouched. The teams that acted were already cost-conscious. The teams that ignored the reports faced no consequences for ignoring them.

The missing piece is a feedback loop that connects cloud spend to team-level financial planning. That feedback loop has two forms: showback and chargeback.

Showback vs Chargeback: What Each Model Actually Does

Showback means teams receive cost reports showing what they consumed. The costs are visible but do not affect team budgets or P&L statements. Think of it as an itemized receipt with no bill attached.

Chargeback means cloud costs are allocated directly to team budgets. The costs reduce available budget, show up in quarterly reviews, and factor into capacity planning. The receipt comes with a bill.

The FinOps Foundation is explicit on this: neither model is inherently more mature than the other. Showback is foundational to every FinOps practice. Chargeback depends on whether your organization has separate P&Ls per team or product line. A company where all engineering runs under one cost center gains little from chargeback mechanics — showback with executive visibility achieves the same behavioral change.

Dimension	Showback	Chargeback
Budget impact	None — informational only	Direct — costs hit team P&L
Behavior change rate	10-20% action on recommendations	40-60% action on recommendations
Data trust requirement	Moderate — directional accuracy sufficient	High — teams will dispute inaccurate charges
Implementation complexity	Low — dashboards and reports	High — allocation rules, GL integration, dispute process
Shared cost handling	Can defer or simplify	Must resolve — every dollar needs an owner
Best fit	Single P&L orgs, early FinOps maturity	Multi-BU orgs with separate budgets

The same financial services firm that saw 18% reduction with showback added chargeback one year later. The additional reduction was 22%. Combined, that is a 40% spend reduction — but the chargeback portion required 12 months of building allocation accuracy and organizational trust first.

The Allocation Problem: Tagging, Shared Costs, and the Unallocated Bucket

Before any cost reaches a team's report, it must be allocated. This is where most chargeback programs stall.

Direct costs are simple. An EC2 instance tagged team:payments costs $420 per month. That $420 goes to the payments team. No ambiguity.

Shared costs are the problem. Your Kubernetes control plane, NAT gateways, enterprise support contract, CI/CD infrastructure, and networking egress serve multiple teams simultaneously. These costs have no single owner and cannot be tagged to one team.

Three allocation methods dominate for shared costs:

Method	How It Works	Accuracy	Overhead	Best When
Even split	Total shared cost divided equally across consuming teams	Low	Minimal	Early maturity, small team count
Proportional split	Allocated by usage proxy — CPU-hours, request count, data volume	High	Significant — requires metering	Teams have measurably different consumption patterns
Fixed proportional	Predetermined percentages, refreshed quarterly	Medium	Low after initial setup	Consumption patterns are relatively stable

The pragmatic guidance from FinOps practitioners: not every shared cost needs allocation. Platform team salaries, enterprise support contracts, and security tooling often belong in a central overhead pool. Allocating them to product teams creates complexity without changing behavior because no team can reduce those costs through their own actions.

The danger is the unallocated bucket. When shared costs are poorly defined, teams learn to shift spend toward untagged or shared categories. The unallocated pool becomes a dumping ground. A telecom provider discovered this pattern when one microservice accounted for 40% of data transfer costs — costs that had been sitting in the "shared networking" bucket for months. Identifying and reassigning that cost saved $45,000 per month.

Target tagging compliance of 85-90% overall and 95%+ for production resources before activating chargeback. With approximately 32% of cloud spend sitting on improperly tagged resources industry-wide, most organizations need 2-3 months of tagging enforcement before the data is trustworthy enough.

The Crawl-Walk-Run Implementation Path

Deploying chargeback on day one is a recipe for organizational friction. The phased approach works because each stage builds the data accuracy and organizational trust required for the next.

Crawl (months 1-3) focuses on data foundation. Enforce tagging standards using AWS SCPs, Azure Policy, or GCP Organization Policies. Map every cost center to an owning team. Identify which costs are direct, which are shared, and which will remain centrally absorbed. The exit criterion: 85%+ tagging compliance across all accounts.

Walk (months 4-6) activates showback. Teams receive weekly cost reports with line-item visibility. This is where data trust gets tested. Expect disputes. Establish a clear dispute process — a shared channel or ticketing queue where teams can flag allocations they believe are incorrect. Resolve disputes within 48 hours. The exit criterion: dispute rate below 5% of total allocations.

Run (months 7-12) transitions to chargeback. Costs now hit team budgets. Quarterly allocation reviews ensure the model stays accurate as team structures and consumption patterns shift. Automation enforces tagging compliance and flags untagged resources before they enter the billing cycle.

The financial impact compounds. Organizations using mature allocation models report 25% better cost optimization outcomes and 40% more accurate departmental budgeting compared to ad-hoc tracking.

Five Failure Modes That Kill Chargeback Programs

Every failure mode below has appeared in production. Knowing them upfront saves months of organizational friction.

Failure Mode	Symptom	Fix
Data trust collapse	Every review meeting starts with "where did this number come from?"	Invest in tagging compliance first; publish methodology documentation; allow 90-day showback period before chargeback
Allocation driver gaming	Teams restructure workloads to minimize their allocation metric rather than actual cost	Audit allocation drivers quarterly; use multiple weighted drivers rather than a single metric
Surprise bills without buy-in	Business units feel ambushed by charges they never agreed to	Socialize the model 60 days before activation; get VP-level sign-off per business unit
Growing unallocated bucket	Shared cost pool increases quarter over quarter as teams dodge attribution	Cap unallocated at 15% of total spend; flag any resource without an owner within 7 days
No automation	Manual tagging, manual reports, manual allocation — the model works for 3 months then collapses	Automate tag enforcement via policy engines; automate cost pipeline from export through report delivery

The most common killer is data trust. When teams cannot trace a charge back to a specific resource, they reject the entire model. This is why the showback phase matters — it builds trust in the allocation methodology before money moves.

Building the Allocation Pipeline on AWS, Azure, and GCP

The allocation pipeline follows four stages regardless of cloud provider: export raw billing data, normalize it into a common schema, apply allocation rules, and post results to financial systems.

AWS provides Cost Categories for rule-based grouping and the Cost and Usage Report (CUR 2.0) for raw data export to S3. Cost Categories handle direct allocation well but require custom logic for proportional shared cost splits. The CUR is the standard data source for any serious allocation pipeline.

Azure offers Cost Management with a cost allocation feature that can redistribute shared subscription costs to other subscriptions. It handles basic showback natively. For chargeback, you will need Azure Exports to a storage account and downstream processing for complex allocation rules.

GCP exports detailed billing records to BigQuery, which means your allocation logic can run as SQL queries. Labels must be applied at resource creation — there is no retroactive labeling. Budget alerts are per-project or per-label but are alerting-only with no enforcement.

All three providers support the FOCUS 1.3 specification, which introduces allocation-specific columns that standardize how costs are split across workloads. If you operate multi-cloud, normalizing to FOCUS format before applying allocation rules eliminates provider-specific transformation logic.

The gap across all three: none of them solve the shared cost problem natively. Proportional allocation of Kubernetes cluster costs, networking egress, or platform team infrastructure requires custom logic — whether that is SQL in BigQuery, Python processing CUR files, or a dedicated FinOps tool.

Chargeback and showback are not reporting features. They are governance mechanisms that connect cloud spend to the teams that control it. Start with showback to build data trust. Graduate to chargeback when your tagging compliance exceeds 85% and your dispute rate drops below 5%. Automate everything between the cloud bill and the team budget. The organizations that treat cost allocation as an engineering problem — not a finance problem — are the ones that actually change spending behavior.

S3 Storage Class Automation: Stop Paying Hot Prices for Cold Data

Muskan — Tue, 05 May 2026 05:16:38 +0000

A typical production S3 bucket at 18 months old has accumulated objects across every feature that ever ran. The initial uploads from your onboarding pipeline. The exports from the analytics job that ran twice and was deprecated. The thumbnails from the old image processing service. The log archives from before you switched to CloudWatch.

Run S3 Inventory on a bucket that has been active for a year and the pattern is always the same: 70-80% of objects have a last-accessed date older than 90 days. Most of them have never been read after the day they were written.

Every one of those objects is sitting in S3 Standard at $0.023 per GB per month. That is the default. AWS does not move them for you.

At 50TB of storage, the gap between Standard pricing and what you should be paying for cold data is roughly $800 per month — $9,600 per year — for a single bucket. Teams with hundreds of buckets multiply that number accordingly.

The Per-GB Math Across All Six Storage Classes

AWS offers six distinct S3 storage classes in us-east-1. The price spread from Standard to Deep Archive is 23x. That spread is the opportunity.

Storage Class	Storage (per GB/month)	Retrieval (per GB)	Min Duration	Min Object Size	Retrieval Latency
Standard	0.023	0	None	None	Milliseconds
Standard-IA	0.0125	0.01	30 days	128 KB	Milliseconds
One Zone-IA	0.01	0.01	30 days	128 KB	Milliseconds
Glacier Instant	0.004	0.03	90 days	128 KB	Milliseconds
Glacier Flexible	0.0036	0.01 (standard)	90 days	40 KB	3-5 hours
Deep Archive	0.00099	0.02	180 days	40 KB	12 hours

The retrieval cost column is where most teams get surprised. Standard-IA looks like a 46% discount over Standard until you read the line that says $0.01 per GB to retrieve. For data accessed once per month at 10TB, that retrieval cost adds $100 back each month — nearly erasing the storage savings.

The math on a 1TB bucket with no retrievals and data older than 90 days:

Standard: $23.55/month
Standard-IA: $12.80/month (save $10.75)
Glacier Instant: $4.10/month (save $19.45)
Deep Archive: $1.01/month (save $22.54)

Intelligent-Tiering: When It Wins and When It Costs You More

Intelligent-Tiering automates transitions between Frequent Access and Infrequent Access tiers based on access patterns. AWS charges $0.0025 per 1,000 objects per month as a monitoring fee — whether or not any tiering occurs.

At 1 million objects: $2.50/month monitoring. At 10 million objects: $25/month. At 100 million objects: $250/month. That fee has to be recouped by IA transition savings.

When IT wins: For a 1MB object going inactive after 30 days, the monthly savings from IA transition is $0.0000105 per object. The monitoring fee per object is $0.0000025. The monitoring fee is recouped in 0.24 months. IT wins clearly for objects larger than 128KB accessed less than once per month.

When IT loses: 50 million objects averaging 10KB. Monthly monitoring fee: $125. Monthly storage in Standard: ~476GB × $0.023 = $10.95. You are paying $125 in monitoring fees on an $11 storage bill — an 11x overhead. Use a lifecycle rule to Standard-IA instead.

The opt-in Archive Access tier activates after 90 consecutive days of inactivity at $0.0045/GB. The Deep Archive Access tier activates after 180 days at $0.00099/GB. Both require activation on the bucket configuration — they are off by default.

Lifecycle Policy Design: The 30/90/180 Framework

Day	Transition	Storage Class	Storage Cost	Retrieval Latency
0-30	None	Standard	0.023/GB	Milliseconds
30	Transition	Standard-IA	0.0125/GB	Milliseconds
90	Transition	Glacier Instant	0.004/GB	Milliseconds
180	Transition	Glacier Flexible	0.0036/GB	3-5 hours
365	Transition	Deep Archive	0.00099/GB	12 hours
2555	Expire	—	0	—

The structure of your S3 prefixes determines whether these rules work. A lifecycle rule applied to the bucket root will transition everything uniformly. If uploads/ contains objects from yesterday alongside objects from two years ago, a 30-day transition rule sweeps both. Separate prefixes by access pattern before writing any lifecycle rules.

Before writing any rule, define the maximum acceptable restore time for each prefix. That constraint — not cost optimization — sets the floor on how deep you can go.

Automating With Guardrails: Storage Lens, Inventory, and Policy Gates

S3 Inventory delivers daily or weekly CSV/Parquet reports per bucket. Each row contains object key, size, storage class, last-modified date. This is the raw material for every cost decision.

Query the Inventory output with Athena: segment objects by storage class and last-modified age. A bucket with 500GB in Standard where 80% of objects have a last-modified date older than 60 days is an immediate transition candidate.

S3 Storage Lens (advanced tier, $0.20 per million objects) shows per-prefix GET and HEAD request rates. A prefix with 200GB and zero GET requests in 30 days: transition to IA immediately. A prefix with daily GET traffic: exclude from all lifecycle rules.

The guardrail is a tag-based override. Any object tagged lifecycle-exempt: true is excluded from all transition rules. Application teams use this for objects that must remain in Standard — primary database backups, active config files, test seed data.

Failure Modes That Will Erase Your Savings

Failure Mode	Root Cause	Dollar Cost Example	Guard Rule
Small objects in Intelligent-Tiering	Objects under 128KB pay monitoring fee with no IA benefit	50M objects at 10KB = $125/month monitoring, $0 savings	Exclude IT from buckets where avg object size < 128KB
Minimum duration charges	Object transitioned to Glacier, deleted before 90-day minimum	100GB deleted at day 10 = 80 extra days × $0.004/GB = $10.24 extra	Set lifecycle rule expiry ≥ minimum duration of target class
Retrieval cost surprise	Bulk restore of large dataset not costed before triggering	10TB restore from Glacier Flexible = $100 retrieval	Require cost approval for any restore above 100GB
Rule at wrong prefix	Hot uploads/ prefix shares root-level rule with cold archive	Recent objects transition to IA at 30 days, causing retrieval fees on every read	Always scope rules to specific prefixes, never bucket root

The retrieval cost calculation is often skipped. Glacier Flexible expedited retrievals cost $0.03/GB plus $0.01 per request. A 50TB archive costs $1,500 in one expedited restore — that erases 8 months of storage savings in a single incident.

Glacier Flexible and Deep Archive are appropriate only when: the acceptable restore SLA is hours or days, restores happen at most once per year, and object lifetime is long enough to amortize minimum duration charges. Everything else belongs in Standard-IA or Glacier Instant.

Run S3 Inventory every 30 days after applying lifecycle rules. If Standard-IA objects are accumulating faster than expected or objects appear in Glacier with last-modified dates more recent than your transition window, a rule is misconfigured. Catch it before minimum duration charges compound.

The 23x price gap between Standard and Deep Archive exists because AWS prices access and durability separately. Most teams leave it on the table by never looking at what their data actually costs. S3 Inventory takes 24 hours to run. The lifetime of a well-designed lifecycle policy is years. The arithmetic on 50TB at $800/month savings is $9,600 per year — and that is one bucket.

The Real Cost of a Service Mesh: Istio Sidecar Overhead in Production

Muskan — Tue, 05 May 2026 05:15:09 +0000

Istio does not appear on your infrastructure budget as a line item. It appears as a gradual expansion of your node count, an unexplained increase in CPU utilization across the cluster, and a growing gap between what your application pods request and what nodes actually deliver.

The mechanism is the sidecar. Every pod in an Istio mesh gets an Envoy proxy injected at admission. That proxy handles mTLS termination, telemetry collection, and traffic management. At idle, it consumes 50-100 millicores of CPU and 50-100MiB of memory per pod. Under load it consumes more.

At 10 pods, the overhead is noise. At 100 pods, it is 10 extra CPU cores running 24/7. At 500 pods, it is a dedicated node tier — infrastructure you are paying for but not using for your application.

The Overhead Math at Scale

The numbers per pod at idle, measured from production Istio 1.20 deployments:

Envoy sidecar CPU: 50-100m (request), 200-500m (under traffic load)
Envoy sidecar memory: 50-100MiB (idle), 150-300MiB (under load)
istiod control plane: 500m CPU, 2GiB memory (base), scales with mesh size
Latency per hop: 1-3ms added per service-to-service call

Total overhead across cluster sizes:

Pod Count	Sidecar CPU (idle)	Sidecar Memory (idle)	Equivalent Nodes (m5.xlarge)	Annual Cost (us-east-1)
10 pods	0.75 cores	750MiB	0.2 nodes	350 USD
50 pods	3.75 cores	3.75GiB	1 node	1,750 USD
100 pods	7.5 cores	7.5GiB	2 nodes	3,504 USD
250 pods	18.75 cores	18.75GiB	5 nodes	8,760 USD
500 pods	37.5 cores	37.5GiB	10 nodes	17,520 USD

These numbers assume 75m CPU and 75MiB memory per sidecar at idle, plus istiod at 500m CPU and 2GiB. Node cost based on m5.xlarge at $0.192/hr. Real clusters running under traffic will see 2-3x these CPU numbers during peak load.

The control plane does not scale linearly. istiod's resource consumption grows with the number of services, endpoints, and configuration changes pushed to Envoy sidecars. A mesh with 500 pods across 100 services will see istiod consuming 2-4 cores and 4-8GiB under active configuration changes — certificate rotation, service discovery updates, traffic policy changes.

What You Actually Get for the Overhead

The question is not whether Istio costs resources. It does. The question is whether the features you use justify those resources.

Feature	What It Provides	Teams That Need It	Teams That Don't
mTLS	Encrypted, authenticated pod-to-pod traffic	PCI-DSS, SOC2, HIPAA environments	Internal clusters with no compliance requirement
L7 observability	Per-service latency, error rate, throughput via Prometheus/Jaeger	Teams without existing APM tooling	Teams already running Datadog, New Relic, or similar
Traffic shifting	Canary deployments, A/B testing at the mesh layer	Teams doing frequent blue/green releases	Teams deploying once per sprint to stable endpoints
Circuit breaking	Automatic fail-open when downstream services degrade	Microservice architectures with complex dependency chains	Monoliths, small service counts
Fault injection	Testing failure modes by injecting delays and errors into traffic	SRE teams running chaos engineering	Teams without active failure testing programs

The honest audit: most teams use mTLS and L7 metrics. Traffic shifting is used occasionally. Circuit breaking is configured but rarely tuned. Fault injection is almost never used in production.

If your actual usage is mTLS plus basic metrics, there are lighter paths to both of those features than running a full sidecar mesh.

Alternatives: Cilium, Ambient Mesh, and No Mesh

Option	Overhead Per Pod	mTLS	L7 Observability	Traffic Shifting	When to Choose
Istio Sidecar	50-100m CPU, 50-100MiB	Yes	Full	Full	Full L7 features needed, compliance requires it
Istio Ambient Mesh	0 per pod (node-level ztunnel)	Yes	L4 by default, L7 opt-in	Limited	mTLS required, want to eliminate per-pod overhead
Cilium eBPF	~5m CPU per pod	Yes (WireGuard)	L4 + limited L7	Basic	CNI already Cilium, want encryption without sidecars
No mesh + mTLS at app layer	0	App-managed	App APM only	App-managed	Small service count, low compliance requirement

Istio Ambient Mesh is the architectural shift that eliminates sidecar injection entirely. Instead of a proxy per pod, ambient mesh uses a per-node ztunnel process for L4 mTLS and an optional waypoint proxy per service account for L7 features. Memory footprint drops from 25-30GiB across a 100-pod cluster to 3-4GiB. The waypoint proxy adds overhead only on services that need L7 features, not on every pod by default. Ambient mesh reached stable in Istio 1.22.

Cilium eBPF enforces network policy and provides encryption at the kernel level using eBPF programs rather than userspace proxies. If Cilium is already your CNI, you already have most of what Istio's sidecar provides for network security. Adding WireGuard encryption to Cilium costs approximately 5m CPU per pod — a 10-15x reduction from Envoy sidecar overhead. L7 observability is more limited than Istio's, but for teams using an external APM the gap is not visible.

When the Tradeoff Is Worth It vs When to Skip

Workload Profile	Recommendation	Reason
100+ microservices, PCI/SOC2 required	Istio sidecar or Ambient	Compliance mandates encryption in transit; L7 observability reduces MTTR
20-50 services, no compliance mandate	Cilium eBPF or Ambient	Gets mTLS without per-pod sidecar cost; ambient is lower overhead
5-15 services, monolith-adjacent	No mesh	Service count too low for mesh overhead to be justified; mTLS at app layer
Batch / ML workloads	No sidecar injection	Sidecars add fixed overhead to pods that run for minutes; benefit near zero
Dev / staging namespaces	Disable injection	Dev workloads do not need mTLS; saving 75m CPU per dev pod adds up

The compliance mandate is the clearest decision signal. If a security audit requires encryption in transit between services and you cannot implement it at the application layer, you need a mesh. The choice between sidecar and ambient is then a cost question, and ambient wins on that question for most new deployments.

If there is no compliance mandate and your team is running APM tooling that already provides service-level metrics, Istio's L7 observability is redundant. The sidecar overhead is paying for a feature that does not add information you do not already have.

Right-Sizing If You Keep Istio

Three changes reduce sidecar overhead without removing the mesh:

Set resource limits on sidecars. By default, Envoy sidecars have no CPU or memory limits. Set them via MeshConfig.defaultConfig.resources: request 50m CPU and 64MiB memory, limit 200m CPU and 256MiB. Sidecars will be throttled if they try to consume more. This prevents a traffic spike from driving sidecar CPU to 2 cores per pod.

Disable injection on namespaces that do not need it. Dev, staging, and batch namespaces can opt out with istio-injection: disabled on the namespace. This eliminates sidecar overhead for workloads where mTLS provides no compliance value.

Disable unused features. If you are not using traffic shifting, set PILOT_ENABLE_VIRTUALSERVICE_DELEGATE=false and remove all VirtualService and DestinationRule resources that are not actively used. Istio still pushes configuration changes to every Envoy sidecar when any xDS resource changes — reducing the number of resources reduces control plane churn and sidecar memory.

The sidecar tax is real and it scales with your pod count. At 100 pods you are running 10 extra CPU cores to support the mesh. That cost is justified if mTLS compliance, L7 observability, or traffic shifting are delivering value your alternative tools cannot. It is not justified if the mesh was installed because it seemed like a good idea and has been running on defaults ever since. Audit what you actually use, compare it against what ambient mesh or Cilium eBPF can provide, and decide whether the overhead is earning its keep.

Closed-Loop FinOps: Detect, Decide, Act, Verify in 5 Minutes

Muskan — Tue, 05 May 2026 05:12:54 +0000

Closed-Loop FinOps: Detect, Decide, Act, Verify in 5 Minutes

A FinOps team produces a recommendation report on Monday morning. It identifies $185,000 of monthly waste across 240 cloud resources. By Friday, 12 of those 240 are remediated. By the end of week 4, another 6. By month 3, the remaining 222 have been quietly dropped, because the engineer who would have owned each fix has shipped two sprints of features since the report was generated. The recommendation isn't wrong. The handoff is broken.

This is not a tooling problem. It is a process problem with a predictable decay curve. 30% action rate in week 1, 5% by week 4, effectively 0% by month 3 on the same recommendations. The fix is structural: close the loop. Detection feeds decision feeds action feeds verification, all under 5 minutes, with no human in the critical path for low-blast-radius remediations.

Why Reports Don't Save Money

The action-rate decay curve is the central problem. A typical recommendation sits in a backlog while the engineer who would address it ships features, attends incidents, and forgets the original context.

Time since report	Typical action rate	What's happening
Week 1	30%	Report fresh; easy ones get done first
Week 2-3	8%	Sprint pressure crowds out non-urgent work
Week 4	5%	Original context cold; engineer not sure why this was flagged
Month 2-3	<2%	Recommendation effectively dead; new report supersedes it

The decay is not laziness. It is the cost of context-switching. Reading a recommendation, verifying it still applies, mapping it to the team that owns the resource, opening a ticket, scheduling the change, executing, and verifying takes 30-90 minutes per recommendation. Multiplied across 240 recommendations, that is 120-360 engineer-hours of work that nobody has on their calendar.

The closed-loop alternative collapses the same workflow into 5 minutes by eliminating context-switching for the safe-tier remediations. The report-and-ticket flow stays in place for the human-tier work. The middle tier (approval-required) keeps a human in the loop but pre-fills the context so the decision takes 30 seconds instead of 30 minutes.

This pattern works when the safe-tier classification is conservative enough that nobody fears the auto-action. It breaks when the classification is sloppy and the loop touches resources it shouldn't, because one bad auto-action damages trust for the next twenty good ones.

The Four-Stage Pipeline

The architecture has four stages with explicit contracts. Each stage has a specific input shape, a specific output shape, and a specific failure mode. The end-to-end target for the safe tier is under 5 minutes from detection to verification complete.

The signal flowing through the pipeline carries the resource ID, the proposed change, the classification tier, the snapshot of pre-state, and a reverse-action definition. A row in this signal is everything needed to execute, verify, and roll back. Every stage either advances the signal or kicks it back with a reason.

Detection: Anomaly + Threshold + Drift

Three input streams feed the loop. Each has a different latency, false-positive rate, and waste pattern it catches.

Detection method	Latency	False positive rate	Catches
Threshold rules (Cloud Custodian, AWS Config)	Minutes	Low	Known waste patterns: idle resources, missing tags, oversized instances
Anomaly detection (Datadog Cost, OpenCost)	Hours	Medium	Sudden spikes, behavior changes, runaway workloads
Drift detection (Terraform refresh, AWS Config)	Hours-days	Low	One-off manual changes that bypass IaC

Cloud Custodian is the most-adopted open-source policy-as-code engine for AWS / Azure / GCP cost remediation. Policies are YAML, run on a schedule, and support modes: report-only, notify, action. Most teams stop at notify; the productivity gain is in switching select policies to action with a defined blast-radius classification.

False positives go to the same queue but get a "needs review" tag. Novel anomalies (not seen in the last 30 days) automatically classify as approval-required, never as auto-safe. This is how the loop tolerates noisy detection without breaking trust: detection precision is fine; classification is what protects production.

Decision: Blast-Radius Classification

The safety architecture has three tiers with clear membership criteria. The Decide stage is a policy-as-code engine evaluating each signal and routing to the right action path.

Tier	Coverage	Examples	Action
Auto-safe	70-80% of value	Idle non-prod termination, log retention reduction, disk class downgrade with rollback	Execute without human approval
Approval-required	15-20% of value	Production VM right-size, reserved instance purchase, schedule change	Pre-filled ticket; one-click approve
Human-only	5-10% of value	Architecture changes, multi-tenant resource modifications	Report and route to owner

Open Policy Agent Rego rules encode the classification declaratively. A rule like auto-allow termination of non-prod resources older than 30 days with no traffic in last 7 days executes deterministically every cycle without re-asking humans. The Rego rule is the source of truth for what counts as auto-safe.

[diagram could not be rendered]

The classification rules need to be reviewed quarterly. Workloads change, new resource types appear, and the line between auto-safe and approval-required moves. Treating the Rego rules as code (versioned, tested, reviewed) is the only sustainable model.

Action: Idempotent Automation

The Act stage executes the change. The technical floor is idempotency: running the same action twice produces the same result as running it once. Without idempotency, retries amplify rather than recover. With idempotency, the loop tolerates network failures, partial executions, and operator restarts.

Idempotent automation has three preconditions. The source of truth (Terraform / Pulumi / kubectl) is updated, not the live resource directly. The action records a snapshot ID and a reverse-action definition before executing. The action is wrapped in a verification check that confirms the resource state matches expectation post-execution.

The wrapper layer that handles snapshot/reverse-action is the operational glue that makes auto-action defensible. Without it, "we made the change" is a leap of faith. With it, "we made the change, here is the snapshot ID to roll back, here is the reverse-action definition" is auditable.

Verification and Rollback

Verification compares the metric the change was meant to affect (cost, utilization, response time) over a 5-15 minute post-action window. A statistically significant regression triggers automatic rollback. The window length is workload-specific.

Workload type	Verification window	Success criteria	Rollback timeout
Stateless service (right-size)	5-10 minutes	p95 latency unchanged, error rate unchanged	<60 seconds
Batch job (downgrade compute)	15-30 minutes	Job completion time within 1.2x baseline	<5 minutes
Stateful system (storage class change)	30-60 minutes	Read latency unchanged, no replication lag	<15 minutes
Cost-only (log retention reduction)	24 hours	No incident reports requiring deeper logs	N/A (revert via re-enable)

Rollback is the safety mechanism that makes auto-action acceptable. The pattern: when verification fails, the loop reads the recorded reverse-action and executes it. The rollback path must complete in under 60 seconds for stateless workloads, under 5 minutes for stateful. If rollback itself fails, the on-call gets paged with full context: original signal, action taken, verification failure, rollback failure, current resource state.

This pattern works when there is a clear metric to verify against. It breaks when the change has no measurable signal in the verification window (e.g. cost reduction that takes a billing day to surface), in which case verification has to run on a longer cycle with explicit rollback approval gates rather than auto-rollback.

A 90-Day Closed-Loop Adoption Plan

Closed-loop adoption sequences cleanly. Each phase produces measurable safety wins, and the data from each phase informs the next.

Phase	Weeks	Action	Effort	Verification criterion
Detection-only	1-4	Deploy Cloud Custodian / OpenCost in report-only mode. Build the unified signal queue. Tag every detection with proposed tier and proposed action.	2 engineer-weeks	100% of detections have a tier and a reverse-action recorded
Classification	5-6	Write OPA Rego rules for auto-safe / approval / human tiers. Review with platform team. Deploy in shadow mode (predicts but doesn't act).	1 engineer-week	Shadow predictions match human classification on 95%+ of historical signals
Auto-safe execution	7-10	Turn on action mode for the top 3 auto-safe rules (idle non-prod, log retention, disk class). Verification window per workload. Auto-rollback on regression.	2 engineer-weeks	Zero verified regressions over 14-day rolling window
Approval-required pipeline	11-12	Pre-fill approval tickets with full context (resource, proposed change, snapshot, reverse-action). Slack-bot approve workflow.	1 engineer-week	Median approval-to-action time under 30 minutes
Drift detection layer	13	Add drift detection to fill the gap between known-pattern threshold rules and statistical anomaly detection. Route most drift to approval-required.	3 days	Drift backlog drains within 7 days of detection

A team starting with 240 unaddressed FinOps recommendations typically lands on 0-15 unaddressed at any given time after 90 days, because the auto-safe tier catches 70-80% of value before a human ever sees the signal. The remaining 20-30% flow through the pre-filled approval pipeline in days, not weeks.

To get started, run Cloud Custodian in report-only mode for one week against your production AWS account. The report itself is illuminating: 60-80% of recommendations will be obvious enough to classify auto-safe on the spot. Pair the loop with a chargeback / showback layer so the auto-actions are visible to the teams whose resources they touch, and the recommendation backlog stops growing while you build the rest of the pipeline.

The Egress Bill: Why Your Multi-Region FinOps Plan Misses $40k/Month

Muskan — Tue, 05 May 2026 05:11:40 +0000

The Egress Bill: Why Your Multi-Region FinOps Plan Misses $40k/Month

Multi-region is the architecture default for serious SaaS in 2026. The justification is real: disaster recovery, latency, sovereignty. The cost is hidden in the bill under "data transfer" and "regional traffic" and gets zero attention because no engineer "owns" data transfer the way they own compute or storage.

A multi-region SaaS spending $200,000 per month on AWS pays $30,000-$50,000 of that bill on data transfer alone. Most of it is AZ-to-AZ replication for stateful systems, plus cross-region database read replicas, plus chatty service-to-service calls that should have stayed regional. The compute team optimizes compute. The storage team optimizes storage. The egress bill keeps growing because it sits in the gap between teams.

FinOps is the engineering practice of bringing financial accountability to variable cloud spend by aligning engineering, finance, and product on continuous cost decisions, per the FinOps Foundation. Applied to egress, the practice has four levers: replication topology, service placement, observability routing, and endpoint coverage. This piece covers each in order of typical impact.

Why the Egress Bill Stays Hidden

The egress bill is invisible because no one team owns data transfer. Compute is owned by the team running the service. Storage is owned by the team that picked the bucket. Data transfer is the result of a thousand independent decisions about where services live, how they communicate, and what their replication policies are. Without a single owner, optimization stalls.

The bill grows organically. Adding a region for compliance adds 10-20% to data transfer. Adding a third microservice that calls the existing two crosses an AZ boundary. Adding observability for the new feature ships another 200GB/day to the central region. Each decision is locally rational. Aggregated, they are the line item nobody can explain.

This pattern works when the team has a designated "platform" or "infrastructure" owner for egress. It breaks when egress is treated as an emergent property of architecture decisions, because by the time the bill grows large enough to investigate, the architectural choices that drove it are already deeply embedded.

The Pricing Gradient: Free to Brutal

AWS, GCP, and Azure all use a four-tier model. Movement within the smallest scope (a single AZ) is free or near-free. Each step out — across AZ, across region, across the public internet — adds an order of magnitude to the cost.

Path	AWS	GCP	Azure
Intra-AZ / intra-zone	Free	Free	Free
Cross-AZ / cross-zone (same region)	$0.01/GB each direction	$0.01/GB	$0.01/GB
Cross-region (same continent)	$0.02/GB	$0.02/GB	$0.02/GB
Public internet egress (first 10TB/month)	$0.09/GB	$0.12/GB (premium tier)	$0.087/GB
Public internet egress (>500TB/month)	$0.05/GB	$0.08/GB	$0.05/GB

The same 10TB of monthly traffic costs $0 intra-AZ, $200 cross-AZ round-trip, $200 cross-region one-way, or $900 to the public internet. The architectural decision of where to place a service and which path its traffic takes drives a 100x cost gap on identical bytes.

Replication Topology: Where the Big Money Hides

Cross-AZ and cross-region replication for stateful systems is the largest waste category. Multi-AZ Postgres or MySQL with synchronous replication doubles every write's egress cost. Cross-region read replicas for global services multiply it again.

A 1TB write workload on synchronous multi-AZ Postgres pays $20/month in cross-AZ replication for the second AZ alone. Add a third AZ for the recommended HA pattern, $40/month. Add a cross-region read replica in another continent, $200/month for the cross-continent egress. The same write workload on a single-AZ primary with async cross-AZ replicas pays near zero, at the cost of recovery point objective (RPO) increasing from 0 seconds to roughly 30 seconds during a failover.

Cross-region replicas for analytics workloads are the easiest fix. Real-time replicas only justify themselves for transactional read paths where latency-to-source matters. Analytics workloads accept a one-day staleness for a 70-90% lower egress cost, by replacing real-time replication with S3 Cross-Region Replication snapshots that refresh nightly.

Replication topology	1TB write workload, 2 replica regions, monthly egress
Synchronous multi-region active-active	$400 (two cross-region copies of every write)
Single primary + sync multi-AZ + async cross-region	$40 (cross-AZ only, async cross-region)
Single primary + async multi-AZ + nightly snapshot to other region	$5 (snapshot deltas only)
Single-AZ primary, snapshots only	$0

The right topology depends on RPO, recovery time objective (RTO), and read locality requirements. Most teams default to the most expensive option (active-active sync) because it sounds safest. The right answer for most workloads is the second or third row.

This pattern works when the team can classify workloads by RPO/RTO. It breaks when every workload is treated as critical because nobody wants to be the one who downgrades replication. The fix is an explicit per-workload classification with sign-off from the workload owner.

Service Placement: Topology-Aware Routing

The microservices version of the same problem. A 30-service architecture deployed naively across AZs (Kubernetes scheduling pods wherever capacity exists) generates 5-10TB/month of cross-AZ service-to-service traffic. At $0.02/GB round-trip, that is $100-$200/month, scaling linearly with service count and traffic.

Topology-aware routing keeps 70-80% of traffic intra-AZ at zero cost. The pattern: services prefer in-AZ peers when available, fall through to cross-AZ only when capacity demands. Kubernetes supports this via topologyKeys on services and pod-anti-affinity for replicas.

The trade-off is uneven load distribution across AZs. Topology-aware routing assumes services have enough capacity in each AZ to handle local demand. For services with low replica counts (1-2 replicas total), forcing in-AZ traffic produces hot AZs and idle AZs. The fix is a minimum replica count of 3 (one per AZ) for any service that participates in topology-aware routing.

Architecture	Cross-AZ traffic	Monthly cost (10TB total)
30 services, naive scheduling	50% cross-AZ	$100 cross-AZ
30 services, topology-aware	25% cross-AZ	$50 cross-AZ
30 services, single-AZ deployment	0% cross-AZ	$0 (but no AZ HA)

This pattern works for stateless services where any replica can serve any request. It breaks for sticky-session workloads where requests must land on a specific replica, because topology-aware routing then competes with session affinity.

Observability Routing: The Quiet Egress Tax

Observability data shipping is 30% of egress in many setups, per the FinOps Foundation 2026 observability report. Every region's logs, metrics, and traces ship to a central observability vendor, and most teams never measure how much that costs in egress alone.

The default deployment of any observability agent (Datadog, New Relic, Splunk, Honeycomb) is "ship every event to the vendor's endpoint." If the vendor's region differs from where the workload runs (which it almost always does in multi-region setups), every byte crosses regions. A 5-region deployment shipping 1TB/day of observability data per region to one central vendor region pays $30,000+/month on observability egress alone.

Region-local aggregators (Vector, Fluent Bit, OpenTelemetry Collector) batch, compress, and sample before shipping cross-region. Compression alone saves 60-80% on log data. Sampling traces at 1-5% (instead of the default 100%) saves another order of magnitude. The trade-off is incident-time access to the dropped data, which most teams accept once they see the bill.

This pattern works when alert latency tolerates a 30-60 second batching delay. It breaks for workloads with sub-second alert SLOs that need real-time event streaming, in which case the unbatched cost is justified.

VPC Endpoints, Direct Connect, and the Endpoint Habit

VPC endpoints (AWS PrivateLink) and Cloud Interconnect (GCP) eliminate egress charges for traffic that would otherwise cross the public internet. A workload calling S3 in the same region pays $0.01/GB without an endpoint and $0.00/GB through a Gateway Endpoint. For a 50TB/month S3 access pattern, that is $500/month savings per region with one one-line Terraform configuration.

The five highest-impact endpoints to enable first:

Service	Endpoint type	Why it pays
S3	Gateway Endpoint	Most workloads read 10-100TB/month from S3; endpoint eliminates per-GB charges
DynamoDB	Gateway Endpoint	Same as S3; high-volume table reads stay private
ECR	Interface Endpoint	Pulling container images from ECR otherwise crosses NAT, paying NAT processing + per-GB egress
Secrets Manager / Parameter Store	Interface Endpoint	Every container start fetches secrets; volume is small but path matters for compliance
CloudWatch Logs	Interface Endpoint	Log shipping bytes that would otherwise cross NAT

Direct Connect and Dedicated Interconnect provide flat-rate egress to on-premises networks at $0.02/GB versus $0.09/GB public internet, for workloads with sustained 1Gbps+ throughput. The breakeven is roughly 50TB/month of sustained on-prem-bound traffic, below which the port-hour fees ($0.30/hour for 1Gbps Direct Connect) exceed the per-GB savings.

This pattern works when the team has visibility into per-service egress paths via the AWS Cost and Usage Report or GCP Billing Export. It breaks when nobody has enabled CUR (which most teams haven't, despite it being free since 2024) and the cost data is invisible at the per-service level.

A 60-Day Egress Cost Reduction Plan

Egress optimization sequences cleanly. Each phase produces measurable savings, and the data from each phase informs the next.

Phase	Weeks	Action	Effort	Expected saving
Visibility	1-2	Enable AWS Cost and Usage Report (or GCP Billing Export). Build a dashboard showing top egress sources by service, source region, destination region.	1 engineer-week	0 (data only)
VPC endpoints	3	Enable Gateway Endpoints for S3 + DynamoDB. Enable Interface Endpoints for ECR, Secrets Manager, CloudWatch Logs.	2 days	15-25% on egress for endpoint-eligible traffic
Topology-aware service routing	4-6	Enable Kubernetes topology-aware routing on the top 10 services by cross-AZ traffic. Verify replica counts.	2 weeks	30-50% on cross-AZ service-to-service traffic
Observability aggregator rollout	7-8	Deploy Vector or Fluent Bit as a region-local aggregator. Configure batching, compression, sampling.	1 week	60-80% on observability egress
Replication topology audit	9-10	Classify databases by RPO/RTO. Move analytics replicas from real-time to nightly snapshots. Move non-critical workloads from sync to async multi-AZ.	2 weeks	40-70% on stateful replication egress
Direct Connect evaluation	11-12	If on-prem-bound traffic exceeds 50TB/month sustained, evaluate Direct Connect. Otherwise skip.	1 week + procurement	Variable, only if breakeven is clear

A team starting at $40,000/month in data transfer charges typically lands at $12,000-$18,000 after 60 days. The work is architectural, not new tooling. Each phase is testable in isolation. Most teams do the first three phases and stop, because by that point the egress bill has dropped enough that the marginal effort on the rest doesn't pay back.

To get started, enable the AWS Cost and Usage Report and look at the Data Transfer line item broken down by source region. The largest 3-5 contributors are almost always cross-AZ replication for one major database, observability shipping for one major service, and ECR image pulls during deployments. Any one of those is a 1-week project with a measurable bill reduction. Pair the work with autonomous remediation so the gains hold once attention shifts to the next architectural problem.

LLM FinOps: Per-Feature Cost Attribution and Token Budgets

Muskan — Tue, 05 May 2026 05:10:26 +0000

LLM FinOps: Per-Feature Cost Attribution and Token Budgets

A B2B SaaS product team ships its first AI feature in 2024. By 2026, the same team has 12 AI features in production: summarization, classification, extraction, search, an AI assistant, three flavors of auto-complete, two analytics features, and the chatbot product engineering still calls "the demo" eight months after launch. The Anthropic bill is $48,000 per month — the same kind of black-box cloud bill that plagued infrastructure spend before FinOps. Nobody can tell you what each feature costs.

The CFO asks "what's our AI cost per customer?" The answer that arrives a week later is wrong because nobody had instrumentation in place. The team that shipped the latest feature with a 4,000-token system prompt and 1M monthly requests doesn't realize until the following month that they alone added $12,000 to the bill.

FinOps is the engineering practice of bringing financial accountability to variable cloud spend by aligning engineering, finance, and product on continuous cost decisions, per the FinOps Foundation. Applied to LLM ops, the practice has four levers: tag every call, count tokens authoritatively, aggregate per feature, enforce per-feature budgets. This piece covers each in implementation order.

Why Your AI Bill Is a Black Box

The model pricing structure makes per-feature accounting essential, not optional. The cost gap between flagship and small models is roughly 18-20x per output token. A feature that runs on Opus when Haiku would suffice costs 18x what it should — but you cannot tell which features those are without per-feature attribution.

Model	Input ($/MTok)	Output ($/MTok)	Use case
Claude Opus 4.5	$15	$75	Complex reasoning, long-form generation
Claude Sonnet 4.6	$3	$15	Production default, balanced quality/cost
Claude Haiku 4.5	$0.80	$4	Classification, extraction, structured output
GPT-4 Turbo	$10	$30	Reasoning, complex agents
GPT-3.5 Turbo	$0.50	$1.50	Simple chat, classification

A typical B2B SaaS feature processes 800-2,000 input tokens and produces 200-600 output tokens per request, per Anthropic case studies. The pattern echoes chargeback / showback frameworks used for cloud cost — same accountability problem, new line item. At Sonnet rates, that is $0.0027 to $0.0150 per request. A feature handling 100,000 requests per month costs $270 to $1,500. With 12 such features and uneven distribution, the bill ranges $5,000 to $25,000 per month — and "uneven distribution" is the part you cannot see without attribution.

Tagging at the Call Site: The One Line That Makes Everything Else Possible

Adding a feature_id tag to every LLM call is the architectural decision that determines whether per-feature accounting is possible at all. Adding it from day one is a single line of code at every call site. Adding it retroactively across a 30-feature codebase is a quarter-long migration through 30 different teams' code.

Both major providers accept metadata that flows through to their consoles and to your usage logs. The pattern:

Anthropic accepts a metadata.user_id string up to 256 chars. OpenAI accepts a user parameter up to 64 chars. Both end up in the provider's console and in any logs your wrapper writes. The tag should encode three things: the feature owner, the request ID, and the tenant.

Field	Example	What it enables
feature_id	`summarize_email_v2`	Per-feature monthly roll-up
request_id	`req_2k4a8f9...`	Trace one request through retries, fallbacks
tenant_id	`tenant_acme_corp`	Per-customer cost (essential for unit economics)
model_used	`claude-sonnet-4-6`	Detect when a feature accidentally upgraded model
cached_tokens	`12000`	Track prompt-cache hit rate per feature

This pattern works when the call site is yours to modify. It breaks when LLM calls flow through a third-party SDK that does not expose a metadata pass-through, in which case the wrapper has to be replaced or proxied.

Counting Tokens From Provider Responses, Not Estimates

Estimating tokens with tiktoken or word-count heuristics drifts 5-15% from authoritative billing. The provider response is the truth. Both Anthropic and OpenAI return token counts in every response.

The Anthropic response surfaces response.usage.input_tokens and response.usage.output_tokens. OpenAI returns usage.prompt_tokens, usage.completion_tokens, and usage.total_tokens. Neither charges for tokens you didn't send or receive. Use these values, not estimates.

The usage log table needs the columns to support all the queries you'll want later:

Column	Type	Notes
timestamp	timestamptz	When the call completed
feature_id	text	The tag from the call site
tenant_id	text	Per-customer attribution
request_id	text	Trace through retries / fallback chain
provider	text	`anthropic` / `openai` / `gemini`
model	text	Specific model used (matters for cost rollup)
input_tokens	int	From response.usage
output_tokens	int	From response.usage
cached_input_tokens	int	If prompt caching is on
latency_ms	int	For p50/p95 dashboards
error	text	null on success, error class on failure

This pattern works when every LLM call goes through one wrapper. It breaks when half the codebase calls the SDK directly and half goes through a wrapper, because the direct calls don't end up in the log. The fix is a lint rule that bans direct SDK imports outside the wrapper module.

Model Routing: The 18x Cost Lever Most Teams Skip

The pricing table above shows an 18-20x cost gap between flagship and small models per output token. Most teams default to flagship for everything because they tested with flagship during prototyping. Auditing each feature against the question "does this need flagship-quality output?" typically shows 60-70% of features tolerate the small model.

The small-model-first pattern routes to Haiku, validates the output, falls back to Sonnet only on low-confidence responses.

For a 10:1 success ratio (Haiku handles 10 requests for every 1 that escalates to Sonnet), the blended cost is roughly 1/10th of running Sonnet for everything. The math:

Routing	Cost per 1M requests (avg 1k in / 300 out)
Sonnet only	$7,500
Haiku only	$1,800
Haiku-first, Sonnet fallback (10:1 ratio)	$2,375
Haiku-first, Sonnet fallback (5:1 ratio)	$2,950

Confidence checks are low-cost and feature-specific. For structured extraction, validate the JSON parses and required fields are present. For classification, check the predicted class against an allowlist. For summarization, count output tokens vs input tokens to flag pathological short responses. The validator runs in microseconds; the savings compound.

Feature class	Recommended model	Fallback policy
Structured extraction (JSON, key-value)	Haiku	Sonnet on JSON parse error or missing field
Classification (single label)	Haiku	Sonnet on low-confidence (logprobs / consensus check)
Summarization	Sonnet	Opus on length > 50k input or "complex source" flag
Creative generation	Sonnet	Opus only when explicitly requested
Complex reasoning, agents	Sonnet	Opus per feature decision, not per request
Free-form chat	Sonnet	No fallback (chat tolerates variance)

This pattern works when the low-cost model can handle the majority of inputs. It breaks when the inputs are uniformly hard (every request is genuinely complex), in which case the fallback rate climbs above 50% and the routing overhead exceeds the savings.

Prompt Caching and System Prompt Diet

Two related cost levers on the input side. Anthropic prompt caching charges $1.25/MTok for the initial cache write and $0.30/MTok for cached reads on Sonnet, against the standard $3/MTok input rate. For a 50,000-token system prompt re-used 1,000 times per day:

Setup	Daily input cost	Monthly cost
No caching	$150	$4,500
Cache write once + 999 cached reads	$0.06 + $14.97	$450
Trim system prompt to 12,000 tokens, no cache	$36	$1,080
Trim to 12,000 tokens + cache	$0.015 + $3.59	$108

The system prompt diet matters independently. Most production system prompts are 2-4x larger than necessary because they accumulate examples and policy text over months without anyone removing the redundant ones. Trimming a 4,000-token system prompt to 1,000 tokens for a feature handling 1M requests/month saves $9,000 monthly at Sonnet rates.

Output token cost dominates for most features. Trimming system prompts matters but capping max_tokens and prompting for terser outputs ("respond in 2 sentences", "JSON only, no explanation") usually saves more. A feature averaging 600 output tokens that drops to 300 with a tighter prompt cuts output cost in half — and at $15/MTok output, that is the larger half of the bill.

This pattern works when the system prompt is stable across requests (same examples, same policy text). It breaks when the prompt varies per-request (per-tenant policy injected, retrieved context appended), because cache hits become rare. The fix is to split the prompt into a stable cached prefix and a variable suffix.

Per-Feature Budgets: From Alerting to Enforcement

Daily aggregation rolls up per-feature spend. Alerts fire at 50%, 80%, and 100% of the monthly budget. Most teams stop there. Most teams also have a story about a runaway feature that burned 10x its budget over a weekend before anyone noticed.

The hard stop is a thin gateway. Track cumulative spend per feature_id in Redis. When a request would push a feature over 100% of its monthly budget, return 429 with a clear error message. The product team controls the budget; the gateway controls the kill switch.

The gateway design has to handle a few real-world wrinkles. Per-tenant carve-outs (an enterprise customer paid for higher limits). Burst tolerance (allow 110% on a single day if the monthly budget is on track). Soft-fail (when in doubt, allow the request and alert; do not block on infrastructure failures of the gateway itself). And a clear out-of-band override path for the on-call to lift the cap during legitimate incidents.

This pattern works when the team owns the call path end-to-end. It breaks when a third-party integration calls the LLM directly without going through the gateway, in which case the budget is enforced only on the routes you control.

A 60-Day LLM FinOps Implementation Plan

The implementation sequences cleanly. Each phase produces measurable savings, and the data from each phase informs the next.

Phase	Weeks	Action	Effort	Expected saving
Tag every call	1-2	Add feature_id, request_id, tenant_id, model_used to every LLM call site. Centralize through one wrapper. Lint against direct SDK imports outside the wrapper.	1 engineer-week	0 (visibility only)
Usage logging	2-3	Build the usage_log table. Write one row per LLM call with provider-returned token counts. Daily aggregation by feature_id.	3 days	0 (visibility only)
Per-feature dashboard	3	Surface per-feature daily spend in Slack or BI tool. Identify the top 3 features by spend.	2 days	Sustains future savings via behavior change
Model routing (top 3 features)	4-6	Implement Haiku-first with Sonnet fallback for the top 3 features. Confidence check per feature class.	2 weeks	50-70% on the routed features
Prompt caching	7	Enable Anthropic prompt caching on features with large stable system prompts. Measure cache hit rate.	3 days	70-85% on input cost for cached features
System prompt diet	8	Audit system prompts for redundancy. Trim examples that don't change quality. Cap max_tokens where outputs run long.	1 week	30-50% on input + output cost
Per-feature budgets	9-10	Set monthly budgets per feature based on observed baseline + 20% buffer. Wire alerts at 50/80%. Document override path.	1 week	Bounds runaway costs

A team starting at $48,000/month in LLM spend typically lands at $18,000-$24,000 after 60 days. The work is implementation discipline, not new architecture. Each phase is testable in isolation; each delivers measurable savings; none requires re-platforming.

To get started, audit your top three AI features. Pull the last 30 days of LLM provider usage from your console, identify which features they map to (this part is already painful without tagging), and decide which two could move from Sonnet to Haiku-first routing. The savings show up in week two. Pair the cost work with autonomous remediation so budget overruns trigger automatic gateway adjustments rather than a Sunday-night Slack thread.

Snowflake FinOps: The Compute Credit Trap and How to Stop It

Muskan — Tue, 05 May 2026 05:08:16 +0000

Snowflake FinOps: The Compute Credit Trap and How to Stop It

A LARGE Snowflake warehouse left running 24/7 costs $11,520 per month on AWS Standard Edition, per the Snowflake pricing documentation. Multiply that by the four warehouses a typical data team runs (ETL, dashboards, ad-hoc, ML feature pipelines) and you are at $46,080 per month before storage, before reservations, before anyone has tuned a single query. Most teams pay this number and assume it is the cost of doing data warehousing at scale.

It is not. The same workload, with auto-suspend tuned, warehouses right-sized, multi-cluster scaling capped, and query-level attribution in place, runs $18,000 to $25,000 per month with identical performance for end users. The 60% gap is not technical complexity. It is configuration discipline applied to four specific levers.

FinOps is the engineering practice of bringing financial accountability to variable cloud spend by aligning engineering, finance, and product on continuous cost decisions, per the FinOps Foundation. Applied to Snowflake, the practice has four levers: auto-suspend, warehouse sizing, multi-cluster caps, and query-level attribution. This piece covers each in order of impact.

Why Snowflake Bills Surprise Engineering Teams

The credit model is straightforward in isolation and brutal in aggregate. Warehouses bill per second with a 60-second minimum charge per start. Warehouse size doubles the per-hour credit consumption at every tier. Multi-cluster warehouses bill each running cluster independently. Idle time bills until auto-suspend fires.

Warehouse	Credits/hour	Standard ($2/credit) monthly always-on	Enterprise ($3/credit)	Business Critical ($4/credit)
XSMALL	1	$1,440	$2,160	$2,880
SMALL	2	$2,880	$4,320	$5,760
MEDIUM	4	$5,760	$8,640	$11,520
LARGE	8	$11,520	$17,280	$23,040
XLARGE	16	$23,040	$34,560	$46,080
2XLARGE	32	$46,080	$69,120	$92,160

The multi-cluster multiplier compounds these numbers. A LARGE warehouse with auto-scaling configured to 10 max clusters bills up to $115,200 per month if all 10 clusters run continuously. The default scaling policy adds clusters aggressively and removes them slowly, so most teams pay for clusters that ran briefly during a lunchtime spike and stayed warm for hours afterward.

This pattern works when traffic is genuinely concurrent and bursty. It breaks when "concurrency" actually means "five analysts ran a query in the same 30-minute window," because Snowflake's queue is fast enough to handle that on a single cluster without user-visible latency. The 10-cluster cap was the wrong signal to send.

Auto-Suspend: The Setting That Saves 30% in One Edit

Auto-suspend is the single highest-impact configuration change in Snowflake FinOps. The default timeout is 600 seconds (10 minutes). Idle time is billed until the timeout fires. For a warehouse with 30 idle gaps per day, the default leaves 270 minutes (4.5 hours) of compute on the bill that produced no query work.

Reducing the timeout to 60 seconds captures most of those minutes back. The cost is a 1-3 second warm-start delay on the first query after a suspend. For analytical workloads (dashboards, ad-hoc queries, BI tools), users do not notice the warm-start. For latency-sensitive serving (a few production lookups via Snowflake), the longer timeout is justified.

Workload type	Recommended auto-suspend	Why
Ad-hoc analytics, BI dashboards	60 seconds	Idle gaps are 5-30 minutes between queries; warm-start is invisible
ETL / batch transforms	30 seconds	Jobs run end-to-end; nothing happens between runs
ML feature pipelines	60 seconds	Scheduled runs with predictable gaps
Production lookup serving	5-10 minutes	Warm-start latency hurts SLO; tolerate higher idle bill
Dev / sandbox warehouses	30 seconds	Queries are sporadic; nobody cares about warm-start

The math is simple. A LARGE warehouse with 30 idle gaps per day at 8 credits per hour costs $0.13 per minute. Default 600s timeout pays for 9 extra minutes per gap, 270 minutes per day, $35 per day, $1,050 per month, just on idle time that produced no useful work. Across four warehouses, $4,200 per month from one configuration setting.

This pattern works when the warehouse is not feeding a latency-critical serving path. It breaks when a 1-3 second warm-start delay violates an SLO, in which case 5-minute timeouts are the right tradeoff for that specific warehouse.

Right-Sizing Warehouses With Query History

Most teams size their warehouse for the worst query they ever run. They notice a slow ETL job, bump the warehouse from MEDIUM to LARGE, and never revisit the decision. The other 80% of queries on that warehouse run on capacity they do not need.

The fix is data-driven. The ACCOUNT_USAGE.QUERY_HISTORY view records every query with execution time, warehouse size, credits consumed, user, and role. Pulling p50, p95, and p99 query duration over 14 days surfaces the actual sizing decision. The 80/20 split shows up immediately: 80% of queries complete in under 30 seconds, 20% take 5-30 minutes.

The split is a routing decision. BI dashboards and quick analyst queries go to a SMALL warehouse with aggressive auto-suspend. The 20% of long-running ETL or analytical queries go to a separate LARGE warehouse, on demand.

Workload	Old setup	New setup	Monthly compute
One LARGE warehouse for everything (12h/day active)	LARGE @ 12h/day	—	$5,760
Split: SMALL for fast queries (12h/day), LARGE for slow (2h/day)	—	SMALL 12h + LARGE 2h	$2,400
Saving			$3,360 (58%)

The right-sizing process takes one analyst-week. Pull the QUERY_HISTORY data, eyeball the duration histogram, set up the routing, watch QUERY_HISTORY for a week to confirm no regressions. The 50%+ savings on warehouse compute are durable as long as the workload mix does not shift dramatically.

This pattern works when query-routing can be done at the SQL layer (BI tools, dbt, Airflow operators all support warehouse hints). It breaks when the application layer hard-codes a single warehouse name with no override path, because then the routing has to be wired into the connection pool.

Multi-Cluster Scaling: The 10-Cluster Myth

Multi-cluster warehouses solve a real problem: too many concurrent queries queue and slow each other down. The default solution is to set max clusters to 10 and walk away. The bill arrives a month later.

The two scaling policies behave very differently. Standard policy adds a cluster as soon as a query queues, removes a cluster only after a long idle period. Economy policy delays adding a cluster (queries queue briefly first), and removes idle clusters faster. For most analytical workloads, Economy is the right default.

Then there is the max-cluster cap. The right way to set it is to measure peak concurrency from QUERY_HISTORY (group by minute, count distinct queries running, take the p99). Most teams find their actual peak is 3-5 concurrent queries, not 50. Capping max clusters at 3-5 produces the same user experience at a fraction of the cost.

Max clusters	Monthly cost (LARGE, all clusters running 12h/day)	Notes
10 (default-ish)	$57,600	Default if you accept the wizard. Overkill for almost everyone.
5	$28,800	Common right-sizing target after measurement.
3	$17,280	Adequate for most analytics teams under 50 daily users.
1 (multi-cluster off)	$5,760	Right answer when concurrency is below 3 most of the time.

The cap works because Snowflake queues briefly when the cap is hit, and queues clear in seconds for typical query mixes. The user impact is a 1-3 second wait for the 5th simultaneous query, not the 30-second wait many teams fear.

Query-Level Cost Attribution Without a Vendor

Most teams cannot tell you what their queries cost per team. They have one big Snowflake bill and a vague sense of who runs what. Without attribution, there is no per-team budget, no incentive to tune queries, and no signal when one team is burning 70% of credits.

The data is already there. The QUERY_HISTORY view records USER_NAME, ROLE_NAME, WAREHOUSE_NAME, WAREHOUSE_SIZE, EXECUTION_TIME, CREDITS_USED_CLOUD_SERVICES, and BYTES_SCANNED. Joined with a role-to-team mapping table, the query produces per-team cost attribution at query granularity.

The aggregation query takes 50 lines of SQL. Run it daily, post the per-team credit consumption to a Slack channel, and within a quarter the highest-cost teams will tune their own queries because the cost is visible.

Team	Credits/day	Monthly cost (Standard $2/credit)	Top query type
Data Science	280	$16,800	Feature engineering scans
BI / Analytics	95	$5,700	Daily dashboard refresh
ETL / Platform	60	$3,600	Hourly transforms
Product Analytics	35	$2,100	Ad-hoc cohort queries
Engineering (debug)	12	$720	Production data lookups

Resource monitors enforce the budget. Configure SUSPEND on dev warehouses when daily credit budgets are exceeded (zero blast radius). Configure NOTIFY on prod warehouses (alerts to Slack, no kill). Most teams never set these up because they fear killing legitimate workloads. The right pattern is dev = enforce, prod = alert, with weekly review.

Storage Cost: Time Travel, Fail-Safe, and the 21TB Footprint

Compute is 70-90% of Snowflake bills. Storage is the rest, and it is consistently mis-tuned. Time Travel retention is the main lever. Fail-Safe is non-configurable and adds 7 days on top of whatever Time Travel is set to.

A 10TB working set with 90-day Time Travel and Fail-Safe occupies roughly 21TB of storage in practice (the working set, plus 90 days of changes, plus 7 days of Fail-Safe). At $23 per TB per month on AWS, that is $483 per month for storage that mostly stores data no one queries.

Time Travel retention	Effective storage (10TB working set)	Monthly cost (AWS, $23/TB)
1 day (Standard default)	~12TB	$276
7 days	~14TB	$322
30 days	~17TB	$391
90 days (Enterprise max)	~21TB	$483

Most production tables need 7 days of Time Travel. Audit-relevant tables justify 30 days. The 90-day setting exists because someone read the docs, set the maximum, and never returned to the question. Reducing to 7 days saves 33% of storage spend without removing anything that gets used.

Zero Copy Clone is the second storage lever. Cloning a 10TB production database to dev costs zero storage initially (clones are metadata-only) and only diverges as writes happen. Most dev teams instead create full copies, paying for the full 10TB twice. One ALTER DATABASE CLONE statement replaces gigabytes of redundant storage.

A 90-Day Snowflake Cost Reduction Plan

Snowflake cost reduction sequences cleanly. Each phase produces measurable savings, and the data from one phase informs the next.

Phase	Weeks	Action	Effort	Expected saving
Baseline	1-2	Tag every warehouse by workload type. Pull QUERY_HISTORY for 14 days. Compute per-warehouse idle ratio, p95 query duration, peak concurrency.	1 analyst-week	0 (data only)
Auto-suspend	3	Reduce timeout to 60s on analytical warehouses, 30s on ETL/dev.	1 day	25-35% on idle warehouse cost
Workload routing	4-6	Split fast vs slow queries. SMALL warehouse for 80%, LARGE for 20%. Update BI tool / dbt / Airflow to use right warehouse per workload.	2 weeks	40-50% on warehouse compute
Multi-cluster cap	7	Switch to Economy policy. Cap max clusters at p99 measured concurrency.	2 days	30-50% on multi-cluster overhead
Query attribution	8-9	Build daily aggregation joining QUERY_HISTORY with role mapping. Post per-team credit consumption to Slack.	1 week	Sustains future savings via behavior change
Resource monitors	10	SUSPEND on dev, NOTIFY on prod with weekly budget review.	2 days	Bounds runaway costs
Storage retention	11	Reduce Time Travel to 7 days on most tables, 30 days on audit. Adopt Zero Copy Clone for dev.	1 week	30-50% on storage cost
Reservation evaluation	12	If 60%+ of compute is steady-state, evaluate Capacity Pre-Purchase.	2 days + procurement	25-40% on baseline compute

A team starting at $50,000 per month in Snowflake spend typically lands at $20,000-$28,000 after 90 days. The work is configuration discipline, not a re-platforming. Each phase is reversible if the savings come at a real performance cost. Most do not.

To get started, pull QUERY_HISTORY for the last 14 days from your busiest warehouse. Compute average idle ratio, p95 query duration, and per-team credit consumption. The numbers will surface the highest-impact fix specific to your workload, which is almost always either auto-suspend or warehouse right-sizing. Pair the reduction work with autonomous remediation so the 90-day savings hold once attention shifts elsewhere.

Kubernetes Multi-Tenancy: Namespace Isolation, RBAC, and Network Policies Explained

Muskan — Mon, 04 May 2026 09:14:44 +0000

Kubernetes Multi-Tenancy: Namespace Isolation, RBAC, and Network Policies Explained

Most teams running shared Kubernetes clusters believe they have isolation. They have namespaces. They have different teams deploying to different namespaces. It feels like separation. It is not.

Kubernetes was designed as a single-tenant system. Multi-tenancy is not a built-in feature. It is a property you construct by layering four controls: namespace scoping, RBAC, network policies, and resource quotas. Miss any one of them, and you do not have multi-tenancy. You have an illusion of it.

This post covers each layer specifically: what it enforces, what it does not, and how to configure it correctly.

Multi-Tenancy in Kubernetes Is Not a Feature: It's a Configuration Problem

Out of the box, a Kubernetes cluster has no tenant separation. Every pod can reach every other pod on any port. Any service account token can be used to query the API server. No namespace has a CPU or memory cap. A single misconfigured workload can OOM a node and take down unrelated services running next to it.

The hardened state requires explicit work. That work has four parts.

Namespaces: Logical Separation With No Security Enforcement

A namespace is an API boundary. It scopes names: two services named api can coexist if they are in different namespaces. It scopes RBAC: a RoleBinding in namespace team-a does not grant access to objects in team-b. It scopes ResourceQuota objects.

That is the full extent of what namespaces enforce.

Namespaces do not restrict network traffic. A pod in team-a can send HTTP requests to a pod in team-b with no configuration required. There is no wall between namespaces at the network layer. The Kubernetes scheduler will also happily place pods from different namespaces on the same node, sharing CPU and memory.

This is the most common misunderstanding we encounter in production cluster reviews. Teams assume namespace separation means network separation. It does not. You need network policies for that.

RBAC Done Right: Least Privilege for Teams, Pipelines, and Operators

Kubernetes RBAC has two scopes: namespace (Role + RoleBinding) and cluster (ClusterRole + ClusterRoleBinding). The distinction matters more than most teams realize.

A ClusterRole bound via ClusterRoleBinding grants access to all namespaces and all cluster-level objects. This is appropriate for platform operators who manage the cluster itself. It is not appropriate for application teams, CI/CD pipelines, or monitoring agents.

RBAC evaluation uses OR logic. If a subject has two RoleBindings and one of them grants pods/exec, the subject can exec into pods even if the other binding does not allow it. There is no way to subtract permissions. Over-permissioning is additive and irreversible without deleting bindings.

Persona	Role Type	Verbs	Scope
Dev team	Role	get, list, create, update, delete on pods/deployments/services	Their namespace only
CI/CD pipeline	Role	get, create, update on deployments/configmaps	Target namespace only
Platform operator	ClusterRole	All verbs, all resources	Cluster-wide
Read-only auditor	Role	get, list, watch on all resources	Specific namespace
Monitoring agent	ClusterRole	get, list, watch on pods/nodes/metrics	Cluster-wide (read-only)

One specific footgun: every pod gets a service account token mounted at /var/run/secrets/kubernetes.io/serviceaccount/token by default. This token can authenticate to the API server. In a default cluster, that token has enough permissions to list pods and services across namespaces. Set automountServiceAccountToken: false on service accounts that do not need API access. That covers most application workloads.

For CI/CD pipelines, create a dedicated service account per namespace with exactly the verbs needed to update deployments. No get on secrets. No exec. No portforward. The pipeline does not need those and they should not have them.

Network Policies: Default Deny First, Then Allow What You Need

A NetworkPolicy is enforced by the CNI plugin. This is the first thing to verify: not all CNI plugins support NetworkPolicy objects. Flannel does not. Kubenet does not. If you are running either of those, NetworkPolicy objects are silently ignored. They exist in etcd, kubectl get networkpolicies returns them, but no traffic is actually blocked.

CNI Plugin	NetworkPolicy Support
Calico	Full support, including global policies
Cilium	Full support, plus L7 HTTP policies
Weave Net	Full support
Flannel	No support
kubenet (GKE basic)	No support
AWS VPC CNI	Supported via Network Policy Controller add-on

The correct pattern is default-deny-all applied at namespace creation, then explicit allow rules for each communication path.

The default-deny NetworkPolicy for a namespace looks like this in concept: it selects all pods in the namespace (podSelector: {}) and specifies no ingress rules. Because there are no rules, no ingress is permitted. Then you add explicit allow rules as separate NetworkPolicy objects, one per communication path.

This approach is additive. Each team adds the allow rules they need. The platform team enforces the default-deny at namespace creation via a bootstrap controller or Helm chart. No allow rule, no traffic. That is the correct default.

Egress is harder to default-deny because pods need DNS (port 53 to kube-dns) and often need to reach the Kubernetes API server. A practical approach: default-deny ingress for all application namespaces on day one. Tackle egress after you have visibility into what each workload actually calls.

ResourceQuota and LimitRange: The Noisy Neighbor Defense

Without quotas, a single namespace can consume every CPU and memory resource on every node in the cluster. This is not hypothetical. A misconfigured batch job with no memory limit will grow until the OOM killer terminates pods, potentially across unrelated namespaces on the same node. The same uncapped resource behavior that causes CPU throttling in Kubernetes also drives evictions at the node level.

ResourceQuota sets limits at the namespace level: total CPU requests, total memory requests, total number of pods, total number of services. When a new pod exceeds the namespace quota, the API server rejects the creation with a 403. The runaway workload stops there.

LimitRange sets defaults and limits at the container level. Without it, a pod spec with no resource requests or limits is valid. The scheduler places it based on available capacity but it has no ceiling. LimitRange solves this by injecting a default request and limit when none is specified.

Control	Scope	What It Enforces
ResourceQuota	Namespace total	Sum of all pods' requests cannot exceed quota
LimitRange	Per container	Default request/limit if not specified; max ceiling per container

The practical starting point: set ResourceQuota on every namespace at creation. Use CPU and memory requests as the primary limits. Set LimitRange defaults at roughly half the quota limit so a single untuned pod cannot exhaust the whole namespace by itself.

Idle Namespaces Are an Open Attack Surface

Multi-tenancy is not just about isolation between active tenants. It is also about reducing the surface area of tenants that are not actively in use.

A dev or staging namespace with running pods has active service account tokens, open TCP connections, and running container processes. If an attacker gains access to one idle pod through a vulnerable image, an exposed debug endpoint, or a misconfigured ingress, they have a live credential to the Kubernetes API. The namespace is "idle" from your team's perspective but fully alive from an attacker's.

This is where automated environment lifecycle management addresses multi-tenancy directly. zopnight suspends non-production namespaces during off-hours by scaling deployments to zero. No running pods means no active service account tokens in memory, no open ports, and no container processes to exploit.

The security benefit compounds with the cost savings. A suspended staging namespace costs nothing and presents no attack surface. Bringing it back up on demand takes under 60 seconds. The tradeoff is zero.

For teams managing multi-tenant clusters, treating idle environment cleanup as a security control, not just a cost control, changes the calculus. It makes the Kubernetes cost management conversation relevant to security engineers, not just finance.

The sequence for a correctly hardened multi-tenant namespace is: create with default-deny NetworkPolicy, bind least-privilege RBAC, apply ResourceQuota and LimitRange, and suspend workloads when not in use. Each step is independently valuable. Together they give you actual isolation, not the appearance of it.

ZopNight v2.0: The Control Layer Your Cloud Bill Has Been Missing

Muskan — Mon, 04 May 2026 09:10:39 +0000

ZopNight v2.0: The Control Layer Your Cloud Bill Has Been Missing

We've been watching cloud bills grow for years. Dashboards got prettier. Alerts got louder. The bills kept climbing. ZopNight v2.0 is our answer to why: the problem was never visibility. It was control.

This release ships the full four-layer stack we believe every multi-cloud team needs: discovery with 14-day metrics, policy that binds, audit that remembers, and action that executes. One platform, three clouds, no agent, 60-second connect. Here's what we built and why each piece exists.

The problem: observation is not control

Industry surveys have put average cloud waste at 32% of spend for years running. That number does not move because the default tooling is reporting tools, not control tools.

A dashboard ends in a human. The human files a ticket. The ticket joins a queue. A control loop closes automatically: evidence triggers policy, policy triggers action, action writes to audit. That is the entire difference.

ZopNight v2.0 closes that loop across AWS, GCP, and Azure without an agent in your cluster.

Feature 1: Schedule anything — no cron required

The most common non-prod waste driver is resources running when nobody is using them. ZopNight's scheduler is a 7-day visual grid. Green means on, red means off. You drag to set the window. ZopNight generates the cron, evaluates per minute, and fires idempotent actions.

Dependency awareness is what makes this production-safe. Real environments have boot order requirements. Resource groups encode that order — ZopNight starts in sequence, tears down in reverse. No script can replicate this safely. Cron can't do it at all.

Every override has a start, an expiresAt, a reason, and an auditable owner. Maintenance windows that break schedules by design get recorded, not silently bypassed.

Feature 2: 337 audit rules — every one shows its evidence

"You could save 25%" gets ignored. "Your m5.xlarge in us-east-1 ran at 8% p95 CPU for 14 days; moving to m5.large saves 93 per month with 96% confidence" gets acted on.

ZopNight v2.0 ships 337 pre-built rules: AWS 155, GCP 75, Azure 107. Seven categories: idle, rightsizing, schedule, orphan, compliance, discount, governance. Every rule carries its evidence window, its metric, its current cost, and its optimised cost. The recommendation shows its work.

Discovery is topology-aware: EKS clusters surface with their node groups, Databricks with instance pools, managed databases with replicas. Right-sizing calls have the full tree. That is why the recommendations are closed-loop, not speculative.

Feature 3: Auto-tag at discovery — not at cleanup

Tag governance enforced at discovery time holds untagged resource rates below 5%. Quarterly cleanup campaigns typically stay above 20% and rising. The difference is timing: cleanup fights existing drift, governance prevents it.

ZopNight predicts prod, staging, dev, and noStop the moment a resource appears. A 1-100 confidence score is assigned and held for human approval before anything persists. The approved tag feeds both scheduling and showback — engineering and finance use the same taxonomy automatically.

Approach	Untagged rate	Finance reconciliation	Cleanup cadence
Quarterly cleanup	20%+	Manual spreadsheet	Every 3 months
Discovery-time governance	Below 5%	Automatic showback	Never needed

Feature 4: MCP server — AI that queries live state

Engineers waste 45 minutes reconstructing what happened to a cloud resource. They screenshot a dashboard, paste it into ChatGPT, describe the error, wait. The answer is as stale as whatever they pasted.

ZopNight v2.0 ships an MCP server with 43 read-only tools over streamable HTTP. Cursor, Claude Code, Codex, and Windsurf connect with a PAT and query live governance context directly. "Which schedules fired in the last two hours and what did they change?" returns a live answer, not a reconstruction.

Also available via 125+ REST endpoints with consistent pagination and retry semantics. PATs support flexible expiry. OIDC covers Azure federation.

Feature 5: Atlas — your infrastructure on a globe

ZopNight's Atlas view renders your infrastructure as a 3D globe with inter-region arcs for VPC peering, transit gateways, and interconnects. Drill from provider to node group on a single canvas. Cross-region blast radius is visible in one click instead of three spreadsheets.

Before you enforce a schedule or apply a right-sizing rule, Atlas shows you where that policy lands. Governance without topology is guesswork. Every action in ZopNight is topology-aware.

Feature 6: Cost reporting — one screen, actionable

Six dashboard widgets. Cost forecasting. Showback by team or tag with reconciliation. Multi-currency. Budget health. Anomaly detection with root-cause breakdown. CSV exports that match what is on screen.

Finance stops rebuilding your report. The labels engineering sets at discovery time are the same labels finance reports against. Showback reconciles at the resource level without a separate tool.

Feature 7: RBAC and audit — governance without gatekeeping

The cloud governance RBAC model in ZopNight v2.0 is built around an operational reality: the person who needs to see history is usually not the person who needs to change policy.

Three tiers: Viewer with 16 policy categories, Editor with 32, Admin with 52. Custom roles fill the gaps. Every start, stop, tag change, and override writes a full audit record: resource, status, source, triggered by, timestamp with timezone. Filterable across all five dimensions.

When an auditor asks who turned that on, when, and why, the answer is one filter away — not a Slack thread and three screenshots.

Try it without talking to anyone

The Playground is loaded with finance, retail, and healthcare datasets. Same UI as production. Read-only, no persistence, no auth required. You get the full ZopNight experience — schedules, audit rules, topology, tagging — against realistic data before you commit to anything.

If you want the architecture behind all seven features, the v2.0 deep dive covers the design decisions behind each layer.

The goal was never a prettier chart. It was a bill that matches what your team actually intended to run. ZopNight v2.0 is the mechanism. Everything else is noise.

The Night Shift Strategy for Cloud Savings

Muskan — Mon, 04 May 2026 09:08:39 +0000

The Night Shift Strategy for Cloud Savings

Your Non-Prod Environments Are Burning Money While You Sleep

A typical engineering team works 8 to 10 hours per day, Monday through Friday. Their dev and staging environments run 24 hours per day, 7 days per week. That means non-production infrastructure sits completely idle for 128 hours every week while still generating charges.

The Flexera State of the Cloud Report found that organizations waste 32% of their total cloud budget. The single largest source of that waste: non-production environments running around the clock with nobody using them. Cloud spending hit 723.4 billion globally in 2025. Apply that 32% waste rate and you get 231 billion burned on idle resources.

The fix is straightforward: shut down non-production resources when nobody is using them. CloudKeeper reports that scheduling auto-shutdown during nights and weekends saves 65-75% on those resources immediately. No architecture changes. No migration projects. Just turning things off when the office is empty.

The Math Behind Night Shift Savings

A standard work week is 50 hours of active use (10 hours per day, 5 days). A full week is 168 hours. That leaves 118 hours where non-production resources run for no reason. Shutting down during those 118 hours saves 70% of the weekly cost for those resources.

Here is what that looks like for a mid-size engineering team running common AWS resources:

Resource	Monthly Always-On Cost	Monthly Scheduled Cost	Annual Savings
20 EC2 instances (m5.xlarge)	5,606	1,682	47,088
5 RDS instances (db.r5.large)	3,285	986	27,588
3 EKS clusters (10 nodes each)	8,410	2,523	70,644
10 ECS services	2,190	657	18,396
Total	19,491	5,848	163,716

That is 163,716 per year saved by doing nothing more than stopping resources at 7 PM and starting them at 8 AM. No rightsizing analysis. No reserved instance planning. No architecture review. Just scheduling.

In the first 30 days, quick-hit actions like instance schedules and snapshot cleanup can recover 5-8% of total cloud spend. Over 12 months, automated scheduling combined with guardrails sustains a 25-30% lower run-rate versus the baseline.

What to Shut Down (And What to Never Touch)

Not every resource is safe to stop. Some lose data. Some take 20 minutes to restart. Some break dependent services when they go offline. Categorizing resources before scheduling prevents outages.

Safe to stop: EC2 instances in dev accounts, ECS tasks, EKS node groups (scale to zero), and load balancers with no active targets. These resources stop and start cleanly with no data loss.

Stop with caution: RDS instances, ElastiCache clusters, and OpenSearch domains. These retain data when stopped, but RDS instances cannot remain stopped for more than 7 days — AWS automatically restarts them. ElastiCache clusters lose their in-memory data on stop. Plan for 10-15 minute warm-up time on restart.

Never stop: Production resources (obvious), CI/CD pipelines (they run builds at all hours), monitoring infrastructure (you need it watching while everything else is off), and shared service meshes that production depends on.

The key principle: tag everything with an Environment tag (dev, staging, production) and a Schedule tag (business-hours, extended-hours, always-on). Resources without tags default to always-on. This prevents accidental production shutdowns.

Implementation in 5 Days, Not 5 Months

Organizations that treat scheduling as a 6-month project never finish. The infrastructure keeps growing, the savings keep compounding, and the project stays in planning. Start small. Ship in a week.

Day 1: Tag every non-production resource. Enforce 4 mandatory tags: Team, Environment, Service, Schedule. Use AWS Tag Policies or GCP Organization Policies to block untagged resource creation. Run a compliance report. Most organizations find 40-60% of resources are untagged on first audit.

Day 2: Map resource dependencies. For each environment, document the startup order. Databases start first (2-5 minutes), then backend services (30-60 seconds), then frontend services, then load balancers. This ordering prevents services from crashing on startup because their database is still initializing.

Day 3: Schedule dev environment shutdown. Set the schedule to stop at 7 PM local time and start at 8 AM. Use a 60-second warm-up buffer between dependency tiers. Monitor the first morning startup. If services come up healthy, the schedule works.

Day 4: Add staging to the schedule. Use a wider window: stop at 9 PM, start at 7 AM. Staging often runs automated test suites in the evening, so the later shutdown avoids interrupting CI pipelines.

Day 5: Set up cost monitoring and alerts. Create a dashboard showing daily spend before and after scheduling. Set an alert if any non-production resource runs outside its schedule for more than 2 hours. This catches resources that were manually overridden and never restored.

By Friday, you have automated scheduling running across dev and staging. The savings appear on next month's bill: 65-75% reduction in non-production compute costs.

The Dependency Trap and How to Avoid It

The most common failure in environment scheduling: services crash every morning because resources start in the wrong order. A backend service tries to connect to its database, the database is still initializing, the connection fails, the health check fails, the service gets terminated, and the auto-scaler gives up after 3 restart attempts.

Approach	What Happens at 8 AM	Result
Naive shutdown (stop everything at once)	All resources start simultaneously. Services fail because databases are not ready	Developers arrive to broken environments. File tickets. Lose trust
Dependency-aware shutdown (tiered startup)	Databases start at 7:55 AM. Services start at 8:00 AM. Load balancers start at 8:02 AM	Everything healthy when developers arrive. No tickets
Ephemeral environments (on-demand only)	Nothing runs until developer triggers. Environment spins up in 3-5 minutes	Maximum savings (70-80%). Slightly longer first-use wait

Dependency-aware sequencing requires knowing 3 things about each resource: what it depends on, how long it takes to become healthy, and what depends on it. A database takes 2-5 minutes to accept connections. A Kubernetes pod takes 30-60 seconds to pass readiness checks. A load balancer takes 15-30 seconds to register healthy targets.

The startup sequence follows the dependency graph: data stores first, then compute, then networking. The shutdown sequence runs in reverse: networking first, then compute, then data stores. This ensures clean connections on startup and graceful draining on shutdown.

For Kubernetes specifically, the challenge is stateful workloads. EKS node groups can scale to zero, but PersistentVolumeClaims and StatefulSets need special handling. The node group must restore with the same availability zone placement so volumes reattach correctly. Without this, pods fail to schedule because their volume is in us-east-1a but the new node launched in us-east-1b.

The organizations that sustain scheduling savings beyond the first month are the ones that invested in dependency mapping upfront. Five hours of dependency documentation saves 200 hours of morning firefighting over the next year.

Azure Firewall Premium Without TLS Inspection: That's $693/Month Wasted

Muskan — Mon, 13 Apr 2026 09:10:50 +0000

Azure Firewall Premium Without TLS Inspection: That's $693/Month Wasted

Azure Firewall Premium costs $2.496 per hour. Azure Firewall Standard costs $1.25 per hour. That gap — $1.246 per hour, $10,915 per year for a single instance — is the price of four features: TLS inspection, IDPS, full URL filtering with path awareness, and web category filtering.

Every one of those features requires explicit configuration after deployment. None of them are active by default. If your team deployed Premium because it appeared in an architecture diagram or a security checklist, and never completed the configuration steps, you are paying $10,915 per year for capabilities your firewall is not using.

This is more common than it should be. TLS inspection requires deploying an intermediate CA certificate, configuring it in Azure Key Vault, and enabling it in the Firewall Policy. IDPS requires switching from the default Alert mode to Alert and Deny. URL filtering requires building category policies. Teams that find this complexity difficult to schedule simply defer it indefinitely, while the Premium billing continues.

What Premium Actually Adds Over Standard

Understanding what you paid for is the starting point for deciding whether to keep it.

Feature	Standard	Premium	Requires Explicit Config	Default State
Network rules (IP, port, protocol)	Yes	Yes	No	Active
Application rules (FQDN filtering)	Yes	Yes	No	Active
Threat intelligence filtering	Yes	Yes	No	Alert mode
DNS proxy	Yes	Yes	No	Active when enabled
TLS inspection (decrypt, inspect, re-encrypt HTTPS)	No	Yes	Yes — requires intermediate CA in Key Vault	Disabled
IDPS (signature-based intrusion detection)	No	Yes	Yes — must set Alert and Deny mode	Alert only
URL filtering (full path, not just FQDN)	No	Yes	Yes — requires URL rules in policy	No rules applied
Web category filtering	No	Yes	Yes — requires category policy	No categories applied

The bottom four rows are what you are paying the Premium premium for. If none of them are configured, your firewall has the same effective security posture as Standard, with threat intelligence filtering in alert mode — which Standard also provides.

The critical detail on IDPS: Alert mode logs suspicious traffic but does not block it. A Premium firewall with IDPS in Alert mode offers no additional protection over Standard for the traffic patterns IDPS is designed to catch. Switching to Alert and Deny mode is what activates the protection. Most deployments never make that switch.

How to Check if Your Firewall Is Actually Using Premium Features

The fastest way to audit your firewall is through the Azure Portal and CLI. This takes under 10 minutes.

Check TLS inspection status:

az network firewall policy show \
  --name <policy-name> \
  --resource-group <rg-name> \
  --query "transportSecurity"

If the output is null or {}, TLS inspection is not configured. You are not inspecting any HTTPS traffic.

Check IDPS mode:

az network firewall policy show \
  --name <policy-name> \
  --resource-group <rg-name> \
  --query "intrusionDetection.mode"

Output will be "Off", "Alert", or "Deny". If it is "Off" or "Alert", the IDPS engine is either disabled or logging only. Neither blocks threats.

Check URL filtering rules:
In the Azure Portal, go to your Firewall Policy, select Application Rules, and look for rules with a rule type of URL (not FQDN). If all your application rules use FQDN, you are not using URL filtering. Standard supports FQDN rules identically.

Check web category policies:
In the same Application Rules view, look for rules using Web Categories. No category rules means web category filtering is not in use.

If every branch of that audit lands on the non-Premium outcome, you are running Standard functionality on Premium billing.

The Real Annual Cost of Unused Premium

A single Azure Firewall Premium instance running 24/7 in East US costs $21,870 per year in fixed instance fees alone ($2.496 x 8,760 hours). The equivalent Standard instance costs $10,950. The difference is $10,920 per year, per firewall.

Organizations with hub-and-spoke network topologies or Azure Virtual WAN deployments often run multiple firewall instances.

Firewall Count	Annual Cost (Premium)	Annual Cost (Standard)	Annual Overspend
1	$21,870	$10,950	$10,920
3	$65,610	$32,850	$32,760
5	$109,350	$54,750	$54,600

Data processing charges ($0.016/GB) are identical between tiers and are excluded from this comparison. The overspend figures are purely from instance pricing.

For an organization running three firewalls in a hub-and-spoke topology — one per region, none with TLS inspection configured — the wasted spend is $32,760 per year. That is not a rounding error. It is a budget line that can fund meaningful engineering work or be redirected to tools that are actually in use.

Three Scenarios Where Premium Is Justified

Premium is the right choice in specific, verifiable circumstances.

Compliance audit requiring TLS inspection. PCI-DSS v4.0 and HIPAA technical safeguard requirements can, depending on auditor interpretation, require inspection of encrypted outbound traffic. If your compliance framework has this requirement and your auditor expects TLS inspection to be demonstrably active, Premium with TLS inspection configured is a compliance necessity, not a cost choice. The key word is demonstrably: the configuration must be active and logs must show inspection events.

IDPS in Deny mode for regulated or high-sensitivity workloads. If your firewall protects workloads that process sensitive data and your security team has explicitly enabled IDPS in Alert and Deny mode with reviewed signature exclusions, Premium is earning its price. The IDPS signature database provides detection coverage that Standard's threat intelligence filtering does not match in depth or granularity.

User-facing environments with web category filtering requirements. If you need to enforce browsing policies for employees or contractor-facing environments — blocking social media, gambling, or high-risk categories — URL and web category filtering is meaningfully easier to manage in Azure Firewall Premium than in third-party solutions. If this use case applies, Premium is the right tool.

If none of those three conditions apply, Standard meets your requirements at half the price.

Downgrading from Premium to Standard: What It Actually Takes

There is no in-place downgrade path for Azure Firewall. You cannot change the SKU of an existing firewall instance. The migration requires creating a new Standard firewall and cutting over traffic.

The process is straightforward but requires a maintenance window.

Key considerations:

The route table swap is the cutover moment. Azure Route Tables point next-hop traffic to the firewall's private IP. Updating the next-hop address from the Premium firewall IP to the Standard firewall IP redirects traffic. This takes effect within seconds of saving the route table change. Plan the swap during a low-traffic window and have rollback steps ready (re-pointing the route table back to the Premium IP).

If you have Premium-only rules in your policy, specifically URL rules or web category rules, those must be converted to FQDN equivalents before migration. Run the firewall in parallel for 24 hours before decommissioning Premium to confirm all traffic patterns are handled correctly.

The migration itself takes 2-4 hours for most environments. The majority of that time is validation and monitoring, not configuration.

If You Stay on Premium, At Least Use It

For teams with a legitimate Premium requirement, the configuration debt is the real problem. Paying for Premium with TLS inspection disabled and IDPS in Alert mode means you have the billing exposure of a premium security tool with the protection level of a basic one.

The minimum configuration that makes Premium worth its cost: TLS inspection enabled with a managed intermediate CA deployed to endpoints, IDPS set to Alert and Deny mode with a reviewed exclusion list for known-safe traffic patterns, and URL category policies applied to at least the highest-risk categories (newly registered domains, malware, phishing).

If your team has not completed that configuration because it has been difficult to schedule, that is the actual problem to solve. The choice is not between Premium and Standard. It is between paying for Premium capabilities and using them, or paying for Standard capabilities at Standard prices.

Running unused Premium is the worst of both options.

When Autoscaling Makes Your Bill Worse, Not Better

Muskan — Mon, 13 Apr 2026 09:10:14 +0000

When Autoscaling Makes Your Bill Worse, Not Better

Autoscaling is sold as the solution to cloud waste. Scale down when traffic drops, scale up when it rises, pay only for what you use. That logic holds when the configuration is correct. When it is not, autoscaling becomes the most expensive mistake in your cluster.

We have seen production clusters where HPA and VPA were both active, Cluster Autoscaler was provisioning nodes on every spike, and the monthly bill was 40% higher than the equivalent fixed-size deployment would have been. The scaling was working as configured. The configuration was wrong.

This is not a rare edge case. The four failure modes below appear consistently across teams that have just enabled autoscaling and assumed the defaults are safe.

How HPA Actually Computes Scale Decisions

Most engineers understand HPA conceptually: when CPU goes up, add replicas. The detail that causes the most misconfiguration is what "CPU goes up" actually means to the controller.

HPA computes utilization as a percentage of the pod's CPU request, not its limit, and not the raw CPU usage in millicores.

If a pod has a CPU request of 100m and a CPU limit of 2000m, and HPA's targetCPUUtilizationPercentage is set to 50, HPA will try to keep actual CPU usage at 50m per pod. A pod that idles at 60m looks perpetually overloaded. HPA adds a replica. The new replica also idles at 60m. HPA adds another. This continues until maxReplicas is hit.

The pod is not overloaded. The cluster is.

The formula HPA uses is: desiredReplicas = ceil(currentReplicas * (currentUtilization / targetUtilization)).

If currentUtilization is already above targetUtilization before any real load arrives, the multiplier is greater than 1 on every reconciliation loop. The cluster scales continuously.

The fix: Set CPU requests to match the pod's actual idle CPU consumption, measured over 7 days. HPA's target should reflect load above idle, not total usage. A pod that idles at 60m and peaks at 300m should have a request near 60m and an HPA target around 70%, triggering scale-out when load reaches 42m above idle.

The Four Failure Modes That Inflate Your Bill

1. Target Set Below Idle CPU

Already described above. The signal is an HPA that shows TARGETS at or above targetCPUUtilizationPercentage even when no traffic is hitting the service. Check with kubectl get hpa -n <namespace> and look for utilization values that match your target even during off-hours.

2. HPA and VPA Running Simultaneously in Auto Mode

VPA in Auto mode evicts pods to apply new resource recommendations. Each eviction causes a brief CPU spike as the replacement pod starts up. HPA reads that spike and adds a replica. VPA then recalculates recommendations against a fleet that now has more replicas than before. The new recommendation is different. VPA evicts again. HPA scales again.

The result is replica count drift upward over time, with no corresponding increase in actual workload.

The fix: Run VPA in Off or Recommendation mode only. Read VPA's output and apply it manually to your deployment's resources.requests. Let HPA handle replica scaling from a stable request baseline. Never run VPA Auto and HPA on the same deployment.

3. Node Scale-Down Lag

Cluster Autoscaler's default scale-down-delay-after-add is 10 minutes. Its default scale-down-unneeded-time is also 10 minutes. A spike that lasts 3 minutes provisions new nodes that remain billable for at least 20 minutes after the spike ends.

For workloads with frequent short spikes, this means your cluster is almost always running with the node count from the last peak, not the current load. At $0.10/node-hour on t3.large, 5 extra nodes during 8 hours of evening quiet costs $4 per night, $120 per month, per service.

The fix: Tune --scale-down-delay-after-add to 3-5 minutes for development and staging clusters. For production, balance cost against the re-provisioning latency your workload can tolerate.

4. Metric Staleness in Custom Autoscalers

KEDA and any custom metrics-based HPA configuration depend on Prometheus (or another metrics source) for their scaling signals. Prometheus default scrape interval is 15 seconds. With metric collection lag, rule evaluation delay, and HPA reconciliation period, the data HPA acts on can be 30-60 seconds old.

For a bursty workload that spikes and drops in under 60 seconds, the autoscaler always reacts after the fact. It adds replicas as the spike is already ending. Those replicas sit idle through the stabilization window (default 5 minutes), then Cluster Autoscaler keeps the nodes alive for another 10 minutes. The spike cost 3 minutes of real load and 15 minutes of real spend.

The fix: For bursty workloads, reduce Prometheus scrape interval to 5 seconds for the relevant metrics. Or use predictive scaling (pre-scale before known peak times) instead of reactive scaling for workloads with predictable traffic patterns.

Detecting Runaway Scaling Before It Hits Your Invoice

The failure modes above are detectable before they become expensive. These are the signals worth watching.

Signal	Warning Condition	How to Check
HPA utilization at target during off-hours	HPA shows utilization at or above target with no active traffic	`kubectl get hpa -A` during off-peak window
Replica count trend	Replicas increasing over days without traffic growth	Prometheus `kube_deployment_spec_replicas` 7-day graph
VPA eviction rate	More than 2 evictions/hour per deployment	`kubectl get events --field-selector reason=Evicted`
Node count vs request count ratio	Node count stable while request rate drops	Prometheus `kube_node_info` vs ingress RPS
HPA scale-up frequency	More than 4 scale-up events per hour during normal load	`kubectl describe hpa <name>` events section
Cluster Autoscaler churn	Nodes provisioned and deleted more than twice per day	Cluster Autoscaler logs: `grep "Scale-up" cluster-autoscaler.log`

Set alerts on the first three. The rest are diagnostic when you suspect a problem.

Configuration Patterns That Eliminate the Failure Modes

VPA as Input to HPA, Not a Parallel Controller

This is the pattern that produces stable scaling. VPA's recommendations are reviewed and applied on a cadence. HPA operates against a request value that reflects actual idle consumption. Cluster Autoscaler provisions nodes against predictable replica counts.

Stabilization Windows Tuned to Your Traffic Shape

HPA's scaleDown.stabilizationWindowSeconds defaults to 300 seconds (5 minutes). For a service with 10-minute traffic cycles, that is reasonable. For a service with 2-hour traffic cycles, replicas will churn constantly. Set the window to match the natural period of your load pattern.

Similarly, scaleUp.stabilizationWindowSeconds defaults to 0 (immediate scale-up). For services where a 30-second spike does not justify a new replica, set this to 60-120 seconds to absorb transient spikes without triggering scale-out.

Cluster Autoscaler Tuning by Environment

For production clusters, the default 10-minute delays are appropriate. For non-production clusters (dev, staging, preview), reduce scale-down-delay-after-add and scale-down-unneeded-time to 2-3 minutes each. Non-prod clusters are typically not latency-sensitive. Aggressive scale-down on non-prod is pure cost reduction with no operational downside.

Going further: for non-prod environments, scheduled scaling (down to zero at end of day, back up at start of day) eliminates the problem entirely. A cluster that is off for 14 hours per day costs 58% less than one running 24/7. Autoscaling on a non-prod cluster is often the wrong tool. Scheduling is simpler and cheaper.

KEDA Scrape Interval Alignment

If you use KEDA with a Prometheus trigger, set pollingInterval in your ScaledObject to match your Prometheus scrape interval. The default pollingInterval is 30 seconds. If Prometheus scrapes every 15 seconds, KEDA sees data that is up to 45 seconds old (scrape age plus polling delay). Reducing both to 5-10 seconds closes the detection gap for bursty workloads.

What Good Autoscaling Actually Looks Like

A well-tuned autoscaling setup has two visible characteristics. First, replica counts are stable during steady-state traffic, moving only when load genuinely changes over a meaningful time window. Second, node count follows replica count with a predictable lag, and drops back to baseline within 15-20 minutes after load normalizes.

If your replica count graph looks like a heartbeat at rest, your autoscaling is calibrated. If it looks like a seismograph, the configuration is fighting your workload rather than tracking it.

The deeper issue is that autoscaling is a tool for production variability. Non-production environments do not have the same variability profile. Dev and staging clusters run at low load most of the day, spike briefly during CI runs or manual testing, then sit idle for hours. Autoscaling on these environments responds to those spikes by provisioning nodes that stay alive long after the spike ends. For non-prod, scheduled environment management eliminates this entirely. zopnight handles this automatically: environments shut down after inactivity and wake on access, without relying on autoscaler heuristics that were designed for production traffic patterns.

The goal is not to autoscale everything. The goal is to pay for what you actually use. Sometimes that means better-tuned HPA. Sometimes it means no autoscaling at all.