You’re paying rent on servers everyone forgot.
Cloud waste just did something it hasn’t done in five years: it went up. Flexera’s 2026 State of the Cloud report puts wasted spend at 29% of the total cloud bill — the first increase since 2021, and it landed after most of the industry staffed up FinOps teams. The discipline got serious. The bill got worse anyway.
Your share of that 29% has a name. Every platform team has a graveyard: a pipeline that still runs every night and writes to a bucket nobody reads, a service account with keys nobody can date, a DNS record pointing at a load balancer that was deleted quarters ago. Nobody built these things wrong. They just never got finished.
The graveyard has better uptime than most production services. It never pages, never deploys, never breaks. It just bills — quietly, forever. The most reliable thing you own, doing absolutely nothing.
The lifecycle diagram everyone draws ends at “deploy” — and the long tail of retirement gets treated as janitorial work for someday. Someday never comes. The graveyard grows: attack surface you forgot you own, bills you can’t attribute, alerts paging rotations that no longer exist.
This post treats decommissioning as a feature: an owner, an exit check, a rollback plan. Below: the five layers every service leaves behind, the seven-phase sequence that retires them safely, the kill chain for credentials, and the verification loop that proves the dead stay dead.
One service, five retirement layers
“Just delete it” fails because a running service is not one thing. It’s a stack of at least five layers, and compute is the smallest of them:
One workload, five layers of things that must be retired with it.
Delete the deployment and call it done, and the other four layers keep billing, authenticating, and paging. Identity is the dangerous one: a service account outlives its service by design, and unlike a pod, nothing alerts when it authenticates. Network is the embarrassing one: stale DNS and firewall rules are how “decommissioned” services keep receiving traffic. Data is the expensive one: orphaned buckets and snapshots accumulate in silence. Observability is the noisy one: dead alerts desensitize the living on-call rotation.
It’s demolishing a house and leaving the gas, water, and electricity connected — then acting surprised at the bills. The house is gone. The meters are still spinning.
Workloads — Retire this: Deployments, CronJobs, data pipelines, CI schedules · Verify with: Label-scoped
kubectl getacross namespacesIdentity — Retire this: Service accounts, keys, federation providers, IAM bindings · Verify with: IAM policy re-read + asset inventory search
Network — Retire this: DNS records, ingress, load balancers, firewall rules · Verify with: DNS lookup returns nothing; no listeners on the ports
Data — Retire this: Buckets, databases, backups, snapshots (after retention clears) · Verify with: Storage inventory shows no orphaned objects
Observability — Retire this: Dashboards, alerts, SLOs, on-call rotations, runbooks · Verify with: Alert rules and dashboard panels reference nothing dead
Deleting the pods is step one of six. The other five layers are where decommissioning actually happens.
Build this table at build time. Written during onboarding it takes minutes; reconstructed a year later, days — and it’s always incomplete.
The lifecycle has seven phases, not four
Retirement goes wrong as one frantic afternoon. It goes right as seven small, boring phases, each with an exit check:
Seven phases. Each has an exit check and a rollback plan.
Three phases deserve emphasis. Deprecate is communication, not technology: announce end-of-life, find every consumer, migrate them. The failure mode is the consumer you never knew about — the spreadsheet reading the warehouse table, the other team’s cron hitting the endpoint. Freeze stops new change: no deploys, no new traffic, schedules paused. A frozen service can be unfrozen, which is exactly the point. Disable is the load-bearing phase: turn off authentication and watch. If anything legitimate still depends on the service, it screams now — while recovery is one command.
Disable before delete. Disabled is reversible for days; deleted is not. Every phase should be boring enough to run on a Friday.
The decommission checklist
Run every retirement through this, in order. The items that feel unnecessary are the ones that bite:
[ ] Owner and record: one named owner and a retirement record — unowned retirements stall at phase two
[ ] Consumer hunt: search code, dashboards, and pipeline configs for references to the service’s endpoints, tables, and topics
[ ] Deprecation notice: consumers informed with a date; usage confirms the decline before you proceed
[ ] Freeze: CI/CD pipelines disabled, schedules paused, deploys blocked
[ ] Traffic drained: load balancer, ingress, and DNS drained — confirm zero requests before touching auth
[ ] Credentials disabled: service accounts disabled, keys listed (kill chain below)
[ ] Bindings removed: IAM grants and RBAC roles stripped; workload-identity federation severed
[ ] Data disposition: retention policy honored — backups kept for the required window, then destroyed
[ ] IaC pruned: Terraform state and manifests updated so the next apply doesn’t resurrect anything
[ ] Observability retired + verified: alerts, dashboards, SLOs, runbooks archived; asset inventory search and IAM policy re-read prove nothing references the dead thing
Before and after: the same estate, two different endings.
Zombies are not a mystery to investigate. They are a backlog to work.
Infrastructure-as-code is a resurrection machine: if the state file still lists the resources, the next apply brings them back. Prune manifests and state (terraform state rm for out-of-band deletions) during retirement, not after.
Killing credentials without breaking the living
Identity cleanup has a safe order and a dangerous one. The dangerous one starts with delete — and you learn about it from an outage. The safe order stops issuance first, removes grants next, deletes last:
The credential kill chain. Reversibility runs top to bottom.
Step one is the whole trick: disabling stops new authentication immediately and is trivially reversible — the “watch for screams” window:
# 1. Disable first — new auth stops now, re-enabling is one command.
gcloud iam service-accounts disable retired-sa@PROJECT_ID.iam.gserviceaccount.com
# 2. List user-managed keys and delete each.
gcloud iam service-accounts keys list \
--iam-account=retired-sa@PROJECT_ID.iam.gserviceaccount.com \
--managed-by=user
gcloud iam service-accounts keys delete KEY_ID \
--iam-account=retired-sa@PROJECT_ID.iam.gserviceaccount.com
# 3. Audit every grant to the retired identity, then strip them.
gcloud projects get-iam-policy PROJECT_ID \
--flatten="bindings[].members" \
--filter="bindings.members:retired-sa" \
--format="table(bindings.role)"
Then sever the federation paths: a bound Kubernetes service account keeps minting tokens until the binding is gone; a workload-identity pool keeps issuing external tokens until the provider is gone:
# 4. Remove the Kubernetes workload-identity binding (GKE: the
# trailing "-" deletes the annotation).
kubectl annotate serviceaccount retired-ksa -n retired-ns \
iam.gke.io/gcp-service-account-
# 5. Delete the federation provider, then the pool.
gcloud iam workload-identity-pools providers delete retired-provider \
--workload-identity-pool=retired-pool --location=global
gcloud iam workload-identity-pools delete retired-pool --location=global
# 6. Only now delete the account itself — the point of no return.
gcloud iam service-accounts delete retired-sa@PROJECT_ID.iam.gserviceaccount.com
Reversibility runs top to bottom. If step one causes no screams after a full business cycle, steps two through seven are formalities.
One caution: the API will let you delete a Kubernetes service account while pods still reference it — and the pods will fail in confusing ways. Disable at the cloud layer first, confirm silence, then remove the cluster-side objects.
Verifying the dead stay dead
Deletion is an assertion; verification is the proof. Two queries do most of the work. First, the asset inventory — search the whole estate for the retired name:
# Anything still referencing the retired identity, anywhere in the project.
gcloud asset search-all-resources \
--scope=projects/PROJECT_ID \
--query="name:retired-sa"
# Re-read the IAM policy and confirm the member appears nowhere.
gcloud projects get-iam-policy PROJECT_ID \
--flatten="bindings[].members" \
--filter="bindings.members:retired-sa" \
--format="value(bindings.role)"
Second, the cluster. Label-scoped queries across all namespaces catch the strays that namespace-scoped habits miss:
# Every namespaced object still carrying the retired label.
kubectl get deploy,sts,ds,cronjob,svc,ingress,sa,configmap,secret \
-A -l app=retired-app
# Namespaces stuck in Terminating usually mean a finalizer is holding
# the deletion. Inspect it — don't force-clear it blindly.
kubectl get ns retired-ns -o jsonpath='{.spec.finalizers}'
# Cluster-scoped bindings are the classic forgotten layer.
kubectl get clusterrolebinding,clusterrole -o name | grep retired
A finalizer exists to guarantee cleanup ordering — detach before volume delete, deregistration before record removal. Force-clearing one skips the cleanup it was guarding. Read it, satisfy it, and let the deletion complete on its own.
Make verification recurring, not ceremonial: a scheduled scan turns “we decommissioned it” into “we keep proving it.” The first run usually finds something. That’s the point of the scan.
A decommission without verification is a hope. Run the queries, read the output, and keep running them on a schedule.
The takeaway
Decommissioning is a feature: phases, exit checks, rollback plan — like everything else you ship. This week, pick one retired thing and run it through the kill chain.
Somewhere in your cloud account, a resource you forgot is billing you right now for doing nothing. Go introduce yourself. Then retire it properly — all five layers, with the checklist. The graveyard is full enough.
So here's my question for you: what is the oldest forgotten resource still billing you, and which of the five layers did its retirement skip?
Disclaimer
This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company's intentions, policies, or internal decisions. Where I do not know something, the post says so.
I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.
Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.
Nothing in this post is professional, financial, legal, or security advice. Examples are illustrative — validate in your own environment before running anything in production.
Corrections are welcome: if a company named here publishes an official statement, or a detail above is wrong, I will update this post and note the correction. The goal is an accurate public record of my experience, not a verdict on anyone.
© 2026 Honey Kiran. All rights reserved.
Honey Kiran · DevOps & Platform Engineering
I write about the unsexy work that keeps systems alive at 3am — platform engineering, Kubernetes, cloud IAM, and the lifecycle practices nobody demos.
New posts regularly. Follow along — and send this to the teammate who deleted the deployment and called the retirement done.
Examples are illustrative — validate in your own environment before running in production.
Top comments (0)