DEV Community: Rick Wise

I Shipped a Redesign That Orphaned My Own Product

Rick Wise — Wed, 19 Aug 2026 13:53:46 +0000

This month I rebuilt CloudWise around an AI agent.

Not a chatbot bolted into a corner. The agent is the product now. You open the app and you're talking to it. "Where's my money going?" — and it pulls your AWS spend, ranks the waste, shows you the dollars per month per finding, and tells you what's safe to fix.

It looked done. The screens were beautiful. The agent answered. I shipped it as the front door.

Then I used it like a customer would. And I discovered I had quietly orphaned half my own product.

This is the story of that month, because it taught me the single most useful lesson I've learned as a solo founder: a redesign isn't done when it looks done.

What I built

For most of this year, CloudWise looked like every other AWS cost tool: a dashboard. Charts, tables, a sidebar of pages. Useful, but passive. You had to know what to look for.

The bet I made for the overhaul was that the interface should be a conversation, not a dashboard. Most people don't want to read a cost dashboard. They want to ask a question and get an answer.

So the centerpiece became a conversational workspace backed by a real tool-calling agent loop on Claude (running on AWS Bedrock). "Real" is the important word. This isn't a model that summarizes a page of text. It's a model with tools — it actually queries your cost data, your findings, your Reserved Instance and Savings Plan coverage, and composes an answer from live numbers. It remembers your account between conversations.

You can ask it:

"Where's my money going?" → it breaks down spend by service, in gold, with the month-over-month delta.
"What's safe to fix?" → it ranks your waste findings: idle NAT Gateways, forgotten SageMaker notebooks, unattached EBS snapshots, each with a dollar figure and a safe-to-fix flag.
"Compare my two accounts." → and it does.

That last one matters more than it sounds, and it's where the trouble started.

The part where it looked finished

By the start of June, the redesign was, on paper, complete. New design system. Dark, native. A conversational workspace. A guided tour. A redesigned dashboard. The agent loop. Cross-session memory. I'd migrated every screen.

I flipped the new workspace to be the post-login front door and moved on.

Here's the uncomfortable truth about building alone: you stop seeing your own product. You navigate it the way the author navigates it — from the inside, knowing every shortcut, never actually starting cold the way a real user does.

So I made myself start cold. I logged in like a brand-new customer and tried to do the boring things. Switch to my other AWS account. Open settings. Change a notification. Log out.

I couldn't.

The front door had hidden the house

None of those features were gone. Every settings page still existed. The account switcher still existed. Logout still existed. The old navigation still existed, sitting in the codebase, fully functional.

They just weren't reachable from the place users now landed.

The new workspace shell had a clean little user footer — a static label with the account name. No menu. No logout. No link to subscription, notifications, alerts, API keys, AWS accounts, or password. From the new front door, there was literally no path to any of it.

Worse, the half of the product that did still have navigation rendered in the old chrome. So a new user would land in a slick dark conversational workspace, click one thing, and get bounced into the previous design — a completely different layout. Two products wearing different clothes, stitched together at a seam the user falls straight through.

I wrote it down plainly in my own audit doc at the time, because I needed to see it without flinching:

The redesign did not delete your features — the cutover orphaned them.

That sentence reframed the entire month. This wasn't a teardown. It was a finishing problem. The work wasn't to rebuild — it was to take ownership of every job the old interface used to do, and make the new one do it better.

The agent was also lying about multi-account

While I was in there, I found a real bug — the kind that only surfaces when you use the product for real, with more than one account.

CloudWise is a multi-account tool. You connect your AWS Organization and it discovers all your accounts. But the agent was collapsing them. When you asked about cost, it blended every account together with no way to scope to one. And when you asked about Reserved Instance and Savings Plan coverage, it reported coverage for exactly one account — the first one it happened to grab — and silently ignored the rest.

That's not a cosmetic miss. That's the tool confidently giving you a wrong answer about money. If you had three accounts and asked "how's my RI coverage," it answered for one-third of your footprint and didn't tell you.

The backend could already filter by account. The old reports page could already do multi-account selection. The agent — the new centerpiece — was strictly less capable than both. The redesign had, in this one spot, made the product worse while looking like it made it better.

So the fix had three parts:

A header account switcher that scopes the entire workspace — single or multi-select, capped by your plan tier.
Per-account arguments on the agent's tools, so the model can honor that switcher and answer in-conversation requests like "compare my two accounts."
Fixing the coverage bug so it reports across all accounts, not the first one.

# Before: commitments reported for whatever account came first.
def get_commitments(accounts):
    first = accounts[0]
    return fetch_coverage(first.account_id)   # silently ignores the rest

# After: the agent scopes to what you asked for — one, some, or all.
def get_commitments(accounts, scope_ids=None):
    targets = [a for a in accounts if a.account_id in scope_ids] if scope_ids else accounts
    return [fetch_coverage(a.account_id) for a in targets]

A small diff. A big difference in whether you can trust the answer.

Thirty days of taking ownership

That's what the month actually was. Not "add features." Re-own the basics, and make the agent the real front door instead of a beautiful demo sitting on top of a product it had stopped being responsible for.

In order, here's what changed:

One app shell, not two. Unified chrome around every authenticated page — one persistent sidebar, the account switcher, every settings link, and logout — so you never fall through the seam between the new workspace and the old layout again. The workspace became one surface inside a consistent app, not a separate world.

The account switcher scopes everything. Pick an account (or several) in the header and the whole workspace re-scopes to it — and the agent answers for exactly that selection.

The agent learned to count accounts. Per-account tool arguments, the coverage bug fixed, and the ability to genuinely compare accounts on request.

A deep, cross-page guided tour. Instead of a checklist nobody reads, a docked "CloudWise agent" companion walks a new user through the real product — navigating the actual screens across seventeen steps, not narrating a slideshow.

The whole product went dark-mode-native. Every deep page — cost reports, remediation, savings plans, settings, the dashboard — rebuilt on one design system. Money is always gold. No more half-migrated screens where one page is dark and the next is white.

A shareable cost-health score. One number, 0–100, for how efficiently you're running AWS — with a public, sanitized share card carrying no account details or PII. Because the first question every engineer actually asks is "are we good, or not?"

Across the month that came out to a few dozen shipped changes and around fifty releases — versions 1.54 through 1.104. Most of them were not glamorous. Most of them were re-owning a job the old interface used to do, quietly, that the new one had dropped on the floor.

The lesson, stated plainly

I've shipped a lot of software in twenty-five years. I still fell for this one, which is why I think it's worth writing down.

"Looks done" and "is the product" are not the same state. They can be a hundred releases apart.

A redesign is not finished when the new screens are beautiful and the happy path works. It's finished when the new thing has quietly taken over every job the old thing did — logout, account switching, the boring settings page nobody screenshots — and does each of them at least as well. Until then you don't have a new product. You have a beautiful front door on a house whose rooms you've locked.

And the cruelest part: users don't grade you on the demo. They grade you on the one ordinary day they need the single feature you forgot to bring across. The day they need to switch accounts, or check their RI coverage across all three, or just sign out. That's the moment your redesign is actually judged — and it's never the moment you rehearsed.

So now I have a rule. Before I call any cutover done, I log in cold and do the boring things. All of them. If I can't sign out, I'm not done. No matter how good the front door looks.

CloudWise is an AWS cost optimization tool for startups — 191 automated waste checks, a real agent that runs against your actual usage, air-gapped mode for security teams, starting at $19/month. If you want to ask an agent where your AWS money is going, it's at cloudcostwise.io.

EBS Snapshot Sprawl: The Waste Cost Explorer Can't Show You

Rick Wise — Wed, 19 Aug 2026 13:51:19 +0000

Last week's short covered the 101 on old EBS snapshots: the 90-day threshold, the $0.05/GB-month rate, aws ec2 delete-snapshot. If you saw it, you already know snapshots are cheap-per-unit and expensive-in-aggregate. What that 42 seconds couldn't fit in is the actual reason snapshot sprawl is so hard to find in the first place — and it isn't the price. It's that Cost Explorer structurally cannot show you who's responsible for it.

Cost Explorer shows you a bill, not a culprit

Group your AWS costs by usage type and EBS snapshots show up as one line: EBS:SnapshotUsage, rolled up per account and region. That's it. Not per-snapshot, not per-volume, not per-AMI. If that line is $340/month, Cost Explorer will tell you the total and nothing about which of your 200 snapshots — or which of your 40 AMIs pinning them — put it there.

Part of why this is so opaque is how the billing actually works. EBS snapshots are incremental: the first snapshot of a volume captures every block, but every snapshot after that only stores blocks that changed since the previous one. Delete an "old" snapshot in the middle of a chain and AWS doesn't just drop its unique blocks — it can merge the still-referenced blocks from that snapshot into the next one to keep the chain valid. The result is a genuinely well-designed storage model that happens to make per-snapshot cost attribution close to meaningless from the outside. You can't look at snapshot #14 in a chain of 20 and know what deleting it actually frees, and Cost Explorer doesn't even try — it just gives you the account-wide sum and moves on.

So the number you see going up every month is real. The reason is invisible from the billing console. You have to go look at the resources directly.

`deregistered ≠ deleted` — the AMI-pinning trap

Here's the gap that costs teams the most and gets found the least: deregistering an AMI does not delete the snapshot backing it.

When you run CreateImage against an EC2 instance, AWS creates an AMI and, silently, one or more EBS snapshots to back it — you'll find the linkage in the snapshot's own description field, something like Created by CreateImage(i-0abc123def456789) for ami-0fedcba987654321. Deregister that AMI later (cleaning up an old release, retiring a pipeline, whatever the reason) and AWS removes the AMI. The snapshot stays. Forever. Nothing in the console flags it, nothing in Cost Explorer changes shape, and nothing tells you the thing that snapshot was created for no longer exists.

This is exactly the pattern our AMI_ORPHANED_SNAPSHOT detector checks for. It's not a guess — it's a direct read of the relationship AWS itself records: the detector regex-matches each snapshot's description against the CreateImage(...) for (ami-...) pattern, pulls the referenced AMI ID, and checks it against the account's currently-registered AMIs. If the AMI isn't there anymore, the snapshot is flagged — at higher priority than a generic "old snapshot" check, because an orphaned-AMI snapshot has a certain reason to be dead, not just an age-based guess.

That priority ordering matters. Our storage detector dedups three overlapping checks against the same snapshot inventory: AMI-orphan first, then volume-orphan (the source volume was deleted), then plain age (over the 90-day default threshold, at the $0.05/GB-month rate from last week's short). A snapshot only gets counted once, under whichever explanation is strongest — an AMI-orphaned snapshot isn't also reported as merely "old," because "old" undersells why it's actually safe to delete.

Lifecycle policies are the actual fix — for snapshots you haven't made yet

None of the above is a criticism of EBS snapshots as a backup mechanism. They're cheap, they're incremental, and they're the right tool. The problem is entirely operational: nothing deletes them automatically unless you tell it to.

AWS Data Lifecycle Manager (DLM) exists for exactly this — attach a policy to a tag or resource type and it will create snapshots on a schedule and expire them on a schedule, so "backup taken 400 days ago for a server that's been gone for 399 of them" stops being possible going forward. If you're not running DLM policies today, that's the highest-leverage 20-minute fix here, full stop.

But DLM only prevents new sprawl. It does nothing for the snapshots already sitting in your account from AMIs someone deregistered two years ago, or backups nobody automated before DLM was set up. That backlog needs to be found once, by hand or by a scan, before a policy can keep it clean going forward. That's the gap our detector is built for — not a replacement for lifecycle policies, the thing that finds what predates them.

Finding yours

If you want to see this in your own account instead of grepping AMI descriptions by hand, a free scan checks this along with 190+ other waste patterns across 40+ AWS services — read-only, five minutes, nothing gets deleted without you clicking it. If you'd rather watch the 90-day/$0.05-per-GB basics first, we posted a 30-second walkthrough of that detector last week.

Or check the AMI-pinning trap yourself right now:

# Snapshots whose description names a CreateImage/AMI pairing
aws ec2 describe-snapshots --owner-ids self \
  --query "Snapshots[?contains(Description, 'CreateImage')].[SnapshotId,Description]" \
  --output table

# Currently-registered AMI IDs
aws ec2 describe-images --owners self \
  --query "Images[].ImageId" --output text

Cross-reference the AMI ID in each snapshot's description against the second list. Anything missing has been quietly billing you since the day someone cleaned up an AMI and assumed the cleanup was done.

The Bug That Turned Every Bad Password Into a Server Outage

Rick Wise — Wed, 19 Aug 2026 13:50:30 +0000

Type your password wrong on CloudWise's login page, and for a while, the server told you it had a nervous breakdown.

Not "invalid credentials." Not even a plain 401. An HTTP 500 — the code reserved for "something on our end is broken" — for the most routine failure mode there is: a human mistyping a password.

This happened on staging. It happened on production. It happened to POST /api/v1/auth/login with wrong creds, an unknown email, an unconfirmed account, or an account mid-password-reset. Four different "you did something ordinary" situations, all reported back as "we did something wrong." It went undetected until an automated gate caught it and refused to let a release ship. This is that bug, the fix, and why the distinction matters more than it sounds like it should.

The bug

_handle_cognito_auth_error, the function that translates AWS Cognito's auth exceptions into an HTTP response, raised a CloudWiseException without ever setting a status_code. No status code means the default. The default is 500.

So every one of these:

Wrong password
Email that isn't registered
Account that hasn't confirmed its email yet
Account stuck in a forced password reset

came back as a server error. Reproduced live on both environments — for example, POST /api/v1/auth/login with bad credentials on staging returned:

HTTP 500
{"detail":"Invalid email or password"}

and on production:

HTTP 500
{"detail":"Authentication failed: User does not exist."}

Read that second one again. Production wasn't just returning the wrong status code — it was telling an anonymous caller whether a given email address had an account. That's a second, smaller bug riding along inside the first one.

Why the status code is the actual bug

It's tempting to shrug this off — the message was right there in the body, "Invalid email or password", so what's the harm in the wrong three-digit prefix?

The harm is that the status code isn't decoration. It's the part of the response that infrastructure reads without understanding a word of the payload. A monitoring dashboard doesn't parse detail. It counts 5xx rates. To CloudWise's own alerting, every single mistyped password looked identical to an actual outage — same bucket, same page-worthy signal, same "something is on fire" pattern, forever, at whatever rate normal users normally fat-finger their passwords. That's not a rare event. It's baseline noise, and the bug was quietly dressing it up as baseline crisis.

It also meant every one of those routine rejections logged at error level — the log level meant for things that need a human to look at them, now firing constantly for people who just needed to try again.

And then it did something worse than annoy a dashboard: it blocked a production release.

How it actually got caught

CloudWise's release pipeline runs an E2E gate against staging before anything is allowed to promote to production. One of the negative-path assertions in that gate — ci-gate-read.spec.ts:93 — logs in with a wrong password and checks that the app shows an error and stays on /auth/login, the way a real login form should behave.

That test didn't expect a 500. Nothing in a normal auth flow should return one for a wrong password. The gate went red, and release 1.105.0 sat there, unable to auto-promote to production, because the pipeline correctly refused to trust a build where the login form's error handling was behaving like a crash.

That's the part worth sitting with: this bug was already live in production, unnoticed, for who knows how long. It took a different release's gate run to surface it — not because the gate was looking for this specific bug, but because it was asserting the right general behavior (bad password → clean error, stay put) and the actual behavior didn't match. A negative-path test doesn't need to know about your bug in advance. It just needs to check that the ordinary failure case looks ordinary.

The fix

The fix, in backend/app/services/cognito_auth_service.py, maps Cognito's exceptions to the status codes they should have had all along:

NotAuthorizedException (wrong password) and UserNotFoundException (unknown email) → 401, both returning the exact same message: "Invalid email or password". Same message for both cases on purpose — the response can no longer be used to tell whether an email is registered, closing that account-enumeration leak.
UserNotConfirmedException and PasswordResetRequiredException → 403 — the account exists, but the request is correctly rejected for a reason that isn't "try a different password."
Anything else — a genuinely unexpected Cognito error — still defaults to 500. That default is correct in that case. An unrecognized failure mode is a server-side concern worth alerting on. The bug was never that 500 existed; it was that the four most common, most expected auth rejections were routed into it by omission.

Expected-rejection logging dropped from error to warning, so the logs now reflect what actually happened: a routine, anticipated rejection, not an incident.

The test suite got the fix that should have caught this the first time. test_handle_cognito_auth_error_* previously asserted on the error message only — never the status code. That's exactly how a 500-instead-of-401 slips through code review and CI both: the message text looked fine, so nobody noticed the number in front of it was wrong. The tests now assert status codes explicitly, plus two cases that weren't covered before: the no-leak UserNotFoundException path, and PasswordResetRequiredException.

This is the sibling of an earlier fix, CLO-48, which cleaned up the same bug class on the refresh-token path — an expired refresh token was logging as ERROR twice when a clean 401 was all that was warranted. That one didn't touch login. This one closes the login path CLO-48 didn't cover.

The takeaway

Status codes are a contract between your API and everything that reads it without understanding it — monitors, alerting rules, retry logic, browsers, other services. A wrong password is a client error. Treating it as a server error doesn't just mislabel one response; it teaches every downstream system that watches your 5xx rate to distrust the signal, right when you need that signal to mean something.

The thing that actually caught this wasn't a code review, a manual QA pass, or a customer complaint. It was an automated negative-path assertion doing exactly what negative-path assertions are for: checking that the boring, expected failure looks boring and expected. It's staying in the gate. Nobody has to remember to test this again — the pipeline already refuses to ship a build that gets it wrong.

If you want to see what else that kind of scrutiny turns up, a free, read-only scan of your AWS account takes about five minutes and changes nothing without you approving it first.

Inside a Real $11,871/mo AWS Waste Scan

Rick Wise — Wed, 19 Aug 2026 13:50:30 +0000

Most cost-optimization content shows you a screenshot of a dashboard and asks you to trust the number. So instead, here's an actual scan CloudWise runs internally as a demo profile — a mid-size SaaS company's AWS account, $28,500/mo in total spend, run through all 191 detectors. It came back with $11,871/mo in identified savings across 18 findings.

I want to walk through what's actually in that list, because the shape of it surprised me even after building the detectors.

Where the 18 findings live

Split by category, not dollar amount:

Database — 9 findings. By far the largest bucket. A mix of idle RDS instances, an oversized analytics database running at 12% CPU, stale manual snapshots, and a cluster of ElastiCache-specific findings (idle replicas, an engine migration opportunity, a serverless-fit opportunity, and one very large data-tiering opportunity — more on that below).
Compute — 2 findings. An idle bastion host nobody SSHs into anymore, and one oversized EC2 fleet running at 15% CPU.
Storage — 2 findings. An unattached EBS volume left over from a migration, and a batch of aging EBS snapshots.
Commitment Management — 2 findings. A Savings Plan expiring in 52 days, and a Convertible Reserved Instance sitting on previous-generation hardware with a free exchange available.
Network — 2 findings. Two idle Global Accelerators — one genuinely idle, one just disabled but still billing.
Purchase Optimization — 1 finding. Production RDS instances that have been running on-demand, 24/7, for 90+ days with no Reserved Instance covering them.

Nine of eighteen findings sitting in "Database" isn't a coincidence — it's the category where the widest range of waste patterns overlap: idle instances, oversized instances, stale snapshots, and now a whole sub-family of ElastiCache-specific checks (engine choice, replication topology, traffic shape, data tiering) that didn't exist as separate detectors a year ago.

The one number that matters most

Of the $11,871/mo total, one single finding accounts for 68% of it: $8,057/mo, from an ElastiCache data-tiering opportunity.

The cluster in question runs 4× cache.r6g.16xlarge nodes — memory-only, no local SSD — for 1,676 GiB of total cache capacity. ElastiCache's R6gd family adds local NVMe storage and tiers data between RAM and SSD automatically, based on access frequency. For a dataset where most of the data isn't accessed on every request (true of almost every real cache), that means the same effective capacity fits on far less hardware: 1× cache.r6gd.16xlarge instead of 4× cache.r6g.16xlarge.

That's not a "you forgot to delete something" finding. It's an architecture-level rightsizing call that requires reading how your own cache is actually used before you touch it — which is exactly why it's flagged as confidence: medium, not high, and comes with an explicit risk note: SSD-resident data has slightly higher latency, so it's a good fit only when a meaningful chunk of your data is cold.

What the next four biggest opportunities have in common

Set aside the $8,057/mo ElastiCache finding and look at what else shows up near the top of this scan: an expiring Savings Plan, an unpurchased Reserved Instance opportunity on production RDS, an oversized analytics database at 12% CPU, and an oversized EC2 fleet at 15% CPU.

None of those are "click delete." Every one of them is a standing decision that has to be made again, deliberately, on some recurring cadence:

A Savings Plan doesn't renew itself — someone has to decide, before it lapses, whether the workload underneath it still looks the same.
An RDS instance rightsized today can be wrong again in six months if the workload grows, shrinks, or changes shape.
"Buy the Reserved Instance" is itself a bet on the next 12 months looking like the last 3 — which is exactly the kind of call CloudWise's Commitment Risk Score exists to check before you make it.

Compare that to the idle-resource findings in this same scan — the bastion host, the unattached EBS volume, the stale snapshots, the disabled accelerators. Those are real money too, and they're the easiest to fix: find it, delete it, done. But they're structurally small, because once you delete something, it stays deleted. It doesn't come back next quarter.

Rightsizing and commitment decisions aren't like that. The right instance size and the right commitment level are correct for right now — and they drift the moment your traffic pattern, team, or architecture changes, which for a growing company is constantly. That's the actual thesis worth taking from a scan like this: waste renews itself. The idle-resource sweep is a one-time cleanup. The rightsizing and commitment layer is a standing job, and it's where most of the money in this particular scan actually lives.

Why this matters for how you should read your scan

If you run a scan and the top finding is "delete this idle volume," fix it and move on — there's nothing recurring about it. But if your biggest findings look more like this account's — an oversized instance, an expiring commitment, a data-tiering opportunity that requires understanding your own traffic — treat that as a signal that the fix isn't a single action, it's a review cadence you don't currently have. That's a different kind of problem, and it's worth naming as one.

CloudWise is an AWS cost optimization tool for startups — 191 automated waste checks across 42 AWS services, read-only by design, starting at $19/month. Run a free scan at cloudcostwise.io.

We Cut Lambda Cold Starts 56% — Three Wrong Turns Before the Real SnapStart Fix

Rick Wise — Wed, 19 Aug 2026 13:43:07 +0000

CloudWise's /dashboard took up to 2.3 seconds to load cold. That's the API Lambda's own cliff — not network, not the frontend. AWS Lambda SnapStart was already turned on. It was already restoring a frozen snapshot in about 600ms, which is the number SnapStart is supposed to deliver. And for two deploys, turning it on made no measurable difference to the thing that actually made the dashboard feel slow. This is the debugging path that got from there to a measured 56% cut — including the two attempts that didn't work, because the wrong-turn part is the part worth reading.

What was actually slow

We measured every layer before touching anything (full numbers in docs/redesign/clo114-dashboard-speed-analysis.md). Four dashboard API calls, already firing in parallel — parallelizing them further was a non-issue, wall-clock already tracked the slowest call. Server compute, warm, was fine: ~200–380ms. SnapStart's own restore was fine: ~550–620ms.

The problem was one specific number from CloudWatch REPORT lines: on a cold (post-restore) invocation, the first request's own Duration — not the restore, the handler work after the restore — was 1,124ms at p50, up to 2,805ms at the tail. SnapStart had already paid for the expensive part (a frozen Python interpreter + imports) and handed back a restored environment in 600ms. Something in the first real request was still doing ~1,100ms of work the snapshot didn't cover.

The diagnosis: init phase vs. handler phase

SnapStart snapshots whatever ran at module import / init. Anything deferred to the request handler is still paid, lazily, on the first real invocation — that's the whole cliff. get_settings() already runs at import in app/main.py, and constructing Settings performs the full AWS Parameter Store load — so config reads were already inside the snapshot, free. What wasn't: the boto3 session and DynamoDB client (built lazily in a factory, inside the request path) and FastAPI's startup_event — Mangum runs lifespan="auto" on the first request, not at import.

The fix looked obvious: move both into the init phase, so they're captured in the snapshot. We rejected the alternative — Lambda provisioned concurrency — outright: it bills a kept-warm 2GB instance 24/7 regardless of traffic, and it's redundant with SnapStart, which already restores in ~600ms. Paying a standing bill to paper over a problem SnapStart half-solves is exactly the kind of waste we built this product to catch in other people's accounts. So: init-phase priming, ~$0 incremental cost.

Wrong turn #1: it shipped clean and did nothing

PR #641 landed the priming code. Deploy went green, gates passed, prod promoted. Re-measured: cold Duration p50 1,142ms — statistically the same as the 1,124ms baseline. Priming had shipped and, as far as the numbers were concerned, changed nothing.

Worse: we couldn't even tell why. Every diagnostic line we'd added — "hooks registered," "SnapStart prime," even FastAPI's own unconditional startup log — was completely absent from CloudWatch across 90 minutes of cold starts. The working theory for a while was that SnapStart's init-phase logs simply don't reliably surface in CloudWatch, which would have made this nearly undebuggable from logs alone.

Wrong turn #2 (well, half a turn): it was a log level

It wasn't a SnapStart logging quirk. It was logging.info(). The Lambda's effective log level was dropping INFO — CloudWatch showed our [WARNING] lines and nothing below them, meaning every one of our priming diagnostics had been silently filtered the whole time. We moved the key diagnostic and priming log lines to WARNING (PR #648) and immediately got a real signal for the first time in this investigation. Lesson, underlined: when a fix "does nothing and also produces no logs," check the log level before you start reasoning about distributed systems semantics. We spent longer on the second hypothesis than the first.

The actual bug: `after_restore` cleared, but never rebuilt

With real diagnostics finally visible, we split the cold REPORT lines into two groups: requests that hit a fresh init (rare — a brand-new execution environment) versus requests that hit a restored snapshot (the common case SnapStart exists for). The restored-and-then-requested group was still averaging ~1,064ms — basically the original cliff, just hiding in a bucket we hadn't isolated before.

The cause was in our own restore hook. A boto3 session captured in a snapshot carries frozen credentials and, once a client exists, frozen TLS connection pools. After a real restore, the execution environment is new: credentials need refreshing, any frozen socket is dead. So reset_for_restore() correctly cleared the cached session and clients on after_restore — that part was right, and necessary for correctness. What it didn't do was rebuild them. So the next request found an empty session, built one lazily, and paid almost exactly the cost priming was supposed to eliminate — just relocated from "first request ever" to "first request after every restore," which for a Lambda that scales to zero between sparse dashboard loads is most of them.

The fix (PR #649): reset_for_restore() clears and eagerly rebuilds — with the freshly-restored credentials — so the next request finds the session already built, not merely uncorrupted.

The result

Post-restore first-request Duration, n=7 samples per version, staging:

Version	mean	p50
Baseline (no priming)	1,183ms	1,124ms
Clear-only (the bug)	1,064ms	1,060ms
Eager re-prime (the fix)	652ms	490ms

p50 1,124 → 490ms, a 56% reduction. Diagnostics on every restored environment now confirm the full chain worked as intended: hooks_registered=True, session_already_built=True, restore_reprimed=True, zero errors.

It's a real win, and it's not a complete one. About 3 of 7 restores in that sample still land at ≥500ms, even with the session confirmed pre-built — most likely the first actual DynamoDB operation opening its own TLS connection, since boto3 connects lazily on first call rather than at client construction. Chasing that down would mean priming a real DynamoDB round-trip inside the restore hook itself, for what's probably diminishing returns against a ~490ms floor that's already good enough. We banked the 56% rather than shipping a seventh iteration.

Alongside this, a frontend session-storage stale-while-revalidate cache (shipped separately, PR #639) hides both the network round-trip and any residual cold cliff on repeat visits — the dashboard's daily-batched data makes brief staleness safe — and a keep-warm EventBridge ping keeps instances hot enough that cold restores stay rare in the first place.

The actual lesson

None of the three wrong turns here were exotic. "The fix looks right but does nothing" was a log level. "The clear-only version regressed" was a hook that did half its job and returned success anyway, because clearing genuinely is correct and necessary — it just isn't sufficient. The instrumentation that finally cracked it wasn't clever; it was making the diagnostic state (hooks_registered, session_already_built, restore_reprimed) observable on the request path instead of trusting init-phase logs that, it turned out, we weren't even looking at correctly.

If you're chasing a Lambda cold start and the fix you shipped isn't moving the number: check what log level is actually filtering your diagnostics before you start doubting your architecture.

This is the same instinct behind the product: measure what's actually happening in your AWS account before acting on it. If you want to see what that looks like pointed at your own bill, a free, read-only scan takes about five minutes and changes nothing without you approving it first.

Why Read-Only Is the Only Safe Way to Let AI Near Your AWS Account

Rick Wise — Wed, 12 Aug 2026 13:54:04 +0000

Every AWS cost tool eventually asks you for the same terrifying thing: an IAM role. And every vendor says the same reassuring word about it: "read-only." I want to tell you exactly what that word means when CloudWise says it, because "read-only" gets used loosely enough in this industry that the word alone shouldn't be enough to trust anyone — including us. So this post is the IAM, not the marketing.

The role you actually grant

When you connect an AWS account through CloudWise's one-click setup, you launch exactly one CloudFormation stack: the monitoring stack. There is no button in onboarding that launches anything else — startConnect('monitoring') is the only path the connect flow calls.

That stack's policy (cloudwise-cur-setup-template.yaml) is a bespoke allow-list we wrote and maintain, not the AWS-managed ReadOnlyAccess policy. That distinction matters more than it sounds like it should: ReadOnlyAccess is enormous and vague — it grants read access to almost every AWS service, including ones CloudWise has no reason to ever look at. Our policy is scoped to what a cost scan actually needs, action by action: ec2:DescribeInstances, rds:DescribeDBInstances, s3:ListAllMyBuckets, ce:GetCostAndUsage, compute-optimizer:GetEC2InstanceRecommendations, and so on — read the whole thing at cloudcostwise.io/security/permissions, which renders the live action count straight off the template, not a number we typed into a page and forgot to update. Every single statement in that policy is a Get*, Describe*, List*, or BatchGet* call — plus sts:GetCallerIdentity and iam:SimulatePrincipalPolicy, which CloudWise uses to check what a role can do without ever calling it. There is no Put, Create, Delete, Update, Attach, or Modify verb anywhere in that role's policy. It cannot make a single write call against your account. Not "shouldn't" — cannot; IAM will reject the attempt at the API layer before it reaches any resource.

(One nuance, for the pedants, because I'd rather you catch it than an auditor: the CUR bucket in that same template does have a bucket policy permitting s3:PutObject — granted to AWS's own billingreports.amazonaws.com service principal, not to CloudWise. That's how AWS itself delivers your Cost and Usage Report into a bucket you own. It's AWS writing to you, not us writing to anything.)

Remediation is a different stack, a different role, and a different click

If you want CloudWise to actually fix waste — stop an idle NAT gateway, delete an orphaned snapshot, right-size an instance — that is never part of onboarding. It requires deploying a second, separate CloudFormation stack (CloudWise-Remediation, using cloudwise-remediation-role.yaml) that you reach only from settings/remediation or setup/permissions, deliberately after the point where you've already seen what a read-only scan finds. Nothing in the sign-up or connect flow grants this role. You have to go looking for it.

That role's policy carries an explicit Deny block that overrides everything else, no matter what any other statement in the policy says:

iam:*, organizations:*, sts:* — CloudWise can never touch identity, org structure, or assume other roles
secretsmanager:GetSecretValue / PutSecretValue / CreateSecret / UpdateSecret — no access to your secrets, ever
cloudtrail:DeleteTrail, cloudtrail:StopLogging, config:DeleteConfigRule, config:StopConfigurationRecorder, guardduty:DeleteDetector — your audit and detection trail can't be turned off
s3:DeleteBucket, ec2:DeleteVpc, ec2:DeleteSubnet, ec2:DeleteSecurityGroup, rds:DeleteDBInstance — no deleting the structural stuff that would actually hurt
ec2:AuthorizeSecurityGroupIngress/Egress, ec2:RevokeSecurityGroupIngress/Egress, ec2:CreateSecurityGroup — CloudWise cannot open, close, or create a network path
kms:CreateKey/CreateGrant/Encrypt/Decrypt/GenerateDataKey*, ssm:* — no touching encryption or Systems Manager

A Deny statement in IAM always wins, regardless of what any Allow statement elsewhere in the same policy — or any other policy — grants. That's not a UI promise. It's how the policy evaluates at the API layer.

You'll also see a small number of Create* actions in that same policy file — things like ec2:RunInstances or rds:CreateDBInstance. Those exist for one purpose: rollback. Before executing an approved action, CloudWise records enough state to reverse it — restart an instance it stopped, recreate a resource it deleted, restore a secret's scheduled deletion. If a fix goes wrong, or you change your mind, there's a way back. That's a narrower and more honest thing than "can create resources," and it's also narrower than "can never create resources" — so I'm not going to round it either direction. Read the actual file if you want the specifics; that's the point of publishing it.

Nothing executes without your click

Here's the part I actually care about you believing: CloudWise proposes fixes. It does not execute them on its own initiative, ever, under any circumstance we ship today.

The execution path (lambdas/remediation_executor/handler.py) checks a status field before it will run a single mutating API call, and that status has to read approved. The only thing that writes approved is a separate approval-gateway Lambda, triggered by a verified, authenticated action from you. There is no code path today that sets an action to approved automatically. If you never click approve, the action sits there, proposed, forever, and nothing happens to your account.

There's also a second, quieter layer under that: every mutating call CloudWise's execution role makes is tagged with a session tag — aws:PrincipalTag/cloudwise-action — that CloudWise itself sets at execution time, and the role's policy can further condition on that tag. I want to be precise about what this is and isn't: it's real defense-in-depth inside our own execution path, not a lever you hold. You don't set that tag; we do. The thing you actually control is upstream of it — the approve click that has to happen before any of this fires at all.

What "read-only" means when we say it

So, precisely:

Onboarding grants exactly one role, scoped to Get/Describe/List/BatchGet actions, verifiable line-by-line at cloudcostwise.io/security/permissions. It cannot write to your account. This is the role every CloudWise customer has, whether or not they ever touch remediation.
Remediation is opt-in, separate, and denies the dangerous stuff outright — identity, org structure, secrets, audit trails, network rules, deletion of anything structural — regardless of what else the policy grants.
Nothing mutates without your explicit approval. Propose, then execute only what you approve. Not "propose, then execute unless you object."

None of that requires you to trust our intentions. It requires you to trust IAM evaluation semantics, which is a much smaller ask, and one you can verify yourself against files we publish rather than a page of prose we wrote about ourselves.

If you want to see what the read-only role actually finds in your account, a free, read-only scan takes about five minutes, grants nothing beyond what's described above, and changes nothing without you approving it first.

NAT Gateway: The Bill Nobody Reads

Rick Wise — Mon, 20 Jul 2026 16:03:01 +0000

Group your AWS bill by service in Cost Explorer and NAT Gateway charges don't get their own row. They're folded into Amazon Virtual Private Cloud, sitting next to VPN connections, Transit Gateway attachments, and PrivateLink endpoints. Unless you break the view down by usage type — NatGateway-Hours, NatGateway-Bytes — the number you're actually paying for a gateway that might be doing nothing is invisible.

We wrote up the mechanics of that bill back in March — the $0.045/hour base charge, the $0.045/GB data processing fee, the multi-AZ multiplication that turns one gateway into three. That post is still accurate and worth reading if you want the full pricing breakdown. This one is about something different: what CloudWise's detector actually checks before it tells you a NAT Gateway is dead weight, and where that check still falls short.

What the detector actually looks at

The idle-NAT-gateway check lives in cloudwise_scan_core's network detector, alongside the unattached-EIP and idle-load-balancer checks — it's one pass over a VPC's network resources, not a dedicated NAT scanner. For every NAT Gateway in available state, it pulls two CloudWatch metrics over a 7-day window: ActiveConnectionCount and BytesOutToDestination. If both are zero for the full week, it flags the gateway as IDLE_NAT_GATEWAY at HIGH confidence.

Two design choices worth calling out:

Why 7 days, not 1. A single quiet day doesn't mean a gateway is unused — it might serve a batch job that runs Sundays, or a staging environment nobody touches on weekends. A full week with zero connections and zero bytes is a much harder signal to explain away.

Why the estimate is conservative. The finding's monthly_savings uses a flat $32.40 — the hourly base charge times a 720-hour month, not the $32.85 you'd get from AWS's actual 730-hour average. It also doesn't add anything for data processing, because by definition a gateway that's flagged idle processed zero bytes in the lookback window. The number CloudWise shows you is the floor, not an estimate padded to look impressive.

The finding's risk field is blunt about the tradeoff, too: deleting a NAT Gateway immediately cuts internet egress for every private-subnet resource routed through it. The action is offered — aws ec2 delete-nat-gateway — but nothing executes it without a human approving first. That's the same read-only-first posture behind every detector we ship, not something special-cased for NAT Gateways.

What it doesn't do yet

Here's the honest gap: the detector tells you a NAT Gateway earned its keep for zero dollars this week. It doesn't tell you what to replace it with.

That decision genuinely depends on what's routing through it:

If the only traffic is S3 or DynamoDB, a Gateway VPC Endpoint replaces it for free — no hourly charge, no per-GB fee, ever.
If it's other AWS services (Secrets Manager, ECR, CloudWatch Logs), an Interface VPC Endpoint runs $0.01/hour per AZ plus $0.01/GB — about 78% cheaper per gigabyte than NAT, though you're paying a small hourly fee per endpoint per AZ instead of one gateway.
If it's a non-production environment with real internet egress needs but low traffic and no requirement for managed HA, a self-managed NAT instance (something like a t4g.nano) can run for a few dollars a month — you trade the 24/7 base charge for patching it yourself.

CloudWise doesn't make that call for you today. It's a fair next detector to build — matching an idle gateway's actual destination traffic against what a Gateway Endpoint could cover for free — but until it exists, the decision after "this is idle" is still on you. We'd rather say that plainly than imply the tool does more than it does.

Finding yours

If you want to see this in your own account without reading CloudWatch dashboards by hand, a free scan checks this along with 190+ other waste patterns across 40+ AWS services — read-only, five minutes, nothing gets deleted without you clicking it. We posted a 30-second walkthrough of this exact detector if you'd rather watch than read.

Or just run the two-metric check yourself:

aws cloudwatch get-metric-statistics \
  --namespace AWS/NATGateway \
  --metric-name BytesOutToDestination \
  --dimensions Name=NatGatewayId,Value=nat-0abc123def456 \
  --start-time $(date -u -v-7d +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 86400 \
  --statistics Sum

Seven zeros in a row is $32.40 a month for a gateway that isn't gating anything.

Why I Built a Risk Score Instead of a Buy Button

Rick Wise — Wed, 08 Jul 2026 17:26:13 +0000

The obvious feature to build here is a button. "You're spending $4,200/month on m5.xlarge — buy a 1-year Savings Plan and save 30%." One click, instant discount, everybody's happy.

I almost built that button. I'm glad I didn't.

Here's the problem with the button: it's only looking at the last three months. It has no idea whether the workload it's telling you to commit to will still exist in month nine. And a Reserved Instance or Savings Plan isn't a coupon — it's a bet, paid up front or amortized monthly, that a specific slice of your infrastructure will look roughly the same for a year or three. Get that bet wrong and the "savings" tool just talked you into a liability.

So instead of a buy button, I built a Commitment Risk Score — a feature whose entire job is to sometimes tell you not to buy anything yet.

The feature that argues with itself

The Commitment Risk Score pulls four signals straight from Cost Explorer for an account: how much your top instance families churn month to month, how volatile your spend is, how long your individual resources actually live, and how well you're already using the commitments you have. It weights them (35/25/25/15) into a single 0–100 score, and that score maps to a recommendation — anywhere from "3-year Convertible RI, take the discount" down to "on-demand and Spot only, buy nothing."

That last outcome is the part that made this feature interesting to build. Most cost-optimization tools are graded on how much they tell you to save. This one is graded on how honest it is about when a "savings" purchase would actually be a mistake. It costs about $0.04/account/month to compute — four GetCostAndUsage calls, refreshed weekly — the recommendation quality has nothing to do with the compute cost, so there was no excuse to cut a signal to save pennies.

I wrote up the full math — the Jaccard-distance churn calculation, the coefficient-of-variation volatility threshold, the worked examples — in a separate deep dive, because the mechanics deserved their own space: The Commitment-Risk Score: should you buy that RI?

What almost went wrong

The instinct that almost got me was averaging. My first pass weighted all four signals close to evenly, because "why not, they're all relevant." It took building the worked examples to see the problem: a team mid-Graviton-migration with high churn but a stable dollar total would score fine on an even-weighted average, because volatility and churn partially cancel out in the wrong direction. The churn signal needed to dominate — 35%, not 25% — because a changing instance mix is structurally the most common way a commitment gets stranded, regardless of what the topline spend number is doing.

The existing-waste signal ended up smallest at 15%, which felt backwards at first — isn't "you're already wasting money" the most damning fact? It is, but it's also the least predictive one. It tells you about the past, not whether the next commitment will strand. It stayed in the score because it's a legitimate red flag, but it doesn't get to drown out the forward-looking signals.

Why this is the right shape for a cost tool

CloudWise's whole premise is read-only: we look at your AWS usage and billing data, we never touch your infrastructure, and every dollar figure we show you needs to survive you checking it against Cost Explorer yourself. A recommendation engine that only ever says "buy more" doesn't survive that scrutiny for long — eventually it recommends a commitment that strands, and you stop trusting the number.

A feature that's willing to say "not yet, here's why" is the one worth trusting the next time it says "yes, buy it." That was true before I shipped this, and building it just made me believe it more.

CloudWise is an AWS cost optimization tool for startups — 191 automated waste checks including commitment-risk scoring, read-only by design, starting at $19/month. Run a free scan at cloudcostwise.io.

EC2 Cost Optimization: Are You Ready to Commit to Reserved Instances?

Rick Wise — Wed, 10 Jun 2026 12:38:36 +0000

RI and Savings Plan utilization rates below 80% are far more common than teams admit. The dashboards show green, the discount applied, the finance team is happy — and meanwhile a chunk of every commitment is being paid for and not used.

Here's how it usually happens. You buy a 1-year EC2 Instance Savings Plan on m5.xlarge. All-upfront, that's roughly $1,075 for the year. Six months in, someone runs a Graviton benchmark, the numbers are great, and you migrate the fleet to m7g. That EC2 Instance Savings Plan was scoped to the m5 family. It doesn't follow you. You now pay for the m7g capacity on-demand and keep paying off the stranded m5 commitment. The "savings" turned into a double bill.

The mistake isn't buying Reserved Instances. RIs and Savings Plans are the single biggest lever on an EC2 bill — 30% to 70% off on-demand. The mistake is buying them before your architecture is stable enough to actually use them for the full term.

And the heuristic most teams use to decide is backwards. "We've been running this instance type for three months, let's commit." That's looking in the rear-view mirror. A commitment is a bet on the next 12 to 36 months, not the last three. The real question is: how likely is this workload to still look like this a year from now?

You can answer that with data you already have. There are four signals, each measurable from Cost Explorer or the AWS CLI, that together tell you whether you're ready to commit.

Four signals that tell you if you're ready

These are the four signals that combine into a single 0–100 risk score. The weights below aren't arbitrary — they reflect how much each one actually predicts a stranded commitment. You can compute every one of them yourself.

Signal 1 — Instance Family Churn (weight: 35%)

This is the heaviest signal, because a changing instance mix is the most common way commitments get stranded. It measures how much your top instance families shift month over month.

The method: pull Cost Explorer grouped by INSTANCE_TYPE_FAMILY at monthly granularity over the last 6 months. You can do exactly that from the CLI — set the dates to your own 6-month window:

aws ce get-cost-and-usage \
  --time-period Start=2026-01-01,End=2026-06-01 \
  --granularity MONTHLY \
  --metrics "UnblendedCost" \
  --group-by Type=DIMENSION,Key=INSTANCE_TYPE_FAMILY \
  --query "ResultsByTime[].Groups[].{Family:Keys[0],Cost:Metrics.UnblendedCost.Amount}" \
  --output table

For each month, take the top 5 families by spend. Then measure how different consecutive months are using Jaccard distance:

Jaccard distance = 1 − |intersection| / |union|

A distance of 0 means the two months had identical top-5 families (no churn). A distance of 1 means they shared nothing (total churn).

Worked example. Say your top families this period are {m5, r5, c5, t3, r6i}, and a few months later, mid-Graviton-migration, they're {m7g, r7g, c7g, t4g, m5}. The only family in both sets is m5. So the intersection is {m5} (size 1) and the union is all 9 distinct families. That gives:

1 − 1/9 = 0.89

0.89 is very high churn. Your workload is structurally different than it was, and any family-specific commitment you bought before the migration is now mostly dead weight.

The score averages the Jaccard distance across each consecutive month-pair over the window.

Risk threshold: an average Jaccard distance above 0.3 over 6 months means your fleet is moving too fast for long-term, family-locked commitments. Graviton migrations, containerizing onto a different family, and ML-pipeline rebuilds all push this number up fast.

Signal 2 — Spend Trend Volatility (weight: 25%)

Churn tells you what you're running; volatility tells you how much. A workload can stay on the same families but swing wildly in size — and a commitment sized to a peak month is wasted in a trough month.

The metric is the coefficient of variation (CV) of your monthly EC2 spend over 6 months:

CV = (standard deviation / mean) × 100

Worked example: six months averaging $4,200/month with a standard deviation of $1,260 gives a CV of 30%. That's a third of your spend bouncing around month to month — too much to safely commit to a fixed hourly floor.

Risk threshold: CV above 25% is too unpredictable to commit. (In the scoring model, CV is doubled and capped, so a 50% CV maxes out this signal at 100.)

One nuance worth internalizing: if your spend is volatile but you still want some commitment, Compute Savings Plans are far safer here than EC2 Instance Savings Plans. Compute SPs apply across families, sizes, regions, and even Fargate/Lambda, so a flexible commitment sized to your baseline survives the swings.

Signal 3 — Resource Lifecycle Duration (weight: 25%)

This one catches ephemeral fleets. If your individual instances live for days, not months, a 1-year commitment is the wrong instrument no matter how stable the aggregate spend looks.

The metric is the median number of days your individual EC2 instances stay alive — median, not mean, because a few long-lived bastion hosts will drag a mean upward and hide a fleet of short-lived workers. You can derive it from daily Cost Explorer data grouped by RESOURCE_ID, or sanity-check it directly against the live fleet with one CLI call:

aws ec2 describe-instances \
  --filters Name=instance-state-name,Values=running \
  --query "Reservations[].Instances[].{ID:InstanceId,Type:InstanceType,Launch:LaunchTime}" \
  --output table

Sort the output by Launch and eyeball the median age.

Risk threshold: a median instance lifetime under 30 days signals an ephemeral workload where RIs are the wrong tool. In the scoring model, a median of 30 days or less scores the maximum, and 180+ days scores zero, with a linear slide between.

Context: ECS on Spot, batch processing, ephemeral ML training jobs, and aggressive scale-in all produce short lifetimes. Steady-state app servers, databases, and background workers produce long ones — and those are exactly the resources that are safe to commit to.

Signal 4 — Existing Commitment Utilization (weight: 15%)

The lowest weight, but the most damning when it's bad — because it's direct evidence you're already wasting commitments. Before buying more, look at how well you're using what you have.

Check Cost Explorer → Savings Plans → Utilization Report, plus the RI Utilization Report. The score is simply the inverse of your average utilization: 100% utilized scores 0 (no waste), 0% utilized scores 100.

Risk threshold: existing utilization under 80% means fix this before buying anything new.

The trap here is subtle and worth stating plainly: buying more commitments does not fix low utilization — it buries the signal. New unused capacity drags the blended utilization number around and makes the underlying waste harder to see. If you're below 80%, the next purchase isn't optimization, it's compounding a mistake.

Mapping your score to a commitment decision

The four signals combine — weighted 35 / 25 / 25 / 15 — into a single 0–100 risk score. Here's how that score maps to an actual purchasing decision, mirroring the labels the model emits.

Low risk — stable architecture, churn < 0.3, CV < 15%, lifecycle > 90 days, utilization > 90%.
→ 3-year Convertible RIs or a 1-year Compute Savings Plan. This is the maximum-savings zone. Your architecture has earned the commitment; go take the discount.

Medium risk — some volatility, a Graviton migration partway done, moderate churn.
→ 1-year Compute Savings Plan only. Avoid EC2 Instance Savings Plans and standard RIs here — they're too specific for a fleet still in motion. A Compute SP applies to any instance family, size, and OS, which makes it the right instrument to capture savings on your stable baseline while the rest of the fleet shifts underneath it.

High risk — actively containerizing, churn > 0.6, short lifecycles, existing utilization under 70%.
→ On-demand plus Spot. No new commitments. Lean on Spot for fault-tolerant and interruptible workloads, and reserve any Compute SP strictly for the genuinely stable baseline portion of your compute, if there is one.

One clarification that trips people up: Convertible RIs can be exchanged for a different family, size, or region, so they're often pitched as "flexible." But the exchange is a manual process with real restrictions, and it's nowhere near as fluid as a Compute Savings Plan that just applies automatically. When in doubt mid-transition, the Compute SP is the safer flexible bet.

The 15-minute audit: check your account right now

You don't need a tool to get a first read. Here's the manual version of all four signals, end to end.

Step 1 — Churn. Cost Explorer → Group by Instance Type Family → Monthly granularity → last 6 months. Compare the top-5 families month over month. If the set is visibly reorganizing — new families entering, old ones dropping out — that's churn. Eyeball whether a typical month-to-month change shares fewer than ~3 of its top 5 families with the prior month (that's roughly the 0.3 distance line).

Step 2 — Volatility. Export your monthly EC2 totals for the same 6 months. Calculate the mean and standard deviation, then CV = std_dev / mean × 100. If CV is above 25%, flag it.

Step 3 — Lifecycle. Run the describe-instances command from Signal 3, sort by LaunchTime, and estimate the median instance age in days. Under 30 days is a red flag.

Step 4 — Utilization. Cost Explorer → Savings Plans → Utilization Report for the last 3 months. If you're under 80%, stop — understand why before you buy anything else.

Fifteen minutes of Cost Explorer and one CLI command will tell you more about your readiness to commit than any "we've been running this for a while" gut check.

Automated detection

If you'd rather have this calculated automatically than pull Cost Explorer data by hand every quarter, CloudWise computes a Commitment Risk Score as part of its RI/SP management view (shipped June 2, 2026). It combines the same four signals using the weights above and returns a 0–100 score, a label (LOW / MEDIUM / HIGH / CRITICAL), and a recommended maximum commitment term — so the buy/don't-buy decision comes with the math already done. You can find it at cloudcostwise.io.

The discount on a Reserved Instance is real, but it's only a discount if you use the whole term. Run the four signals first. If your architecture is stable, commit with confidence and take the savings. If it's moving, stay flexible — a Compute Savings Plan on your baseline, Spot for the rest — and revisit when the numbers settle. The worst outcome isn't paying on-demand a little longer. It's locking in a year of spend on a fleet you've already left behind.

ElastiCache Pricing Breakdown: Where the Money Actually Goes

Rick Wise — Thu, 16 Apr 2026 14:29:01 +0000

ElastiCache looks straightforward on the bill. You pick a node type, maybe add a replica for high availability, and move on. Then the invoice arrives and the number is bigger than the mental math suggested.

The gap usually comes from one of five places: engine choice, replication topology, extended support surcharges, idle clusters, or oversized nodes nobody ever right-sized. Let's break down exactly how ElastiCache charges — and where teams get surprised.

Three Engines, Three Price Points

ElastiCache supports three engines: Valkey, Redis OSS, and Memcached. They don't cost the same.

Valkey is 20% cheaper than Redis OSS and Memcached for node-based clusters, and 33% cheaper on ElastiCache Serverless. This isn't a promotional rate — it's the permanent pricing structure AWS launched with Valkey.

For context, a cache.r7g.xlarge in us-east-1:

Engine	Hourly Rate	Monthly (730 hrs)
Valkey	$0.3496	~$255
Redis OSS	$0.437	~$319
Memcached	$0.437	~$319

Prices shown for us-east-1, On-Demand.

That's a $64/month difference per node on a single instance type. Multiply that across a 12-node cluster and you're looking at $768/month — just from engine choice. If you're running Redis OSS and don't need Redis-specific features that Valkey doesn't support, the migration saves real money.

Node-Based Pricing: You Pay Whether the Cache Is Hit or Not

ElastiCache charges per node-hour from the moment a node is launched until it's terminated. Partial hours are billed as full hours. There is no scale-to-zero.

A few common node types and what they cost:

Node Type	Memory	Hourly Rate	Monthly (730 hrs)
cache.t3.micro	0.5 GiB	$0.017	~$12
cache.m5.large	6.38 GiB	$0.156	~$114
cache.r7g.xlarge	26.32 GiB	$0.437	~$319
cache.r6g.16xlarge	419.09 GiB	$5.254	~$3,835

Prices shown for Redis OSS / Memcached in us-east-1, On-Demand. Valkey is 20% lower.

The important thing to internalize: a cache.t3.micro sitting idle costs the same $12/month as one handling thousands of requests per second. The meter runs on time, not usage.

AWS recommends reserving 25% of a node's memory for non-data use (replication buffers, OS overhead, etc.), so the usable capacity of a cache.r7g.xlarge is roughly 19.74 GiB, not 26.32 GiB.

Replication Multiplies the Bill

Most production deployments use replication for high availability. With Redis OSS or Valkey, you configure a replication group with a primary node and one or more replica nodes per shard.

Every replica is a full node charged at the same hourly rate.

A three-shard cluster with one replica per shard using cache.r7g.xlarge (Valkey):

3 shards × 2 nodes per shard = 6 nodes
6 × $0.3496/hr = $2.10/hr → ~$1,531/month

Add a second replica for read scaling:

3 shards × 3 nodes per shard = 9 nodes
9 × $0.3496/hr = $3.15/hr → ~$2,297/month

Plus, multi-AZ replication generates cross-AZ data transfer at $0.01/GiB in each direction. For a high-throughput cache doing 100,000 requests/second with 500-byte objects, that's roughly 167 GiB/hour of traffic. If 50% crosses AZ boundaries, that's an extra $0.84/hour — about $613/month in data transfer alone.

Teams often enable multi-AZ replication on dev and staging environments where a single node would be fine.

Serverless: Simpler, But Not Always Cheaper

ElastiCache Serverless removes the node sizing decision entirely. You pay for two things:

Data stored — billed in GB-hours
ElastiCache Processing Units (ECPUs) — a unit combining vCPU time and data transferred

Dimension	Valkey	Redis OSS	Memcached
Data storage	$0.084/GB-hr	$0.125/GB-hr	$0.125/GB-hr
ECPUs	$0.0023/M	$0.0034/M	$0.0034/M
Minimum data stored	100 MB	1 GB	1 GB

Prices shown for us-east-1.

A simple GET or SET transferring under 1 KB consumes 1 ECPU. A command transferring 3.2 KB consumes 3.2 ECPUs. Commands that use more vCPU time (like SORT or ZADD) consume proportionally more.

Serverless can be cheaper for spiky workloads because you don't over-provision for peaks. But for stable, high-throughput workloads, node-based clusters are often significantly cheaper. AWS's own Example 2 shows a spiky workload costing $2.92/hour serverless vs. $5.66/hour on-demand nodes — but for steady traffic, the math can flip the other way.

The minimum charge matters too. A Serverless cache for Redis OSS or Memcached is metered for at least 1 GB of data stored — roughly $91/month minimum even if you're storing almost nothing. Valkey's 100 MB minimum brings that floor down to about $6/month.

Extended Support: The Surcharge Nobody Budgets For

When a Redis OSS or Memcached engine version reaches end-of-life, AWS continues providing security patches through Extended Support — at a steep premium.

Period	Surcharge
Year 1–2 after EOL	80% premium on node-hour rate
Year 3 after EOL	160% premium on node-hour rate

A cache.m5.large running Redis 5 (EOL January 31, 2026) at $0.156/hour becomes:

Year 1–2: $0.156 + ($0.156 × 80%) = $0.281/hour (~$205/month)
Year 3: $0.156 + ($0.156 × 160%) = $0.406/hour (~$296/month)

That's nearly triple the base cost by year three. Teams that don't track engine versions can drift into Extended Support without realizing their bill just jumped 80%.

Backup Storage and Data Transfer

Two cost categories that don't appear under the main "ElastiCache" line:

Backup storage: $0.085/GiB per month for all regions. No data transfer charges for creating or restoring backups. This is generally small unless you're snapshotting large clusters frequently.

Data transfer:

Path	Cost
Same AZ (EC2 ↔ ElastiCache)	Free
Cross-AZ (same Region)	$0.01/GiB each way
Cross-Region (Global Datastore)	$0.02/GiB

The cross-AZ charge is easy to miss because it shows up as EC2 data transfer on the bill, not ElastiCache. You're only charged for the EC2 side — there's no ElastiCache data transfer charge for traffic in or out of the node itself.

Data Tiering: The Cost Saver Most Teams Don't Know About

R6gd nodes combine memory and NVMe SSD, automatically moving least-frequently-accessed data to SSD. You get nearly 5× the total storage capacity compared to memory-only R6g nodes.

AWS's example: a 1 TiB dataset needs 1 cache.r6gd.16xlarge node ($9.98/hour) vs. 4 cache.r6g.16xlarge nodes ($21.01/hour) — a 52% cost reduction.

The trade-off: SSD-resident data has slightly higher latency on first access. If your workload regularly accesses less than 20% of the dataset, data tiering is worth evaluating.

Data tiering is not available with ElastiCache Serverless.

Reserved Nodes: Up to 55% Off

If your ElastiCache usage is stable, reserved nodes offer steep discounts:

Commitment	Discount vs. On-Demand
1-year, No Upfront	Up to 48.2%
1-year, Partial Upfront	Up to 52%
3-year, All Upfront	Up to 55%

Reserved nodes are size-flexible — you can apply the discount across different node sizes within the same family. If you buy a reservation for cache.r7g.xlarge, it can cover cache.r7g.large nodes proportionally.

One useful detail: Redis OSS reservations automatically apply to Valkey nodes in the same family and region. Since Valkey is 20% cheaper, you get 20% more value from existing reservations after migrating.

The Real Problem: Idle Caches

Here's what actually burns money: caches nobody is using.

ElastiCache has no scale-to-zero for node-based clusters. A cache with zero hits costs exactly the same as one handling millions of requests. This is the pattern we see most often:

A team provisions a cache for a microservice, then the service is deprecated
Dev/staging caches left running after the project ends
A "temporary" cache for a migration that became permanent infrastructure
A replicated cluster in non-production where a single node would suffice

A three-node cache.r7g.xlarge cluster running idle for a year at Valkey on-demand rates: $9,186 wasted.

Over-Provisioned Caches Are Nearly as Bad

Beyond idle caches, oversized nodes are the second biggest source of waste. Teams pick a large node type during initial setup, the workload stabilizes at a fraction of capacity, and nobody revisits the sizing.

A cache.r6g.xlarge running at 6% CPU with active connections is doing real work — but it's doing it on a node that's 3–4× larger than needed. Downsizing from cache.r6g.xlarge to cache.r6g.large can cut costs by 40–50% with no performance impact.

How to Spot the Waste

Check these CloudWatch metrics for each cluster:

CacheHits: Zero for 14+ days means nothing is reading from this cache
CurrConnections: Zero means nothing is even connecting
EngineCPUUtilization: Consistently under 10% with active connections means the node is oversized

Quick CLI inventory of all your ElastiCache clusters:

aws elasticache describe-cache-clusters \
  --show-cache-node-info \
  --query 'CacheClusters[*].{
    ClusterId:CacheClusterId,
    Engine:Engine,
    EngineVersion:EngineVersion,
    NodeType:CacheNodeType,
    NumNodes:NumCacheNodes,
    Status:CacheClusterStatus
  }' \
  --output table

If any of those clusters show an engine version approaching EOL, you're on the clock for an Extended Support surcharge.

CloudWise detects idle ElastiCache clusters by analyzing CloudWatch cache hit metrics over 14 days, flags oversized nodes running under 10% CPU, and alerts you when clusters are approaching or already incurring Extended Support surcharges. Three detectors, one scan.

CloudWise automates AWS cost analysis across 180+ waste detectors. Try it at cloudcostwise.io.

How Timestream Actually Bills: A Breakdown for Engineers

Rick Wise — Thu, 09 Apr 2026 13:56:59 +0000

Timestream can look simple on the bill until you break down the line items. Most teams think in terms of "stored data," but Amazon Timestream for LiveAnalytics is billed across multiple meters that move independently.

If you only watch one number, you can miss where most of the spend actually comes from.

First: Know Which Timestream Product You Are Using

AWS now has two Timestream offerings:

Amazon Timestream for LiveAnalytics (serverless, billed by writes, query compute, memory store, magnetic store)
Amazon Timestream for InfluxDB (managed InfluxDB, billed by DB instance-hours and storage)

This post focuses on Timestream for LiveAnalytics, where most billing misunderstandings happen.

The LiveAnalytics Billing Model (What Actually Ticks)

For Timestream for LiveAnalytics, AWS charges separately for:

Writes: billed by amount of data written (rounded to nearest KiB), often shown in pricing examples as a per-million 1 KiB write unit.
Queries: billed by Timestream Compute Units (TCUs) consumed over time (TCU-hours), not by a flat per-query fee.
Memory store: billed by GB-hour.
Magnetic store: billed by GB-month (with account/region minimums for magnetic storage usage).

So the statement "Timestream is $0.10/GB-month" is not accurate for LiveAnalytics. That kind of single storage rate framing is incomplete and often wrong.

Why Teams Get Surprised

1) Query charges are compute-time based, not per query count

A dashboard running one heavy query every few seconds can cost more than many lightweight queries. Query cost follows TCU consumption and duration.

2) Memory retention is expensive relative to magnetic retention

Keeping a long retention period in memory store drives GB-hour charges. Moving older data to magnetic store usually lowers storage cost.

3) High write frequency amplifies ingestion cost fast

Small records at high frequency still add up, especially without batching and schema optimization.

4) "Idle table" thinking can be misleading

In LiveAnalytics, empty or unused tables are not the main cost driver. Spend usually comes from write volume, query compute, and retained data in memory/magnetic tiers.

A Better Mental Model for Retention

Retention decisions directly shape spend:

Short memory retention for hot, low-latency workloads
Longer magnetic retention for historical analysis
Keep only what is needed in memory store

If your query patterns are mostly historical and not sub-second operational reads, memory retention is often set too high.

Practical Cost Review Checklist

Run this monthly:

Writes: are records batched efficiently? Are you writing unnecessary dimensions/measures?
Queries: which workloads consume most TCU time? Any dashboards refreshing too frequently?
Memory store: is hot retention longer than real operational need?
Magnetic store: is long-term retention aligned with compliance and analytics requirements?
Table lifecycle: are stale datasets still retained without business need?

CLI: Inventory Retention Settings Across Tables

Use this to review memory/magnetic retention quickly:

aws timestream-write list-databases --query 'Databases[].DatabaseName' --output text | tr '\t' '\n' | while read db; do
  [ -z "$db" ] && continue
  aws timestream-write list-tables --database-name "$db" --query 'Tables[].TableName' --output text | tr '\t' '\n' | while read tbl; do
    [ -z "$tbl" ] && continue
    aws timestream-write describe-table \
      --database-name "$db" \
      --table-name "$tbl" \
      --query '{Database:Table.DatabaseName,Table:Table.TableName,MemoryHours:Table.RetentionProperties.MemoryStoreRetentionPeriodInHours,MagneticDays:Table.RetentionProperties.MagneticStoreRetentionPeriodInDays}'
  done
done

This does not prove usage by itself, but it gives you the retention map you need before optimizing query and write behavior.

Bottom Line

Timestream for LiveAnalytics billing is multi-dimensional:

writes
query compute (TCUs)
memory store
magnetic store

If you treat it as a single storage bill, you will miss the biggest optimization levers.

CloudWise helps teams surface these hidden cost patterns and prioritize the fastest savings opportunities across AWS data services.

CloudWise automates AWS cost analysis and waste detection. Try it at cloudcostwise.io.

The Hidden Costs of Idle EMR Clusters (And How to Stop the Bleed)

Rick Wise — Thu, 02 Apr 2026 14:36:43 +0000

EMR looks simple on the bill. You spin up a cluster, run your Spark jobs, and shut it down. But most teams don't shut it down — and that's where the money disappears.

EMR Has Two Price Tags

Every EMR instance carries two charges:

EC2 instance cost — the standard on-demand rate for the instance type
EMR surcharge — an additional per-instance-hour fee, typically 20–25% of the EC2 price

For a common analytics instance like m5.xlarge (4 vCPUs, 16 GB RAM) in us-east-1:

Component	Hourly Cost
EC2	$0.192
EMR surcharge	$0.048
Total	$0.240/hr

A 5-node cluster of m5.xlarge instances costs $1.20/hr — roughly $876/month if left running. That's just compute. Storage is extra.

Most teams focus on the EC2 line item and completely miss the EMR surcharge. It doesn't show up as a separate line — it's bundled into the "Amazon Elastic MapReduce" charge on your bill, and it adds up fast across multiple clusters.

The EBS Trap

Every EMR node gets EBS volumes attached. The default root volume is typically 10–15 GB, but core and task nodes often get larger volumes for HDFS or local shuffle storage.

Current EBS pricing in us-east-1:

Volume Type	Cost
gp3 (default for new clusters)	$0.08/GB-month
gp2 (legacy default)	$0.10/GB-month
io1 (provisioned IOPS)	$0.125/GB-month + $0.065/IOPS

A 5-node cluster with 100 GB gp3 per node adds $40/month in storage alone. Not huge — but it never stops charging, even when the cluster is idle.

The real problem isn't the per-GB rate. It's that EBS charges continue as long as the cluster exists, regardless of whether any jobs are running.

The Idle Cluster Problem

Here's the scenario that burns money: a cluster in WAITING state.

EMR clusters have three relevant states:

RUNNING — actively executing steps (Spark, Hive, Presto jobs)
WAITING — cluster is up, all steps are complete, waiting for new work
TERMINATED — shut down, no charges

The WAITING state is the silent budget killer. The cluster is fully provisioned — all EC2 instances running, all EBS volumes attached, EMR surcharge ticking — but doing zero work. It's an idle engine burning fuel in a parked car.

This happens more often than you'd think:

Dev/test clusters spun up for debugging, never terminated
Scheduled pipelines where the cluster outlives the job by hours or days
"Keep-alive" clusters left running for ad-hoc queries that happen once a week
Failed termination where auto-termination was configured but a step error left the cluster hanging

A 10-node r5.2xlarge cluster in WAITING state costs roughly $5,256/month — EC2 ($0.504/hr × 10 × 730) plus EMR surcharge ($0.126/hr × 10 × 730) plus EBS. For processing zero bytes of data.

What to Actually Check

If you want to audit your EMR spend, focus on three things:

1. Clusters in WAITING state for more than 24 hours

aws emr list-clusters --active --query 'Clusters[?Status.State==`WAITING`]'

Any cluster that's been waiting more than a day is almost certainly forgotten. Check the ReadyDateTime in the timeline to see how long it's been idle.

2. Long-running clusters with no recent steps

Some teams run "persistent" EMR clusters for interactive workloads (Jupyter, Presto). These are valid — but they should be right-sized. Check list-steps to see when the last step actually ran.

aws emr list-steps --cluster-id j-XXXXX --step-states COMPLETED \
  --query 'Steps[0].Status.Timeline.EndDateTime'

If the last step completed weeks ago, the cluster is waste.

3. Auto-termination configuration

EMR supports auto-termination after idle timeout. If your clusters don't have this enabled, you're one forgotten SSH session away from a surprise bill.

aws emr describe-cluster --cluster-id j-XXXXX \
  --query 'Cluster.AutoTerminationPolicy'

The Fix

For batch workloads, the answer is straightforward: use transient clusters. Spin up, process, terminate. EMR's step execution mode does this automatically — the cluster terminates after the last step completes.

For interactive workloads, set aggressive auto-termination policies (1–2 hours of idle time) and right-size instance types based on actual utilization, not peak estimates from six months ago.

And tag everything. You can't optimize what you can't attribute. Use aws:elasticmapreduce:editor-id and custom cost allocation tags to tie clusters back to teams and projects.

CloudWise detects idle and long-running EMR clusters automatically, flags the monthly waste, and generates remediation plans. Try it at cloudcostwise.io.

DEV Community: Rick Wise

I Shipped a Redesign That Orphaned My Own Product

What I built

The part where it looked finished

The front door had hidden the house

The agent was also lying about multi-account

Thirty days of taking ownership

The lesson, stated plainly

EBS Snapshot Sprawl: The Waste Cost Explorer Can't Show You

Cost Explorer shows you a bill, not a culprit

deregistered ≠ deleted — the AMI-pinning trap

Lifecycle policies are the actual fix — for snapshots you haven't made yet

Finding yours

The Bug That Turned Every Bad Password Into a Server Outage

The bug

Why the status code is the actual bug

How it actually got caught

The fix

The takeaway

Inside a Real $11,871/mo AWS Waste Scan

Where the 18 findings live

The one number that matters most

What the next four biggest opportunities have in common

Why this matters for how you should read your scan

We Cut Lambda Cold Starts 56% — Three Wrong Turns Before the Real SnapStart Fix

What was actually slow

The diagnosis: init phase vs. handler phase

Wrong turn #1: it shipped clean and did nothing

Wrong turn #2 (well, half a turn): it was a log level

The actual bug: after_restore cleared, but never rebuilt

The result

The actual lesson

Why Read-Only Is the Only Safe Way to Let AI Near Your AWS Account

The role you actually grant

Remediation is a different stack, a different role, and a different click

Nothing executes without your click

What "read-only" means when we say it

NAT Gateway: The Bill Nobody Reads

What the detector actually looks at

What it doesn't do yet

Finding yours

Why I Built a Risk Score Instead of a Buy Button

The feature that argues with itself

What almost went wrong

Why this is the right shape for a cost tool

EC2 Cost Optimization: Are You Ready to Commit to Reserved Instances?

Four signals that tell you if you're ready

Signal 1 — Instance Family Churn (weight: 35%)

Signal 2 — Spend Trend Volatility (weight: 25%)

Signal 3 — Resource Lifecycle Duration (weight: 25%)

Signal 4 — Existing Commitment Utilization (weight: 15%)

Mapping your score to a commitment decision

The 15-minute audit: check your account right now

Automated detection

ElastiCache Pricing Breakdown: Where the Money Actually Goes

Three Engines, Three Price Points

Node-Based Pricing: You Pay Whether the Cache Is Hit or Not

Replication Multiplies the Bill

Serverless: Simpler, But Not Always Cheaper

Extended Support: The Surcharge Nobody Budgets For

Backup Storage and Data Transfer

Data Tiering: The Cost Saver Most Teams Don't Know About

Reserved Nodes: Up to 55% Off

The Real Problem: Idle Caches

Over-Provisioned Caches Are Nearly as Bad

How to Spot the Waste

How Timestream Actually Bills: A Breakdown for Engineers

First: Know Which Timestream Product You Are Using

The LiveAnalytics Billing Model (What Actually Ticks)

Why Teams Get Surprised

1) Query charges are compute-time based, not per query count

2) Memory retention is expensive relative to magnetic retention

3) High write frequency amplifies ingestion cost fast

4) "Idle table" thinking can be misleading

A Better Mental Model for Retention

Practical Cost Review Checklist

CLI: Inventory Retention Settings Across Tables

Bottom Line

The Hidden Costs of Idle EMR Clusters (And How to Stop the Bleed)

EMR Has Two Price Tags

`deregistered ≠ deleted` — the AMI-pinning trap

The actual bug: `after_restore` cleared, but never rebuilt