DEV Community: Rick Wise

SR&ED documentation as a solo founder: log technological uncertainty while you build

Rick Wise — Thu, 24 Sep 2026 13:27:37 +0000

The SR&ED claim took me longer than the feature I was claiming for.

Not the form. The reconstruction. CRA wants the technological uncertainty documented: what you didn't know, what you tried, why the obvious approach failed. I had none of that written down. I had commits.

Rebuilding that narrative from about 14 months of git history and AI-assisted coding sessions took roughly three weeks I had not budgeted. This post is about the habit that would have made it a few days: a running log, two lines per dead end, written on the day the dead end happens.

Context, so you can weigh this properly. I'm the solo founder of CloudWise, an AWS cost optimization tool: 189 waste checks across 40+ AWS services, built by one person, pre-revenue, no paying customers. I paused feature development in August 2026 to work on distribution. I filed the SR&ED claim myself, without a consultant, and later received a CRA T2 notice of reassessment that resulted in a refund. I have filed exactly once. Everything below is one founder's experience, not advice on whether your work is eligible.

The problem: reconstructing SR&ED documentation after the fact

If you are mid-build, your record of the work is probably the same as mine was: a git log, some pull request descriptions, a ticket tracker, and a pile of chat sessions with an AI coding assistant.

That record answers one question well: what changed. It answers a different question badly: what did you not know at the time, and how did you find out.

A commit message like fix: handle untagged resources in attribution tells you a fix landed. It does not tell you whether that fix was the first approach or the fourth, why the earlier ones failed, or what each failure taught you. The failed approaches are often not in the history at all. They lived on a branch you deleted, or in a session you never saved, or only in your head.

So the three weeks were not writing time. They were archaeology: reading my own diffs in order, working out which ones marked a change of approach, then trying to recover why I had abandoned the previous one. Fourteen months later, the why is the part that has decayed the most.

What CRA actually asks you to document on the T661

I am going to stay close to the source here, because this is the part where founder blog posts tend to freelance.

The claim form is the T661. Its project section asks, in order, for three things: the scientific or technological uncertainties you faced, the work you did to try to overcome them, and the advancement you achieved or attempted. CRA's guide to Form T661 (T4088) walks through those lines and their word limits. The same guide has you indicate what supporting evidence exists for the work, and notes that you keep that evidence rather than submit it with the claim.

CRA's guidelines on the eligibility of work describe technological uncertainty as a situation where it is unknown whether, or how, a result can be achieved because the available technological knowledge is insufficient. The same page tells claimants to keep the evidence that is generated as the work progresses.

Read those two pages yourself before you read anyone's summary of them, including mine. They are shorter than you expect.

The practical consequence for technological uncertainty in an SR&ED claim is this: the form asks for a narrative with a specific shape. Uncertainty, then systematic work, then what was learned. A git log has a different shape. It is a list of things that worked well enough to commit.

What I had vs. what I needed

What the form asks about	What I had	What I needed
What was uncertain at the start	Nothing written. Memory.	A dated note stating what I did not know and why existing approaches did not settle it
What I tried, in order	Commits for the approaches that survived	A record of the approaches that did not survive, with the reason each was dropped
Why the obvious approach failed	Occasionally a PR description	One sentence, written the day it failed
What was learned	The final code	A plain statement of the conclusion, including "this does not work"
When the work happened	Commit timestamps	The same, which was the one thing git gave me for free

Timestamps were the only column where my existing record was enough. Everything else I rebuilt.

The AI-assisted coding sessions were the other source. Where a session survived, it sometimes held the reasoning I was missing, because I had explained the problem to the assistant in plain language before asking for help. But I had never kept them with this purpose in mind, so coverage was patchy and finding the relevant ones was slow.

One more thing I learned during the reconstruction: most of what I had built did not belong in the claim at all. Standard integration work, UI, and infrastructure plumbing did not qualify. What qualified was a narrow slice, cross-account cost attribution where tagging is incomplete and the answer has to be inferred, because that was where I did not know whether the approach would work. Sorting fourteen months of work into those two piles was part of the reconstruction. A log would have done that sorting for me as I went.

The lightweight SR&ED documentation habit: two lines per dead end

Here is the whole system. One file in the repo, docs/uncertainty-log.md, append-only. An entry only when an approach fails or you hit something you do not know how to do. Not a journal. Not a standup.

Each entry is two lines:

## 2026-03-14
Tried: inferring the owning account for untagged resources from the creating IAM principal.
Failed because: a shared deploy role creates resources for several accounts, so the principal does not identify the owner. Next: try usage-pattern correlation. (branch: attribution-v2, commit abc1234)

That entry is an illustration of the format in my problem area, not a record from my actual claim.

The rules I would follow:

Write it the day it fails. The reason is obvious today and gone in a month.
Line one is what you tried. Line two is why it failed and what that told you. If you cannot write line two, you have not understood the failure yet, which is worth knowing anyway.
Link the commit or branch. The log carries the reasoning, git carries the dates and the diffs. Together they cover each other's gaps.
Log the question before the answer. When you start on something you do not know how to do, write one line saying so. That line is the start of the narrative the form asks for.
Do not log routine work. If the path was known and it was just effort, it does not go in. This keeps the file short and gives you, or whoever prepares your claim, a first rough sort of uncertain work from routine work.
Save the AI sessions that contain reasoning. Export them into a folder next to the log, named by date. They are timestamped explanations of what you did not know, written at the time. I had these by accident. Have them on purpose.

Cost: a couple of minutes per dead end. I would have traded that for three weeks without thinking about it.

What I'd do differently

Start the log on day one of the build, not when I first hear the word SR&ED.
Stop deleting failed branches. Tag them and leave them. A failed approach with a commit history is evidence. A deleted one is an anecdote.
Write PR descriptions that say what was ruled out, not only what was done.
Read the T661 guide before the build, not after. Knowing the shape of the questions changes what you bother to write down.
Budget the time. Even with a good log, the claim is real work. Without one, it was three weeks.
Separate uncertain work from routine work as I go, because that line decided what went into the claim, and drawing it afterwards was slow.

What this post is not

I am a founder who filed once. I am not an SR&ED advisor, I cannot assess your work, and I cannot tell you whether you qualify. If you are a Canadian founder working on SR&ED for startups and you think some of your work involved real technological uncertainty, talk to a qualified claim preparer and read CRA's own pages linked above.

What I can tell you is narrower and I am confident in it: whatever you end up claiming, and whoever prepares it, the record you keep while building is the raw material. Two lines per dead end. Start today.

Rick Wise is the solo founder of CloudWise, an AWS cost optimization tool with 189 waste checks across 40+ AWS services. He filed one SR&ED claim, himself, and is not a tax advisor. He helps technical founders reconstruct the engineering narrative from their repo and AI coding sessions at SR&ED Lab. Your accountant or claim preparer still files.

How We Prove an AWS Waste Check Works: Fire on a Real Cluster, Stay Silent on Its Twin

Rick Wise — Wed, 23 Sep 2026 13:00:00 +0000

A unit test tells you your code does what you think AWS does. It can't tell you what AWS actually does.

For a tool that reads CloudWatch metrics and billing data to decide whether a resource is wasted, that gap is the whole product. So we test each check the same way: build the wasted thing for real, and make the check prove itself on it.

The rule is two halves:

Fixture: a real resource that should trigger the check. The check has to fire on it.
Control: the healthy version of the same thing. The check has to stay silent on it.

Fire without silence and you've built an alarm that always rings. Silence without fire and you've built nothing. You need both, measured against live AWS, before you trust the check.

Here is what that looked like for one check: idle DocumentDB.

The check, and the real cluster we pointed it at

The rule: flag a DocumentDB cluster with zero connections and combined read plus write IOPS under 10 per second, averaged over 7 days.

We stood up a real db.t3.medium cluster in a separate test AWS account and left it alone. It costs $1.87 a day and it's a short-lived test resource, not something we keep. Measured on 2026-09-19:

Metric (7-day window)	Value	Gate
DatabaseConnections	0.0000	must be 0
Combined read + write IOPS	5.46 / s	under 10 / s

That is an idle cluster by any reasonable reading. The check should fire.

What the real cluster told us that the unit tests couldn't

It did not fire. Not on the first try, and not for one reason. Three separate bugs were stacked in front of it, and every one of them lived in the gap between our code and AWS's behavior.

Layer	What was wrong	How it hid
Wiring	The detector called a metrics method that didn't exist on the live data provider, so it always got nothing back. Fixed 2026-07-09.	A missing answer reads as "no data", which reads as "nothing wrong here".
Units	CloudWatch reports DocumentDB IOPS as a rate (Count per Second). We read it with `Statistics=Sum` over 7 days, which adds a rate up as if it were a running total. A steady ~5.15 per second over 10,080 one-minute datapoints sums to roughly 52,000, against a gate expecting a small number.	Our regression tests pinned the same `Sum`-shaped mock, so they enshrined the bug.
Naming	Cost data calls the product "Amazon DocumentDB (with MongoDB compatibility)". Our service-to-detector map only knew "Amazon DocumentDB" and "AmazonDocDB", and the lookup is exact, so on every scheduled scan the DocumentDB detector was skipped, even on accounts whose costs listed DocumentDB.	A skipped detector reports nothing, which looks exactly like an account with no waste.

The second row is the one worth sitting with. The tests were not sloppy. They were written against a mock that matched the code's assumption, so they could only ever confirm the assumption. Only a real cluster reporting a real rate could disagree, and it did.

The fix for the second bug reads the average of the rate instead of the sum, and gates on combined IOPS averaging under 10 per second with zero connections.

The first real firing, and why we almost missed it

With all three fixed, the 2026-09-19 06:01 UTC scan ran the DocumentDB detector on that cluster for the first time and it fired: idle cluster, list price $56.94 a month for one db.t3.medium.

You would not have seen it in the findings list, though. Our scan reconciles every finding against the account's actual billing before showing it, and the test account was on DocumentDB's free trial, so the finding reconciled to $0.00 and was dropped from the customer-facing list on purpose. It only existed in the pre-floor debug output we keep for exactly this kind of question.

That's the system working as designed, and it's also why "look at the findings page" is not a test. We had to read the layer beneath it.

The control: the same cluster, connected

A healthy twin usually means a second resource. For this check the healthy state is simply "someone is using it", and the two states are mutually exclusive by construction: the check requires zero connections, so any connection at all rules idle out.

So the control was the same cluster with one variable changed. On 2026-09-19 we opened a handful of connections from a short-lived Lambda inside the VPC and deleted it the same day.

	Fixture (idle)	Control (connected)
DatabaseConnections	0	0 to 6 per minute while connected, 7-day average 0.0169
Combined IOPS	5.46 / s	5.44 / s
Scan on 2026-09-20	fired the day before	detector ran, 0 findings

Nothing moved except the connections. IOPS stayed under the gate, and one afternoon of traffic was enough to pull the 7-day average off zero and silence the check. Fired on the fixture, silent on the control.

One honest caveat, because it would be easy to leave out: those two observations are separated in time, not simultaneous. It's one cluster measured in two states, and the code guarantees the states can't both be true. That's a weaker claim than "two clusters side by side", and we say so in the write-up.

Controls need checking too

The same sweep caught the opposite failure. A Glue job we built as a healthy control set off the "missing timeout" check. We had left the timeout out when creating it, and Glue quietly defaults to 2,880 minutes, which is exactly the number the check looks for. The control was never healthy.

A control that fires isn't a control. We now set the timeout explicitly, and we verify every control is silent across all checks, not only its own.

What we took from it

A passing unit test is a statement about your assumptions. Real resources are the only thing that can contradict them.
Test the whole path, not the function. All three bugs lived outside the detector's decision logic: in how metrics were fetched and read, and in how the check got selected to run.
Prove silence as hard as you prove firing. A check nobody has watched stay quiet is a guess.
It's cheap. One small cluster for roughly a week costs about what a lunch does, and it found bugs our mock-based tests couldn't.

That's the method we use when we validate a check against real AWS. CloudWise runs 189 waste checks across 40+ AWS services, and if you want to see what a read-only pass finds in your own account, it takes about five minutes.

Free, read-only AWS waste scan: cloudcostwise.io — five minutes, no card.

Your Container Tag Is Lying to You: A Mutable ECR Tag Put the Wrong Build in Production

Rick Wise — Wed, 09 Sep 2026 13:00:00 +0000

On 2026-08-21, a routine docs-only PR merged to main. CI built the frontend image, pushed it, and called start-deployment on the App Runner service — and got rejected. The pipeline went red. "Deploy to Production" failed.

That part is normal; pipelines go red. What isn't normal is what CloudTrail and ECR showed once we went looking: production wasn't broken by the red step. It was broken by the deploy that succeeded 3 seconds earlier — from a laptop, not CI — and the pipeline's red error message never once mentioned it.

The timeline, from CloudTrail and ECR, not from memory

Time (EDT)	Event
10:52:25	`StartDeployment` on `cloudwise-production-frontend`, called by an IAM user from a local machine (`aws-cli/2.34.2`, macOS, arm64) — not CI. At this instant the `production` tag points at the previous build.
10:52:43	CI pushes the new frontend image and moves the `production` tag to point at it.
10:52:44	CI calls `StartDeployment` as its own deploy role → `InvalidRequestException: Can't start a deployment … because it isn't in RUNNING state` → exit 254 → pipeline red.
10:55:49	The manual deployment from 10:52:25 succeeds — having resolved the `production` tag back at 10:52:25, before CI moved it. It deploys the old image.

Net state: the production tag pointed at the image CI had just built. That image was never deployed. App Runner was RUNNING on the previous build, and the pipeline was red for a reason that named none of this.

18 seconds is the whole story. That's the window between the manual deploy resolving the tag and CI moving it. Move a mutable tag inside that window and the registry and the running container permanently disagree about which image "production" means.

Why a mutable tag is the actual defect

ECR_TAG=production for the frontend repository always points at the newest push. App Runner doesn't pin a digest — it resolves whatever production points to at the moment a deployment starts, not at the moment the container swap completes. Those are two different instants, and normally nothing happens in between.

Two deployments racing to start closes that gap. One of them read the tag old; the other wrote it new. Whichever deployment actually executes the swap runs whatever it resolved, and there's no signal anywhere in App Runner's own state that says "the tag has moved since I started."

This had already been flagged as a hazard in the abstract — two CI runs landing close together can serve an older image, all green, no laptop involved. This incident reached the same failure by a different route: the second actor wasn't CI at all, so none of the existing concurrency guards between pipeline runs applied.

The fix: verify the digest, don't trust the tag (PR #1181)

Two independent changes, both in .github/workflows/deploy-environment.yml:

1. Bounded retry on start-deployment. A RUNNING-state rejection now retries for up to 20 minutes instead of failing on the first attempt — a wait can only delay a deploy, never corrupt one, so it's a safe response to "something else is deploying right now." But a retry alone doesn't fix the race; it just makes CI patient. The actual fix is next.

2. Re-read the tag's digest on every retry attempt, and refuse to deploy if it moved. Before each start-deployment call, CI re-reads what production currently points to and compares it against the digest of the image this run pushed. If they no longer match — because something else moved the tag in the interim — CI stops and errors loudly instead of deploying an image it can no longer identify:

"The production tag no longer points at the image this run built. Built <digest>, tag now <digest>. Something pushed to this tag outside this pipeline; deploying now would ship an unknown image."

That single check closes the 18-second window: CI will now only ever call start-deployment when it can prove the tag still means what it thinks it means.

3. A post-swap check — warn-only, on purpose. After the deployment completes, CI reads the digest App Runner actually deployed and compares it against what CI pushed. If they don't match, it logs a warning and keeps going rather than failing the pipeline.

That's a deliberate asymmetry, and it's the design decision worth explaining: the pre-swap digest check is a hard gate, because it's checkable before anything ships and a mismatch there means CI is about to deploy the wrong thing. The post-swap check can't be a hard gate, because by the time it runs, the running container already is what it is — the thing that ran ahead was the registry tag, not production. Failing the pipeline red at that point would block a legitimate deploy for a discrepancy that already happened and that the running service had no part in causing. Warn, don't block, on a fact you can no longer change.

The part that makes it worth writing about

The merge that triggered this was docs-only. The image that never got deployed differed from the one that did by markdown and release notes — nothing user-visible. Production was healthy and serving the whole time.

That's luck, not design, and it's the actual point: the same race on a real frontend change ships a red pipeline and production quietly serving the old build, and the red error points at App Runner deployment state — not at "prod is stale." You'd fix the wrong problem. InvalidRequestException: isn't in RUNNING state is a true statement about App Runner. It was never a true statement about what mattered, which was that the tag we were about to deploy had already been consumed by someone else. A pipeline that goes red for the right reason but tells you the wrong story is worse than one that goes red honestly — you close the ticket on the error you can see instead of the one that actually happened.

Boring fixes are the ones worth shipping: retry with a bound, verify a digest instead of trusting a tag, and only fail hard on the check that happens before anything ships.

We hit this on our own production App Runner service, running the app you'd be scanning your AWS bill with. Curious what a read-only pass over your own account finds? Free, read-only AWS waste scan: cloudcostwise.io — five minutes, no card.

Catching a Cost Spike the Same Day It Happens: Inside Our Z-Score Anomaly Detector

Rick Wise — Thu, 27 Aug 2026 13:03:55 +0000

AWS Cost Explorer's own data can lag up to 24 hours behind real usage, and nobody's on call for "the bill." A forgotten load test, a runaway Auto Scaling event, a script that didn't tear down its own infrastructure — by the time a human opens the console, whatever ran already ran.

CloudWise's anomaly detector is built to close that gap: a daily scheduled scan, not a dashboard you have to remember to check. Here's the actual statistics behind it — the real thresholds from lambdas/anomaly_detector/handler.py — and the exact test case, already in our shipped suite, that proves it catches a spike.

The mechanic: z-score, not a percentage

The naive version of this feature checks "is today more than X% above yesterday?" That breaks immediately: a service that costs $2/day naturally swings 200% day to day out of pure noise, while a service that costs $4,000/day moving 15% is a real four-figure problem. Percentage-of-yesterday has no sense of a service's normal variance.

The detector instead computes a z-score per AWS service, per account, every day:

# lambdas/anomaly_detector/handler.py
SPIKE_THRESHOLD_STD_DEV = 2.0   # standard deviations from the mean
MIN_ABSOLUTE_CHANGE_USD = 5.0   # minimum dollar change to trigger, at all
MIN_DATA_POINTS = 7             # minimum days of history required

mean = statistics.mean(historical)
std_dev = statistics.stdev(historical) if len(historical) > 1 else 0
change = today_cost - mean
z_score = (today_cost - mean) / std_dev if std_dev > 0 else 0

if z_score >= SPIKE_THRESHOLD_STD_DEV:
    # ...flag it

historical is the trailing daily costs for that service, excluding today. If a service doesn't have at least 7 days of history yet, it's skipped — no history means no baseline, and a false "anomaly" on day one of a brand-new resource is worse than useless. And before z-score even runs, there's a $5 absolute-change floor: if today's cost moved by less than five dollars, the detector doesn't care how many standard deviations that represents. A service that normally costs $0.02/day moving to $0.11/day is a 450% swing, a huge z-score, and completely irrelevant to anyone's bill. The dollar floor is what keeps a statistically-driven detector from paging you about noise.

Severity isn't binary

Once something clears the 2.0 z-score bar, it gets bucketed:

SEVERITY_THRESHOLDS = {
    'CRITICAL': 4.0,
    'HIGH': 3.0,
    'MEDIUM': 2.0,
    'LOW': 1.5,
}

(Yes, LOW is below the 2.0 trigger threshold in the table — that band exists for a second call site that widens the net for manual/on-demand scans. The scheduled daily job only ever emits MEDIUM and up, because the scheduled job's trigger is 2.0.) The severities aren't decoration — they're what a Shield-tier customer's Slack alert leads with, and what determines whether the message reads as "worth a look this week" or "look now."

The test that proves it

Rather than a hypothetical dollar figure, here's the exact fixture from our own shipped test suite (lambdas/anomaly_detector/test_handler.py::test_detects_anomaly_with_high_z_score) — real, committed, running in CI on every change to this file:

An EC2 line item costs $9–$11/day for seven straight days: $10, $11, $9, $10.5, $9.5, $10, $10. Mean $10.00, standard deviation $0.65. Then it jumps to $100 in a day.

change     = 100 - 10        = $90
z_score    = (100 - 10) / 0.65 = 139.4
percentage = 90 / 10 * 100     = 900%

A z-score of 139.4 isn't a borderline call by any measure — the test asserts the detector flags it as CRITICAL or HIGH severity, and it does, deterministically, every run. The absolute dollars here are small on purpose: it's a unit test, not a customer account. The statistical shape — a service ten times its normal cost, standard deviations off its own baseline — is exactly what the 2.0 threshold exists to catch on day one, whether the line item is $10 or $10,000.

Why this is Shield-tier, and why it's read-only

The detector runs once a day via EventBridge, over the last 30 days of each Shield-tier account's cost data, and writes findings to an alerts table before dispatching Slack and email notifications. It doesn't touch anything in your AWS account — no remediation, no API calls beyond reading Cost Explorer data CloudWise already ingests for the dashboard. That's deliberate: detecting a spike and deciding what to do about it are different problems, and this one only does the first.

The mechanic is boring on purpose — mean, standard deviation, a threshold, a dollar floor. Boring is what you want from something that's deciding whether to interrupt your weekend.

I Shipped a Redesign That Orphaned My Own Product

Rick Wise — Wed, 19 Aug 2026 13:53:46 +0000

This month I rebuilt CloudWise around an AI agent.

Not a chatbot bolted into a corner. The agent is the product now. You open the app and you're talking to it. "Where's my money going?" — and it pulls your AWS spend, ranks the waste, shows you the dollars per month per finding, and tells you what's safe to fix.

It looked done. The screens were beautiful. The agent answered. I shipped it as the front door.

Then I used it like a customer would. And I discovered I had quietly orphaned half my own product.

This is the story of that month, because it taught me the single most useful lesson I've learned as a solo founder: a redesign isn't done when it looks done.

What I built

For most of this year, CloudWise looked like every other AWS cost tool: a dashboard. Charts, tables, a sidebar of pages. Useful, but passive. You had to know what to look for.

The bet I made for the overhaul was that the interface should be a conversation, not a dashboard. Most people don't want to read a cost dashboard. They want to ask a question and get an answer.

So the centerpiece became a conversational workspace backed by a real tool-calling agent loop on Claude (running on AWS Bedrock). "Real" is the important word. This isn't a model that summarizes a page of text. It's a model with tools — it actually queries your cost data, your findings, your Reserved Instance and Savings Plan coverage, and composes an answer from live numbers. It remembers your account between conversations.

You can ask it:

"Where's my money going?" → it breaks down spend by service, in gold, with the month-over-month delta.
"What's safe to fix?" → it ranks your waste findings: idle NAT Gateways, forgotten SageMaker notebooks, unattached EBS snapshots, each with a dollar figure and a safe-to-fix flag.
"Compare my two accounts." → and it does.

That last one matters more than it sounds, and it's where the trouble started.

The part where it looked finished

By the start of June, the redesign was, on paper, complete. New design system. Dark, native. A conversational workspace. A guided tour. A redesigned dashboard. The agent loop. Cross-session memory. I'd migrated every screen.

I flipped the new workspace to be the post-login front door and moved on.

Here's the uncomfortable truth about building alone: you stop seeing your own product. You navigate it the way the author navigates it — from the inside, knowing every shortcut, never actually starting cold the way a real user does.

So I made myself start cold. I logged in like a brand-new customer and tried to do the boring things. Switch to my other AWS account. Open settings. Change a notification. Log out.

I couldn't.

The front door had hidden the house

None of those features were gone. Every settings page still existed. The account switcher still existed. Logout still existed. The old navigation still existed, sitting in the codebase, fully functional.

They just weren't reachable from the place users now landed.

The new workspace shell had a clean little user footer — a static label with the account name. No menu. No logout. No link to subscription, notifications, alerts, API keys, AWS accounts, or password. From the new front door, there was literally no path to any of it.

Worse, the half of the product that did still have navigation rendered in the old chrome. So a new user would land in a slick dark conversational workspace, click one thing, and get bounced into the previous design — a completely different layout. Two products wearing different clothes, stitched together at a seam the user falls straight through.

I wrote it down plainly in my own audit doc at the time, because I needed to see it without flinching:

The redesign did not delete your features — the cutover orphaned them.

That sentence reframed the entire month. This wasn't a teardown. It was a finishing problem. The work wasn't to rebuild — it was to take ownership of every job the old interface used to do, and make the new one do it better.

The agent was also lying about multi-account

While I was in there, I found a real bug — the kind that only surfaces when you use the product for real, with more than one account.

CloudWise is a multi-account tool. You connect your AWS Organization and it discovers all your accounts. But the agent was collapsing them. When you asked about cost, it blended every account together with no way to scope to one. And when you asked about Reserved Instance and Savings Plan coverage, it reported coverage for exactly one account — the first one it happened to grab — and silently ignored the rest.

That's not a cosmetic miss. That's the tool confidently giving you a wrong answer about money. If you had three accounts and asked "how's my RI coverage," it answered for one-third of your footprint and didn't tell you.

The backend could already filter by account. The old reports page could already do multi-account selection. The agent — the new centerpiece — was strictly less capable than both. The redesign had, in this one spot, made the product worse while looking like it made it better.

So the fix had three parts:

A header account switcher that scopes the entire workspace — single or multi-select, capped by your plan tier.
Per-account arguments on the agent's tools, so the model can honor that switcher and answer in-conversation requests like "compare my two accounts."
Fixing the coverage bug so it reports across all accounts, not the first one.

# Before: commitments reported for whatever account came first.
def get_commitments(accounts):
    first = accounts[0]
    return fetch_coverage(first.account_id)   # silently ignores the rest

# After: the agent scopes to what you asked for — one, some, or all.
def get_commitments(accounts, scope_ids=None):
    targets = [a for a in accounts if a.account_id in scope_ids] if scope_ids else accounts
    return [fetch_coverage(a.account_id) for a in targets]

A small diff. A big difference in whether you can trust the answer.

Thirty days of taking ownership

That's what the month actually was. Not "add features." Re-own the basics, and make the agent the real front door instead of a beautiful demo sitting on top of a product it had stopped being responsible for.

In order, here's what changed:

One app shell, not two. Unified chrome around every authenticated page — one persistent sidebar, the account switcher, every settings link, and logout — so you never fall through the seam between the new workspace and the old layout again. The workspace became one surface inside a consistent app, not a separate world.

The account switcher scopes everything. Pick an account (or several) in the header and the whole workspace re-scopes to it — and the agent answers for exactly that selection.

The agent learned to count accounts. Per-account tool arguments, the coverage bug fixed, and the ability to genuinely compare accounts on request.

A deep, cross-page guided tour. Instead of a checklist nobody reads, a docked "CloudWise agent" companion walks a new user through the real product — navigating the actual screens across seventeen steps, not narrating a slideshow.

The whole product went dark-mode-native. Every deep page — cost reports, remediation, savings plans, settings, the dashboard — rebuilt on one design system. Money is always gold. No more half-migrated screens where one page is dark and the next is white.

A shareable cost-health score. One number, 0–100, for how efficiently you're running AWS — with a public, sanitized share card carrying no account details or PII. Because the first question every engineer actually asks is "are we good, or not?"

Across the month that came out to a few dozen shipped changes and around fifty releases — versions 1.54 through 1.104. Most of them were not glamorous. Most of them were re-owning a job the old interface used to do, quietly, that the new one had dropped on the floor.

The lesson, stated plainly

I've shipped a lot of software in twenty-five years. I still fell for this one, which is why I think it's worth writing down.

"Looks done" and "is the product" are not the same state. They can be a hundred releases apart.

A redesign is not finished when the new screens are beautiful and the happy path works. It's finished when the new thing has quietly taken over every job the old thing did — logout, account switching, the boring settings page nobody screenshots — and does each of them at least as well. Until then you don't have a new product. You have a beautiful front door on a house whose rooms you've locked.

And the cruelest part: users don't grade you on the demo. They grade you on the one ordinary day they need the single feature you forgot to bring across. The day they need to switch accounts, or check their RI coverage across all three, or just sign out. That's the moment your redesign is actually judged — and it's never the moment you rehearsed.

So now I have a rule. Before I call any cutover done, I log in cold and do the boring things. All of them. If I can't sign out, I'm not done. No matter how good the front door looks.

CloudWise is an AWS cost optimization tool for startups — 191 automated waste checks, a real agent that runs against your actual usage, air-gapped mode for security teams, starting at $19/month. If you want to ask an agent where your AWS money is going, it's at cloudcostwise.io.

EBS Snapshot Sprawl: The Waste Cost Explorer Can't Show You

Rick Wise — Wed, 19 Aug 2026 13:51:19 +0000

Last week's short covered the 101 on old EBS snapshots: the 90-day threshold, the $0.05/GB-month rate, aws ec2 delete-snapshot. If you saw it, you already know snapshots are cheap-per-unit and expensive-in-aggregate. What that 42 seconds couldn't fit in is the actual reason snapshot sprawl is so hard to find in the first place — and it isn't the price. It's that Cost Explorer structurally cannot show you who's responsible for it.

Cost Explorer shows you a bill, not a culprit

Group your AWS costs by usage type and EBS snapshots show up as one line: EBS:SnapshotUsage, rolled up per account and region. That's it. Not per-snapshot, not per-volume, not per-AMI. If that line is $340/month, Cost Explorer will tell you the total and nothing about which of your 200 snapshots — or which of your 40 AMIs pinning them — put it there.

Part of why this is so opaque is how the billing actually works. EBS snapshots are incremental: the first snapshot of a volume captures every block, but every snapshot after that only stores blocks that changed since the previous one. Delete an "old" snapshot in the middle of a chain and AWS doesn't just drop its unique blocks — it can merge the still-referenced blocks from that snapshot into the next one to keep the chain valid. The result is a genuinely well-designed storage model that happens to make per-snapshot cost attribution close to meaningless from the outside. You can't look at snapshot #14 in a chain of 20 and know what deleting it actually frees, and Cost Explorer doesn't even try — it just gives you the account-wide sum and moves on.

So the number you see going up every month is real. The reason is invisible from the billing console. You have to go look at the resources directly.

`deregistered ≠ deleted` — the AMI-pinning trap

Here's the gap that costs teams the most and gets found the least: deregistering an AMI does not delete the snapshot backing it.

When you run CreateImage against an EC2 instance, AWS creates an AMI and, silently, one or more EBS snapshots to back it — you'll find the linkage in the snapshot's own description field, something like Created by CreateImage(i-0abc123def456789) for ami-0fedcba987654321. Deregister that AMI later (cleaning up an old release, retiring a pipeline, whatever the reason) and AWS removes the AMI. The snapshot stays. Forever. Nothing in the console flags it, nothing in Cost Explorer changes shape, and nothing tells you the thing that snapshot was created for no longer exists.

This is exactly the pattern our AMI_ORPHANED_SNAPSHOT detector checks for. It's not a guess — it's a direct read of the relationship AWS itself records: the detector regex-matches each snapshot's description against the CreateImage(...) for (ami-...) pattern, pulls the referenced AMI ID, and checks it against the account's currently-registered AMIs. If the AMI isn't there anymore, the snapshot is flagged — at higher priority than a generic "old snapshot" check, because an orphaned-AMI snapshot has a certain reason to be dead, not just an age-based guess.

That priority ordering matters. Our storage detector dedups three overlapping checks against the same snapshot inventory: AMI-orphan first, then volume-orphan (the source volume was deleted), then plain age (over the 90-day default threshold, at the $0.05/GB-month rate from last week's short). A snapshot only gets counted once, under whichever explanation is strongest — an AMI-orphaned snapshot isn't also reported as merely "old," because "old" undersells why it's actually safe to delete.

Lifecycle policies are the actual fix — for snapshots you haven't made yet

None of the above is a criticism of EBS snapshots as a backup mechanism. They're cheap, they're incremental, and they're the right tool. The problem is entirely operational: nothing deletes them automatically unless you tell it to.

AWS Data Lifecycle Manager (DLM) exists for exactly this — attach a policy to a tag or resource type and it will create snapshots on a schedule and expire them on a schedule, so "backup taken 400 days ago for a server that's been gone for 399 of them" stops being possible going forward. If you're not running DLM policies today, that's the highest-leverage 20-minute fix here, full stop.

But DLM only prevents new sprawl. It does nothing for the snapshots already sitting in your account from AMIs someone deregistered two years ago, or backups nobody automated before DLM was set up. That backlog needs to be found once, by hand or by a scan, before a policy can keep it clean going forward. That's the gap our detector is built for — not a replacement for lifecycle policies, the thing that finds what predates them.

Finding yours

If you want to see this in your own account instead of grepping AMI descriptions by hand, a free scan checks this along with 190+ other waste patterns across 40+ AWS services — read-only, five minutes, nothing gets deleted without you clicking it. If you'd rather watch the 90-day/$0.05-per-GB basics first, we posted a 30-second walkthrough of that detector last week.

Or check the AMI-pinning trap yourself right now:

# Snapshots whose description names a CreateImage/AMI pairing
aws ec2 describe-snapshots --owner-ids self \
  --query "Snapshots[?contains(Description, 'CreateImage')].[SnapshotId,Description]" \
  --output table

# Currently-registered AMI IDs
aws ec2 describe-images --owners self \
  --query "Images[].ImageId" --output text

Cross-reference the AMI ID in each snapshot's description against the second list. Anything missing has been quietly billing you since the day someone cleaned up an AMI and assumed the cleanup was done.

The Bug That Turned Every Bad Password Into a Server Outage

Rick Wise — Wed, 19 Aug 2026 13:50:30 +0000

Type your password wrong on CloudWise's login page, and for a while, the server told you it had a nervous breakdown.

Not "invalid credentials." Not even a plain 401. An HTTP 500 — the code reserved for "something on our end is broken" — for the most routine failure mode there is: a human mistyping a password.

This happened on staging. It happened on production. It happened to POST /api/v1/auth/login with wrong creds, an unknown email, an unconfirmed account, or an account mid-password-reset. Four different "you did something ordinary" situations, all reported back as "we did something wrong." It went undetected until an automated gate caught it and refused to let a release ship. This is that bug, the fix, and why the distinction matters more than it sounds like it should.

The bug

_handle_cognito_auth_error, the function that translates AWS Cognito's auth exceptions into an HTTP response, raised a CloudWiseException without ever setting a status_code. No status code means the default. The default is 500.

So every one of these:

Wrong password
Email that isn't registered
Account that hasn't confirmed its email yet
Account stuck in a forced password reset

came back as a server error. Reproduced live on both environments — for example, POST /api/v1/auth/login with bad credentials on staging returned:

HTTP 500
{"detail":"Invalid email or password"}

and on production:

HTTP 500
{"detail":"Authentication failed: User does not exist."}

Read that second one again. Production wasn't just returning the wrong status code — it was telling an anonymous caller whether a given email address had an account. That's a second, smaller bug riding along inside the first one.

Why the status code is the actual bug

It's tempting to shrug this off — the message was right there in the body, "Invalid email or password", so what's the harm in the wrong three-digit prefix?

The harm is that the status code isn't decoration. It's the part of the response that infrastructure reads without understanding a word of the payload. A monitoring dashboard doesn't parse detail. It counts 5xx rates. To CloudWise's own alerting, every single mistyped password looked identical to an actual outage — same bucket, same page-worthy signal, same "something is on fire" pattern, forever, at whatever rate normal users normally fat-finger their passwords. That's not a rare event. It's baseline noise, and the bug was quietly dressing it up as baseline crisis.

It also meant every one of those routine rejections logged at error level — the log level meant for things that need a human to look at them, now firing constantly for people who just needed to try again.

And then it did something worse than annoy a dashboard: it blocked a production release.

How it actually got caught

CloudWise's release pipeline runs an E2E gate against staging before anything is allowed to promote to production. One of the negative-path assertions in that gate — ci-gate-read.spec.ts:93 — logs in with a wrong password and checks that the app shows an error and stays on /auth/login, the way a real login form should behave.

That test didn't expect a 500. Nothing in a normal auth flow should return one for a wrong password. The gate went red, and release 1.105.0 sat there, unable to auto-promote to production, because the pipeline correctly refused to trust a build where the login form's error handling was behaving like a crash.

That's the part worth sitting with: this bug was already live in production, unnoticed, for who knows how long. It took a different release's gate run to surface it — not because the gate was looking for this specific bug, but because it was asserting the right general behavior (bad password → clean error, stay put) and the actual behavior didn't match. A negative-path test doesn't need to know about your bug in advance. It just needs to check that the ordinary failure case looks ordinary.

The fix

The fix, in backend/app/services/cognito_auth_service.py, maps Cognito's exceptions to the status codes they should have had all along:

NotAuthorizedException (wrong password) and UserNotFoundException (unknown email) → 401, both returning the exact same message: "Invalid email or password". Same message for both cases on purpose — the response can no longer be used to tell whether an email is registered, closing that account-enumeration leak.
UserNotConfirmedException and PasswordResetRequiredException → 403 — the account exists, but the request is correctly rejected for a reason that isn't "try a different password."
Anything else — a genuinely unexpected Cognito error — still defaults to 500. That default is correct in that case. An unrecognized failure mode is a server-side concern worth alerting on. The bug was never that 500 existed; it was that the four most common, most expected auth rejections were routed into it by omission.

Expected-rejection logging dropped from error to warning, so the logs now reflect what actually happened: a routine, anticipated rejection, not an incident.

The test suite got the fix that should have caught this the first time. test_handle_cognito_auth_error_* previously asserted on the error message only — never the status code. That's exactly how a 500-instead-of-401 slips through code review and CI both: the message text looked fine, so nobody noticed the number in front of it was wrong. The tests now assert status codes explicitly, plus two cases that weren't covered before: the no-leak UserNotFoundException path, and PasswordResetRequiredException.

This is the sibling of an earlier fix, CLO-48, which cleaned up the same bug class on the refresh-token path — an expired refresh token was logging as ERROR twice when a clean 401 was all that was warranted. That one didn't touch login. This one closes the login path CLO-48 didn't cover.

The takeaway

Status codes are a contract between your API and everything that reads it without understanding it — monitors, alerting rules, retry logic, browsers, other services. A wrong password is a client error. Treating it as a server error doesn't just mislabel one response; it teaches every downstream system that watches your 5xx rate to distrust the signal, right when you need that signal to mean something.

The thing that actually caught this wasn't a code review, a manual QA pass, or a customer complaint. It was an automated negative-path assertion doing exactly what negative-path assertions are for: checking that the boring, expected failure looks boring and expected. It's staying in the gate. Nobody has to remember to test this again — the pipeline already refuses to ship a build that gets it wrong.

If you want to see what else that kind of scrutiny turns up, a free, read-only scan of your AWS account takes about five minutes and changes nothing without you approving it first.

Inside a Real $11,871/mo AWS Waste Scan

Rick Wise — Wed, 19 Aug 2026 13:50:30 +0000

Most cost-optimization content shows you a screenshot of a dashboard and asks you to trust the number. So instead, here's an actual scan CloudWise runs internally as a demo profile — a mid-size SaaS company's AWS account, $28,500/mo in total spend, run through all 191 detectors. It came back with $11,871/mo in identified savings across 18 findings.

I want to walk through what's actually in that list, because the shape of it surprised me even after building the detectors.

Where the 18 findings live

Split by category, not dollar amount:

Database — 9 findings. By far the largest bucket. A mix of idle RDS instances, an oversized analytics database running at 12% CPU, stale manual snapshots, and a cluster of ElastiCache-specific findings (idle replicas, an engine migration opportunity, a serverless-fit opportunity, and one very large data-tiering opportunity — more on that below).
Compute — 2 findings. An idle bastion host nobody SSHs into anymore, and one oversized EC2 fleet running at 15% CPU.
Storage — 2 findings. An unattached EBS volume left over from a migration, and a batch of aging EBS snapshots.
Commitment Management — 2 findings. A Savings Plan expiring in 52 days, and a Convertible Reserved Instance sitting on previous-generation hardware with a free exchange available.
Network — 2 findings. Two idle Global Accelerators — one genuinely idle, one just disabled but still billing.
Purchase Optimization — 1 finding. Production RDS instances that have been running on-demand, 24/7, for 90+ days with no Reserved Instance covering them.

Nine of eighteen findings sitting in "Database" isn't a coincidence — it's the category where the widest range of waste patterns overlap: idle instances, oversized instances, stale snapshots, and now a whole sub-family of ElastiCache-specific checks (engine choice, replication topology, traffic shape, data tiering) that didn't exist as separate detectors a year ago.

The one number that matters most

Of the $11,871/mo total, one single finding accounts for 68% of it: $8,057/mo, from an ElastiCache data-tiering opportunity.

The cluster in question runs 4× cache.r6g.16xlarge nodes — memory-only, no local SSD — for 1,676 GiB of total cache capacity. ElastiCache's R6gd family adds local NVMe storage and tiers data between RAM and SSD automatically, based on access frequency. For a dataset where most of the data isn't accessed on every request (true of almost every real cache), that means the same effective capacity fits on far less hardware: 1× cache.r6gd.16xlarge instead of 4× cache.r6g.16xlarge.

That's not a "you forgot to delete something" finding. It's an architecture-level rightsizing call that requires reading how your own cache is actually used before you touch it — which is exactly why it's flagged as confidence: medium, not high, and comes with an explicit risk note: SSD-resident data has slightly higher latency, so it's a good fit only when a meaningful chunk of your data is cold.

What the next four biggest opportunities have in common

Set aside the $8,057/mo ElastiCache finding and look at what else shows up near the top of this scan: an expiring Savings Plan, an unpurchased Reserved Instance opportunity on production RDS, an oversized analytics database at 12% CPU, and an oversized EC2 fleet at 15% CPU.

None of those are "click delete." Every one of them is a standing decision that has to be made again, deliberately, on some recurring cadence:

A Savings Plan doesn't renew itself — someone has to decide, before it lapses, whether the workload underneath it still looks the same.
An RDS instance rightsized today can be wrong again in six months if the workload grows, shrinks, or changes shape.
"Buy the Reserved Instance" is itself a bet on the next 12 months looking like the last 3 — which is exactly the kind of call CloudWise's Commitment Risk Score exists to check before you make it.

Compare that to the idle-resource findings in this same scan — the bastion host, the unattached EBS volume, the stale snapshots, the disabled accelerators. Those are real money too, and they're the easiest to fix: find it, delete it, done. But they're structurally small, because once you delete something, it stays deleted. It doesn't come back next quarter.

Rightsizing and commitment decisions aren't like that. The right instance size and the right commitment level are correct for right now — and they drift the moment your traffic pattern, team, or architecture changes, which for a growing company is constantly. That's the actual thesis worth taking from a scan like this: waste renews itself. The idle-resource sweep is a one-time cleanup. The rightsizing and commitment layer is a standing job, and it's where most of the money in this particular scan actually lives.

Why this matters for how you should read your scan

If you run a scan and the top finding is "delete this idle volume," fix it and move on — there's nothing recurring about it. But if your biggest findings look more like this account's — an oversized instance, an expiring commitment, a data-tiering opportunity that requires understanding your own traffic — treat that as a signal that the fix isn't a single action, it's a review cadence you don't currently have. That's a different kind of problem, and it's worth naming as one.

CloudWise is an AWS cost optimization tool for startups — 191 automated waste checks across 42 AWS services, read-only by design, starting at $19/month. Run a free scan at cloudcostwise.io.

We Cut Lambda Cold Starts 56% — Three Wrong Turns Before the Real SnapStart Fix

Rick Wise — Wed, 19 Aug 2026 13:43:07 +0000

CloudWise's /dashboard took up to 2.3 seconds to load cold. That's the API Lambda's own cliff — not network, not the frontend. AWS Lambda SnapStart was already turned on. It was already restoring a frozen snapshot in about 600ms, which is the number SnapStart is supposed to deliver. And for two deploys, turning it on made no measurable difference to the thing that actually made the dashboard feel slow. This is the debugging path that got from there to a measured 56% cut — including the two attempts that didn't work, because the wrong-turn part is the part worth reading.

What was actually slow

We measured every layer before touching anything (full numbers in docs/redesign/clo114-dashboard-speed-analysis.md). Four dashboard API calls, already firing in parallel — parallelizing them further was a non-issue, wall-clock already tracked the slowest call. Server compute, warm, was fine: ~200–380ms. SnapStart's own restore was fine: ~550–620ms.

The problem was one specific number from CloudWatch REPORT lines: on a cold (post-restore) invocation, the first request's own Duration — not the restore, the handler work after the restore — was 1,124ms at p50, up to 2,805ms at the tail. SnapStart had already paid for the expensive part (a frozen Python interpreter + imports) and handed back a restored environment in 600ms. Something in the first real request was still doing ~1,100ms of work the snapshot didn't cover.

The diagnosis: init phase vs. handler phase

SnapStart snapshots whatever ran at module import / init. Anything deferred to the request handler is still paid, lazily, on the first real invocation — that's the whole cliff. get_settings() already runs at import in app/main.py, and constructing Settings performs the full AWS Parameter Store load — so config reads were already inside the snapshot, free. What wasn't: the boto3 session and DynamoDB client (built lazily in a factory, inside the request path) and FastAPI's startup_event — Mangum runs lifespan="auto" on the first request, not at import.

The fix looked obvious: move both into the init phase, so they're captured in the snapshot. We rejected the alternative — Lambda provisioned concurrency — outright: it bills a kept-warm 2GB instance 24/7 regardless of traffic, and it's redundant with SnapStart, which already restores in ~600ms. Paying a standing bill to paper over a problem SnapStart half-solves is exactly the kind of waste we built this product to catch in other people's accounts. So: init-phase priming, ~$0 incremental cost.

Wrong turn #1: it shipped clean and did nothing

PR #641 landed the priming code. Deploy went green, gates passed, prod promoted. Re-measured: cold Duration p50 1,142ms — statistically the same as the 1,124ms baseline. Priming had shipped and, as far as the numbers were concerned, changed nothing.

Worse: we couldn't even tell why. Every diagnostic line we'd added — "hooks registered," "SnapStart prime," even FastAPI's own unconditional startup log — was completely absent from CloudWatch across 90 minutes of cold starts. The working theory for a while was that SnapStart's init-phase logs simply don't reliably surface in CloudWatch, which would have made this nearly undebuggable from logs alone.

Wrong turn #2 (well, half a turn): it was a log level

It wasn't a SnapStart logging quirk. It was logging.info(). The Lambda's effective log level was dropping INFO — CloudWatch showed our [WARNING] lines and nothing below them, meaning every one of our priming diagnostics had been silently filtered the whole time. We moved the key diagnostic and priming log lines to WARNING (PR #648) and immediately got a real signal for the first time in this investigation. Lesson, underlined: when a fix "does nothing and also produces no logs," check the log level before you start reasoning about distributed systems semantics. We spent longer on the second hypothesis than the first.

The actual bug: `after_restore` cleared, but never rebuilt

With real diagnostics finally visible, we split the cold REPORT lines into two groups: requests that hit a fresh init (rare — a brand-new execution environment) versus requests that hit a restored snapshot (the common case SnapStart exists for). The restored-and-then-requested group was still averaging ~1,064ms — basically the original cliff, just hiding in a bucket we hadn't isolated before.

The cause was in our own restore hook. A boto3 session captured in a snapshot carries frozen credentials and, once a client exists, frozen TLS connection pools. After a real restore, the execution environment is new: credentials need refreshing, any frozen socket is dead. So reset_for_restore() correctly cleared the cached session and clients on after_restore — that part was right, and necessary for correctness. What it didn't do was rebuild them. So the next request found an empty session, built one lazily, and paid almost exactly the cost priming was supposed to eliminate — just relocated from "first request ever" to "first request after every restore," which for a Lambda that scales to zero between sparse dashboard loads is most of them.

The fix (PR #649): reset_for_restore() clears and eagerly rebuilds — with the freshly-restored credentials — so the next request finds the session already built, not merely uncorrupted.

The result

Post-restore first-request Duration, n=7 samples per version, staging:

Version	mean	p50
Baseline (no priming)	1,183ms	1,124ms
Clear-only (the bug)	1,064ms	1,060ms
Eager re-prime (the fix)	652ms	490ms

p50 1,124 → 490ms, a 56% reduction. Diagnostics on every restored environment now confirm the full chain worked as intended: hooks_registered=True, session_already_built=True, restore_reprimed=True, zero errors.

It's a real win, and it's not a complete one. About 3 of 7 restores in that sample still land at ≥500ms, even with the session confirmed pre-built — most likely the first actual DynamoDB operation opening its own TLS connection, since boto3 connects lazily on first call rather than at client construction. Chasing that down would mean priming a real DynamoDB round-trip inside the restore hook itself, for what's probably diminishing returns against a ~490ms floor that's already good enough. We banked the 56% rather than shipping a seventh iteration.

Alongside this, a frontend session-storage stale-while-revalidate cache (shipped separately, PR #639) hides both the network round-trip and any residual cold cliff on repeat visits — the dashboard's daily-batched data makes brief staleness safe — and a keep-warm EventBridge ping keeps instances hot enough that cold restores stay rare in the first place.

The actual lesson

None of the three wrong turns here were exotic. "The fix looks right but does nothing" was a log level. "The clear-only version regressed" was a hook that did half its job and returned success anyway, because clearing genuinely is correct and necessary — it just isn't sufficient. The instrumentation that finally cracked it wasn't clever; it was making the diagnostic state (hooks_registered, session_already_built, restore_reprimed) observable on the request path instead of trusting init-phase logs that, it turned out, we weren't even looking at correctly.

If you're chasing a Lambda cold start and the fix you shipped isn't moving the number: check what log level is actually filtering your diagnostics before you start doubting your architecture.

This is the same instinct behind the product: measure what's actually happening in your AWS account before acting on it. If you want to see what that looks like pointed at your own bill, a free, read-only scan takes about five minutes and changes nothing without you approving it first.

Why Read-Only Is the Only Safe Way to Let AI Near Your AWS Account

Rick Wise — Wed, 12 Aug 2026 13:54:04 +0000

Every AWS cost tool eventually asks you for the same terrifying thing: an IAM role. And every vendor says the same reassuring word about it: "read-only." I want to tell you exactly what that word means when CloudWise says it, because "read-only" gets used loosely enough in this industry that the word alone shouldn't be enough to trust anyone — including us. So this post is the IAM, not the marketing.

The role you actually grant

When you connect an AWS account through CloudWise's one-click setup, you launch exactly one CloudFormation stack: the monitoring stack. There is no button in onboarding that launches anything else — startConnect('monitoring') is the only path the connect flow calls.

That stack's policy (cloudwise-cur-setup-template.yaml) is a bespoke allow-list we wrote and maintain, not the AWS-managed ReadOnlyAccess policy. That distinction matters more than it sounds like it should: ReadOnlyAccess is enormous and vague — it grants read access to almost every AWS service, including ones CloudWise has no reason to ever look at. Our policy is scoped to what a cost scan actually needs, action by action: ec2:DescribeInstances, rds:DescribeDBInstances, s3:ListAllMyBuckets, ce:GetCostAndUsage, compute-optimizer:GetEC2InstanceRecommendations, and so on — read the whole thing at cloudcostwise.io/security/permissions, which renders the live action count straight off the template, not a number we typed into a page and forgot to update. Every single statement in that policy is a Get*, Describe*, List*, or BatchGet* call — plus sts:GetCallerIdentity and iam:SimulatePrincipalPolicy, which CloudWise uses to check what a role can do without ever calling it. There is no Put, Create, Delete, Update, Attach, or Modify verb anywhere in that role's policy. It cannot make a single write call against your account. Not "shouldn't" — cannot; IAM will reject the attempt at the API layer before it reaches any resource.

(One nuance, for the pedants, because I'd rather you catch it than an auditor: the CUR bucket in that same template does have a bucket policy permitting s3:PutObject — granted to AWS's own billingreports.amazonaws.com service principal, not to CloudWise. That's how AWS itself delivers your Cost and Usage Report into a bucket you own. It's AWS writing to you, not us writing to anything.)

Remediation is a different stack, a different role, and a different click

If you want CloudWise to actually fix waste — stop an idle NAT gateway, delete an orphaned snapshot, right-size an instance — that is never part of onboarding. It requires deploying a second, separate CloudFormation stack (CloudWise-Remediation, using cloudwise-remediation-role.yaml) that you reach only from settings/remediation or setup/permissions, deliberately after the point where you've already seen what a read-only scan finds. Nothing in the sign-up or connect flow grants this role. You have to go looking for it.

That role's policy carries an explicit Deny block that overrides everything else, no matter what any other statement in the policy says:

iam:*, organizations:*, sts:* — CloudWise can never touch identity, org structure, or assume other roles
secretsmanager:GetSecretValue / PutSecretValue / CreateSecret / UpdateSecret — no access to your secrets, ever
cloudtrail:DeleteTrail, cloudtrail:StopLogging, config:DeleteConfigRule, config:StopConfigurationRecorder, guardduty:DeleteDetector — your audit and detection trail can't be turned off
s3:DeleteBucket, ec2:DeleteVpc, ec2:DeleteSubnet, ec2:DeleteSecurityGroup, rds:DeleteDBInstance — no deleting the structural stuff that would actually hurt
ec2:AuthorizeSecurityGroupIngress/Egress, ec2:RevokeSecurityGroupIngress/Egress, ec2:CreateSecurityGroup — CloudWise cannot open, close, or create a network path
kms:CreateKey/CreateGrant/Encrypt/Decrypt/GenerateDataKey*, ssm:* — no touching encryption or Systems Manager

A Deny statement in IAM always wins, regardless of what any Allow statement elsewhere in the same policy — or any other policy — grants. That's not a UI promise. It's how the policy evaluates at the API layer.

You'll also see a small number of Create* actions in that same policy file — things like ec2:RunInstances or rds:CreateDBInstance. Those exist for one purpose: rollback. Before executing an approved action, CloudWise records enough state to reverse it — restart an instance it stopped, recreate a resource it deleted, restore a secret's scheduled deletion. If a fix goes wrong, or you change your mind, there's a way back. That's a narrower and more honest thing than "can create resources," and it's also narrower than "can never create resources" — so I'm not going to round it either direction. Read the actual file if you want the specifics; that's the point of publishing it.

Nothing executes without your click

Here's the part I actually care about you believing: CloudWise proposes fixes. It does not execute them on its own initiative, ever, under any circumstance we ship today.

The execution path (lambdas/remediation_executor/handler.py) checks a status field before it will run a single mutating API call, and that status has to read approved. The only thing that writes approved is a separate approval-gateway Lambda, triggered by a verified, authenticated action from you. There is no code path today that sets an action to approved automatically. If you never click approve, the action sits there, proposed, forever, and nothing happens to your account.

There's also a second, quieter layer under that: every mutating call CloudWise's execution role makes is tagged with a session tag — aws:PrincipalTag/cloudwise-action — that CloudWise itself sets at execution time, and the role's policy can further condition on that tag. I want to be precise about what this is and isn't: it's real defense-in-depth inside our own execution path, not a lever you hold. You don't set that tag; we do. The thing you actually control is upstream of it — the approve click that has to happen before any of this fires at all.

What "read-only" means when we say it

So, precisely:

Onboarding grants exactly one role, scoped to Get/Describe/List/BatchGet actions, verifiable line-by-line at cloudcostwise.io/security/permissions. It cannot write to your account. This is the role every CloudWise customer has, whether or not they ever touch remediation.
Remediation is opt-in, separate, and denies the dangerous stuff outright — identity, org structure, secrets, audit trails, network rules, deletion of anything structural — regardless of what else the policy grants.
Nothing mutates without your explicit approval. Propose, then execute only what you approve. Not "propose, then execute unless you object."

None of that requires you to trust our intentions. It requires you to trust IAM evaluation semantics, which is a much smaller ask, and one you can verify yourself against files we publish rather than a page of prose we wrote about ourselves.

If you want to see what the read-only role actually finds in your account, a free, read-only scan takes about five minutes, grants nothing beyond what's described above, and changes nothing without you approving it first.

NAT Gateway: The Bill Nobody Reads

Rick Wise — Mon, 20 Jul 2026 16:03:01 +0000

Group your AWS bill by service in Cost Explorer and NAT Gateway charges don't get their own row. They're folded into Amazon Virtual Private Cloud, sitting next to VPN connections, Transit Gateway attachments, and PrivateLink endpoints. Unless you break the view down by usage type — NatGateway-Hours, NatGateway-Bytes — the number you're actually paying for a gateway that might be doing nothing is invisible.

We wrote up the mechanics of that bill back in March — the $0.045/hour base charge, the $0.045/GB data processing fee, the multi-AZ multiplication that turns one gateway into three. That post is still accurate and worth reading if you want the full pricing breakdown. This one is about something different: what CloudWise's detector actually checks before it tells you a NAT Gateway is dead weight, and where that check still falls short.

What the detector actually looks at

The idle-NAT-gateway check lives in cloudwise_scan_core's network detector, alongside the unattached-EIP and idle-load-balancer checks — it's one pass over a VPC's network resources, not a dedicated NAT scanner. For every NAT Gateway in available state, it pulls two CloudWatch metrics over a 7-day window: ActiveConnectionCount and BytesOutToDestination. If both are zero for the full week, it flags the gateway as IDLE_NAT_GATEWAY at HIGH confidence.

Two design choices worth calling out:

Why 7 days, not 1. A single quiet day doesn't mean a gateway is unused — it might serve a batch job that runs Sundays, or a staging environment nobody touches on weekends. A full week with zero connections and zero bytes is a much harder signal to explain away.

Why the estimate is conservative. The finding's monthly_savings uses a flat $32.40 — the hourly base charge times a 720-hour month, not the $32.85 you'd get from AWS's actual 730-hour average. It also doesn't add anything for data processing, because by definition a gateway that's flagged idle processed zero bytes in the lookback window. The number CloudWise shows you is the floor, not an estimate padded to look impressive.

The finding's risk field is blunt about the tradeoff, too: deleting a NAT Gateway immediately cuts internet egress for every private-subnet resource routed through it. The action is offered — aws ec2 delete-nat-gateway — but nothing executes it without a human approving first. That's the same read-only-first posture behind every detector we ship, not something special-cased for NAT Gateways.

What it doesn't do yet

Here's the honest gap: the detector tells you a NAT Gateway earned its keep for zero dollars this week. It doesn't tell you what to replace it with.

That decision genuinely depends on what's routing through it:

If the only traffic is S3 or DynamoDB, a Gateway VPC Endpoint replaces it for free — no hourly charge, no per-GB fee, ever.
If it's other AWS services (Secrets Manager, ECR, CloudWatch Logs), an Interface VPC Endpoint runs $0.01/hour per AZ plus $0.01/GB — about 78% cheaper per gigabyte than NAT, though you're paying a small hourly fee per endpoint per AZ instead of one gateway.
If it's a non-production environment with real internet egress needs but low traffic and no requirement for managed HA, a self-managed NAT instance (something like a t4g.nano) can run for a few dollars a month — you trade the 24/7 base charge for patching it yourself.

CloudWise doesn't make that call for you today. It's a fair next detector to build — matching an idle gateway's actual destination traffic against what a Gateway Endpoint could cover for free — but until it exists, the decision after "this is idle" is still on you. We'd rather say that plainly than imply the tool does more than it does.

Finding yours

If you want to see this in your own account without reading CloudWatch dashboards by hand, a free scan checks this along with 190+ other waste patterns across 40+ AWS services — read-only, five minutes, nothing gets deleted without you clicking it. We posted a 30-second walkthrough of this exact detector if you'd rather watch than read.

Or just run the two-metric check yourself:

aws cloudwatch get-metric-statistics \
  --namespace AWS/NATGateway \
  --metric-name BytesOutToDestination \
  --dimensions Name=NatGatewayId,Value=nat-0abc123def456 \
  --start-time $(date -u -v-7d +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 86400 \
  --statistics Sum

Seven zeros in a row is $32.40 a month for a gateway that isn't gating anything.

Why I Built a Risk Score Instead of a Buy Button

Rick Wise — Wed, 08 Jul 2026 17:26:13 +0000

The obvious feature to build here is a button. "You're spending $4,200/month on m5.xlarge — buy a 1-year Savings Plan and save 30%." One click, instant discount, everybody's happy.

I almost built that button. I'm glad I didn't.

Here's the problem with the button: it's only looking at the last three months. It has no idea whether the workload it's telling you to commit to will still exist in month nine. And a Reserved Instance or Savings Plan isn't a coupon — it's a bet, paid up front or amortized monthly, that a specific slice of your infrastructure will look roughly the same for a year or three. Get that bet wrong and the "savings" tool just talked you into a liability.

So instead of a buy button, I built a Commitment Risk Score — a feature whose entire job is to sometimes tell you not to buy anything yet.

The feature that argues with itself

The Commitment Risk Score pulls four signals straight from Cost Explorer for an account: how much your top instance families churn month to month, how volatile your spend is, how long your individual resources actually live, and how well you're already using the commitments you have. It weights them (35/25/25/15) into a single 0–100 score, and that score maps to a recommendation — anywhere from "3-year Convertible RI, take the discount" down to "on-demand and Spot only, buy nothing."

That last outcome is the part that made this feature interesting to build. Most cost-optimization tools are graded on how much they tell you to save. This one is graded on how honest it is about when a "savings" purchase would actually be a mistake. It costs about $0.04/account/month to compute — four GetCostAndUsage calls, refreshed weekly — the recommendation quality has nothing to do with the compute cost, so there was no excuse to cut a signal to save pennies.

I wrote up the full math — the Jaccard-distance churn calculation, the coefficient-of-variation volatility threshold, the worked examples — in a separate deep dive, because the mechanics deserved their own space: The Commitment-Risk Score: should you buy that RI?

What almost went wrong

The instinct that almost got me was averaging. My first pass weighted all four signals close to evenly, because "why not, they're all relevant." It took building the worked examples to see the problem: a team mid-Graviton-migration with high churn but a stable dollar total would score fine on an even-weighted average, because volatility and churn partially cancel out in the wrong direction. The churn signal needed to dominate — 35%, not 25% — because a changing instance mix is structurally the most common way a commitment gets stranded, regardless of what the topline spend number is doing.

The existing-waste signal ended up smallest at 15%, which felt backwards at first — isn't "you're already wasting money" the most damning fact? It is, but it's also the least predictive one. It tells you about the past, not whether the next commitment will strand. It stayed in the score because it's a legitimate red flag, but it doesn't get to drown out the forward-looking signals.

Why this is the right shape for a cost tool

CloudWise's whole premise is read-only: we look at your AWS usage and billing data, we never touch your infrastructure, and every dollar figure we show you needs to survive you checking it against Cost Explorer yourself. A recommendation engine that only ever says "buy more" doesn't survive that scrutiny for long — eventually it recommends a commitment that strands, and you stop trusting the number.

A feature that's willing to say "not yet, here's why" is the one worth trusting the next time it says "yes, buy it." That was true before I shipped this, and building it just made me believe it more.

CloudWise is an AWS cost optimization tool for startups — 191 automated waste checks including commitment-risk scoring, read-only by design, starting at $19/month. Run a free scan at cloudcostwise.io.

DEV Community: Rick Wise

SR&ED documentation as a solo founder: log technological uncertainty while you build

The problem: reconstructing SR&ED documentation after the fact

What CRA actually asks you to document on the T661

What I had vs. what I needed

The lightweight SR&ED documentation habit: two lines per dead end

What I'd do differently

What this post is not

How We Prove an AWS Waste Check Works: Fire on a Real Cluster, Stay Silent on Its Twin

The check, and the real cluster we pointed it at

What the real cluster told us that the unit tests couldn't

The first real firing, and why we almost missed it

The control: the same cluster, connected

Controls need checking too

What we took from it

Your Container Tag Is Lying to You: A Mutable ECR Tag Put the Wrong Build in Production

The timeline, from CloudTrail and ECR, not from memory

Why a mutable tag is the actual defect

The fix: verify the digest, don't trust the tag (PR #1181)

The part that makes it worth writing about

Catching a Cost Spike the Same Day It Happens: Inside Our Z-Score Anomaly Detector

The mechanic: z-score, not a percentage

Severity isn't binary

The test that proves it

Why this is Shield-tier, and why it's read-only

I Shipped a Redesign That Orphaned My Own Product

What I built

The part where it looked finished

The front door had hidden the house

The agent was also lying about multi-account

Thirty days of taking ownership

The lesson, stated plainly

EBS Snapshot Sprawl: The Waste Cost Explorer Can't Show You

Cost Explorer shows you a bill, not a culprit

deregistered ≠ deleted — the AMI-pinning trap

Lifecycle policies are the actual fix — for snapshots you haven't made yet

Finding yours

The Bug That Turned Every Bad Password Into a Server Outage

The bug

Why the status code is the actual bug

How it actually got caught

The fix

The takeaway

Inside a Real $11,871/mo AWS Waste Scan

Where the 18 findings live

The one number that matters most

What the next four biggest opportunities have in common

Why this matters for how you should read your scan

We Cut Lambda Cold Starts 56% — Three Wrong Turns Before the Real SnapStart Fix

What was actually slow

The diagnosis: init phase vs. handler phase

Wrong turn #1: it shipped clean and did nothing

Wrong turn #2 (well, half a turn): it was a log level

The actual bug: after_restore cleared, but never rebuilt

The result

The actual lesson

Why Read-Only Is the Only Safe Way to Let AI Near Your AWS Account

The role you actually grant

Remediation is a different stack, a different role, and a different click

Nothing executes without your click

What "read-only" means when we say it

NAT Gateway: The Bill Nobody Reads

What the detector actually looks at

What it doesn't do yet

Finding yours

Why I Built a Risk Score Instead of a Buy Button

The feature that argues with itself

What almost went wrong

Why this is the right shape for a cost tool

`deregistered ≠ deleted` — the AMI-pinning trap

The actual bug: `after_restore` cleared, but never rebuilt