DEV Community: Rick Wise

A Client Cost Review With the Credentials You Already Have

Rick Wise — Fri, 23 Oct 2026 13:00:00 +0000

You already have a profile for the client's account. An SSO role, a read-only role you assume, a ReadOnlyAccess user someone set up for the audit.

A cost review should not need anything more than that.

Most waste tools start by asking the client for something new: a cross-account role for the vendor, a stack to deploy, a data export. That is a security review and a week of email before you've looked at a single volume. The tool below skips it. It runs on your laptop, with the credentials you already hold, and sends nothing to us.

This is a walk-through of that review, including the places where it can mislead you.

Run it against the client profile

pipx install cloudcostwise
cloudcostwise scan --profile client-acme --regions all --format json > waste.json

--profile is the AWS profile name from your own config. Without it the tool uses the standard credential chain: environment variables, SSO, an instance role.

--regions all scans every enabled region, a few at a time (the default is four in parallel). The README puts a full multi-region run at roughly two to three minutes rather than ten. Leave --regions off and you get us-east-1 plus the profile's own region.

Pick the output for the audience. table is for you, markdown pastes into a ticket, json is the one to keep.

What it needs

AWS's managed ReadOnlyAccess is enough. If you'd rather scope a role down, the minimal policy the checks use ships in the repository as docs/iam-policy.json. It was derived from a scan of an account with most of the supported resource types, so if a check lacks a permission, the scan says so in its data warnings instead of failing.

That matters when the client's role is narrower than you expected. You'll get a gap, labelled as a gap.

What it will not do

Every AWS call the scanner makes goes through a guard that refuses any operation not starting with Describe, List, Get, Search or Lookup, whatever your credentials would allow. I wrote up the mechanism, with the code, last week in Read-Only by Construction, so I won't retell it.

Two things worth saying to a client who asks:

The guard is a second layer. Your credentials should be read-only too. Two layers that fail differently beat one.
It trusts AWS's naming. A few read-named operations have side effects, and the guard guards its own process, not a hostile one. The post above lists them.

The code is source-available under FSL-1.1-ALv2, so the client's engineer can read the guard instead of taking your word, or mine.

The one cost: Cost Explorer

The Cost Explorer API is not free. AWS bills it at $0.01 per request, and a scan makes about 13 of them, so roughly $0.13. That lands on the account the calls run in.

Those requests feed the Reserved Instance and Savings Plans checks. If the client has a cost-sensitive view of any charge appearing in their bill, skip them:

cloudcostwise scan --profile client-acme --regions all --no-cost-explorer

You lose the commitment checks, and extended-support surcharges fall back to estimates. The rest of the scan is unchanged. The output says which one you got.

What it checks

The local CLI runs 20 AWS services: EC2, Lambda, SageMaker, WorkSpaces, Lightsail, EBS, S3, EFS, ECR, RDS, DynamoDB, ElastiCache, Elastic IPs, NAT gateways and load balancers, VPC endpoints, dangling DNS records, CloudWatch logs and dashboards, security posture, RI and Savings Plans opportunities, and Compute Optimizer.

It is not a full audit of the account. It is a first pass at the things that bill quietly: unattached volumes, log groups that never expire, container images nobody cleaned up, idle load balancers. For idle EC2, RDS and NAT gateways, and for rightsizing, AWS's own tools cover the same ground, and you should use them too.

Reading the result honestly

This is the part that decides whether the review is worth the client's time.

The savings figure is an estimate. Each finding carries an estimated monthly saving and a confidence. Treat the total as "worth a look", not "will appear on the next invoice". Some waste types are marked advisory: they are listed, but never added to the total, because they could not be validated against a real AWS resource.

A failed read is reported as missing, not zero. If the credentials can't read a metric, the check says so in the data warnings (run with --verbose to see them). A finding that is absent because the read failed is not a clean bill of health. Look at warnings, errors and permission_errors in the JSON before you tell a client a service is clean.

A finding is a lead, not an instruction. Whether that volume is really unattached, or that load balancer really idle, is a question for the people who own the workload. Check before you delete anything. The tool never deletes anything, which is the point, but you will.

What you hand the client

The JSON is the useful artifact, and not only for the findings:

findings: waste type, region, resource, estimated monthly saving, confidence and the suggested action.
aws_api_calls: every AWS call the scan made, by service. This is the audit trail for the client's security person. They can check it against their own CloudTrail.
warnings, errors, permission_errors, coverage_notes: what the scan could not see.
checks_run: which checks ran at all.

Pair it with a short note of your own: which findings you'd act on first, which need the owner's input, and what the scan didn't cover. That note is the part a client is paying for. The tool just saves you the afternoon of clicking through consoles.

Hand over the file, and the account's resource names stay between you and them. Nothing was sent to CloudWise.

If you find a wrong answer

A false positive, a miss, or a dollar figure that looks off: post it in issue #1 on the repository, or email cloudwise@cloudcostwise.io. The waste type and counts are enough; leave out account IDs and resource names. A confirmed false positive gets fixed in the next release.

Try it

Free, read-only, source-available. Nothing is sent to CloudWise.

pipx install cloudcostwise

Details and the install line: cloudcostwise.io. If you run reviews like this for several clients, there is a page for consultancies at cloudcostwise.io/partners.

Read-Only by Construction: The Guard in Front of Every AWS Call

Rick Wise — Thu, 15 Oct 2026 13:00:00 +0000

Originally published on the CloudWise blog.

My launch post says the scanner "cannot change anything."

That is a big claim from one person you've never met, about a tool you're about to point at your AWS account. The fair reaction is: how do you know?

I'd rather you not take my word for it. So here is the mechanism: a small guard that sits in front of every AWS call the scanner makes. You can copy the pattern for anything you ship that touches someone else's cloud.

"Read-only" is usually a promise

Look at how most tools earn the word:

The docs say it.
The IAM role you create for them only allows reads.

The second one is real protection. But it protects you from the vendor's permissions, not from the vendor's code. If the role is wrong, or a later release asks for more, the promise and the behavior drift apart and nobody notices until something is deleted.

I wanted a second, independent layer that lives in the code path itself. If the tool is ever wrong about what it's about to do, the call should die on the way out.

One chokepoint

Every boto3 client call, for every AWS service, ends up in one method: BaseClient._make_api_call in botocore. That is the chokepoint. Wrap it and you see every operation by name, before any request is signed or sent.

The guard in the scanner is a context manager around that method. Trimmed:

READ_ONLY_PREFIXES = ("Describe", "List", "Get", "Search", "Lookup")

@contextmanager
def read_only_guard(blocked_services=frozenset()):
    log = GuardLog(calls=Counter(), violations=[], blocked=Counter())
    original = botocore.client.BaseClient._make_api_call

    def guarded(client, operation_name, api_params):
        service = client.meta.service_model.service_name
        if not operation_name.startswith(READ_ONLY_PREFIXES):
            log.violations.append(f"{service}:{operation_name}")
            raise ReadOnlyViolation(
                f"refused non-read AWS call {service}:{operation_name}"
            )
        log.calls[service] += 1
        return original(client, operation_name, api_params)

    botocore.client.BaseClient._make_api_call = guarded
    try:
        yield log
    finally:
        botocore.client.BaseClient._make_api_call = original

(The real version also handles blocked_services, which I'll get to.)

That's the whole idea. Four things in it are doing the work.

1. An allow-list, not a deny-list. The guard doesn't try to list every dangerous verb (Put, Create, Delete, Update, Modify, Start, Stop, Tag, and whatever AWS invents next year). It lists the five prefixes that mean "read" and refuses everything else. A new write operation is blocked by default without anyone remembering to add it.

2. Refuse before send. The check runs before the original method, so a refused call never leaves the machine. It doesn't matter what the credentials would have allowed.

3. The refusal is loud, twice. This is the easy part to miss. The scan's detectors catch their own exceptions, because one failing check shouldn't kill a scan of 20 services. That means a raised ReadOnlyViolation can be swallowed and logged as a warning, and the scan would carry on looking fine. So the guard also appends every refusal to log.violations, and the caller checks that list when each region finishes. If it is not empty, the whole scan fails:

if violations:
    raise ReadOnlyViolation(
        f"refused non-read AWS call(s): {', '.join(sorted(set(violations)))}"
    )

A safety check that a try/except can quietly turn into a warning is not a safety check. There is a test for exactly this: a refused write in any region fails the whole scan.

4. Scoped, and always restored. The patch exists only inside the with block and is undone in finally. Nothing else in the process keeps a patched client.

The same hook does a second job

Because the guard sees every call by service and operation, it counts them. The JSON output includes the number of calls made to each service, so you can see what the scan actually did against your account.

It also answers one service locally. The Cost Explorer checks cost $0.01 per request on AWS's side, about $0.13 per scan, so --no-cost-explorer adds ce to blocked_services. Those calls are never sent. The guard answers them with AccessDenied, exactly what the scanner would see if your role had no Cost Explorer permission, and the scan falls back the same way. They are counted separately from violations, because skipping a paid read is not a safety event.

What it does not protect you from

This is the part that matters for trust, so, plainly:

It trusts AWS's naming. The guard is a prefix allow-list. An operation called Get... is allowed because its name says it is a read, and that is a rule about names, not a proof about behavior. Some read-named operations do have side effects: STS GetSessionToken mints temporary credentials, and Cognito Identity's GetId can generate a new identity ID. Neither changes your infrastructure, but the guard cannot tell the difference, and it would not catch a Get... that did. The scan's JSON output lists the calls it actually made per service, which is how you audit what ran.
It guards the scanner's own process. It wraps botocore inside the same Python process as the code it protects. It stops mistakes: a detector that tries to be helpful, a future change, a bad merge. It is not a sandbox against hostile code, which could remove the patch. That is why the code is source-available (FSL-1.1-ALv2): the guard is a few lines you can read before you run anything.
Read-only is not "reads nothing." Reads can still return sensitive data if the credentials allow it. Only give the scanner a profile that can see what you're comfortable having it see.
It is not a substitute for a read-only role. Use a read-only IAM role anyway. The guard and the role fail in different ways, which is the point of having both: the role stops the guard's bugs, and the guard stops the role's drift.

One more check sits outside the guard. When I recorded every network connection during a full scan, it was 48 hosts, all of them *.amazonaws.com. Nothing is sent to CloudWise. The guard answers "will it change anything?" and the network capture answers "who is it talking to?"

Copy this for your own tool

If you ship something that runs inside a customer's cloud account, the pattern generalizes:

Find the one function every call goes through.
Allow-list the safe operations by name. Refuse everything else by default.
Refuse before the request is sent, not after.
Record the refusal somewhere an exception handler can't swallow it, and fail the run.
Scope the hook to the work, and restore it in finally.
Keep the role read-only too. Two layers that fail differently beat one.

Try it

The scanner is free and runs on your machine, with your own credentials. It checks 20 AWS services, and --format json lists how many calls it made to each.

pipx install cloudcostwise
cloudcostwise scan --profile my-profile --regions all

Free, read-only, a few minutes: cloudcostwise.io. The guard lives in runtime.py in the repo. If you can find a way past it, open an issue. I'd rather hear it from you.

Zero Strangers Connected an AWS Account. So the Scanner Runs on Your Machine.

Rick Wise — Wed, 14 Oct 2026 13:00:00 +0000

For fourteen months I built CloudWise, a hosted tool that finds AWS waste and fixes it safely.

This summer I stopped building and ran an experiment with exactly one metric: would ten strangers connect an AWS account by October 12?

The answer was zero.

Why zero is the right answer

It took me too long to see it, but it's obvious from the other side of the table. Connecting a cost tool means creating an IAM role in your account and handing it to a vendor. When the vendor is one person you've never heard of, the sensible answer is no. It doesn't matter how good the scans are. Nobody gets far enough to see them.

I had been trying to fix a trust problem with features. Fourteen months of detectors, dashboards and an AI copilot, aimed at a step nobody would take.

So the scanner comes to you

Today the scanner is free and source-available (FSL-1.1-ALv2) as cloudcostwise. It runs on your machine, with your own read-only credentials:

pipx install cloudcostwise
cloudcostwise scan --profile my-profile --regions all

It checks 20 AWS services, including EC2, EBS, S3, RDS, DynamoDB, ElastiCache, Lambda, NAT gateways and load balancers, Elastic IPs, VPC endpoints, CloudWatch logs and dashboards, ECR, EFS, SageMaker, Lightsail, WorkSpaces, dangling DNS records, RI and Savings Plans opportunities, and Compute Optimizer. All 17 regions take a few minutes, four regions at a time.

It's also an MCP server. With uv installed, add it to Claude Code or Codex and ask "where am I wasting money on AWS?":

/plugin marketplace add cloudwise-app/cloudcostwise
/plugin install cloudcostwise@cloudcostwise

Three promises, enforced in code

It cannot change anything. Every AWS call goes through a guard that refuses any operation that isn't Describe, List, Get, Search or Lookup, before the request leaves your machine. If one were ever attempted, the scan stops with an error. Read-only by construction, not by policy.

Nothing is sent to CloudWise. No signup, no telemetry, no update checks. I recorded every network connection during a full scan: 48 hosts, all of them *.amazonaws.com. If you use it through Claude Code or Codex, the findings go to that assistant like any other tool output, so treat them the way you treat the rest of your session.

It tells you what it costs you. The RI and Savings Plans checks call Cost Explorer, which AWS bills at $0.01 per request, about $0.13 a scan. The tool says so before it runs, and --no-cost-explorer skips them.

What AWS's free tools don't flag

The fair question: AWS already has Cost Optimization Hub and Compute Optimizer for free, and Trusted Advisor's cost checks if you pay for a Business or Enterprise support plan. Use them. For idle EC2, RDS and NAT gateways and for rightsizing, this tool overlaps with them.

So I went through AWS's own lists of what those three cover (Trusted Advisor, Cost Optimization Hub, Compute Optimizer) and compared every check this tool has proven on real resources. Of its 45 proven checks, 17 flag waste that none of AWS's tools flag, and 10 more go further than the closest AWS check. A few:

Log groups set to never expire. They grow every month. Trusted Advisor does check log retention, but it flags retention that is too short, for compliance, so "never expire" passes.
Old and untagged ECR images. Trusted Advisor checks that a repository has a lifecycle policy. Nothing flags the images already piling up.
EFS file systems with no mount targets. Nothing can read them and they still bill. Trusted Advisor's EFS checks look at redundancy, throughput and backups, not this.
Lambda functions that would be about 20% cheaper on arm64, Redis clusters that would be about 20% cheaper on Valkey, ElastiCache versions paying the Extended Support surcharge, and anything in Lightsail. None of the three looks at these.

The full list, with the closest AWS check for each and the AWS page that says so, is in the README. The hosted product adds 16 more proven checks none of AWS's tools flag, across Glue, Step Functions, AWS Backup, AppSync, Elastic Beanstalk and KMS.

On my test account, where the resources are deliberately tiny, the scan reports some of them like this:

old_ecr_images  x1  $0.06/mo
    us-east-1       cwfx-old-ecr-images                         $0.06/mo
untagged_ecr_images  x1  $0.02/mo
    us-east-1       cwfx-untagged-ecr-images                    $0.02/mo
idle_efs  x1  $0.01/mo
    us-east-1       fs-090bfb32bd49de8b7                        $0.01/mo
no_retention_log_group  x1  $0.01/mo
    us-east-1       /aws/lambda/cwfx-clo464-docdb-connect       $0.01/mo

How I know the checks are right

Cost tools have a credibility problem: a wrong finding is worse than no finding. So every waste type is tracked in a validation ledger with levels:

L1: reviewed against current AWS documentation and pricing
L2: fired on a real AWS resource built to trigger it, and stayed silent on a healthy twin
L3: its finding rendered correctly in the product

Of the 91 waste types this tool can report, 51 are proven at L2 or above against real resources in a test account. The other 40 are desk-reviewed. Findings that couldn't be validated within budget are labelled advisory and never added to a savings total. The full table is public: VALIDATION.md.

I wrote most of this with Claude Code. The ledger is how I keep that honest.

What's free and what isn't

The free scanner is 20 of the 46 service checks in CloudWise. The other 26 (Glue, ECS, Step Functions, Backup, CloudFront, commitments and more) stay in the hosted product, along with history, alerts, savings reconciled against your actual bill, and fixes you approve, with rollback. If the local scan earns your trust, that's where the rest lives.

The license is the Functional Source License: read it, run it, use it at work. The one thing you can't do is sell it as a competing product. Each release becomes Apache 2.0 two years after it ships.

The ask

Run it on a real account and tell me where it's wrong. A false positive in your account is the most useful thing you could send me. Open an issue, or reply wherever you found this.

The repo: github.com/cloudwise-app/cloudcostwise

A Cost-Anomaly Job Can Run Green and Check Nothing: CUR Lag and a date == today Gate

Rick Wise — Wed, 07 Oct 2026 13:00:00 +0000

Our cost-anomaly sweep runs every day at 08:00 UTC. It reads daily cost rows built from AWS billing data, and it only evaluated a service if that service had a row dated today.

Billing data lags real usage. At 08:00 UTC a row dated today almost never exists. So the sweep evaluated zero services on every run, with no error.

This is the full story behind a correction we added to one of our own posts, and the three things we would tell anyone building on Cost and Usage Report (CUR) data.

The honest link first

In August we published Catching a Cost Spike the Same Day It Happens. It walked through the z-score math behind our anomaly detector. The math was right. The promise in the title was not.

On September 21 we added a correction note to the top of that post. It says, in short: when the post went out, the scheduled sweep did not catch spikes the same day, because it only looked at rows dated today, and those rows almost never existed yet. It also says a second bug stopped the sweep from resolving Shield-tier users at all.

That note is two sentences long. This post is the rest of it.

Bug 1: a `date == today` gate on lagged data

Here is the shape of the old code, trimmed:

today = datetime.now(timezone.utc).date().isoformat()

for service, daily_costs in service_costs.items():
    today_cost = next(
        (d['cost'] for d in daily_costs if d['date'] == today),
        None
    )

    if today_cost is None:
        continue

    # ... z-score against the trailing history ...

Read the continue line with lagged data in mind. At 08:00 UTC the newest cost row for an account is usually yesterday or the day before. No row matches today. Every service hits continue. The loop finishes, the function returns an empty list, and the Lambda exits cleanly.

Nothing about that run looks wrong from the outside. No exception, no timeout, no error log. An empty list of anomalies is also exactly what a quiet, healthy account produces. The job was indistinguishable from a job that checked everything and found nothing.

That is the trap. A scheduled job's success status tells you it finished. It says nothing about how much it covered.

Bug 2: a lookup on an attribute no row ever had

The second bug sat one step earlier. The sweep resolved each Shield-tier user through a cognito_user_sub attribute. No row in our users table has ever had that attribute: the table's key is id, which holds the same Cognito identifier.

So every user's ID resolved to None, and the sweep could not process a single Shield user.

Our tests didn't catch it because the test fixtures gave fake users a cognito_user_sub field. The fixtures agreed with the code, so the tests passed. The fix changed the lookup to id, and the tests now use users shaped like real rows (an id, no cognito_user_sub). Each new or updated test was confirmed to fail against the pre-fix handler before we trusted it.

Two bugs, stacked. Either one alone was enough to make the sweep evaluate nothing.

The fix: evaluate the newest complete day, and say so when it's too old

We fixed both on September 15 and 16, 2026. The date fix landed in both places that run detection, the scheduled sweep and the on-demand path, and it changed four things.

1. Evaluate the newest date that has data. Instead of today, the sweep picks the newest date with a cost row for the account, across all of its services, and evaluates that day. Detection now runs behind real usage by the billing lag, typically a day or two, not on the same day.

2. Skip accounts whose data is too old, loudly. If an account's newest cost row is more than 3 days old, the sweep skips it instead of evaluating a stale day. Two things record the skip:

A log line at WARNING. Not INFO, because INFO is dropped in our API Lambda, which runs the same logic when someone triggers detection on demand. We learned that one separately, and it applies here too: a log you never see is not a signal.
A CloudWatch metric, CloudWise/Anomaly / StaleCostData, with an Environment dimension only. The commit priced it at about $0.30 a month per environment, about $0.60 across staging and production.

3. Alert once per evaluated day. Evaluating a lagged day means two consecutive daily runs can look at the same date. A separate dedupe keyed on (account, service, evaluated_date) makes sure that day alerts exactly once.

4. Keep newer rows out of the baseline. The baseline now only uses rows before the evaluated day (< target_date, not != target_date). Otherwise a newer, partial CUR row could slip into the history and manufacture a spike.

The z-score thresholds and the $5 floor from the original post are unchanged.

What we would tell anyone building on CUR

Know your data's lag before you write "today". Billing data is not a live feed. Pick the evaluation date from the data you actually have, not from the clock. If your code has date == today anywhere near cost data, check what time the job runs and what the newest row is at that moment.

Count what you evaluated. The number that would have exposed this bug on day one was not "anomalies found". It was "services evaluated". A run that evaluates zero services is not a quiet day. It is a broken job, and it should look different from one.

Alarm on zero coverage, not only on errors. An error alarm would not have caught this, because there was no error. Coverage needs its own signal.

To be straight about where we are with that last one: today the sweep emits the StaleCostData metric, and it logs the skip at WARNING. There is no alarm on the metric yet, and the sweep does not yet publish a count of services evaluated per run. That is the gap still open, and it's the one we would close first in anyone else's system too.

Make test data look like production data. The Shield bug survived because the fixtures had a field no real row has. A fixture shaped from a real row, with the real keys and nothing extra, would have failed on the first run.

Why this matters for a cost tool

A cost tool that errors is annoying. A cost tool that runs, reports success, and checks nothing is worse, because it hands you confidence you didn't earn. The fix here was small. Finding it required asking a question the job's status could never answer: how many things did you actually look at?

If you want to see what a read-only pass over your own account finds, CloudWise runs 187 waste checks across 40+ AWS services, and it takes about five minutes.

Free, read-only AWS waste scan: cloudcostwise.io. Five minutes, no card.

SR&ED documentation as a solo founder: log technological uncertainty while you build

Rick Wise — Thu, 24 Sep 2026 13:27:37 +0000

The SR&ED claim took me longer than the feature I was claiming for.

Not the form. The reconstruction. CRA wants the technological uncertainty documented: what you didn't know, what you tried, why the obvious approach failed. I had none of that written down. I had commits.

Rebuilding that narrative from about 14 months of git history and AI-assisted coding sessions took roughly three weeks I had not budgeted. This post is about the habit that would have made it a few days: a running log, two lines per dead end, written on the day the dead end happens.

Context, so you can weigh this properly. I'm the solo founder of CloudWise, an AWS cost optimization tool: 189 waste checks across 40+ AWS services, built by one person, pre-revenue, no paying customers. I paused feature development in August 2026 to work on distribution. I filed the SR&ED claim myself, without a consultant, and later received a CRA T2 notice of reassessment that resulted in a refund. I have filed exactly once. Everything below is one founder's experience, not advice on whether your work is eligible.

The problem: reconstructing SR&ED documentation after the fact

If you are mid-build, your record of the work is probably the same as mine was: a git log, some pull request descriptions, a ticket tracker, and a pile of chat sessions with an AI coding assistant.

That record answers one question well: what changed. It answers a different question badly: what did you not know at the time, and how did you find out.

A commit message like fix: handle untagged resources in attribution tells you a fix landed. It does not tell you whether that fix was the first approach or the fourth, why the earlier ones failed, or what each failure taught you. The failed approaches are often not in the history at all. They lived on a branch you deleted, or in a session you never saved, or only in your head.

So the three weeks were not writing time. They were archaeology: reading my own diffs in order, working out which ones marked a change of approach, then trying to recover why I had abandoned the previous one. Fourteen months later, the why is the part that has decayed the most.

What CRA actually asks you to document on the T661

I am going to stay close to the source here, because this is the part where founder blog posts tend to freelance.

The claim form is the T661. Its project section asks, in order, for three things: the scientific or technological uncertainties you faced, the work you did to try to overcome them, and the advancement you achieved or attempted. CRA's guide to Form T661 (T4088) walks through those lines and their word limits. The same guide has you indicate what supporting evidence exists for the work, and notes that you keep that evidence rather than submit it with the claim.

CRA's guidelines on the eligibility of work describe technological uncertainty as a situation where it is unknown whether, or how, a result can be achieved because the available technological knowledge is insufficient. The same page tells claimants to keep the evidence that is generated as the work progresses.

Read those two pages yourself before you read anyone's summary of them, including mine. They are shorter than you expect.

The practical consequence for technological uncertainty in an SR&ED claim is this: the form asks for a narrative with a specific shape. Uncertainty, then systematic work, then what was learned. A git log has a different shape. It is a list of things that worked well enough to commit.

What I had vs. what I needed

What the form asks about	What I had	What I needed
What was uncertain at the start	Nothing written. Memory.	A dated note stating what I did not know and why existing approaches did not settle it
What I tried, in order	Commits for the approaches that survived	A record of the approaches that did not survive, with the reason each was dropped
Why the obvious approach failed	Occasionally a PR description	One sentence, written the day it failed
What was learned	The final code	A plain statement of the conclusion, including "this does not work"
When the work happened	Commit timestamps	The same, which was the one thing git gave me for free

Timestamps were the only column where my existing record was enough. Everything else I rebuilt.

The AI-assisted coding sessions were the other source. Where a session survived, it sometimes held the reasoning I was missing, because I had explained the problem to the assistant in plain language before asking for help. But I had never kept them with this purpose in mind, so coverage was patchy and finding the relevant ones was slow.

One more thing I learned during the reconstruction: most of what I had built did not belong in the claim at all. Standard integration work, UI, and infrastructure plumbing did not qualify. What qualified was a narrow slice, cross-account cost attribution where tagging is incomplete and the answer has to be inferred, because that was where I did not know whether the approach would work. Sorting fourteen months of work into those two piles was part of the reconstruction. A log would have done that sorting for me as I went.

The lightweight SR&ED documentation habit: two lines per dead end

Here is the whole system. One file in the repo, docs/uncertainty-log.md, append-only. An entry only when an approach fails or you hit something you do not know how to do. Not a journal. Not a standup.

Each entry is two lines:

## 2026-03-14
Tried: inferring the owning account for untagged resources from the creating IAM principal.
Failed because: a shared deploy role creates resources for several accounts, so the principal does not identify the owner. Next: try usage-pattern correlation. (branch: attribution-v2, commit abc1234)

That entry is an illustration of the format in my problem area, not a record from my actual claim.

The rules I would follow:

Write it the day it fails. The reason is obvious today and gone in a month.
Line one is what you tried. Line two is why it failed and what that told you. If you cannot write line two, you have not understood the failure yet, which is worth knowing anyway.
Link the commit or branch. The log carries the reasoning, git carries the dates and the diffs. Together they cover each other's gaps.
Log the question before the answer. When you start on something you do not know how to do, write one line saying so. That line is the start of the narrative the form asks for.
Do not log routine work. If the path was known and it was just effort, it does not go in. This keeps the file short and gives you, or whoever prepares your claim, a first rough sort of uncertain work from routine work.
Save the AI sessions that contain reasoning. Export them into a folder next to the log, named by date. They are timestamped explanations of what you did not know, written at the time. I had these by accident. Have them on purpose.

Cost: a couple of minutes per dead end. I would have traded that for three weeks without thinking about it.

What I'd do differently

Start the log on day one of the build, not when I first hear the word SR&ED.
Stop deleting failed branches. Tag them and leave them. A failed approach with a commit history is evidence. A deleted one is an anecdote.
Write PR descriptions that say what was ruled out, not only what was done.
Read the T661 guide before the build, not after. Knowing the shape of the questions changes what you bother to write down.
Budget the time. Even with a good log, the claim is real work. Without one, it was three weeks.
Separate uncertain work from routine work as I go, because that line decided what went into the claim, and drawing it afterwards was slow.

What this post is not

I am a founder who filed once. I am not an SR&ED advisor, I cannot assess your work, and I cannot tell you whether you qualify. If you are a Canadian founder working on SR&ED for startups and you think some of your work involved real technological uncertainty, talk to a qualified claim preparer and read CRA's own pages linked above.

What I can tell you is narrower and I am confident in it: whatever you end up claiming, and whoever prepares it, the record you keep while building is the raw material. Two lines per dead end. Start today.

Rick Wise is the solo founder of CloudWise, an AWS cost optimization tool with 189 waste checks across 40+ AWS services. He filed one SR&ED claim, himself, and is not a tax advisor. He helps technical founders reconstruct the engineering narrative from their repo and AI coding sessions at SR&ED Lab. Your accountant or claim preparer still files.

How We Prove an AWS Waste Check Works: Fire on a Real Cluster, Stay Silent on Its Twin

Rick Wise — Wed, 23 Sep 2026 13:00:00 +0000

A unit test tells you your code does what you think AWS does. It can't tell you what AWS actually does.

For a tool that reads CloudWatch metrics and billing data to decide whether a resource is wasted, that gap is the whole product. So we test each check the same way: build the wasted thing for real, and make the check prove itself on it.

The rule is two halves:

Fixture: a real resource that should trigger the check. The check has to fire on it.
Control: the healthy version of the same thing. The check has to stay silent on it.

Fire without silence and you've built an alarm that always rings. Silence without fire and you've built nothing. You need both, measured against live AWS, before you trust the check.

Here is what that looked like for one check: idle DocumentDB.

The check, and the real cluster we pointed it at

The rule: flag a DocumentDB cluster with zero connections and combined read plus write IOPS under 10 per second, averaged over 7 days.

We stood up a real db.t3.medium cluster in a separate test AWS account and left it alone. It costs $1.87 a day and it's a short-lived test resource, not something we keep. Measured on 2026-09-19:

Metric (7-day window)	Value	Gate
DatabaseConnections	0.0000	must be 0
Combined read + write IOPS	5.46 / s	under 10 / s

That is an idle cluster by any reasonable reading. The check should fire.

What the real cluster told us that the unit tests couldn't

It did not fire. Not on the first try, and not for one reason. Three separate bugs were stacked in front of it, and every one of them lived in the gap between our code and AWS's behavior.

Layer	What was wrong	How it hid
Wiring	The detector called a metrics method that didn't exist on the live data provider, so it always got nothing back. Fixed 2026-07-09.	A missing answer reads as "no data", which reads as "nothing wrong here".
Units	CloudWatch reports DocumentDB IOPS as a rate (Count per Second). We read it with `Statistics=Sum` over 7 days, which adds a rate up as if it were a running total. A steady ~5.15 per second over 10,080 one-minute datapoints sums to roughly 52,000, against a gate expecting a small number.	Our regression tests pinned the same `Sum`-shaped mock, so they enshrined the bug.
Naming	Cost data calls the product "Amazon DocumentDB (with MongoDB compatibility)". Our service-to-detector map only knew "Amazon DocumentDB" and "AmazonDocDB", and the lookup is exact, so on every scheduled scan the DocumentDB detector was skipped, even on accounts whose costs listed DocumentDB.	A skipped detector reports nothing, which looks exactly like an account with no waste.

The second row is the one worth sitting with. The tests were not sloppy. They were written against a mock that matched the code's assumption, so they could only ever confirm the assumption. Only a real cluster reporting a real rate could disagree, and it did.

The fix for the second bug reads the average of the rate instead of the sum, and gates on combined IOPS averaging under 10 per second with zero connections.

The first real firing, and why we almost missed it

With all three fixed, the 2026-09-19 06:01 UTC scan ran the DocumentDB detector on that cluster for the first time and it fired: idle cluster, list price $56.94 a month for one db.t3.medium.

You would not have seen it in the findings list, though. Our scan reconciles every finding against the account's actual billing before showing it, and the test account was on DocumentDB's free trial, so the finding reconciled to $0.00 and was dropped from the customer-facing list on purpose. It only existed in the pre-floor debug output we keep for exactly this kind of question.

That's the system working as designed, and it's also why "look at the findings page" is not a test. We had to read the layer beneath it.

The control: the same cluster, connected

A healthy twin usually means a second resource. For this check the healthy state is simply "someone is using it", and the two states are mutually exclusive by construction: the check requires zero connections, so any connection at all rules idle out.

So the control was the same cluster with one variable changed. On 2026-09-19 we opened a handful of connections from a short-lived Lambda inside the VPC and deleted it the same day.

	Fixture (idle)	Control (connected)
DatabaseConnections	0	0 to 6 per minute while connected, 7-day average 0.0169
Combined IOPS	5.46 / s	5.44 / s
Scan on 2026-09-20	fired the day before	detector ran, 0 findings

Nothing moved except the connections. IOPS stayed under the gate, and one afternoon of traffic was enough to pull the 7-day average off zero and silence the check. Fired on the fixture, silent on the control.

One honest caveat, because it would be easy to leave out: those two observations are separated in time, not simultaneous. It's one cluster measured in two states, and the code guarantees the states can't both be true. That's a weaker claim than "two clusters side by side", and we say so in the write-up.

Controls need checking too

The same sweep caught the opposite failure. A Glue job we built as a healthy control set off the "missing timeout" check. We had left the timeout out when creating it, and Glue quietly defaults to 2,880 minutes, which is exactly the number the check looks for. The control was never healthy.

A control that fires isn't a control. We now set the timeout explicitly, and we verify every control is silent across all checks, not only its own.

What we took from it

A passing unit test is a statement about your assumptions. Real resources are the only thing that can contradict them.
Test the whole path, not the function. All three bugs lived outside the detector's decision logic: in how metrics were fetched and read, and in how the check got selected to run.
Prove silence as hard as you prove firing. A check nobody has watched stay quiet is a guess.
It's cheap. One small cluster for roughly a week costs about what a lunch does, and it found bugs our mock-based tests couldn't.

That's the method we use when we validate a check against real AWS. CloudWise runs 189 waste checks across 40+ AWS services, and if you want to see what a read-only pass finds in your own account, it takes about five minutes.

Free, read-only AWS waste scan: cloudcostwise.io — five minutes, no card.

Your Container Tag Is Lying to You: A Mutable ECR Tag Put the Wrong Build in Production

Rick Wise — Wed, 09 Sep 2026 13:00:00 +0000

On 2026-08-21, a routine docs-only PR merged to main. CI built the frontend image, pushed it, and called start-deployment on the App Runner service — and got rejected. The pipeline went red. "Deploy to Production" failed.

That part is normal; pipelines go red. What isn't normal is what CloudTrail and ECR showed once we went looking: production wasn't broken by the red step. It was broken by the deploy that succeeded 3 seconds earlier — from a laptop, not CI — and the pipeline's red error message never once mentioned it.

The timeline, from CloudTrail and ECR, not from memory

Time (EDT)	Event
10:52:25	`StartDeployment` on `cloudwise-production-frontend`, called by an IAM user from a local machine (`aws-cli/2.34.2`, macOS, arm64) — not CI. At this instant the `production` tag points at the previous build.
10:52:43	CI pushes the new frontend image and moves the `production` tag to point at it.
10:52:44	CI calls `StartDeployment` as its own deploy role → `InvalidRequestException: Can't start a deployment … because it isn't in RUNNING state` → exit 254 → pipeline red.
10:55:49	The manual deployment from 10:52:25 succeeds — having resolved the `production` tag back at 10:52:25, before CI moved it. It deploys the old image.

Net state: the production tag pointed at the image CI had just built. That image was never deployed. App Runner was RUNNING on the previous build, and the pipeline was red for a reason that named none of this.

18 seconds is the whole story. That's the window between the manual deploy resolving the tag and CI moving it. Move a mutable tag inside that window and the registry and the running container permanently disagree about which image "production" means.

Why a mutable tag is the actual defect

ECR_TAG=production for the frontend repository always points at the newest push. App Runner doesn't pin a digest — it resolves whatever production points to at the moment a deployment starts, not at the moment the container swap completes. Those are two different instants, and normally nothing happens in between.

Two deployments racing to start closes that gap. One of them read the tag old; the other wrote it new. Whichever deployment actually executes the swap runs whatever it resolved, and there's no signal anywhere in App Runner's own state that says "the tag has moved since I started."

This had already been flagged as a hazard in the abstract — two CI runs landing close together can serve an older image, all green, no laptop involved. This incident reached the same failure by a different route: the second actor wasn't CI at all, so none of the existing concurrency guards between pipeline runs applied.

The fix: verify the digest, don't trust the tag (PR #1181)

Two independent changes, both in .github/workflows/deploy-environment.yml:

1. Bounded retry on start-deployment. A RUNNING-state rejection now retries for up to 20 minutes instead of failing on the first attempt — a wait can only delay a deploy, never corrupt one, so it's a safe response to "something else is deploying right now." But a retry alone doesn't fix the race; it just makes CI patient. The actual fix is next.

2. Re-read the tag's digest on every retry attempt, and refuse to deploy if it moved. Before each start-deployment call, CI re-reads what production currently points to and compares it against the digest of the image this run pushed. If they no longer match — because something else moved the tag in the interim — CI stops and errors loudly instead of deploying an image it can no longer identify:

"The production tag no longer points at the image this run built. Built <digest>, tag now <digest>. Something pushed to this tag outside this pipeline; deploying now would ship an unknown image."

That single check closes the 18-second window: CI will now only ever call start-deployment when it can prove the tag still means what it thinks it means.

3. A post-swap check — warn-only, on purpose. After the deployment completes, CI reads the digest App Runner actually deployed and compares it against what CI pushed. If they don't match, it logs a warning and keeps going rather than failing the pipeline.

That's a deliberate asymmetry, and it's the design decision worth explaining: the pre-swap digest check is a hard gate, because it's checkable before anything ships and a mismatch there means CI is about to deploy the wrong thing. The post-swap check can't be a hard gate, because by the time it runs, the running container already is what it is — the thing that ran ahead was the registry tag, not production. Failing the pipeline red at that point would block a legitimate deploy for a discrepancy that already happened and that the running service had no part in causing. Warn, don't block, on a fact you can no longer change.

The part that makes it worth writing about

The merge that triggered this was docs-only. The image that never got deployed differed from the one that did by markdown and release notes — nothing user-visible. Production was healthy and serving the whole time.

That's luck, not design, and it's the actual point: the same race on a real frontend change ships a red pipeline and production quietly serving the old build, and the red error points at App Runner deployment state — not at "prod is stale." You'd fix the wrong problem. InvalidRequestException: isn't in RUNNING state is a true statement about App Runner. It was never a true statement about what mattered, which was that the tag we were about to deploy had already been consumed by someone else. A pipeline that goes red for the right reason but tells you the wrong story is worse than one that goes red honestly — you close the ticket on the error you can see instead of the one that actually happened.

Boring fixes are the ones worth shipping: retry with a bound, verify a digest instead of trusting a tag, and only fail hard on the check that happens before anything ships.

We hit this on our own production App Runner service, running the app you'd be scanning your AWS bill with. Curious what a read-only pass over your own account finds? Free, read-only AWS waste scan: cloudcostwise.io — five minutes, no card.

Catching a Cost Spike the Same Day It Happens: Inside Our Z-Score Anomaly Detector

Rick Wise — Thu, 27 Aug 2026 13:03:55 +0000

AWS Cost Explorer's own data can lag up to 24 hours behind real usage, and nobody's on call for "the bill." A forgotten load test, a runaway Auto Scaling event, a script that didn't tear down its own infrastructure — by the time a human opens the console, whatever ran already ran.

CloudWise's anomaly detector is built to close that gap: a daily scheduled scan, not a dashboard you have to remember to check. Here's the actual statistics behind it — the real thresholds from lambdas/anomaly_detector/handler.py — and the exact test case, already in our shipped suite, that proves it catches a spike.

The mechanic: z-score, not a percentage

The naive version of this feature checks "is today more than X% above yesterday?" That breaks immediately: a service that costs $2/day naturally swings 200% day to day out of pure noise, while a service that costs $4,000/day moving 15% is a real four-figure problem. Percentage-of-yesterday has no sense of a service's normal variance.

The detector instead computes a z-score per AWS service, per account, every day:

# lambdas/anomaly_detector/handler.py
SPIKE_THRESHOLD_STD_DEV = 2.0   # standard deviations from the mean
MIN_ABSOLUTE_CHANGE_USD = 5.0   # minimum dollar change to trigger, at all
MIN_DATA_POINTS = 7             # minimum days of history required

mean = statistics.mean(historical)
std_dev = statistics.stdev(historical) if len(historical) > 1 else 0
change = today_cost - mean
z_score = (today_cost - mean) / std_dev if std_dev > 0 else 0

if z_score >= SPIKE_THRESHOLD_STD_DEV:
    # ...flag it

historical is the trailing daily costs for that service, excluding today. If a service doesn't have at least 7 days of history yet, it's skipped — no history means no baseline, and a false "anomaly" on day one of a brand-new resource is worse than useless. And before z-score even runs, there's a $5 absolute-change floor: if today's cost moved by less than five dollars, the detector doesn't care how many standard deviations that represents. A service that normally costs $0.02/day moving to $0.11/day is a 450% swing, a huge z-score, and completely irrelevant to anyone's bill. The dollar floor is what keeps a statistically-driven detector from paging you about noise.

Severity isn't binary

Once something clears the 2.0 z-score bar, it gets bucketed:

SEVERITY_THRESHOLDS = {
    'CRITICAL': 4.0,
    'HIGH': 3.0,
    'MEDIUM': 2.0,
    'LOW': 1.5,
}

(Yes, LOW is below the 2.0 trigger threshold in the table — that band exists for a second call site that widens the net for manual/on-demand scans. The scheduled daily job only ever emits MEDIUM and up, because the scheduled job's trigger is 2.0.) The severities aren't decoration — they're what a Shield-tier customer's Slack alert leads with, and what determines whether the message reads as "worth a look this week" or "look now."

The test that proves it

Rather than a hypothetical dollar figure, here's the exact fixture from our own shipped test suite (lambdas/anomaly_detector/test_handler.py::test_detects_anomaly_with_high_z_score) — real, committed, running in CI on every change to this file:

An EC2 line item costs $9–$11/day for seven straight days: $10, $11, $9, $10.5, $9.5, $10, $10. Mean $10.00, standard deviation $0.65. Then it jumps to $100 in a day.

change     = 100 - 10        = $90
z_score    = (100 - 10) / 0.65 = 139.4
percentage = 90 / 10 * 100     = 900%

A z-score of 139.4 isn't a borderline call by any measure — the test asserts the detector flags it as CRITICAL or HIGH severity, and it does, deterministically, every run. The absolute dollars here are small on purpose: it's a unit test, not a customer account. The statistical shape — a service ten times its normal cost, standard deviations off its own baseline — is exactly what the 2.0 threshold exists to catch on day one, whether the line item is $10 or $10,000.

Why this is Shield-tier, and why it's read-only

The detector runs once a day via EventBridge, over the last 30 days of each Shield-tier account's cost data, and writes findings to an alerts table before dispatching Slack and email notifications. It doesn't touch anything in your AWS account — no remediation, no API calls beyond reading Cost Explorer data CloudWise already ingests for the dashboard. That's deliberate: detecting a spike and deciding what to do about it are different problems, and this one only does the first.

The mechanic is boring on purpose — mean, standard deviation, a threshold, a dollar floor. Boring is what you want from something that's deciding whether to interrupt your weekend.

I Shipped a Redesign That Orphaned My Own Product

Rick Wise — Wed, 19 Aug 2026 13:53:46 +0000

This month I rebuilt CloudWise around an AI agent.

Not a chatbot bolted into a corner. The agent is the product now. You open the app and you're talking to it. "Where's my money going?" — and it pulls your AWS spend, ranks the waste, shows you the dollars per month per finding, and tells you what's safe to fix.

It looked done. The screens were beautiful. The agent answered. I shipped it as the front door.

Then I used it like a customer would. And I discovered I had quietly orphaned half my own product.

This is the story of that month, because it taught me the single most useful lesson I've learned as a solo founder: a redesign isn't done when it looks done.

What I built

For most of this year, CloudWise looked like every other AWS cost tool: a dashboard. Charts, tables, a sidebar of pages. Useful, but passive. You had to know what to look for.

The bet I made for the overhaul was that the interface should be a conversation, not a dashboard. Most people don't want to read a cost dashboard. They want to ask a question and get an answer.

So the centerpiece became a conversational workspace backed by a real tool-calling agent loop on Claude (running on AWS Bedrock). "Real" is the important word. This isn't a model that summarizes a page of text. It's a model with tools — it actually queries your cost data, your findings, your Reserved Instance and Savings Plan coverage, and composes an answer from live numbers. It remembers your account between conversations.

You can ask it:

"Where's my money going?" → it breaks down spend by service, in gold, with the month-over-month delta.
"What's safe to fix?" → it ranks your waste findings: idle NAT Gateways, forgotten SageMaker notebooks, unattached EBS snapshots, each with a dollar figure and a safe-to-fix flag.
"Compare my two accounts." → and it does.

That last one matters more than it sounds, and it's where the trouble started.

The part where it looked finished

By the start of June, the redesign was, on paper, complete. New design system. Dark, native. A conversational workspace. A guided tour. A redesigned dashboard. The agent loop. Cross-session memory. I'd migrated every screen.

I flipped the new workspace to be the post-login front door and moved on.

Here's the uncomfortable truth about building alone: you stop seeing your own product. You navigate it the way the author navigates it — from the inside, knowing every shortcut, never actually starting cold the way a real user does.

So I made myself start cold. I logged in like a brand-new customer and tried to do the boring things. Switch to my other AWS account. Open settings. Change a notification. Log out.

I couldn't.

The front door had hidden the house

None of those features were gone. Every settings page still existed. The account switcher still existed. Logout still existed. The old navigation still existed, sitting in the codebase, fully functional.

They just weren't reachable from the place users now landed.

The new workspace shell had a clean little user footer — a static label with the account name. No menu. No logout. No link to subscription, notifications, alerts, API keys, AWS accounts, or password. From the new front door, there was literally no path to any of it.

Worse, the half of the product that did still have navigation rendered in the old chrome. So a new user would land in a slick dark conversational workspace, click one thing, and get bounced into the previous design — a completely different layout. Two products wearing different clothes, stitched together at a seam the user falls straight through.

I wrote it down plainly in my own audit doc at the time, because I needed to see it without flinching:

The redesign did not delete your features — the cutover orphaned them.

That sentence reframed the entire month. This wasn't a teardown. It was a finishing problem. The work wasn't to rebuild — it was to take ownership of every job the old interface used to do, and make the new one do it better.

The agent was also lying about multi-account

While I was in there, I found a real bug — the kind that only surfaces when you use the product for real, with more than one account.

CloudWise is a multi-account tool. You connect your AWS Organization and it discovers all your accounts. But the agent was collapsing them. When you asked about cost, it blended every account together with no way to scope to one. And when you asked about Reserved Instance and Savings Plan coverage, it reported coverage for exactly one account — the first one it happened to grab — and silently ignored the rest.

That's not a cosmetic miss. That's the tool confidently giving you a wrong answer about money. If you had three accounts and asked "how's my RI coverage," it answered for one-third of your footprint and didn't tell you.

The backend could already filter by account. The old reports page could already do multi-account selection. The agent — the new centerpiece — was strictly less capable than both. The redesign had, in this one spot, made the product worse while looking like it made it better.

So the fix had three parts:

A header account switcher that scopes the entire workspace — single or multi-select, capped by your plan tier.
Per-account arguments on the agent's tools, so the model can honor that switcher and answer in-conversation requests like "compare my two accounts."
Fixing the coverage bug so it reports across all accounts, not the first one.

# Before: commitments reported for whatever account came first.
def get_commitments(accounts):
    first = accounts[0]
    return fetch_coverage(first.account_id)   # silently ignores the rest

# After: the agent scopes to what you asked for — one, some, or all.
def get_commitments(accounts, scope_ids=None):
    targets = [a for a in accounts if a.account_id in scope_ids] if scope_ids else accounts
    return [fetch_coverage(a.account_id) for a in targets]

A small diff. A big difference in whether you can trust the answer.

Thirty days of taking ownership

That's what the month actually was. Not "add features." Re-own the basics, and make the agent the real front door instead of a beautiful demo sitting on top of a product it had stopped being responsible for.

In order, here's what changed:

One app shell, not two. Unified chrome around every authenticated page — one persistent sidebar, the account switcher, every settings link, and logout — so you never fall through the seam between the new workspace and the old layout again. The workspace became one surface inside a consistent app, not a separate world.

The account switcher scopes everything. Pick an account (or several) in the header and the whole workspace re-scopes to it — and the agent answers for exactly that selection.

The agent learned to count accounts. Per-account tool arguments, the coverage bug fixed, and the ability to genuinely compare accounts on request.

A deep, cross-page guided tour. Instead of a checklist nobody reads, a docked "CloudWise agent" companion walks a new user through the real product — navigating the actual screens across seventeen steps, not narrating a slideshow.

The whole product went dark-mode-native. Every deep page — cost reports, remediation, savings plans, settings, the dashboard — rebuilt on one design system. Money is always gold. No more half-migrated screens where one page is dark and the next is white.

A shareable cost-health score. One number, 0–100, for how efficiently you're running AWS — with a public, sanitized share card carrying no account details or PII. Because the first question every engineer actually asks is "are we good, or not?"

Across the month that came out to a few dozen shipped changes and around fifty releases — versions 1.54 through 1.104. Most of them were not glamorous. Most of them were re-owning a job the old interface used to do, quietly, that the new one had dropped on the floor.

The lesson, stated plainly

I've shipped a lot of software in twenty-five years. I still fell for this one, which is why I think it's worth writing down.

"Looks done" and "is the product" are not the same state. They can be a hundred releases apart.

A redesign is not finished when the new screens are beautiful and the happy path works. It's finished when the new thing has quietly taken over every job the old thing did — logout, account switching, the boring settings page nobody screenshots — and does each of them at least as well. Until then you don't have a new product. You have a beautiful front door on a house whose rooms you've locked.

And the cruelest part: users don't grade you on the demo. They grade you on the one ordinary day they need the single feature you forgot to bring across. The day they need to switch accounts, or check their RI coverage across all three, or just sign out. That's the moment your redesign is actually judged — and it's never the moment you rehearsed.

So now I have a rule. Before I call any cutover done, I log in cold and do the boring things. All of them. If I can't sign out, I'm not done. No matter how good the front door looks.

CloudWise is an AWS cost optimization tool for startups — 191 automated waste checks, a real agent that runs against your actual usage, air-gapped mode for security teams, starting at $19/month. If you want to ask an agent where your AWS money is going, it's at cloudcostwise.io.

EBS Snapshot Sprawl: The Waste Cost Explorer Can't Show You

Rick Wise — Wed, 19 Aug 2026 13:51:19 +0000

Last week's short covered the 101 on old EBS snapshots: the 90-day threshold, the $0.05/GB-month rate, aws ec2 delete-snapshot. If you saw it, you already know snapshots are cheap-per-unit and expensive-in-aggregate. What that 42 seconds couldn't fit in is the actual reason snapshot sprawl is so hard to find in the first place — and it isn't the price. It's that Cost Explorer structurally cannot show you who's responsible for it.

Cost Explorer shows you a bill, not a culprit

Group your AWS costs by usage type and EBS snapshots show up as one line: EBS:SnapshotUsage, rolled up per account and region. That's it. Not per-snapshot, not per-volume, not per-AMI. If that line is $340/month, Cost Explorer will tell you the total and nothing about which of your 200 snapshots — or which of your 40 AMIs pinning them — put it there.

Part of why this is so opaque is how the billing actually works. EBS snapshots are incremental: the first snapshot of a volume captures every block, but every snapshot after that only stores blocks that changed since the previous one. Delete an "old" snapshot in the middle of a chain and AWS doesn't just drop its unique blocks — it can merge the still-referenced blocks from that snapshot into the next one to keep the chain valid. The result is a genuinely well-designed storage model that happens to make per-snapshot cost attribution close to meaningless from the outside. You can't look at snapshot #14 in a chain of 20 and know what deleting it actually frees, and Cost Explorer doesn't even try — it just gives you the account-wide sum and moves on.

So the number you see going up every month is real. The reason is invisible from the billing console. You have to go look at the resources directly.

`deregistered ≠ deleted` — the AMI-pinning trap

Here's the gap that costs teams the most and gets found the least: deregistering an AMI does not delete the snapshot backing it.

When you run CreateImage against an EC2 instance, AWS creates an AMI and, silently, one or more EBS snapshots to back it — you'll find the linkage in the snapshot's own description field, something like Created by CreateImage(i-0abc123def456789) for ami-0fedcba987654321. Deregister that AMI later (cleaning up an old release, retiring a pipeline, whatever the reason) and AWS removes the AMI. The snapshot stays. Forever. Nothing in the console flags it, nothing in Cost Explorer changes shape, and nothing tells you the thing that snapshot was created for no longer exists.

This is exactly the pattern our AMI_ORPHANED_SNAPSHOT detector checks for. It's not a guess — it's a direct read of the relationship AWS itself records: the detector regex-matches each snapshot's description against the CreateImage(...) for (ami-...) pattern, pulls the referenced AMI ID, and checks it against the account's currently-registered AMIs. If the AMI isn't there anymore, the snapshot is flagged — at higher priority than a generic "old snapshot" check, because an orphaned-AMI snapshot has a certain reason to be dead, not just an age-based guess.

That priority ordering matters. Our storage detector dedups three overlapping checks against the same snapshot inventory: AMI-orphan first, then volume-orphan (the source volume was deleted), then plain age (over the 90-day default threshold, at the $0.05/GB-month rate from last week's short). A snapshot only gets counted once, under whichever explanation is strongest — an AMI-orphaned snapshot isn't also reported as merely "old," because "old" undersells why it's actually safe to delete.

Lifecycle policies are the actual fix — for snapshots you haven't made yet

None of the above is a criticism of EBS snapshots as a backup mechanism. They're cheap, they're incremental, and they're the right tool. The problem is entirely operational: nothing deletes them automatically unless you tell it to.

AWS Data Lifecycle Manager (DLM) exists for exactly this — attach a policy to a tag or resource type and it will create snapshots on a schedule and expire them on a schedule, so "backup taken 400 days ago for a server that's been gone for 399 of them" stops being possible going forward. If you're not running DLM policies today, that's the highest-leverage 20-minute fix here, full stop.

But DLM only prevents new sprawl. It does nothing for the snapshots already sitting in your account from AMIs someone deregistered two years ago, or backups nobody automated before DLM was set up. That backlog needs to be found once, by hand or by a scan, before a policy can keep it clean going forward. That's the gap our detector is built for — not a replacement for lifecycle policies, the thing that finds what predates them.

Finding yours

If you want to see this in your own account instead of grepping AMI descriptions by hand, a free scan checks this along with 190+ other waste patterns across 40+ AWS services — read-only, five minutes, nothing gets deleted without you clicking it. If you'd rather watch the 90-day/$0.05-per-GB basics first, we posted a 30-second walkthrough of that detector last week.

Or check the AMI-pinning trap yourself right now:

# Snapshots whose description names a CreateImage/AMI pairing
aws ec2 describe-snapshots --owner-ids self \
  --query "Snapshots[?contains(Description, 'CreateImage')].[SnapshotId,Description]" \
  --output table

# Currently-registered AMI IDs
aws ec2 describe-images --owners self \
  --query "Images[].ImageId" --output text

Cross-reference the AMI ID in each snapshot's description against the second list. Anything missing has been quietly billing you since the day someone cleaned up an AMI and assumed the cleanup was done.

The Bug That Turned Every Bad Password Into a Server Outage

Rick Wise — Wed, 19 Aug 2026 13:50:30 +0000

Type your password wrong on CloudWise's login page, and for a while, the server told you it had a nervous breakdown.

Not "invalid credentials." Not even a plain 401. An HTTP 500 — the code reserved for "something on our end is broken" — for the most routine failure mode there is: a human mistyping a password.

This happened on staging. It happened on production. It happened to POST /api/v1/auth/login with wrong creds, an unknown email, an unconfirmed account, or an account mid-password-reset. Four different "you did something ordinary" situations, all reported back as "we did something wrong." It went undetected until an automated gate caught it and refused to let a release ship. This is that bug, the fix, and why the distinction matters more than it sounds like it should.

The bug

_handle_cognito_auth_error, the function that translates AWS Cognito's auth exceptions into an HTTP response, raised a CloudWiseException without ever setting a status_code. No status code means the default. The default is 500.

So every one of these:

Wrong password
Email that isn't registered
Account that hasn't confirmed its email yet
Account stuck in a forced password reset

came back as a server error. Reproduced live on both environments — for example, POST /api/v1/auth/login with bad credentials on staging returned:

HTTP 500
{"detail":"Invalid email or password"}

and on production:

HTTP 500
{"detail":"Authentication failed: User does not exist."}

Read that second one again. Production wasn't just returning the wrong status code — it was telling an anonymous caller whether a given email address had an account. That's a second, smaller bug riding along inside the first one.

Why the status code is the actual bug

It's tempting to shrug this off — the message was right there in the body, "Invalid email or password", so what's the harm in the wrong three-digit prefix?

The harm is that the status code isn't decoration. It's the part of the response that infrastructure reads without understanding a word of the payload. A monitoring dashboard doesn't parse detail. It counts 5xx rates. To CloudWise's own alerting, every single mistyped password looked identical to an actual outage — same bucket, same page-worthy signal, same "something is on fire" pattern, forever, at whatever rate normal users normally fat-finger their passwords. That's not a rare event. It's baseline noise, and the bug was quietly dressing it up as baseline crisis.

It also meant every one of those routine rejections logged at error level — the log level meant for things that need a human to look at them, now firing constantly for people who just needed to try again.

And then it did something worse than annoy a dashboard: it blocked a production release.

How it actually got caught

CloudWise's release pipeline runs an E2E gate against staging before anything is allowed to promote to production. One of the negative-path assertions in that gate — ci-gate-read.spec.ts:93 — logs in with a wrong password and checks that the app shows an error and stays on /auth/login, the way a real login form should behave.

That test didn't expect a 500. Nothing in a normal auth flow should return one for a wrong password. The gate went red, and release 1.105.0 sat there, unable to auto-promote to production, because the pipeline correctly refused to trust a build where the login form's error handling was behaving like a crash.

That's the part worth sitting with: this bug was already live in production, unnoticed, for who knows how long. It took a different release's gate run to surface it — not because the gate was looking for this specific bug, but because it was asserting the right general behavior (bad password → clean error, stay put) and the actual behavior didn't match. A negative-path test doesn't need to know about your bug in advance. It just needs to check that the ordinary failure case looks ordinary.

The fix

The fix, in backend/app/services/cognito_auth_service.py, maps Cognito's exceptions to the status codes they should have had all along:

NotAuthorizedException (wrong password) and UserNotFoundException (unknown email) → 401, both returning the exact same message: "Invalid email or password". Same message for both cases on purpose — the response can no longer be used to tell whether an email is registered, closing that account-enumeration leak.
UserNotConfirmedException and PasswordResetRequiredException → 403 — the account exists, but the request is correctly rejected for a reason that isn't "try a different password."
Anything else — a genuinely unexpected Cognito error — still defaults to 500. That default is correct in that case. An unrecognized failure mode is a server-side concern worth alerting on. The bug was never that 500 existed; it was that the four most common, most expected auth rejections were routed into it by omission.

Expected-rejection logging dropped from error to warning, so the logs now reflect what actually happened: a routine, anticipated rejection, not an incident.

The test suite got the fix that should have caught this the first time. test_handle_cognito_auth_error_* previously asserted on the error message only — never the status code. That's exactly how a 500-instead-of-401 slips through code review and CI both: the message text looked fine, so nobody noticed the number in front of it was wrong. The tests now assert status codes explicitly, plus two cases that weren't covered before: the no-leak UserNotFoundException path, and PasswordResetRequiredException.

This is the sibling of an earlier fix, CLO-48, which cleaned up the same bug class on the refresh-token path — an expired refresh token was logging as ERROR twice when a clean 401 was all that was warranted. That one didn't touch login. This one closes the login path CLO-48 didn't cover.

The takeaway

Status codes are a contract between your API and everything that reads it without understanding it — monitors, alerting rules, retry logic, browsers, other services. A wrong password is a client error. Treating it as a server error doesn't just mislabel one response; it teaches every downstream system that watches your 5xx rate to distrust the signal, right when you need that signal to mean something.

The thing that actually caught this wasn't a code review, a manual QA pass, or a customer complaint. It was an automated negative-path assertion doing exactly what negative-path assertions are for: checking that the boring, expected failure looks boring and expected. It's staying in the gate. Nobody has to remember to test this again — the pipeline already refuses to ship a build that gets it wrong.

If you want to see what else that kind of scrutiny turns up, a free, read-only scan of your AWS account takes about five minutes and changes nothing without you approving it first.

Inside a Real $11,871/mo AWS Waste Scan

Rick Wise — Wed, 19 Aug 2026 13:50:30 +0000

Most cost-optimization content shows you a screenshot of a dashboard and asks you to trust the number. So instead, here's an actual scan CloudWise runs internally as a demo profile — a mid-size SaaS company's AWS account, $28,500/mo in total spend, run through all 191 detectors. It came back with $11,871/mo in identified savings across 18 findings.

I want to walk through what's actually in that list, because the shape of it surprised me even after building the detectors.

Where the 18 findings live

Split by category, not dollar amount:

Database — 9 findings. By far the largest bucket. A mix of idle RDS instances, an oversized analytics database running at 12% CPU, stale manual snapshots, and a cluster of ElastiCache-specific findings (idle replicas, an engine migration opportunity, a serverless-fit opportunity, and one very large data-tiering opportunity — more on that below).
Compute — 2 findings. An idle bastion host nobody SSHs into anymore, and one oversized EC2 fleet running at 15% CPU.
Storage — 2 findings. An unattached EBS volume left over from a migration, and a batch of aging EBS snapshots.
Commitment Management — 2 findings. A Savings Plan expiring in 52 days, and a Convertible Reserved Instance sitting on previous-generation hardware with a free exchange available.
Network — 2 findings. Two idle Global Accelerators — one genuinely idle, one just disabled but still billing.
Purchase Optimization — 1 finding. Production RDS instances that have been running on-demand, 24/7, for 90+ days with no Reserved Instance covering them.

Nine of eighteen findings sitting in "Database" isn't a coincidence — it's the category where the widest range of waste patterns overlap: idle instances, oversized instances, stale snapshots, and now a whole sub-family of ElastiCache-specific checks (engine choice, replication topology, traffic shape, data tiering) that didn't exist as separate detectors a year ago.

The one number that matters most

Of the $11,871/mo total, one single finding accounts for 68% of it: $8,057/mo, from an ElastiCache data-tiering opportunity.

The cluster in question runs 4× cache.r6g.16xlarge nodes — memory-only, no local SSD — for 1,676 GiB of total cache capacity. ElastiCache's R6gd family adds local NVMe storage and tiers data between RAM and SSD automatically, based on access frequency. For a dataset where most of the data isn't accessed on every request (true of almost every real cache), that means the same effective capacity fits on far less hardware: 1× cache.r6gd.16xlarge instead of 4× cache.r6g.16xlarge.

That's not a "you forgot to delete something" finding. It's an architecture-level rightsizing call that requires reading how your own cache is actually used before you touch it — which is exactly why it's flagged as confidence: medium, not high, and comes with an explicit risk note: SSD-resident data has slightly higher latency, so it's a good fit only when a meaningful chunk of your data is cold.

What the next four biggest opportunities have in common

Set aside the $8,057/mo ElastiCache finding and look at what else shows up near the top of this scan: an expiring Savings Plan, an unpurchased Reserved Instance opportunity on production RDS, an oversized analytics database at 12% CPU, and an oversized EC2 fleet at 15% CPU.

None of those are "click delete." Every one of them is a standing decision that has to be made again, deliberately, on some recurring cadence:

A Savings Plan doesn't renew itself — someone has to decide, before it lapses, whether the workload underneath it still looks the same.
An RDS instance rightsized today can be wrong again in six months if the workload grows, shrinks, or changes shape.
"Buy the Reserved Instance" is itself a bet on the next 12 months looking like the last 3 — which is exactly the kind of call CloudWise's Commitment Risk Score exists to check before you make it.

Compare that to the idle-resource findings in this same scan — the bastion host, the unattached EBS volume, the stale snapshots, the disabled accelerators. Those are real money too, and they're the easiest to fix: find it, delete it, done. But they're structurally small, because once you delete something, it stays deleted. It doesn't come back next quarter.

Rightsizing and commitment decisions aren't like that. The right instance size and the right commitment level are correct for right now — and they drift the moment your traffic pattern, team, or architecture changes, which for a growing company is constantly. That's the actual thesis worth taking from a scan like this: waste renews itself. The idle-resource sweep is a one-time cleanup. The rightsizing and commitment layer is a standing job, and it's where most of the money in this particular scan actually lives.

Why this matters for how you should read your scan

If you run a scan and the top finding is "delete this idle volume," fix it and move on — there's nothing recurring about it. But if your biggest findings look more like this account's — an oversized instance, an expiring commitment, a data-tiering opportunity that requires understanding your own traffic — treat that as a signal that the fix isn't a single action, it's a review cadence you don't currently have. That's a different kind of problem, and it's worth naming as one.

CloudWise is an AWS cost optimization tool for startups — 191 automated waste checks across 42 AWS services, read-only by design, starting at $19/month. Run a free scan at cloudcostwise.io.

DEV Community: Rick Wise

A Client Cost Review With the Credentials You Already Have

Run it against the client profile

What it needs

What it will not do

The one cost: Cost Explorer

What it checks

Reading the result honestly

What you hand the client

If you find a wrong answer

Try it

Read-Only by Construction: The Guard in Front of Every AWS Call

"Read-only" is usually a promise

One chokepoint

The same hook does a second job

What it does not protect you from

Copy this for your own tool

Try it

Zero Strangers Connected an AWS Account. So the Scanner Runs on Your Machine.

Why zero is the right answer

So the scanner comes to you

Three promises, enforced in code

What AWS's free tools don't flag

How I know the checks are right

What's free and what isn't

The ask

A Cost-Anomaly Job Can Run Green and Check Nothing: CUR Lag and a date == today Gate

The honest link first

Bug 1: a date == today gate on lagged data

Bug 2: a lookup on an attribute no row ever had

The fix: evaluate the newest complete day, and say so when it's too old

What we would tell anyone building on CUR

Why this matters for a cost tool

SR&ED documentation as a solo founder: log technological uncertainty while you build

The problem: reconstructing SR&ED documentation after the fact

What CRA actually asks you to document on the T661

What I had vs. what I needed

The lightweight SR&ED documentation habit: two lines per dead end

What I'd do differently

What this post is not

How We Prove an AWS Waste Check Works: Fire on a Real Cluster, Stay Silent on Its Twin

The check, and the real cluster we pointed it at

What the real cluster told us that the unit tests couldn't

The first real firing, and why we almost missed it

The control: the same cluster, connected

Controls need checking too

What we took from it

Your Container Tag Is Lying to You: A Mutable ECR Tag Put the Wrong Build in Production

The timeline, from CloudTrail and ECR, not from memory

Why a mutable tag is the actual defect

The fix: verify the digest, don't trust the tag (PR #1181)

The part that makes it worth writing about

Catching a Cost Spike the Same Day It Happens: Inside Our Z-Score Anomaly Detector

The mechanic: z-score, not a percentage

Severity isn't binary

The test that proves it

Why this is Shield-tier, and why it's read-only

I Shipped a Redesign That Orphaned My Own Product

What I built

The part where it looked finished

The front door had hidden the house

The agent was also lying about multi-account

Thirty days of taking ownership

The lesson, stated plainly

EBS Snapshot Sprawl: The Waste Cost Explorer Can't Show You

Cost Explorer shows you a bill, not a culprit

deregistered ≠ deleted — the AMI-pinning trap

Lifecycle policies are the actual fix — for snapshots you haven't made yet

Finding yours

The Bug That Turned Every Bad Password Into a Server Outage

The bug

Why the status code is the actual bug

How it actually got caught

The fix

The takeaway

Inside a Real $11,871/mo AWS Waste Scan

Where the 18 findings live

The one number that matters most

What the next four biggest opportunities have in common

Why this matters for how you should read your scan

Bug 1: a `date == today` gate on lagged data

`deregistered ≠ deleted` — the AMI-pinning trap