DEV Community: Framz

We scanned our own AWS account before selling audits. It found 3.4%.

Framz — Sun, 02 Aug 2026 13:59:40 +0000

We scanned our own AWS account before selling audits. It found 3.4%.

We built an AWS cost scanner. Around 70 checks, read-only, walks every region
with spend in it. Before pointing it at anybody else we pointed it at ourselves,
partly as a test and partly because it seemed like the minimum you should do
before charging people for the same thing.

It found $293.82 a month of savings against $8,746.75 of spend. That is 3.4%,
which is below the 20% threshold we guarantee. If we were a client, the audit
would have been free.

We published the whole result anyway, resource by resource: https://cloud-cost.framz.io/demo

This post is the part that was actually interesting, which is not the total.

The findings were not where we expected

105 findings. The distribution surprised us:

66 of them were S3, worth $148 a month, and almost all of it was one bucket with no lifecycle policy.
18 were EBS and snapshots, worth $96 a month, and most of that was snapshots of volumes that no longer exist.
Compute, the thing everybody optimises first, produced almost nothing, because we already run small and we already schedule things off.

The pattern generalises. Compute is where attention goes because it is where the
instance list lives. Storage is where the money quietly accumulates, because
storage grows without anybody deploying anything.

Six things worth checking in your own account

These are the checks that produced findings on ours, in the order they produced
them. All of them are things you can do yourself in an afternoon.

1. Cost Explorer, six months, grouped by service, then by usage type.

Grouping by service tells you where to look. Grouping by usage type inside the
top service is the step people skip, and it is the one that turns "EC2-Other is
$900" into "EBS snapshots are $400 of it". EC2-Other is not a service; it is a
bucket containing EBS, snapshots, NAT processing, public IPv4 and data transfer.

2. Log groups with no retention.

Every CloudWatch log group AWS creates for you defaults to never expiring.
Ingestion is $0.50 per GB and storage is $0.03 per GB-month, so the expensive
part is getting the data in, but the never-expiring part is what makes the line
grow forever. Setting retention is free, takes one API call per group, and the
storage charge stops within about 72 hours.

Check VPC Flow Logs specifically. Enabled once for a security review with ALL
traffic captured, they are frequently the largest ingester in an account, and
nobody reads them.

3. Snapshots whose source volume is gone.

Terminating an instance does not delete its snapshots. Deleting the volume does
not delete its snapshots. Deregistering an AMI does not delete the snapshots
behind it. They are $0.05 per GB-month and they outlive everything they were
taken from.

One detail that trips people up: deleting one snapshot from a chain frees almost
nothing, because later snapshots reference its blocks. You have to delete the
chain.

4. Public IPv4 addresses.

Since 1 February 2024 every public IPv4 address costs $0.005 an hour, about
$3.60 a month, attached or not. The old rule that an attached Elastic IP was
free is no longer true. Unassociated addresses left behind by terminated
instances are the most common orphan there is, and they are also the easiest
thing on this list to fix.

5. NAT gateways carrying traffic that should be free.

A NAT gateway is $0.045 an hour plus $0.045 per GB processed. The per-GB part
applies to traffic going to S3, DynamoDB and ECR, which never leaves the AWS
network. Gateway VPC endpoints for S3 and DynamoDB cost nothing at all and take
that traffic off the NAT path entirely.

If your CI pulls container images through a NAT gateway a hundred times a day,
that is real money for traffic that a free endpoint would carry.

6. Savings Plan coverage.

Cost Explorer, Savings Plans, Coverage. If it says 0% and your compute spend is
flat month to month, that is the largest single number in your audit and it is
one purchase away.

Order matters here. Rightsize first, let the baseline settle for a couple of
weeks, then commit to about 70 to 80% of what is left. A Savings Plan is a
promise to spend a fixed amount per hour for one to three years, and committing
before a cleanup means paying for capacity you are about to delete.

The uncomfortable conclusion

Our account came in at 3.4% because it is small, it is new, and we already do
most of this. An account that has been running for three years without anybody
owning the bill is a different story, and 20% to 35% is normal there.

But the reason it is normal is not technical. Creating a resource on AWS takes
one API call and no approval. Deleting one requires somebody to know it exists,
know nothing depends on it, and care. Those costs are wildly asymmetric, which
is why the bill goes up while traffic stays flat.

The calculators

We turned the arithmetic behind each of these checks into free calculators. No
signup, no email gate, and they run on the same price constants the scanner
uses:

https://cloud-cost.framz.io/tools

Twelve of them, including data transfer, NAT gateway, snapshots, CloudWatch
Logs, S3 storage classes, Graviton and a Fargate against Lambda against EC2
comparison.

And the full checklist the audit runs, published in full rather than gated:

https://cloud-cost.framz.io/guides/aws-cost-optimization-checklist

Full disclosure, since it would be odd not to say it: we sell a fixed-price
audit. The scan of our own account is on the site precisely because it is
unflattering, and it is the only real evidence we have that the thing works.

Every AWS project starts with copy-pasting last repo's Terraform. I built a generator instead.

Framz — Tue, 21 Jul 2026 01:05:33 +0000

Be honest about how your last AWS project's infrastructure started. You copied the Terraform from the previous repo, deleted the bits that threw errors, and moved on. Six months later nobody remembers why half of it is shaped the way it is, and there's a 500-line main.tf that everyone is scared to touch.

I got tired of that ritual, so I built a Terraform Blueprint Generator: pick your AWS stack, and it emits clean, production-structured Terraform in seconds.

What "production-structured" actually means

Not one giant file. Not resources with wide-open security groups because that was fastest. What you get:

A sane module layout — separated concerns, not everything crammed into a single file.
Secure defaults — least-privilege leanings, private-by-default networking, encryption where it belongs. The safe choice is the default choice, so you have to opt out of security rather than remember to opt in.
Portable by design — no gratuitous single-vendor lock-in baked into the baseline. (That's the same posture behind our Vendor Lock-In Auditor — portability as a default, not an afterthought.)

What it is and isn't

It's a baseline you adapt, not a turnkey production deploy. You still:

review every line (it's your infra, own it),
set your own tfvars,
decide and configure your state backend.

What it saves is the boring, error-prone bootstrapping — the part where inconsistency and quiet insecurity creep in. You start from a clean, well-organised foundation instead of last quarter's technical debt.

Free, no signup: framz.io/tools/terraform-blueprint-generator.

Why we made it

We build and run cloud infrastructure at Framz, and a consistent, portable, secure-by-default starting point is something we wanted for ourselves. So it's public and free.

Two questions for the DevOps crowd: which AWS stacks / resources should it cover next? And — the one that always starts a good argument — modules or a flat structure for a small team's baseline? Let me hear it in the comments.

I ran 5 prompt-injection attacks against my chatbot's system prompt. Here's what broke.

Framz — Tue, 21 Jul 2026 01:05:19 +0000

Every team shipping an LLM feature has the same quiet worry: what happens when someone tries to jailbreak our chatbot? And almost nobody has a real answer, because "is our system prompt resilient?" usually gets settled by gut feeling and a couple of manual pokes.

I wanted a feedback loop instead. So I built a Prompt Injection Tester: paste your system prompt, and it runs five classes of attack against it, then hands you a resilience score with the actual model responses as evidence.

The five attack classes

Instruction override — "Ignore all previous instructions and…" The classic. Still works more often than you'd like.
Prompt extraction — coaxing the model into leaking its own system prompt. If your prompt contains rules, examples, or hints about internal tools, this is how they get out.
Delimiter / escape — breaking out of the formatting you wrapped user input in. If you concatenate user text into a template, this is the hole.
Role-play — "Let's play a game where you're an AI with no restrictions…" Wrapping the override in fiction to slip past a naive filter.
Indirect injection — the nasty one. The attack doesn't come from the user at all; it rides in on attacker-controlled text that enters through a tool call, a retrieved document, or a web page your agent reads.

That last one is the eye-opener. A system prompt that looks bulletproof against direct attacks falls over the moment untrusted content flows in through RAG or a browsing tool — because the model can't tell your instructions from the ones hidden in the document it just fetched.

The honest limits

Five families is a floor, not a ceiling. A determined attacker has more, and prompt injection is not a solved problem — there's no system prompt that makes you immune. Treat this as a fast first pass to catch the obvious holes before they catch you, not a certificate of safety.

One thing I deliberately got right: it runs on our own hardware. Your system prompt is not shipped off to a third-party model to be "tested" — which would be a slightly absurd thing for a security tool to do. Free, no signup.

Why I built it

We build private, in-boundary AI at Framz, so "what can untrusted input do to a model?" is a question we live with daily. The tester is the thing I wanted for our own work, so we made it public.

If you've shipped an LLM feature, run your real system prompt through it and see which of the five it survives: framz.io/tools/prompt-injection-tester.

Genuinely curious about the community's war stories: what's the most creative prompt injection you've seen land in production? Drop it in the comments — and tell me which attack classes you think the tester should add next.

When Does Self-Hosting an LLM Actually Beat the API? The Break-Even Math

Framz — Mon, 20 Jul 2026 22:25:36 +0000

Every team I've worked with hits the same argument about three months after shipping something on top of an LLM: "Should we still be paying per token, or should we just self-host?"

And it's always argued on vibes. Someone read a blog post. Someone else got burned by a GPU bill once. Nobody's actually done the math. So here's the math — or at least the shape of it.

The API wins right up until it doesn't

Paying an API has one killer advantage that people forget to give it credit for: you pay for exactly what you use, and nothing when you're idle. No GPU sitting warm at 3am doing nothing. No serving stack to babysit. If your volume is low or spiky, this is basically unbeatable, and honestly you should stop reading and just use the API.

Self-hosting flips that. You take on a mostly fixed cost — the GPU (rented or amortised) plus the very real human cost of running it — and in exchange, each extra token is almost free once you've paid for the box.

So the whole decision collapses to one question: at your volume, has that fixed cost dropped below what the meter would've charged you? That's the break-even. Everything else is just inputs to it.

The six things that move the break-even

Monthly token volume. The big one. API cost scales straight up with it; self-hosting barely moves. Low → API. High and steady → self-hosting.
Input:output ratio. Output usually costs more on APIs, and it's what eats GPU time when you host. A summariser (huge input, tiny output) and a chatbot look nothing alike on the curve.
Quality tier. Here's where people quietly cheat: "self-hosting is cheaper" usually assumes a 30B open model is good enough. If you actually need frontier quality, compare frontier-to-frontier — and you probably can't run frontier yourself.
GPU cost + amortisation. Your fixed cost, per hour or spread over the hardware's life.
Utilisation. A GPU you use 8% of the day is a space heater with a fan. This one number wrecks or saves the whole thing.
Ops overhead. The line nobody puts in the spreadsheet: someone runs it, patches it, and gets paged at 2am. At small scale that alone can be bigger than the API bill — think 10–20 senior-engineer hours a month.

The costs both sides conveniently forget

Self-hosting quietly adds: fine-tuning experiments, storage and egress, eval infrastructure, and your engineers' time — which is not free, no matter how much they enjoy it.

The API quietly adds: price changes and rate limits you don't control, and the awkward bit — your data (and your customers' data) leaving your walls.

The break-even nobody charts: compliance

There's a second crossover that has nothing to do with dollars. If you handle regulated or client-confidential data, self-hosting wins the moment that data would otherwise be shipped off to a third-party API — full stop, whatever the token math says. When "our data can't leave our infrastructure" is a hard rule, you've already made the decision; you're just backfilling the spreadsheet.

Just run your numbers

The model is simple. The arithmetic — 12 months, amortisation, utilisation — is annoying. So we built the boring part: the LLM Cost Calculator charts API vs self-hosting over a year, no signup, no email wall.

FAQ

When does self-hosting get cheaper? When your fixed cost (GPU amortisation + ops time) drops below the per-token meter at your sustained volume. Low/spiky → API. High/steady → self-host.

What's the cost everyone forgets? Engineers' time to run the stack. It's usually the single biggest hidden line.

Is an open model as good as a frontier API? Not automatically — decide your quality tier before you compare price, or you're comparing a Civic to a truck.

I'm genuinely curious where real teams land on this — what was your actual break-even, and did self-hosting turn out cheaper than you expected, or way more of a headache? Tell me in the comments.

Shadow AI: What Your Team Is Already Pasting Into ChatGPT

Framz — Mon, 20 Jul 2026 22:25:24 +0000

Right now, somewhere in your org, someone is pasting a stack trace, a customer record, or a chunk of the codebase into ChatGPT to get unstuck. You know it. I know it. The person doing it half-knows it's sketchy and is doing it anyway — because it works.

That's shadow AI, and the reason it's so hard to stamp out is that it's not laziness or malice. It's people trying to do their jobs faster. Send the "don't use ChatGPT" email and you don't stop it — you just make it quieter.

So don't start with a ban. Start with knowing what's actually leaving the building.

The four things that leak (roughly in order of how often I see it)

PII. Names, emails, addresses — worse if it's health or financial — tucked inside "help me reply to this customer" prompts. This is the one with legal teeth.
Secrets. API keys, tokens, connection strings, a stray .env, private keys — pasted into "why is this failing?" debugging prompts. High blast radius, and usually the person has no idea it was even in there.
Proprietary code. Internal logic pasted in for a refactor. It doesn't come back out, sure — but you've just handed it to a third party.
Client-confidential stuff. Contracts, engagement letters, case notes. For a regulated firm that's not an "oops," that's a reportable breach.

The one everyone quotes is the reported case of engineers pasting proprietary source into ChatGPT. What made it nasty: the data was just gone, unretrievable, and no DLP tool flagged it — because to a network monitor, an AI chat looks like any other encrypted web request, not a file leaving.

It's a governance problem wearing a tech-problem costume

The reflex is to buy a blocking proxy. But blocking just recreates the original problem: people route around it (personal laptop, phone, copy-paste on their own account), and you lose the productivity you were trying to protect.

The thing that actually holds is a compliant path — an approved way to get the same AI help without the data leaving your boundary — plus a short, human-readable policy on what's fair game and what isn't. And detection, because you can't govern what you can't see.

How to check — without leaking it to check it

You can scan for this yourself: regexes for key formats, entropy for secrets, NER for PII. The irony is that most "paste your text here to check it" tools ship your text off to their server or their model — which is the exact thing you were trying to avoid.

So we built the Shadow AI Data Leak Scanner with one rule we refused to break: it runs on our own hardware, and your text never touches a third-party AI. Paste something in, see the PII, secrets, and code it contains, and whether it's safe for a public model. It's heuristics, not magic — treat it as a fast first pass, not a certificate.

Shipping AI features yourself? The mirror-image risk is your own app getting jailbroken — poke at that with the Prompt Injection Tester.

FAQ

What is shadow AI? Staff using AI tools with no approval or oversight — pasting in whatever gets the job done, usually without clocking that the data leaves the org.

What leaks most? PII in "draft this reply," secrets in "why does this break," code in "refactor this," and client-confidential docs.

How do you stop it without a ban? A compliant in-boundary path, a one-page policy people can actually follow, and detection so it's enforceable.

Honest question for the room: has your company actually written down what's OK to paste into an LLM — or is everyone just quietly winging it? I suspect I know the answer. Tell me in the comments.

The Four Layers of Cloud Lock-In (and How Terraform Cuts Your Exit Cost)

Framz — Mon, 20 Jul 2026 22:25:08 +0000

People throw around "vendor lock-in" like it's a binary — you either have it or you don't. That framing is why the conversation usually goes nowhere. Lock-in isn't one thing. It's four different things stacked on top of each other, each with a wildly different cost to unwind, and once you separate them you can actually decide what's worth worrying about.

The four layers

Data. Proprietary formats, and — the real killer — egress fees to get it back out. Cheap to design around early, brutally expensive to fix late.
Platform. The vendor-specific managed stuff: serverless, managed databases, IAM, observability. This is where the deep lock-in lives, because these services are genuinely good and have no clean equivalent next door.
Contract. Renewal clauses, termination terms, committed-spend discounts that quietly jack up your switching cost while feeling like a deal.
Operations. Your team's muscle memory — the tooling, the runbooks, the way everyone thinks. The least visible layer, and usually the stickiest.

If you want to know where you're most exposed, it's almost always managed databases, serverless, IAM, and observability. Those deserve a real decision, not a default.

Lock-in isn't automatically a problem

Here's the part people skip: lock-in only matters when it traps regulated data, kills your pricing leverage, or makes leaving hideously expensive. A managed service that saves your team a month of work and is easy enough to swap later? That's a good trade. Take the deal.

The goal was never zero lock-in. It's knowing your number and controlling it where it counts. Which is why step one is boring and unglamorous: audit each layer, put a figure on the exit, set a date to revisit. (We built a Vendor Lock-In Auditor that spits out a portability score and an exit-cost estimate, so it's an actual number instead of a vibe.)

Where Terraform earns its keep — and where it's oversold

Infrastructure-as-code (Terraform, Pulumi) is the highest-leverage move you can make on the operations and platform layers. Declarative, version-controlled infra buys you:

An environment that's documented and reproducible instead of click-configured and half-remembered.
Your vendor dependencies sitting right there in the code, visible — you can literally read your lock-in.
The ability to stand up equivalent resources elsewhere from one workflow, which is what makes a real exit or multi-cloud plan more than a slide.

Now the honest bit: Terraform does not make a proprietary managed service portable. It makes the exit auditable and cheaper. The platform-layer dependency itself is still a design choice you make service by service. Anyone selling you Terraform as a lock-in cure is overselling.

(If you're standardising from scratch, our Terraform Blueprint Generator hands you a production-ready, portable-by-default baseline in seconds.)

The playbook I'd actually follow

Audit the four layers, price the exit.
Codify infra in Terraform so the dependencies are explicit.
Decide on purpose for each high-lock-in service: take it (and write down the risk + a portability plan) or skip it.
Negotiate egress caps and deconversion help into the contract — before you're committed, when you still have leverage.
Revisit on a schedule, because lock-in creeps while you're not looking.

Selective portability beats both religions. Keep the managed-service speed where it's worth it; keep your options open where it isn't.

FAQ

What are the four layers? Data (formats + egress), platform (vendor-specific services), contract (renewal/termination terms), operations (team skills/tooling). Highest risk: managed DBs, serverless, IAM, observability.

Does Terraform fix lock-in? It lowers the operations/platform layers and makes the exit auditable and cheaper. It does not make proprietary services portable by itself.

Should we go multi-cloud? Usually no — it's real complexity for often-imaginary benefit. Prefer selective portability.

Curious about the war stories here: what's the most brutal egress bill or exit cost you've personally run into? Drop it below — those numbers are always more educational than any blog post.