DEV Community: Patrick Hughes

How to Tune llama.cpp --n-gpu-layers: A Practical VRAM Guide (2026)

Patrick Hughes — Tue, 09 Jun 2026 14:45:13 +0000

You already know what --n-gpu-layers does. It moves transformer layers onto your GPU. This post is the next step: how to actually pick the number.

If you want the basics first, read the original: llama.cpp n-gpu-layers explained. This is the tuning guide that follows it.

The one rule that matters

A model has a fixed number of layers. A 7B model might have 32. A 70B might have 80. The --n-gpu-layers flag (often shortened to ngl) says how many of those go on the GPU. The rest stay on the CPU and run in system RAM.

Full GPU means fast. Full CPU means slow. Partial means somewhere in between, and it scales close to linearly. Offload half the layers and you get roughly half the speedup.

So the goal is simple. Put as many layers on the GPU as your VRAM allows. Not one more.

The VRAM math

Each layer costs roughly the same amount of VRAM. You can estimate it.

Take the model file size on disk. Divide by the layer count. That gives you a rough per-layer cost.

A 7B model quantized to Q4 is around 4 GB. Split across 32 layers, that is about 125 MB per layer. Offload 24 layers and you spend roughly 3 GB on weights.

This is an estimate, not a promise. Attention layers and embedding layers differ slightly. But the per-layer average holds well enough to plan with.

Do not forget the KV cache

Weights are only part of the bill. The KV cache also lives on the GPU when you offload, and it grows with context length.

Longer context means a bigger cache. Double the context window and you roughly double the cache size. On a tight card, a long context can push you into OOM even when the weights fit.

So budget VRAM in two buckets. Weights first. Then leave headroom for the KV cache at the context length you actually plan to run.

Reading OOM symptoms

When you ask for too many layers, llama.cpp fails at load time with a CUDA out of memory error. It will not silently fall back. It stops.

The fix is to drop ngl by a few and reload. Step down until it loads. If you are right at the edge, shave 2 or 3 layers and try again.

Watch your VRAM with nvidia-smi while the model loads. You want a buffer left over, not a card pinned at 100 percent. Other apps, your desktop, and the KV cache all want a slice.

A fast tuning loop

You do not need to calculate everything. You can probe.

Start with ngl set to a high number. Many people use 99 to mean "offload everything." If it loads, you are done. The whole model fits.

If it OOMs, step down. Try 28, then 24, then 20. Each reload tells you where the ceiling is. Five minutes of trial beats an hour of spreadsheet math.

Once it loads cleanly, run a real prompt at your target context length. If that OOMs mid-generation, the KV cache pushed you over. Drop a few more layers and leave room.

Quick starting points by card

These are rough anchors, not guarantees. Your quant, context, and model size all move the number.

On an 8 GB card, a 7B Q4 model usually offloads fully. A 13B will only fit partially.

On a 12 GB card, 13B models fit comfortably and you have room for context.

On 16 GB or more, you can run larger models or push context length hard. A 24 GB card handles most single-GPU local work without much tuning at all.

How quant choice feeds in

Smaller quant means smaller weights means more layers fit. If you cannot offload a model fully, dropping from Q5 to Q4 might get you there. That tradeoff is its own decision, and it pairs directly with this one.

If you are weighing which quant to run, read the companion post: which GGUF quant should you actually pick. Tune ngl and quant together. They share the same VRAM budget.

The instinct underneath all of this

Running models locally is a cost move. Every token you serve on your own GPU is a token you did not pay an API for. Tuning ngl is just squeezing more value out of hardware you already own.

That same instinct, watching the meter and refusing to overspend, is what AgentGuard does for AI agents. It caps token spend, rate limits calls, and stops a runaway loop before it burns your budget. Local inference cuts your fixed cost. AgentGuard caps your variable cost.

If you are running agents and want a hard ceiling on spend, check out AgentGuard.

Which GGUF Quant Should You Actually Pick? Q4 vs Q5 vs Q6 vs Q8 (2026)

Patrick Hughes — Tue, 09 Jun 2026 14:45:10 +0000

You know what Q4, Q5, and Q8 mean. Now the real question: which one do you actually download?

If you need the background on what these numbers represent, start with the original: GGUF quantization Q4 Q5 Q8 explained. This post is the decision guide.

The tradeoff in one line

Lower number means smaller file, less VRAM, faster load, and slightly worse output. Higher number means bigger file, more VRAM, and output closer to the original model.

That is the whole game. You are trading quality for size. The trick is knowing how much quality you actually lose, and the answer is: less than you think at the high end, more than you think at the low end.

Start with what fits

The first filter is not quality. It is VRAM.

Pick the largest quant that fits on your GPU with room for context. A model you can fully offload at Q4 will run faster than the same model at Q5 that spills onto the CPU. Fit beats precision when fit decides speed.

So measure your VRAM, subtract headroom for the KV cache, and see which quant lands under that ceiling. That narrows the choice fast.

What the K-quants give you

You will see names like Q4_K_M and Q5_K_M. The K means K-quants. They are smarter than the old flat quants.

K-quants spend more bits on the parts of the model that matter most and fewer bits on the rest. For the same file size, a K-quant holds quality better than a plain one. This is why Q4_K_M became the default many people reach for.

The M and S suffixes mean medium and small. M keeps more quality. S shrinks further. When in doubt, pick M.

The practical ladder

Here is how the common options stack up for someone on a consumer GPU.

Q4_K_M is the workhorse. Smallest footprint that still feels like the real model. If you are tight on VRAM, start here.

Q5_K_M is the safe upgrade. A noticeable quality bump over Q4 for a modest size increase. If it fits, many people prefer it.

Q6_K is close to lossless for most tasks. Bigger, but the quality gap to the full model is small. Good when you have VRAM to spare and want margin.

Q8_0 is near the original. The difference from full precision is hard to notice in normal use. It is large, so you only pick it when size is not a concern.

How quality actually falls off

Think of it as a curve, not a line.

Going from Q8 down to Q5, the quality loss is small. The model barely changes for most prompts. You get a big size win for almost no cost.

Going below Q4, the loss grows fast. Q3 and Q2 start making real mistakes: weaker reasoning, more repetition, shakier instruction following. They exist for cases where a model simply will not fit otherwise.

So the sweet spot for most people sits between Q4_K_M and Q6_K. Above that you pay size for little gain. Below that you lose quality faster than you save space.

A simple decision flow

Can you fit Q6_K with your context? Take it. Near-lossless, done.

Cannot fit Q6 but can fit Q5_K_M? Take that. Strong quality, smaller.

Tight on VRAM? Q4_K_M is the floor that still feels right.

Cannot even fit Q4? Drop to a smaller model at Q4 before you drop to Q3 of a bigger one. A smaller model at a healthy quant usually beats a big model crushed too hard.

Quant and offload are the same budget

Your quant choice and your --n-gpu-layers setting pull from the same VRAM pool. A smaller quant frees room to offload more layers, which is what makes the model fast.

If you have not tuned your layer offload yet, read the companion post: the n-gpu-layers tuning guide. Pick the quant and the layer count together. They are one decision wearing two hats.

Why any of this matters

Running local models is a cost play. Every prompt you answer on your own card is a prompt you did not pay an API to handle. Picking the right quant means more capable output per dollar of hardware you already bought.

That same discipline, getting the most value while keeping spend capped, is what AgentGuard enforces for AI agents. It sets hard limits on tokens, cost, and call rate so a loop cannot run up a bill while you sleep. Local inference trims your fixed cost. AgentGuard caps your variable cost.

If you run agents and want a real ceiling on spend, check out AgentGuard.

How to Tune --n-gpu-layers for Your VRAM Budget

Patrick Hughes — Mon, 08 Jun 2026 14:45:11 +0000

How to Tune --n-gpu-layers for Your VRAM Budget

I wrote an explainer on llama.cpp's --n-gpu-layers flag and it keeps pulling traffic. The explainer covers what the flag does. This post covers the part people actually struggle with: how to pick the right number, do the offload math, split across two GPUs, and stop the out-of-memory crashes.

What the flag really controls

A model is a stack of transformer layers. --n-gpu-layers (or -ngl) tells llama.cpp how many of those layers to put on the GPU. The rest run on the CPU.

Layers on the GPU run fast. Layers on the CPU run slow. So your goal is simple: put as many layers on the GPU as will fit, and not one more. One layer too many and you get an out-of-memory crash or a silent spill that tanks your speed.

If the whole model fits, just set -ngl 99 and forget it. The number only matters when the model is bigger than your VRAM.

The offload math

Each layer takes roughly the same amount of memory. So the math is:

vram-per-layer = model-weights-GB / total-layers
layers-that-fit = (free-vram-GB - overhead) / vram-per-layer

Work an example. A 13B model at Q4 is about 7.5 GB of weights across 40 layers. That is roughly 0.19 GB per layer.

You have an 8 GB card. Reserve about 1.5 GB for the KV cache and overhead. That leaves 6.5 GB for layers.

6.5 / 0.19 = ~34 layers

So start at -ngl 34 for that model on that card. The other 6 layers run on CPU. You get most of the speed of a full GPU load without the crash.

Find the real number fast

The math gets you close. Then you tune by hand. Watch VRAM in one terminal and step the number in another.

# terminal 1
nvidia-smi -l 1

# terminal 2: start lower than the math says, then climb
./llama-cli -m model-q4.gguf -ngl 30 -p "test prompt" -c 4096

Climb by 2 or 3 layers each run. Watch nvidia-smi. When VRAM hits about 90 percent, stop. Leave headroom. The KV cache grows as the context fills, so a load that fits an empty prompt can crash 3000 tokens later.

That last point is the number one cause of OOM crashes. People tune with a tiny prompt, see it fit, ship it, then crash on a real long input. Always tune at the context length you will actually use, set with -c.

Common OOM mistakes

You set -ngl too high and forgot the KV cache. The cache is not free. At an 8K context it can eat a couple of GB on a 13B. Reserve for it.

You raised the context length and kept the old -ngl. Bigger context means a bigger cache means less room for layers. Re-tune when you change -c.

You loaded a second model on the same card. Two models share one pool of VRAM. The first one's -ngl no longer fits.

You assumed Q8 fits because Q4 did. Q8 is nearly double the weight memory. The layer math changes completely.

Splitting across two GPUs

If you have two cards, llama.cpp can split the model across both. Use --tensor-split to set the ratio.

# two cards, 24 GB and 32 GB: weight the bigger card heavier
./llama-cli -m big-model-q5.gguf -ngl 99 --tensor-split 24,32 -p "test"

The numbers are a ratio, not gigabytes, but matching them to your VRAM sizes is a good start. With -ngl 99 and a split, llama.cpp puts all layers on the GPUs and divides them by the ratio. Now a 34B that fits on neither card alone fits across both.

One catch. Splitting adds a little cross-GPU traffic, so two 16 GB cards are a touch slower than one 32 GB card at the same total memory. Still far faster than spilling to CPU.

When this runs inside an agent

Tuning -ngl gets a single run fast. But if a local model sits behind an agent that calls it in a loop, a stuck loop can peg both GPUs for hours and run your power bill up overnight. Local does not mean free.

That is why I built AgentGuard. It is an open-source runtime budget, token, and rate limiter for AI agents, and it caps your agent loop whether the model is a cloud API or a local GGUF on your own cards. pip install agentguard, wrap the loop, set a cap, and a runaway agent stops before it costs you a night of compute.

Do the layer math, tune at your real context length, leave headroom for the cache, and split across cards when one is not enough. That is the whole game.

llama.cpp Multi-GPU: Splitting a Model Across Cards with --tensor-split

Patrick Hughes — Mon, 08 Jun 2026 14:45:08 +0000

If you are still tuning a single card, start here first: llama.cpp n-gpu-layers explained. That post covers how --n-gpu-layers moves layers onto one GPU and the VRAM math behind it.

This one is the next step. Once a model no longer fits on one card, you split it across several. A 70B model at Q4 needs about 40 GB on disk plus a few GB for the KV cache. No single consumer card has that much VRAM. But three cards together do. This is where --tensor-split comes in.

Most people who go multi-GPU are not running a one-off prompt. They are standing up a local model to feed an agent: a loop that calls the model over and over to read, plan, and act. That changes what matters. A bigger model on more cards is the capacity side of the problem. The other side is keeping that agent loop from running the rig hot all night on a task that should have stopped. Both sides need a plan before you load the weights. We will set up the hardware here and come back to the loop control AgentGuard gives you once the model is running.

The core idea

With one GPU, you only decide how many layers go on the card. With multiple GPUs, you decide how the layers get divided between them. llama.cpp loads the model once and spreads the weights across every GPU you point it at. Each card holds a slice. During inference, activations pass from one card to the next over the PCIe bus.

Three flags control this:

--tensor-split sets the ratio of the model that lands on each GPU.
--main-gpu picks which card holds the KV cache and coordinates the run.
--split-mode chooses how the work is divided: by layer or by row.

--tensor-split: the ratio

--tensor-split takes a comma-separated list, one number per GPU. The numbers are weights, not gigabytes. llama.cpp normalizes them into proportions.

Say you have a 24 GB card and a 16 GB card. You want the bigger card to hold more of the model. A split of 24,16 puts 60 percent on GPU 0 and 40 percent on GPU 1. The exact integers do not matter, only the ratio. 24,16 and 3,2 do the same thing.

The goal is to match the split to each card's free VRAM. If you overload the smaller card, you get an out-of-memory crash mid-load. Leave headroom on whichever card holds the KV cache, because that buffer grows with context length.

--main-gpu and the KV cache

One card has to hold the KV cache and run the orchestration. That is --main-gpu. It defaults to GPU 0. Point it at your largest card so the KV cache has room to grow as the context fills up.

A long context can add several gigabytes to the main GPU on top of its model slice. If your main GPU is also your smallest card, you will hit OOM at high context even though the model loaded fine. Put the cache on the big card.

--split-mode: layer vs row

--split-mode layer is the default and the right choice on consumer hardware. Each GPU owns a contiguous block of layers. Cross-card traffic happens only at the layer boundaries, so PCIe bandwidth stays low.

--split-mode row splits individual tensors across cards. It can be faster, but only when the GPUs talk over a fast link like NVLink. Consumer cards do not have NVLink. On a plain PCIe rig, row mode floods the bus and usually runs slower than layer mode. Stick with layer.

A worked example on a real rig

Here is a mixed consumer setup: an RTX 5090 (32 GB), an RTX 5070 Ti (16 GB), and an RTX 3070 (8 GB). Total is about 56 GB of VRAM. That is enough to run a 70B model at Q4 fully on GPU, with the ~40 GB of weights spread across all three cards and room left for the cache.

GPU	Index	VRAM	Split weight	Role
RTX 5090	0	32 GB	32	main-gpu, holds KV cache
RTX 5070 Ti	1	16 GB	16	model slice
RTX 3070	2	8 GB	6	model slice (leave headroom)

Note the 3070 gets a weight of 6, not 8. Leave a margin on the smallest card so a context spike does not push it over.

The command:

./llama-cli \
  --model models/llama-3.1-70b-instruct-Q4_K_M.gguf \
  --n-gpu-layers 999 \
  --tensor-split 32,16,6 \
  --main-gpu 0 \
  --split-mode layer \
  --ctx-size 8192 \
  --prompt "Explain tensor parallelism in one paragraph."

--n-gpu-layers 999 still means "all layers on GPU." The difference now is that --tensor-split decides which GPU each layer lands on. Watch the load logs. llama.cpp prints how much VRAM it assigns to each device. If one card is near its limit, adjust the ratio down and rerun.

When multi-GPU helps and when it hurts

Multi-GPU helps when a model does not fit on your largest single card. Splitting a 70B across three cards turns "impossible" into "runs at usable speed." That is the whole point.

It hurts when the model already fits on one card. Splitting it then adds PCIe round-trips between cards for no benefit, and throughput drops. If your model fits on the 5090 alone, run it there and leave the other cards free.

Expect throughput to scale below linear. Three cards do not give you 3x the speed of one. Activations still hop across PCIe between layer blocks, and that adds latency. The win is capacity, not raw speed. You are trading some tokens per second for the ability to run a much larger model at all. A 70B at Q4 across this rig lands in a usable interactive range, slower than an 8B on a single card but far smarter.

That slower throughput is exactly where an agent loop gets dangerous. A 70B across three cards might do 10 to 15 tokens a second. An agent that retries, re-reads context, and re-plans can run for hours and you would not notice until the room is warm. The model is free to run, but the time and power are not, and a stuck loop produces nothing. This is the operational cost of local inference: not a per-token bill, but wasted hours on a runaway task. AgentGuard sits around the calls and enforces a hard ceiling on tokens and wall-clock spend, so the loop stops itself instead of grinding until morning. Set the cap before you point an agent at a multi-GPU model, not after the first overnight surprise.

The short version

Single-card --n-gpu-layers is step one: how many layers fit on one GPU. --tensor-split is how you scale past one card: the ratio of the model each GPU holds. Set the ratio to match free VRAM, put --main-gpu on your largest card so the KV cache has room, and keep --split-mode layer unless you have NVLink. For the broader picture on running local models in production, see local LLM inference on consumer GPUs.

That gets the model running. The capacity side is solved. Before you wire the model into an agent, solve the loop side too: put AgentGuard around the calls so a runaway agent on your slow-but-smart 70B stops at a budget instead of burning the whole night.

What Uber's $1,500/Developer AI Cap Tells You About Your Own Bill

Patrick Hughes — Sun, 07 Jun 2026 14:45:13 +0000

Bloomberg reported this week that Uber now caps every employee at $1,500 per month, per AI coding tool. Simon Willison picked it up on June 3. The number matters less than the move. A Fortune 50 company with a real finance team just admitted it cannot predict what its developers spend on AI.

This is the follow-up to a story I wrote about in May. Uber burned its entire 2026 AI coding budget in four months, running at roughly 3x the naive projection. The $1,500 cap is the response. Burn first, cap second. That order tells you everything.

The math is looser than it sounds

Read the policy carefully. The cap is per tool, per employee. Not per person. Not per month total.

So one developer running Claude Code, Cursor, and Copilot can spend $1,500 on each before any limit fires. That is $4,500 a month, per head, inside policy. Multiply by a team. The org-level number is still wide open.

A per-tool cap is a speed bump, not a wall. It slows the worst offenders. It does not give you a real budget. Uber knows this. It is the best they could ship fast, and fast was the constraint.

What this means if you are not Uber

Here is the part that hits small shops and solo builders.

If a company with thousands of engineers and a dedicated FinOps function cannot forecast coding-agent spend, you cannot either. Not because you are bad at it. Because the spend is non-deterministic. An agent in a loop, a long context window, a retry storm, a contractor who left a script running over the weekend. None of that shows up until the bill does.

You probably have one of these problems right now:

A $200 Claude Code subscription that quietly metered out and is now billing API rates.
Three contractors sharing one API key with no per-person ceiling.
An automation agent that retries on failure and occasionally retries forever.

A policy memo does not stop any of these. Uber's lesson is that the fix has to live at the call site, not in a spreadsheet. You want the limit enforced in code, before the request goes out, per identity.

What Uber's policy looks like as runtime code

This is the whole idea behind AgentGuard. It is the open-source primitive Uber is reinventing in-house, except you get it in three lines.

from agentguard import BudgetGuard

guard = BudgetGuard(monthly_usd=1500)

with guard:
    response = client.messages.create(
        model="claude-opus-4-8",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )

When the identity behind that guard crosses $1,500 for the month, the next call raises instead of charging you. No memo. No quarterly surprise. The limit is a fact of the runtime, not a guideline somebody might ignore.

You can scope it per developer, per contractor, per agent, or per customer if you resell access. That is the difference between Uber's blunt per-tool cap and what you actually want: a per-identity ceiling that knows who is spending.

Why in-process beats a proxy

Most cost tools sit in front of your calls as a proxy or router. That works until it does not. A proxy adds a hop, a single point of failure, and another thing to operate. It also cannot see intent. It sees traffic.

AgentGuard runs in your process, around your client. It counts real spend against the identity making the call and stops before the request leaves. No extra infrastructure. No second bill to control your first bill.

Uber proved the demand. A hard per-identity dollar cap on AI tooling is now something the largest engineering orgs ship by hand. You do not have to build it from scratch.

The takeaway

Two cost-cap stories landed in 24 hours this week: Uber's $1,500 cap and Copilot moving to usage-based pricing. The direction is set. AI coding spend is going from flat-rate to metered, and metered means someone has to own the meter.

If you wait for the bill to tell you, you have already lost the month. Put the cap in the code.

AgentGuard is free and open source. Three lines to a hard budget that actually fires. Try it here.

Your AI Agent's Retry Loop Is a Cost Bug Waiting to Happen

Patrick Hughes — Sun, 07 Jun 2026 14:45:09 +0000

This morning a small piece of my own automation got stuck. A repair agent tried to fix one blog draft. It failed the same check 27 times in a row. Each attempt was a full model call. The loop never asked the obvious question: if 26 tries did not work, why would the 27th?

That is a retry loop with no circuit breaker. And it is one of the most common ways AI agents quietly waste money.

Retries are not free

In normal code, a retry is cheap. You hit a flaky network call, you try again, you move on. The cost of one extra attempt is a few milliseconds.

Agent retries are different. Every attempt is a model call. Every model call costs tokens. A loop that retries 27 times is 27 paid attempts at the same task. If the task is impossible the way it is framed, you pay 27 times to learn nothing.

The error handling looks responsible. It catches the failure. It tries again. It logs the attempt. But "try again" without "and stop at some point" is not error handling. It is a slow leak.

A quick cost example

Say each attempt is a 4,000 token call. At a few dollars per million tokens, one attempt is a fraction of a cent. Twenty-seven attempts is still small. Now scale it. A loop that runs on a 40,000 token context, across ten agents, several times a day, adds up. The per-attempt cost hides the total. That is the danger. Small numbers times a loop with no ceiling become a large number you never chose to spend.

Why agents thrash

Traditional retries assume the failure is transient. The server was busy. The connection dropped. Wait a beat and the same input works.

Agent failures are often not transient. The model produced output that breaks a hard rule. A paragraph too long. A forbidden word. A schema mismatch. Feed the same prompt back and you get a slightly different wrong answer. The constraint that blocked attempt one blocks attempt 27.

So the loop runs until something else stops it. A timeout. A token budget. A human noticing the bill. None of those are good stopping conditions.

The fix is a counter and a ceiling

You do not need anything fancy. You need two things most retry loops skip.

First, count the attempts. Not just "did it fail," but "how many times in a row." A simple integer.

Second, set a ceiling. After N tries, stop. Escalate. Write the failure somewhere a human will see it. Hand the work to a different path. Anything except trying again forever.

In my case the right move was obvious in hindsight. After three failed repairs, the draft should have been flagged for a human and the loop should have stopped. Instead it ground through 27 attempts before a separate check caught it.

Make the ceiling visible

The trap is that an uncapped loop looks fine from the outside. The agent is busy. Logs are filling. Work appears to be happening. Nobody sees the problem until the invoice arrives or a quota runs dry.

So make the cap explicit and loud. Log when you hit it. Count retries as their own metric, separate from successes and failures. A spike in retries is an early warning that something is stuck, long before it shows up as cost.

If you run more than one agent, track this across all of them. One stuck loop is annoying. Ten stuck loops at once is a real bill.

This is the same idea as a budget

A retry ceiling and a token budget are the same instinct. Both say an autonomous process should have a hard stop it cannot cross. The agent does not get to decide it needs one more try, one more call, one more dollar. The ceiling decides.

That is why I built loop and budget limits into AgentGuard in the first place. Agents are good at doing things over and over. They are bad at knowing when to quit. The stop has to come from outside the agent.

If your agents retry, audit those loops today. Find the ones with no counter. Add a ceiling. Make the ceiling visible. The 27-attempt loop I hit cost me cents because the task was tiny. The next one might not be.

Want hard caps on retries, tokens, and spend for your agents? Start here: https://bmdpat.com/tools/agentguard

When JPMorgan Turns On AI Bank-Wide, Who Controls the Bill?

Patrick Hughes — Sat, 06 Jun 2026 14:45:09 +0000

JPMorgan just turned on AI for its entire global investment bank. Every employee, all 250,000 of them, now has access to AI tools. Microsoft says pitch deck generation dropped from four hours to about thirty seconds. Jamie Dimon put it plainly: more AI people and fewer bankers.

This is the moment a lot of us have been waiting on. Not the layoffs part. The cost part. When a bank that size flips AI on for everyone, the bill stops being a rounding error. It becomes a line item the board asks about.

The bill nobody budgeted for

Here is the quiet story under the headline. Bloomberg reported that bankers' Claude usage is racking up fees. Not a pilot. Not a sandbox. Real people doing real work, sending real tokens, all day, every day.

That is the part most enterprise AI plans skip. You approve the rollout. You celebrate the four-hours-to-thirty-seconds win. Then the invoice shows up and nobody can explain why it is what it is. Who used what. Which team. Which workflow. Which prompt got run 40,000 times because someone wired it into a loop.

JPMorgan is the first big bank to go bank-wide, but the pattern is everywhere now. Goldman rolled an AI assistant to more than 10,000 workers. Morgan Stanley's AskResearchGPT sits on top of 70,000 research reports. Standard Chartered is cutting 8,000 jobs by 2030. A Citigroup study found 54% of financial jobs have high potential for automation.

Every one of those numbers is a usage number waiting to happen. More seats means more calls. More calls means more spend.

Why enterprise AI spend runs away

Token spend is not like a software license. A license is a fixed cost. You pay for the seat whether the person logs in or not.

AI is the opposite. You pay for what gets used, and usage is invisible until you measure it. A single power user can cost more than a whole team. A badly written automation can spend a month of budget in a weekend. Nobody is being reckless. The meter just runs faster than anyone expects.

Three things make it worse in a big company:

Fan-out. One workflow gets adopted by a department. Now it runs thousands of times a day instead of ten.
No per-team visibility. The bill comes in as one number. You cannot tell sales from research from ops.
No ceiling. Most teams have no hard cap. Spend climbs until someone notices the invoice, which is always after the fact.

That last one is the real problem. By the time finance flags it, the money is gone.

What cost control actually looks like

You do not fix this with a spreadsheet review once a quarter. You fix it at the point where the tokens get spent, in the code, before the call goes out.

That means a few concrete things:

A budget per agent, per team, per workflow. Not a suggestion. A real limit the code respects.
A stop. When a workflow hits its cap, it stops instead of quietly spending the next department's money.
Visibility you can read without a data team. Who spent what, broken down the way your org is actually shaped.

This is plumbing, not strategy. But it is the plumbing that decides whether your AI rollout looks smart in six months or shows up as a surprise on the wrong side of a budget meeting.

The lesson for the rest of us

You are not JPMorgan. You are not turning on AI for 250,000 people. But the math is the same at every scale.

If you are a small business wiring an AI agent into customer support, or a solo builder running a research pipeline overnight, the failure mode is identical. Usage you cannot see. A bill you did not predict. A single bad loop that burns a week of budget while you sleep.

The banks are just hitting it first, and bigger. They have the seats and the volume to make the problem loud. Watch what they do next, because the cost-control tooling they buy is the tooling everyone ends up needing.

The good news: you can put the controls in before your bill gets loud. A budget cap, a hard stop, and per-workflow visibility cost you an afternoon to wire up. A runaway invoice costs you a lot more than that, and it costs you the trust of whoever signed off on the rollout.

Dimon wants more AI people. Fine. Be the AI person who also knows where the money goes. That is the one who keeps getting to build.

If you are running AI agents and you want a budget cap and a hard stop before the bill surprises you, AgentGuard does exactly that. It is a runtime budget, token, and rate limiter for AI agents. Set a ceiling, and the agent stops instead of spending past it.

What Anthropic's MITRE ATT&CK Report Means for Teams Running AI Agents

Patrick Hughes — Sat, 06 Jun 2026 14:45:06 +0000

Anthropic just published a year of threat intel on AI-enabled attacks. It covers March 2025 to March 2026. They banned 832 accounts for malicious cyber activity and mapped what those accounts did to MITRE ATT&CK, the same framework enterprise security teams use to describe attacker behavior. They co-released it with Verizon's 2026 DBIR.

If you run AI agents in production, this is primary-source data. Not a vendor scare deck. Here is what actually matters for the people building and shipping agents.

The attack work moved past writing code

The headline number: 560 of those 832 accounts, about 67 percent, used Claude for malware writing. That is the expected one. Models are good at code, including bad code.

The number that should change how you think is smaller. 54 accounts, about 6.5 percent, used Claude for lateral movement. That is a kill-chain stage that used to be hand-driven. Lateral movement is what an attacker does after they are already inside, hunting for the next box to compromise. Account discovery went up 8.9 percent. AI-assisted phishing actually dropped 8.6 percent.

Read that shift plainly. The work moved away from initial access and toward post-compromise. Attackers are not just generating payloads. They are using AI to make real-time decisions deeper inside systems they already breached.

For you, the takeaway is direct. Your input filter is not the main event. Your agent's blast radius is. The question is not only "can a bad prompt get in." It is "if this agent is compromised or coerced, how far can it reach and how long does it run before anyone notices."

The risk mix is getting worse, not just bigger

Anthropic split the year into two six-month windows. Medium-or-higher-risk actors went from 33 percent of cases to 56 percent. The pool is not just growing. It is concentrating toward serious operators.

One more finding worth sitting with. Technique count and platform type stopped predicting how dangerous an actor is. The high-risk ones do not spray a hundred techniques. They put AI on the operationally hard stuff and skip the easy parts. So "we saw a lot of weird activity" is no longer a clean severity signal. Volume tells you less than it used to.

MITRE ATT&CK does not yet cover agentic orchestration

Here is the part that matters most if you ship agents. Anthropic says the attackers are chaining ATT&CK stages with minimal human input. Autonomous orchestration. And they say it directly: MITRE ATT&CK does not yet capture agentic orchestration. The framework that underwrites enterprise security operations is being outgrown by the threat. Anthropic is working with MITRE to evolve it.

If you build agents, sit with that. The standard model of how attacks work was written for human-paced kill chains. Your own agents already run faster than that model assumes. So do the malicious ones. You are operating in a place the reference frameworks have not fully described yet.

What enterprises running agents should take from this

Three concrete moves.

First, treat agentic orchestration as its own threat category, not a footnote on your existing controls. An agent that can call tools, read data, and act in a loop is not a chatbot with extra steps. The thing that makes it useful, autonomy across many steps, is the thing that makes a compromise expensive.

Second, get value from inference-time safeguards. Anthropic detects malware development and exfiltration patterns at the model layer. If you build on a frontier API, you inherit that floor for free. That is a real reason to keep production agent work on a monitored frontier model instead of an uncensored local model where no one is watching the traffic.

Third, and this is the one most builders skip: cap the blast radius at runtime. The expensive failure with an agent is rarely a single bad call. It is an agent that runs unattended for hours, burning tokens, hitting APIs, doing the wrong thing at machine speed while you sleep. Lateral movement, in attacker terms. Runaway spend, in yours. Same shape.

The cheapest control you can install today

You cannot stop a nation-state actor with a config file. That is not the goal. The goal is limiting how much a compromised or misbehaving agent can cost you before you notice.

That means hard limits at runtime. A budget cap so a loop cannot burn your whole month in a night. A token cap per task. A rate limit so one agent cannot hammer an API into a five-figure bill. These are boring controls. They are also the ones that actually save you, because they work whether the cause is an attacker, a bad prompt, or your own buggy code.

That is exactly what AgentGuard does: runtime budget, token, and rate limits for AI agents, in a few lines. If you run agents in production, put a ceiling on them before you need one. Start here: https://bmdpat.com/tools/agentguard

What GitHub Copilot Users Wish They Had a Week Ago

Patrick Hughes — Fri, 05 Jun 2026 14:45:15 +0000

GitHub Copilot moved to usage-based pricing, and the bills landed fast. Ars Technica covered the reaction: developers reporting that they burned through a full month of credits in a single day under the new model (Ars Technica).

If you have shipped anything with an AI coding tool lately, you know the feeling. The flat monthly fee was predictable. You knew the number. Now the number moves with how hard you work, and nobody told you where the ceiling is.

What actually changed

Copilot went from flat-rate to usage-based. There is no firm public quota tier you can point at and say "I will never pass this." Power users hit a wall they could not see coming. Thirty days of nominal credits, gone in one focused day of work.

The same week, Microsoft pushed small cheap models as the stay-under-budget option inside the same product. That is the tell. When a vendor ships a "use this to spend less" feature alongside a pricing change, the cost problem is real and they know it.

This is the failure mode, not a Copilot problem

Copilot is the headline. The pattern is bigger. Every AI dev tool that charges per token or per request has this shape now. You write code, the meter runs, and you find out the total at the end of the cycle.

Uber reportedly caps AI coding spend at $1,500 per developer. That cap exists because without it, the number runs. A big company can absorb a surprise. A solo builder or a small shop cannot.

The fix is not "use the tool less" or "switch to the cheap model and hope." The fix is a budget envelope at the call site. A hard limit that lives in your code, watches spend in real time, and stops the run before it overshoots.

What a runtime budget cap looks like

This is the wedge for AgentGuard, the open-source budget and rate limiter I maintain (pip install agentguard). It wraps your AI calls and enforces a ceiling you set. When you hit it, the run stops. No surprise invoice.

Here is the shape of it:

from agentguard import BudgetGuard

# Hard ceiling for this session. Pick a number you can defend.
guard = BudgetGuard(
    max_usd=5.00,
    max_tokens=500_000,
    on_exceeded="stop",  # kill the run, do not keep spending
)

@guard.track
def call_model(prompt: str) -> str:
    # your normal AI call, any provider
    return client.complete(prompt)

# Run your agent loop as usual.
# When spend crosses $5.00, the guard raises and stops.
for task in work_queue:
    try:
        result = call_model(task.prompt)
    except guard.BudgetExceeded:
        print(f"Hit the cap. Spent {guard.spent_usd:.2f}. Stopping.")
        break

Thirty lines, give or take. The point is not the syntax. The point is that the limit lives in your code, not in a billing dashboard you check after the damage is done.

Why the call site matters

You can set alerts in a vendor console. Alerts tell you after the fact. By the time the email arrives, the credits are spent. A runtime cap is different. It runs in the same loop as your work, counts every call, and refuses to make the call that would put you over.

It is also cross-provider. Copilot today, some other tool next quarter. If your budget logic lives in your code instead of one vendor's settings page, you do not start over every time you switch.

The takeaway

Usage-based pricing is not going away. It is the default direction for AI dev tools, and Copilot just made the cost real for a lot of people in one news cycle. Predictable spend is now something you build, not something the vendor hands you.

Put the limit at the call site. Set a number. Let the code enforce it. That is the difference between a tool you control and a meter you watch.

If you want the runtime budget cap without writing it yourself, that is exactly what AgentGuard does: bmdpat.com/tools/agentguard.

When Not to Use an AI Agent

Patrick Hughes — Fri, 05 Jun 2026 14:45:11 +0000

When everyone is shipping AI agents, the useful question is the opposite one: when should you not?

I run a one-person operation with a fleet of agents doing real work every day. Writing drafts, scoring leads, checking that scheduled jobs actually did their job. They earn their keep. But I have also wired up agents for tasks that a plain script would have done better, cheaper, and with less drama. Here is where I draw the line now.

Use a script, not an agent, when the steps never change

An agent is worth it when the input is messy and the right next step depends on judgment. If the steps are fixed, you do not need a model in the loop. You need a function.

I once had an agent renaming files and moving them into dated folders. It worked. It also cost tokens, took ten seconds instead of ten milliseconds, and failed in a new creative way about once a week. A six-line script replaced it and has not been touched since. The rule: if you can write the if-statements, write the if-statements.

Skip the agent when a wrong answer is expensive and silent

Agents are confident even when they are wrong. That is fine when a human reviews the output before it matters. It is dangerous when the output flows straight into something irreversible.

Money movement, production deletes, sending email to real customers. I let agents draft those actions. I do not let them commit those actions without a gate. A wrong trade or a wrong DELETE does not announce itself. By the time you notice, the damage is done. Put a human or a hard rule between the agent and the irreversible step.

Do not reach for an agent to dodge a decision you have not made

This is the one that bit me most. When I did not actually know what I wanted, I would hand the fuzzy problem to an agent and hope it would figure out the goal. It cannot. It will pick a goal, usually a plausible wrong one, and pursue it with energy.

An agent amplifies a clear intent. It does not supply one. If you cannot write the success condition in one sentence, the agent is not your problem. The missing decision is.

Watch the cost of the loop, not the cost of one call

A single model call is cheap. An agent that retries, reflects, and calls tools in a loop is not. The cost is the loop, and loops can run away.

I learned this the boring way. A draft in my own publishing pipeline failed a length check, so the repair agent fixed it, resubmitted, failed again, and did that twenty-five times before anything flagged it. No single run looked expensive. The total was real, and the task was dead on arrival. Unbounded retries are how a helpful agent quietly burns your budget.

That last failure mode is exactly why I built AgentGuard. It caps spend, token use, and call counts per run, so a stuck loop stops itself instead of running until you happen to look. An agent should fail loud and cheap, not silent and expensive.

A short test before you build one

Ask three questions. Does the task need judgment, or just rules? If a step goes wrong, will someone notice before it hurts? Can I state the goal in one sentence?

If the answers are rules, no, and no, you do not want an agent. You want a script with a human nearby. Agents are good. They are not the answer to every box on the board, and pretending otherwise is the fastest way to a surprising bill and a quiet mistake.

Build the agent when the task is genuinely fuzzy and the stakes are reviewable. Bound every loop. And if you want hard budget, token, and rate limits around your agents so a runaway loop cannot drain your account, that is what I built AgentGuard for: https://bmdpat.com/tools/agentguard

llama.cpp ngl: when -ngl 99 still runs on your CPU

Patrick Hughes — Thu, 04 Jun 2026 23:25:21 +0000

llama.cpp ngl: when -ngl 99 still runs on your CPU

You passed -ngl 99. You expected the GPU to light up. Instead llama.cpp generates at 9 tokens per second and your fans stay quiet. The flag did nothing.

I have hit this on every machine in my rig: a 3070, a 5070 Ti, and a 5090, all serving Llama 3.1 8B through llama.cpp. The ngl flag is almost never the problem. The build you are running, the wheel pip installed, or the VRAM you do not actually have is the problem. Here is how to find out which, in about 30 seconds, then the five real causes ranked by how often they bite.

If you do not yet know what the flag controls, read the companion post first: llama.cpp n-gpu-layers explained. This post assumes you know what -ngl is supposed to do and want to know why it is not doing it.

The 30-second diagnostic: read the load log

Stop guessing. llama.cpp tells you exactly what it offloaded. When the model loads, look for this line:

load_tensors: offloaded 33/33 layers to GPU

That is the entire diagnosis. Two numbers:

33/33 means every layer is on the GPU. If you are still slow, your problem is downstream (context, sampling, a CPU-bound KV cache).
0/33 means nothing offloaded. Your -ngl flag was accepted and ignored. This is a build problem.
22/33 means partial offload. The model did not fit. This is a VRAM problem.

Everything below is just "which of those three did you get, and what to do about it."

Cause 1: pip installed a CPU-only wheel (the most common by far)

This one cost me the most time. pip install llama-cpp-python ships a CPU-only wheel by default. No CUDA. The -ngl argument is still accepted, parsed, and then silently does nothing because there is no GPU backend compiled in. You get offloaded 0/33.

The fix is to reinstall with the CUDA backend turned on:

CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --force-reinstall --no-cache-dir

On Windows PowerShell:

$env:CMAKE_ARGS="-DGGML_CUDA=on"
pip install llama-cpp-python --force-reinstall --no-cache-dir

If you do not have the CUDA toolkit set up for a source build, use the prebuilt wheel index instead:

pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124

After reinstall, the load log on my 5070 Ti went from offloaded 0/33 at 9 tokens per second to offloaded 33/33 at 95 tokens per second on Llama 3.1 8B Q4_K_M. Same flag. Same model. The only thing that changed was the backend.

Cause 2: you built llama.cpp from source without a GPU flag

Same failure, raw binary version. If you cloned the repo and ran cmake -B build with no backend flag, you built CPU-only. -ngl is ignored.

Rebuild with the backend for your hardware:

# NVIDIA
cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release

# Apple Silicon
cmake -B build -DGGML_METAL=ON && cmake --build build --config Release

# AMD (ROCm)
cmake -B build -DGGML_HIPBLAS=ON && cmake --build build --config Release

Confirm it took: ./llama-cli --version should print your backend, and the startup banner lists the GPU device. If it says CPU only, the build flag did not apply.

Cause 3: the model does not fit, so llama.cpp offloads what it can

You set -ngl 99 but the load log says offloaded 22/33. The flag worked. The VRAM did not. llama.cpp loaded as many layers as fit and put the rest on the CPU. Those CPU layers drag the whole generation down because every token still waits on them.

Two things eat VRAM you forgot about: the KV cache and the context buffer. A 32K context window on an 8B model can cost more than a gigabyte before a single layer loads.

Quick wins, in order:

Shrink the context: --ctx-size 4096 instead of 32768.
Drop a quant level: Q4_K_M to Q3_K_M frees roughly 20 percent.
Check what actually fits before you download anything with the quant comparison tool. Pick GPU, model, and quant, and it does the VRAM math for you.

On my 3070 (8 GB) a Llama 3.1 8B Q4_K_M fits fully only if I keep the context under about 8K. Push it higher and I am back to partial offload without changing the flag at all.

Cause 4: the wrong binary is first on your PATH

If you have ever installed Ollama, LM Studio, and a hand-built llama.cpp on the same box, you have three llama binaries. The one your shell finds first might be the CPU build.

which llama-cli        # macOS / Linux
where.exe llama-cli    # Windows

If that path is not the CUDA build you just compiled, call the right one with a full path or fix your PATH order. I lost an afternoon to this once because a stale binary in ~/.local/bin shadowed the new one.

Cause 5: a CUDA or driver mismatch falls back to CPU

A wheel built for CUDA 12.4 against a driver that only supports 12.0 can fail to initialize the GPU and quietly fall back to CPU. The tell is a warning during load, often swallowed in noisy logs.

Run nvidia-smi and check the "CUDA Version" in the top right. Match your wheel or build to that or lower. When in doubt, the prebuilt cu121 wheel is the most forgiving across older drivers.

The order I actually debug this

Read the load log. 0/33, 22/33, or 33/33 decides everything.
0/33: rebuild or reinstall with the GPU backend. This is the answer 80 percent of the time.
22/33: shrink context, drop a quant level, or use a smaller model.
33/33 and still slow: the flag is innocent. Look at context size, batch size, and whether your KV cache is on CPU.

The -ngl flag is one of the most blamed and least guilty settings in local inference. It is a passthrough. When it looks broken, something upstream of it is.

If you run local models inside an agent loop, the next thing that will surprise you is cost, not speed. A retry storm or a runaway loop can burn tokens and watts long after you stopped watching. That is why I built AgentGuard: a budget and rate limiter you wrap around your agent so a bad night caps out instead of running until morning. Free to install, and it works the same whether the model is local or an API.

What is your load log telling you: 0/33, partial, or full?

I made my blog API reject its own writer

Patrick Hughes — Thu, 04 Jun 2026 14:45:08 +0000

I made my blog API reject its own writer

I run a small content pipeline. A Codex reviewer reads drafts at 08:30 CT. A repair loop fixes the rejections at 08:55. A publisher ships approved drafts at 09:30. The whole thing lives in scheduled tasks on a Windows box in my office.

It worked. Right up until I added a same-day heal path that skipped the reviewer.

The heal path

When brain think finds no post went live today and there is nothing approved in the review folder, it fires a heal agent. That agent picks a topic, writes the post, and ships it. Same hour.

The whole point of the heal path is speed. The whole problem with the heal path is also speed.

If the reviewer is the only thing stopping bad posts, and the heal path skips the reviewer, then on any day the regular pipeline stalls I have a fast lane straight to production with no QA in front of it. Not great.

The first instinct (wrong)

My first instinct was to add stronger instructions to the heal prompt. "Run a QA check before publishing. No exceptions." Capital letters. Three reminders. The works.

This is the prompt-engineer version of yelling at a developer to remember to run the tests. It works until it does not.

A few days later I caught a post that shipped without the QA step. The model decided the post was clean and skipped its own gate. The instructions said do not skip. The model skipped anyway.

The fix that actually held

I moved the gate out of the prompt and into the API.

POST /api/blog now requires two extra fields on any published: true request:

{
  "title": "...",
  "content": "...",
  "published": true,
  "qa_status": "approved",
  "qa_reviewer": "codex"
}

If either field is missing, the server returns HTTP 422 with code qa_provenance_required. The post does not get written.

The fields do not prove the QA actually happened. A lying client can send them anyway. But that is not the threat. The threat is a careless writer who forgets the gate exists. Forgetting now produces a loud error instead of a silent shipped post.

Then I updated the heal prompt to require the QA subagent run BEFORE the POST, with the result written to those two fields. The prompt and the API now agree on what a publishable post looks like.

Why this works when prompt instructions did not

A prompt is a request. An API contract is a wall.

When the gate lives in the prompt, the model is the judge of whether the gate was met. Models are generous judges of their own work. They will skip the gate and tell you the gate was met. You will not know until you read the post.

When the gate lives in the API, the model is no longer the judge. The server is. The server does not care how confident the model is. The server checks the payload. If the fields are not there, the post does not exist.

This is the same reason --no-verify exists on git hooks and why disabling it in CI matters. The local check is a courtesy. The remote check is the law.

The general pattern

If you are running agents that produce real artifacts (blog posts, PRs, emails, code commits), look at where your quality gates live.

If your gates live in the prompt, you are trusting the agent to be its own judge. This works most of the time. The times it does not work are the times you find out by reading the production output.

If your gates live in the system the agent talks to (the API, the CI, the deploy pipeline), the gate runs regardless of what the agent thinks. The agent can be careless, distracted, or wrong. The gate still fires.

Move the rules out of the writer and into the reader. Anything else is a trust system masquerading as a control system.

Cost of the change

Maybe 40 lines of code in the API handler. One new error code. Two new fields on the POST payload. Five minutes to update the heal prompt to include those fields.

The pipeline still ships posts at roughly the same speed. The only difference is that the day the model forgets the gate, the post does not silently go live with no review. The POST returns 422. The heal agent has to actually run the QA step to ship.

That is the entire change. It is small. It holds.

If you build agents that touch production, push the quality checks down the stack into systems the agent cannot route around. Trust no writer, including the one you wrote.

If you want hard budget limits and loop guards for your own agents, start here: https://bmdpat.com/tools/agentguard