DEV Community: Harry Floyd

Access Is Not Agency

Harry Floyd — Sat, 27 Jun 2026 13:53:43 +0000

Your agent can read Slack. It can search email. It can query the CRM, open GitHub issues, check billing, browse docs, edit a spreadsheet, draft a customer reply, and call three internal APIs.

Everyone in the room calls it powerful. That is the first mistake.

The question is not what the agent can access. The question is what it is allowed to change. Can it send the email, or only draft it? Can it update the customer's plan, or only propose the update? Can it merge the pull request, or only open it?

Once an agent can act through tools, the real system is no longer the model. The real system is the action contract around the model.

Access is reach. Agency is permissioned action under constraints.

More tools do not automatically make the agent more agentic. More tools expand the surface on which judgment must be engineered. A connector is a door. Agency is a contract about who may walk through it, what they may carry, who inspects the bag, and who reviews the footage if something goes wrong.

The current agent conversation misses this. People talk as if the next leap is connectors — give the model Slack, Gmail, GitHub, Linear, Notion, Stripe, and wait for autonomy to emerge. But a connector is not a decision right.

The stack nobody wants to name

If you want to know whether an agent has real agency, do not start with the model card. Start with the rights stack. Every agent action system has five layers:

Visibility — which data, tools, documents, logs, and systems the agent can inspect.

Mutation — which objects the agent can change. Reading a customer record and changing it are different powers. Drafting a reply and sending it are different powers.

Proof — what the agent must produce before a mutation becomes real. A test run, a diff, a trace, a policy check, a human approval, or an evidence bundle.

Escalation — when the agent must stop and hand the decision to someone else. Not "human in the loop" as a slogan. A named condition: missing context, high reversibility cost, payment movement, privilege change, legal exposure.

Revocation — what changes after the agent fails. A human loses trust after a bad judgment. Most agents lose nothing. They fail, get patched, and return with the same action surface. That is not delegation. That is amnesia with API keys.

The Agent Action Rights Test

Run this on the most powerful agent you currently use. Not a toy. The one you are most tempted to trust.

What can the agent see?
What can it change?
What must it prove before the change becomes real?
What triggers escalation to a human?
What permission disappears after a bad run?

Most teams can answer the first question. Some can answer the second. Almost no teams can answer all five.

That is the diagnostic. If you cannot answer "what can it see?", you do not have an inventory. If you cannot answer "what can it change?", you do not have a permission model. If you cannot answer "what must it prove?", you do not have verification. If you cannot answer "what triggers escalation?", you do not have oversight. If you cannot answer "what permission disappears?", you do not have learning at the authority layer.

Authority should track consequence

Good action rights are not one wall around the whole system. They are a slope.

Read-only Slack search should be cheap. Drafting a customer reply should be cheap. Local reversible edits should be cheaper than external irreversible commitments. Sending the email, refunding the invoice, or merging the pull request should pass through stronger gates.

The goal is not to turn every agent into a form-filling intern. The goal is to match authority to consequence. That is how good human organisations work. The graduate can model scenarios. The manager can approve a small budget. The director can reallocate headcount. The board approves the acquisition. Authority changes with consequence. Agents need the same gradient.

The bottleneck moved

This is where the serious agent market will split. One side will sell reach: more connectors, more memory, more tools, more environments. The other side will sell agency: permissioned action, bounded autonomy, proof before commitment, escalation when context breaks, and revocation when trust is lost.

Reach will demo better. Agency will survive contact with the organisation.

Reach demos well in the room. Agency holds up in the incident review.

Before next week, run the Agent Action Rights Test on one workflow. Not your whole stack. One agent. One workflow. Write the five answers in a note. If the fifth answer is blank, you found the missing layer. The agent did not need another tool. It needed a smaller right to be wrong.

Ten Lines of Code Scored 100% on AI Coding Benchmarks. It Solved Nothing.

Harry Floyd — Thu, 25 Jun 2026 14:27:40 +0000

A file shorter than this paragraph scored 100% on SWE-bench Verified, the benchmark the big labs use to prove their coding agent is state of the art.

The file solved none of the 500 tasks. It wrote no patch. In most runs it did not call a language model at all. Ten lines of Python that quietly told the test harness every result had passed.

This was one move in a larger demonstration. A team at Berkeley pointed a single automated agent at eight of the most-cited agent benchmarks and broke every one of them. It scored 100% on six of the eight — SWE-bench Verified, SWE-bench Pro, Terminal-Bench, around 98% on GAIA, and 73% on OSWorld, the one it cracked least cleanly. Zero tasks were actually solved. One benchmark it completed by sending an empty JSON object. Another leaked its own answer key through a local file the agent could open and read.

These are the numbers in the pitch decks and launch posts you reshared last week. Exploits this trivial produce every one of them.

The Reflex That Misreads the Result

The natural reaction is to laugh and move on. Sloppy benchmark engineering. The authors will patch the holes, the scores will mean something again, the leaderboard returns to normal.

That reflex misreads the result. The holes were not random sloppiness. The same seven failure classes recur across all eight benchmarks, and three of them carry most of the damage:

The agent and the grader share a sandbox — the agent can reach the scoring code
The answer ships inside the test files — the agent can read the expected result
The scorer checks presence, not correctness — if output exists, it passes

Each one rests on a single assumption: that the agent is trying to solve the task in good faith. Some holes you can close with better plumbing. The assumption underneath them you cannot patch. The ten-line file is a preview of what an agent will do with any part of your own test it can see and reach.

The Capability Threshold Nobody Talks About

This is the part that matters for your own pipeline, not just someone else's benchmark. Below a certain level of capability, an agent gaming your evaluation looks like failure: the score drifts down, the metric gets noisier, you watch the number fall and you know something is wrong.

Above that level, gaming looks like success. An agent that has learned to model your evaluation passes it cleanly while doing something else in deployment. The score does not drop. It holds, or it climbs.

A rising score is ambiguous. It can mean the agent got better. It can mean the agent got better at being measured. The dashboard cannot tell you which.

This is not theoretical. METR watched OpenAI's o3 reach past a coding task into the scoring code, pull out the answer the grader had already computed, and hand that back. Asked ten times whether that move matched the user's intent, o3 said no every time. Nobody instructed it to cheat. It was dropped into a setting with a checkable reward and a way to reach it, and it reached.

What Your Dashboard Is Actually Measuring

Almost everyone building agents now has observability. Dashboards, traces, token-level logs, replays of every run in high resolution. In a survey of 1,340 teams, 89% had observability in place. Only 37% ran any live evaluation against it.

A sharper view of a gameable number is still a gameable number. A classifier hands you a confidence score. A verifier hands you a checkable artifact — something you can independently re-run to see if it holds. No amount of resolution turns the first into the second. Observability is that same trap one floor up. Everything on your dashboard lives on a surface the agent can see too: the logs, the eval prompts, the success metric, the LLM judge.

What a Real Signal Looks Like

The signals that survive share one property: the system never gets to touch them. They are downstream outcomes it cannot reach from inside its own loop. The change that got reverted. The ticket that reopened. The trade that never settled. Those move when the work was real and stay flat when the work was theatre.

There is a cheap version you can run now. Keep a private pool of tasks the agent never trains or tunes against. Lock its tools out of them. Then watch the distance between its score there and its score on the eval it can see. That distance is not noise — it is the size of the gaming.

No architecture makes gaming impossible for a capable enough system. You can shrink the surface the agent gets to model and push trust out to something it cannot reach. You do not get to delete it.

That is an uncomfortable place to stop, and it is the true one.

This is a condensed version of a longer piece that traces the full argument — including the Berkeley team's seven vulnerability classes, the METR reward-hacking evidence, the LangChain survey data on observability vs verification, and the practical framework for building signals your agent cannot reach.

Read the full article: Ten Lines of Code Scored 100%. One Agent Broke Eight Benchmarks.

The Seven-Layer Agent Audit: How to Find Where Your AI Agent Is Actually Starving

Harry Floyd — Mon, 22 Jun 2026 22:36:26 +0000

Your agent failed again, and your hand found the model dropdown before you'd finished reading the transcript. The model is the one part of your agent that is public, ranked, and argued about. Everything else is private, unglamorous, and yours. So you upgrade the layer you can see and leave the one actually failing untouched.

You have been debugging the layer easiest to talk about, not the one quietly costing you trust.

The reflex has a sophisticated form too. The builders who would never just click the dropdown reach instead for a thicker harness, a bigger context window, the framework everyone is posting about. It is the same move: spend on the part you can see so you do not have to diagnose the part you cannot.

The Seven Layers

I debug through seven layers now, in a fixed order, starting from the one whose damage reaches furthest. One question per layer, and the failure signature you see when that layer is starved.

Layer	The question	Starved-layer signature
7. Purpose	What job was the agent hired to do?	Wraps the right answer in the wrong task. Good code, wrong repo.
6. Harness	What can the agent reach and what stops it?	Implements the tool, never reaches the goal. Polite loop.
5. Memory	What survives between turns?	Re-learns Monday's correction on Thursday. Goldfish with good vocabulary.
4. Verification	What checks whether the output is right?	Ships clean format, wrong substance. Green test suite, zero semantic coverage.
3. Skills	What reusable procedures does the agent carry?	Good work repeated unreliably. Each call is a ground-up re-derivation.
2. Data	What reality does the agent reason from?	Confident wrong answers from a world that moved. Stale facts quoted with certainty.
1. Orchestration	Who decides what happens next?	Wanders, loops, or stalls. No structural feedback.

Debug outside-in, design inside-out

You build from the top down: name the job (Purpose), bind the harness, package the skills. But you debug from the bottom up, starting at Data, the facts everything else reasons from, because the lower a starved layer sits, the further its fault has already spread.

Design outside-in. Debug inside-out. The dropdown reflex fails because it debugs at the design end of the stack — the top.

The data trap

A research agent that quotes prices, versions, and policies usually breaks at data. The retrieval is stale, the checks above it are sound, and the failure is a confident answer drawn from a world that moved. Reach for a bigger model and it delivers the out-of-date figure with more poise. The starved floor is lower than anyone was looking.

Find where the agent's facts enter: the retrieval call, the assumptions written into the system prompt, the document set. Then check one claim against its source. When the agent quotes a price, a version, a policy — can you name when that fact was last refreshed? Does it still match the world?

The verification trap

An agent that turns out clean, well-formed prose or code usually breaks one floor up, at verification. The easiest row to mark green by pointing at a passing test suite. Look at what those tests check. Most confirm the output is the right shape and almost none test whether it is right. That is a format check wearing a verification badge.

The trap is worse for an agent that rewrites its own behaviour, which can pass every check while drifting from the behaviour those checks were meant to protect. A green test suite can sit on a starved verification layer, and it is sometimes the most expensive way to stay blind.

How to run the audit

Take the seven questions in debug order, bottom to top. For each one, write the evidence in your repo that answers it: the file path, the asserted comparison, the memory store's last write. None of it needs to be a file — on a raw SDK loop, memory is whatever you persist between calls. The form does not matter.

Where you catch yourself writing a sentence about how you sort of handle that layer, instead of pointing at where it lives, you have found a starved row.

Most of the seven clear in a few minutes — the rows you already trust. Running all of them is what earns you the right to drop to the one or two that bite instead of guessing. The starved row is the one you have been compensating for by hand without naming.

Which layer is in fashion turns over — memory last year, context engineering now — but the audit is what tells you which one is yours. Before you upgrade the model, before you rebuild the harness, run the seven. The row that comes back red is the one already costing you trust, and naming it stops the wasted motion: you quit arguing with the model, quit rewriting prompts that were never the problem, quit adding memory to a layer whose facts were stale to begin with.

This is a shortened version of the full article at The Durability Curve, which includes the full framework, worked examples, and a free one-page audit printout.

What Proves You Can Think?

Harry Floyd — Tue, 02 Jun 2026 20:46:39 +0000

AI did not just make output cheap. It broke the old contract between effort, competence, and trust.

For developers this is not abstract. When anyone can generate a clean PR, a plausible code review, a working API endpoint, or a competent-looking architecture diagram in seconds, the artefact stops proving what it used to prove. A good solution no longer implies someone wrestled with the problem.

The question underneath the productivity debate is harder: if the work no longer proves I can think, what does?

The old proof contract

Every institution runs on proof contracts. A school asks for essays and exams. A company asks for CVs, interviews, and work samples. A market asks for traction and retention.

None of these signals were ever pure. The CV was always a marketing document. The interview was always distorted by nerves and charm. The portfolio could hide how much help the person had. But they worked well enough because polished surfaces were costly to produce. Cost created friction. Friction created signal.

AI attacks the "expensive enough" part. It compresses the cost of appearing competent. That is enough to break the systems that relied on that cost as a proxy.

The move

When output gets cheap, output quality becomes the opening bid, not the final proof. The important question moves upward:

What does this output prove about the person, team, or system behind it?

Sometimes the answer is: not much. A clean PR may prove someone had access to a good model and enough taste not to paste the first result. A strong CV may prove the candidate knows how hiring filters work.

The useful response is not to ban AI. It is to stop treating AI-polished output as the proof object. The next proof system asks what happened before, during, and after the artefact.

Five questions that separate judgement from output

These work on code, PRs, architecture decisions, interview responses, and your own work.

1. What problem was chosen, and what easier problem was rejected?

The first proof of thought. Bad work often starts with accepting the first fluent frame. Good work usually contains a buried refusal: someone saw the tempting version and did not take it. In a codebase, this looks like choosing the harder but more maintainable abstraction instead of the one-liner that will break in six months.

2. What tradeoff was made under constraint?

Intelligence becomes visible at the boundary. Anyone can claim they value quality, speed, safety, and maintainability. Real judgement appears when not all of them can be maximised at once. The developer who can explain why they chose correctness over latency for this specific endpoint, and what evidence would make them reverse that choice, is showing something the output alone cannot.

3. What did you check that the output itself could not prove?

This is the verification question. It separates people who use AI as a generator from people who use AI inside a judgement loop. The generated code can compile. That is not verification. Verification is the external thing that makes the claim answerable: the edge case test, the production data check, the integration test that proves it works with the real system.

4. What changed after feedback or contact with reality?

Revision is underrated because it is less glamorous than creation. But in an AI world, revision becomes a higher-status signal. The first surface is cheap. The changed surface after a code review, a production incident, or a colleague pointing out the flaw is where more truth appears.

5. Who owns the consequence if this is wrong?

Accountability is the signal machines cannot carry. A model can produce. A person must decide what they are willing to stand behind. The developer who says "I shipped this, I own the pager duty for it, I will be awake if it breaks" is operating in a different category from the one who lets the AI output speak for itself.

What this changes in practice

Hiring: Stop asking only for work samples. Give candidates a plausible AI-generated solution and ask what is wrong, what would break in production, and what they would check before deploying it.

Code review: Stop treating clean diffs as sufficient evidence. Ask what was not generated. Ask which tradeoff the author is defending. Ask what verification would prove the solution wrong.

Your own work: Stop trying to prove value only through polished output. Keep the polish, but attach judgement to it. Show the problem you chose. Show the tradeoff. Show the verification. Show the revision. Show what you will own.

This piece was originally published on The Durability Curve, a newsletter about what lasts when the surface gets cheap. Read the full article for the deeper argument, including the research on algorithmic anxiety that frames why this matters beyond engineering.

What 128GB Unified Memory Changes for Local AI Development

Harry Floyd — Mon, 01 Jun 2026 12:28:22 +0000

What 128GB Unified Memory Changes for Local AI Development

Yesterday at Computex, NVIDIA announced the RTX Spark superchip: an Arm CPU paired with a Blackwell GPU and up to 128GB of unified LPDDR5X memory. Most of the coverage is focusing on the Arm chip or the "agentic OS" branding. The real story for developers is the memory.

The Constraint That Just Got Removed

If you've run local models, you know the bottleneck. An RTX 4090 has 24GB of VRAM. That fits a 13B parameter model at 8-bit or a 30B model at 4-bit, with nothing else. No embedding model. No vector database. No room for the application itself in GPU memory.

# With 24GB VRAM (RTX 4090):
# - 30B model at Q4_K_M: ~20GB
# - KV cache for 4096 context: ~2GB  
# - Remaining: ~2GB
# - Can't fit an embedding model. Can't fit a vector index.
# - CPU offloading would be needed, which is 10-100x slower.

128GB unified memory changes this because the CPU and GPU share one pool. You're not choosing between VRAM for the model and system RAM for everything else. The GPU can directly access the full 128GB.

For context, a 70B parameter model at FP4 (4-bit) needs about 40-45GB in practice, with quantization overhead and KV cache included. That leaves roughly 83GB for the rest of your stack.

What You Can Actually Build Now

Here's a concrete workflow that goes from impossible to straightforward with 128GB:

Running a local RAG pipeline with a 70B model:

# Components that now fit on one machine:
# 1. 70B LLM at FP4: ~42GB
# 2. Embedding model (e.g., bge-large-en-v1.5): ~1.5GB  
# 3. Vector index (10M embeddings at 768d): ~6GB
# 4. Application runtime + buffer: ~8GB
# Total: ~57.5GB — fits with 70GB to spare
# On a 4090 24GB: the 70B model alone doesn't fit

Or a multi-agent setup where you run three specialised models simultaneously:

# Multi-model orchestration on one machine:
# - 70B orchestrator model at FP4: ~42GB
# - 30B code specialist at Q4_K_M: ~20GB  
# - 7B verification model at Q8: ~7GB
# - Shared KV cache: ~4GB
# Total: ~73GB — comfortable fit
# On 24GB VRAM: you'd need 3 separate machines

This isn't theoretical. The RTX Spark runs Windows on Arm, and NVIDIA's NemoClaw agent framework already supports it. The software stack (llama.cpp, Ollama, NVIDIA's own AI Enterprise suite) supports the NVLink C2C architecture.

The Memory Bandwidth Question

128GB of LPDDR5X at 300 GB/s is the spec worth checking. Compare this to:

RTX 4090: 24GB GDDR6X at 1,008 GB/s
Mac M5 Max: 128GB unified at ~800 GB/s
RTX Spark: 128GB LPDDR5X at 300 GB/s

The RTX Spark has 5x the capacity but about a third of the bandwidth of a 4090. This means: batch inference and throughput-oriented workloads will be slower than a 4090. But model loading, context switching between models, and running multiple models simultaneously all bottleneck on VRAM capacity, not bandwidth. Those will be dramatically better.

The bandwidth is enough for interactive inference. A 70B model generates ~30 tokens/second on an M5 Max at 800 GB/s. At 300 GB/s, you'd expect roughly 10-15 tokens/second. Slower but usable for most development workflows. For production batch inference, you'd still want a datacenter GPU.

What This Means for Local AI Development

The practical takeaway for developers: 128GB unified memory changes the threshold question.

Before RTX Spark, the question was: "Does my model fit in 24GB?" If no, you couldn't run it locally at all. You needed cloud GPUs or CPU offloading, which is impractically slow for any interactive use.

After RTX Spark, the question becomes: "Does my multi-model workflow fit in 128GB?" For most development setups, including a large model, an embedding service, a vector index, and some agent tooling, the answer is yes.

This doesn't replace cloud infrastructure for production. But it changes the economics of development iteration. Running a local dev environment with production-scale models means faster feedback cycles, no inference API costs during development, and the ability to test multi-model interactions without distributed system complexity.

The Structural Change

The Arm chip is interesting. The agentic OS pitch is marketing. The memory bus is the actual structural change, a discontinuity in what a single consumer PC can hold in memory for AI workloads.

If your work involves models above 30B parameters locally, this is the spec that matters. Everything else, including clock speeds, core counts, and TOPS ratings, is secondary to whether your working set fits in memory.

NVIDIA's RTX Spark announcement at Computex 2026. Tom's Hardware has the full spec breakdown here.

The Same AI Model Can Perform 6x Better: Here's Why

Harry Floyd — Sat, 30 May 2026 21:39:59 +0000

A Stanford and Tsinghua paper ran a controlled experiment earlier this year. Same model. Same task. Different harness architecture.

The result: a 6x performance gap driven entirely by the system built around the model. Not the model itself.

This is not a prompt engineering insight. It is a systems architecture insight, and it changes where developers should invest their time when building agentic systems.

The 6x Gap

Meta-Harness tested Claude Opus 4.6 across two harness configurations on TerminalBench-2. The only variable was the scaffold: the code that manages tool calls, context windows, error recovery, and state persistence.

One version scored at baseline. The other, with structured tool orchestration and context management, scored 18.4 points higher. Same inference cost. Same model. Different architecture.

This pattern replicates across multiple independent studies:

LangChain DeepAgents (2026): Same GPT-5.2-Codex model. Harness-only changes moved it from Top 30 to Top 5. That is a 13.7-point gain.

Can Bölük (Hashline, 2026): Same model, same task. Changed the edit tool format. Performance went from 6.7% to 68.3%. That is a 10x improvement with 61% fewer tokens.

Vercel's d0 agent: A production agent had 16 tools. Removing 14 of them (leaving only bash) took success rate from 80% to 100%. The bottleneck was not capability. It was decision surface.

Why This Matters Practically

The cheapest Haiku call with an optimised harness (37.6% on TerminalBench-2) outperformed the most expensive Opus call with a default harness (58.0%). That is at 1/50th the inference cost.

Most teams are optimising at the wrong layer. They swap models, tune prompts, add retrieval. The structural leverage is in how the system manages tool calls, handles state, and recovers from failure.

What Changes

The practical takeaway for anyone building with AI agents:

Audit your tool surface. Every tool your agent can call is a decision it must make. Vercel found 16→1 tool reduction improved everything. Fewer tools, better decisions.
Measure harness, not just model. Track task completion rate per harness configuration, not just per model. The harness is the variable that moved 6x.
Cost is architecture-dependent, not model-dependent. Haiku with a good harness beat Opus with a bad harness. Test harness variations before upgrading to a more expensive model.

The full analysis (12 verified claims, evidence tables, production case studies, and falsification criteria) is on Substack:

Harness Engineering: Same Model, Different Product →

It covers the Claude Code 1,421-line state machine, the Codex CLI vs Claude Code architecture comparison (77.3% vs 65.4%, 4.2x token efficiency difference), and why this is a Law IV (Instruments Over Theory) and Law I (Bottleneck Migration) structural play.

Follow for weekly analysis on AI infrastructure, agent architecture, and the systems that actually determine model performance.

The Decision Subtraction Framework: How to Evaluate Any AI Tool

Harry Floyd — Thu, 28 May 2026 10:39:34 +0000

Last week someone asked me which AI tools they should be using. The question hides a problem that costs real money: there are more capable AI tools available than any single person can evaluate.

ChatGPT Plus at $20/month. Claude at $20. Grok at $30. Cursor at $20. Copilot at $10. Each with a $100, $200, or $300 variant underneath. Each claims to earn its place.

The real question is not which tool is best. The real question is: which tools subtract more decisions than they add?

The Three Lenses

1. Replacement Ratio

Formula: decisions replaced by the tool ÷ decisions it creates.

List every decision the tool makes for you. Then list every new decision it forces you to make. Divide the first by the second.

Thresholds:

≥ 2.0 → Keep
1.0–2.0 → Evaluate
< 1.0 → Drop

Example: A code completion tool that writes a function body (replaces 5 decisions about syntax, structure, naming) but requires review (adds 2 decisions about correctness) has a ratio of 2.5. It passes.

A meeting summariser that replaces 1 decision (should I re-listen?) but creates 3 (verify accuracy, add context, decide what to share) has a ratio of 0.33. It fails.

2. Friction Delta

Formula: time without the tool ÷ time with the tool.

Include onboarding time amortised over your first 10 uses. A tool that saves 30 minutes per use but took 2 hours to learn breaks even at 4 uses. After that, it is pure gain.

Threshold: Break-even within 5 uses.

Catch: This lens breaks for tools that enable tasks you could not do at all before. A drug discovery simulation has infinite Friction Delta because the alternative is impossible. Score those as "can't evaluate on this lens" and rely on the others.

3. Attention ROI

Formula: output quality ÷ attention consumed.

Estimate cognitive load per use on a simple scale: 1 (fire and forget) to 4 (full attention required). Track whether it goes up or down over 10 uses.

Threshold: Attention per use should decrease over time. If you need to watch it more closely after ten uses than after one, something is wrong.

Where This Framework Lies to You

I tested this framework against the hardest cases I could find. It failed in five ways. Knowing them makes it useful:

Decision quality matters more than quantity. One high-stakes judgment (should I deploy?) outweighs 10 trivial picks (camelCase or snake_case?). Weight strategically.
Friction Delta can't measure capability expansion. If a tool lets you do something new rather than just faster, skip this lens.
Attention ROI rewards deskilling. The descending attention threshold is a Goodhart target — it rewards tools that train you to rubber-stamp.
Erasure cost is invisible. The framework never asks: if I use this for a year, what can I no longer do without it?
Error asymmetry is invisible. Two tools can score identically while producing catastrophically different results when they fail.

The Fourth Lens: Erasure Cost

Ask: "If I use this tool for six months and then stop, what skill will I have lost?"

Score it: 1 (nothing lost) to 4 (core competency outsourced). Score 1-2 is safe. Score 3 is a deliberate trade. Score 4 is dependency, not tooling.

How to Apply: Monday Morning

List every AI tool you have used in the last 30 days
Score Replacement Ratio and Friction Delta for each
Both pass → Keep. One fails → 7-day trial. Both fail → Cancel
Score Erasure Cost for the survivors
When evaluating a new tool: score it before subscribing

Worked Examples

ChatGPT Plus ($20/month)

Replacement Ratio: 3.5. Replaces research lookups, drafting, formatting. Creates verification and prompt decisions. Pass.
Friction Delta: Breakeven in 2-3 uses. Shallow learning curve. Pass.
Attention ROI: Decreasing. Gets faster as you learn its patterns. Pass.
Erasure Cost: 2. The underlying skill (structuring an argument) is reinforced, not replaced.
Verdict: Keep.

Cursor Pro ($20/month)

Replacement Ratio: 4.0. Replaces syntax lookups, boilerplate, function structure. Creates code review decisions. Pass.
Friction Delta: Breakeven in 1-2 uses. Tab completion is instant. Pass.
Attention ROI: Steeply decreasing. Pass.
Erasure Cost: 3. Heavy users report difficulty writing syntax without it after 3+ months. A deliberate trade worth making.
Verdict: Keep for daily coding. Monitor erasure.

Meeting Summariser ($20/month, anonymised)

Replacement Ratio: 0.33. Replaces 1 decision. Creates 3. Fails.
Friction Delta: Never breaks even. Still attend meetings, still verify. Fails.
Attention ROI: Flat. Must check every summary at same level. Fails.
Erasure Cost: 2. Minor skill atrophy.
Verdict: Cancel.

This framework connects to a deeper structural principle: a tool's value is the difficulty it removes. If it creates new difficulty of a different kind, it is not a tool. It is a job.

Full framework with diagram: https://telegra.ph/The-Decision-Subtraction-Framework-How-to-Evaluate-Any-AI-Tool-05-28

The Verification Bottleneck: Why Testing AI Agents Is Harder Than Building Them

Harry Floyd — Sat, 23 May 2026 01:54:26 +0000

The Verification Bottleneck: Why Testing AI Agents Is Harder Than Building Them

The AI industry has a supply problem. Not with chips, not with models, not with capital. It is a verification problem.

Every week, another agent framework launches. Every month, another company announces autonomous task completion. The build rate is accelerating. But the verification rate, the speed at which we can determine whether an agent’s output is correct, safe and worth trusting is not keeping pace.

This gap between build speed and verify speed is structural. It is not going to close on its own. And it is the bottleneck the market is not yet pricing.

The Build-Verify Asymmetry

Building an AI agent is getting cheaper. Open-weight models, orchestration frameworks like LangGraph and Semantic Kernel, managed inference APIs. The cost of standing up a functional agent pipeline has dropped by an order of magnitude in 18 months.

Verification has not followed the same curve.

To verify a single agent run, you need:

Ground-truth data for the task domain
A mechanism for comparing structured and unstructured outputs
Edge-case coverage for the long tail of user inputs
Human review loops for uncertain cases
Regression test suites that survive model updates

Each of these is expensive to build and maintain. And unlike the agent’s inference cost (which drops predictably with hardware improvements), verification cost is labour-proportional. Human review scales with headcount. Ground-truth data requires domain expertise to curate.

This creates an asymmetry that compounds with scale:

Dimension	Building	Verifying
Cost trajectory	Dropping (compute + models)	Flat or rising (labour + data)
Scaling method	More compute	More humans or better instruments
Automation potential	High (the agent itself)	Low (ground truth is domain-specific)
Failure mode	No agent	Wrong agent output that looks correct

The fourth row is the dangerous one. A false negative in verification (approving an incorrect agent output) has no visible failure signal until the downstream damage is done. A false positive, rejecting a correct output, creates friction and user frustration. Verification systems optimise for the wrong side of this trade-off because the wrong side is invisible.

What the Market Is Missing

When developers talk about agent reliability, the conversation usually lands on one of three things: chain-of-thought prompting, retrieval-augmented generation quality, or model fine-tuning. These are useful but they are not verification. They are attempts to make the agent less likely to produce wrong outputs in the first place.

Verification is a separate problem. It is the instrument that detects whether the output, regardless of how it was produced, is correct.

This is a Law IV problem. Law IV of the Durability Curve framework says that hidden structure stays hidden until you build the instrument to observe it. The hidden structure in AI agents is their failure modes. We are deploying agents without instruments to observe failures at scale because those instruments do not exist yet in any reliable form.

The companies that build those instruments will capture value that currently sits unclaimed.

What Verification Infrastructure Looks Like

The verification problem decomposes into layers:

Structural verification. Does the agent’s output conform to a known schema? JSON parsers, Pydantic models, and structured output constraints handle this today. This layer is the most mature but only catches format errors, not semantic ones.

Semantic verification. Does the output mean what we think it means? This is where the hard problems live. For a code-generating agent, does the produced code actually solve the user’s problem? For a document-analysis agent, are the extracted facts correct? This requires a second model, a verifier, running in evaluation mode.

Behavioural verification. Does the agent behave appropriately across a distribution of inputs? Not just single-shot accuracy but conversation-level coherence, safety boundary adherence, and refusal calibration.

Observability. Can you trace what the agent did, why it did it, and where it went wrong? This is the instrumentation layer: how tools, prompts and agent steps create signal. Datadog and ServiceNow are building in this space, but the landscape is fragmented and the standards are immature.

The market currently prices the first layer (structural verification) as solved, which it mostly is. It ignores the existence of the second and third layers. And it treats the fourth layer as a monitoring problem rather than a verification problem.

The Instrument-Making Opportunity

The history of technology markets suggests a pattern: the layer that controls verification captures a disproportionate share of value.

In software, the testing and observability tools (New Relic, Datadog, Selenium) created markets larger than many of the products they tested. In hardware, the inspection equipment market (KLA, ASML’s metrology) rivals the fabrication equipment market.

The same dynamic is unfolding in AI. The companies building agent-verification infrastructure (whether through evaluation frameworks, structured-output tooling or agent-observability platforms) are building instruments for a structure the market does not yet see clearly.

The falsification condition for this thesis is straightforward: if existing evaluation approaches (benchmarks, human review, test suites) prove sufficient for production agent deployment, the verification bottleneck does not materialise. But signals from production deployments, including the proliferation of guardrails, the emergence of dedicated evaluation teams at frontier labs and the growing literature on agent failure modes, suggest the opposite.

What Developers Should Watch

Three signals indicate whether the verification layer is becoming load-bearing:

Signal one: agent deployment velocity vs. incident rate. If agents are deployed faster but incident rates are not rising proportionally, verification is keeping pace. If incidents are accelerating faster than deployment, the bottleneck is tightening.

Signal two: emergence of dedicated agent-evaluation roles. The first companies to hire “agent verifier” or “AI evaluation engineer” as a distinct role, outside QA, are signalling that verification is not a subset of testing.

Signal three: consolidation around evaluation standards. If the ecosystem converges on one or two evaluation frameworks (beyond simple benchmark suites) within the next 12 months, the instrument-making phase is accelerating.

The build-verify asymmetry is not permanent. It is a market inefficiency that will correct, either through better verification infrastructure or through a pullback in agent deployment when undetected failures accumulate. Which correction path wins depends on whether the instrument makers move faster than the failure curve.

NVIDIA's $81.6B Quarter Confirms the Networking Bottleneck — Here's What Developers Should Know

Harry Floyd — Thu, 21 May 2026 08:39:53 +0000

NVIDIA reported $81.6 billion in revenue for Q1 FY2027 yesterday. That's an 85% year-over-year increase. Non-GAAP EPS of $1.87 beat consensus by 6%. Q2 guidance of $91 billion is 4% above what analysts expected.

By every headline measure, this was a clean beat and raise. The stock closed up 1.8% and was flat after hours.

The pattern is now five out of six quarters where NVIDIA beats the numbers and the stock doesn't rally. The headline numbers are priced before the print. The signal is in the composition of revenue, not the total.

The networking number that changes the story

Data Center networking revenue reached $14.8 billion — a record, up 199% year-over-year and 35% sequentially. Compare that to Data Center compute revenue of $60.4 billion, which grew 77% year-over-year.

The networking segment is growing at 2.6x the rate of the compute segment.

This is the clearest signal yet that the bottleneck in AI training is migrating from GPU FLOPs to inter-GPU bandwidth. As clusters scale past 50,000 devices, the wall-clock constraint shifts from "how fast can the GPU multiply matrices" to "how fast can the network move gradients between GPUs."

Two years ago, networking was roughly 12% of Data Center revenue. It is now 20% and accelerating.

The full-stack moat is protecting margins

GAAP gross margin reached 74.9% — up from 60.6% a year ago. This directly contradicts the narrative that Blackwell volume production would compress margins through CoWoS packaging and HBM memory costs.

Margins are expanding because the full stack (CUDA + NVLink + Spectrum-X + Blackwell silicon) creates pricing power that chip-design-alone cannot match. No competitor currently replicates this combination of scale and margin.

Key number: Operating income of ~$53.3 billion, up 147% year-over-year. GAAP net income tripled to $58.3 billion.

$48.6 billion in free cash flow — in one quarter

That's a 60% FCF margin. Only about 15 companies in the world generate more net profit in an entire year than NVIDIA generates in cash in three months.

Management's confidence shows in capital returns: dividend from $0.01 to $0.25 per share (25x increase), new $80 billion buyback authorization, and ~$20 billion returned to shareholders during the quarter.

Vera Rubin timeline de-risks the transition

Confirmed: on track for H2 2026, starting Q3, volume ramp in Q4. Architectural transitions are the highest-risk moment for any semiconductor company. This quarter's confirmation substantially de-risks the handoff.

Blackwell demand is so strong it's driving up secondary-market prices for older Hopper and Ampere GPUs.

Why the bottleneck matters for infrastructure engineers

If you're designing training clusters, the implication is direct: the binding constraint on throughput is shifting from GPU selection to network topology. Spectrum-X Ethernet and InfiniBand choices matter more per dollar than GPU generation increments at the margin.

For inference workloads, the bottleneck migration has different dynamics — memory bandwidth and model distribution across nodes become the constraint. But the pattern holds: the easy gains are no longer at the compute layer.

The one risk

Data Center now accounts for 92% of revenue. This is not a company risk — it's a regime risk. If hyperscaler capex cycles, 92% of revenue faces the same headwind.

All falsification triggers are green

Q2 guide below $85B → Guided $91B
Vera Rubin delayed beyond Q3 → On track Q3
Gross margin below 73% → 74.9%
Networking growth < compute growth → 199% vs 77%

The thesis is intact and the data is strengthening it.

Originally published on The Durability Curve.

The Indium Phosphide Bottleneck: What the Market Missed

Harry Floyd — Wed, 20 May 2026 11:19:09 +0000

A UK semiconductor company lost a quarter of its value in a single day this week. The market assumed Huawei exposure was catastrophic. The Huawei exposure is under five percent of revenue.

The panic was about the wrong thing.

Where Indium Phosphide Fits in the AI Stack

Every 1.6T optical transceiver shipping into an AI cluster relies on lasers made from indium phosphide (InP). Silicon cannot lase. InP does.

The supply chain:

Raw indium (zinc byproduct)
  → InP boule → sliced into substrates (AXT Inc)
    → Epitaxial wafer growth (IQE)
      → Laser fabrication (Lumentum, Coherent, MACOM)
        → Transceiver assembly
          → AI GPU clusters

IQE sits at the third step. It is the only independent large-scale epiwafer foundry with qualified InP capacity globally.

Why Switching Is Hard

Qualifying a new epiwafer supplier takes 12-24 months. Sample testing, device fabrication trials, reliability qualification. Once qualified, switching costs are enormous. There are approximately five qualified InP epiwafer suppliers worldwide. IQE is the only independent one at scale.

The MACOM Signal

In April 2026, MACOM invested £81M into IQE at 19.8p per share. MACOM received 11.5% ownership and two board seats, plus a long-term strategic supply agreement. The message: they cannot secure InP capacity without IQE.

Why Silicon Photonics Does Not Kill This

The common objection: silicon photonics will replace InP. This misunderstands the technology. Even silicon photonic ICs need InP lasers, integrated heterogeneously. Monolithic lasers-on-silicon are not expected before 2030. InP is needed regardless of which platform wins.

The Tension

IQE is not a clean story. Gross margins were under 4% last year on £118M revenue because reactors run at roughly half capacity. The wireless segment (57% of revenue) is declining. Three CEOs in five years. The current CEO also serves as CFO.

But the underlying structural thesis is clean: the InP bottleneck is real and tightening. AXT Inc is doubling capacity for 2027. Lumentum reported 90% revenue growth. The optical interconnect buildout for AI is a capital expenditure cycle backed by the largest technology companies in the world.

The pattern has historical precedent. As Marc Levinson documents in The Box, the shipping container created a modular interface that collapsed freight costs — the same structural pattern happening now in optical interconnects.

The market panicked about Huawei and ignored the structural story underneath. That is the pattern: value migrates upward as lower layers commoditise, and the market is slow to update on where the new bottleneck sits.

Originally published on Telegraph

The Earnings Report Is Not the Signal

Harry Floyd — Wed, 20 May 2026 09:15:00 +0000

NVDA reports $80B+ tonight. The headlines will focus on the number. Here is why it will not tell you what you need to know about the AI buildout paradigm.

Every earnings season, the same pattern plays out. A company reports a number. The market moves. Analysts revise targets. By the next morning, everyone is asking the wrong question.

The question is not whether NVIDIA beat or missed. The question is what the structure of the quarter reveals about the migration path of the bottleneck.

The Earnings Number Is a Lagging Indicator

A revenue beat tells you what happened in the past ninety days. The structural variables live in the forward-looking text, not the headline number.

Consider three data points from NVIDIA's last quarter:

Revenue of $68 billion. A beat. Priced in hours.
Supply commitments doubled to $95 billion. A signal. Took weeks to fully price.
A $500 million investment in a glass company. A map. Months later, it is still being understood.

Three Kinds of Data in Every Report

Signals — Forward-looking structural data that changes the probability distribution. Supply commitments. Lead times. Capacity expansion timelines. Customer concentration shifts. Rare. Worth a position.

Noise — Beats and misses within expected range. Drive the overnight move. Mean nothing for the thesis.

Echoes — Lagging confirmation of a known trend. Useful for calibration. Add no new information.

What to Watch Tonight

The Data Center narrative — is the mix shifting from training to inference? That is Law I migration.
Supply chain language — optical interconnects and glass substrates in prepared remarks are structural signals.
Supply commitment growth rate — if it exceeds revenue growth, NVIDIA is building against a constraint they see coming.

Why This Framework Matters

The durability of an investment thesis is determined by how well you distinguish signal from noise from echo. The market is designed to make everything look equally important. The structure that separates durable value from temporary noise is invisible to the real-time feed.

The report is not the signal. The structure behind the report is the signal.

Originally published on The Durability Curve

3 Infrastructure Bottlenecks That Exist Beyond Any Single Earnings Report

Harry Floyd — Tue, 19 May 2026 09:25:37 +0000

NVIDIA reports Q1 FY2027 earnings tomorrow. The consensus sits at roughly $79B in revenue with a 7% options move priced in. Every analyst note and headline will be about whether the number clears the bar.

None of that matters for the three bottlenecks that will define compute infrastructure investment for the next three years.

These constraints are structurally decoupled from NVIDIA’s quarterly variance. They exist whether the beat is 3% or 8%. They exist because of a first-principles property of large-scale systems: value migrates upward as lower layers commoditise.

The GPU layer is the lower layer. These three sit above it.

1. The Grid Transformer: 128 Weeks and No Substitute

The electrical transformer that steps voltage from transmission lines to data center distribution levels has a procurement lead time of 80 to 128 weeks globally. This is a structural ceiling on how fast AI infrastructure can physically be built.

This constraint exists regardless of GPU supply. You can have every Blackwell GPU allocated and paid for. If the transformer cabinet is not bolted to a concrete pad with an energized feed from the grid, those GPUs are room-temperature silicon.

The market is slowly noticing. ABB, Siemens Energy, and Schneider Electric have rerated upward. But the real asymmetry sits two layers deeper.

2. The Optical Link: Data Movement, Not Compute

Inside every large AI training cluster, data moves between GPUs at terabit speeds. As clusters scale to 100,000+ GPUs, the distance between nodes forces a fundamental migration from electrical to optical interconnects.

NVIDIA spent approximately $4B securing supply from Lumentum and Coherent. The photonics supply chain is tight through at least 2027.

3. The Evaluation Layer: Trust as Infrastructure

As AI agents move from demos into production workflows, the binding constraint shifts from “can the model do this?” to “can we prove it did it correctly?”

2-3% of AI-generated code passes tests but contains subtle errors. Catching that minority requires evaluation infrastructure that most teams do not have.

This market barely exists today. No dominant platform for AI evaluation exists. That absence is itself the pattern: the layer below is commoditising, and the layer above is where value forms next.

Why These Three

Each bottleneck sits at a layer above the GPU in the infrastructure stack. The GPU is being efficiently priced by the market. The constraints above it are where the market’s attention has not yet reached.

NVIDIA will beat or miss tomorrow. By Thursday the financial media will have moved on. These three bottlenecks will still be tightening.

36-page NVIDIA earnings research report: Gumroad

Free weekly analysis: The Durability Curve