DEV Community: Ferhat Atagün

The failures that don't throw

Ferhat Atagün — Tue, 01 Sep 2026 15:24:22 +0000

You ask a model for {"risk": "high" | "medium" | "low", "score": 0-100}. You run it, you get exactly that, you wire it up and ship.

Six weeks later somebody notices that a handful of claims were auto-approved that obviously shouldn't have been. Nothing errored. There's no exception in the logs, no failed request, no alert. The rows just have the wrong value in them, and they've had it for six weeks.

What happened is that one call in forty came back with "HIGH" instead of "high", and your code did this:

switch (result.risk) {
  case "high":   return escalate(claim);
  case "medium": return queue(claim);
  case "low":    return autoApprove(claim);
  default:       return autoApprove(claim);
}

The default branch is almost always the permissive one, because that's what keeps the queue moving. So the model said maximum risk and your system read it as approve automatically. The failure didn't degrade the decision. It inverted it.

That's the class of bug guard-lab exists to find: define a schema, run a prompt against it N times, and look at the distribution of ways it breaks.

TL;DR

Schema failures are distributional. One clean run tells you nothing, and five tell you almost nothing.
A single "94% compliant" number is worse than no number, because the interesting question isn't how many failed — it's what your code does about each one.
Three buckets: crash (throws, already in your error rate), recoverable (right payload, wrapped), and silent (parses, is accepted, is wrong). Only the third is invisible, and it's the only one that needs a tool.
Zero failures in fifty runs is not a zero failure rate. At n = 50 the data is still consistent with about 7%. Ruling out 1% takes 381 clean runs.
Constrained decoding removes most of this and is the right first move. It does not tell you your rate, and it constrains shape rather than sense.

Counting failures is the wrong instrument

The obvious way to build this is a compliance percentage. Run it fifty times, count how many matched, print 94%.

I built that first and it was useless within an hour, for a reason that took me longer to articulate than it should have: the failures are not equally dangerous, and a single number averages over the only distinction that matters.

Consider two runs that both report 94%.

In the first, the 6% is markdown fences — the model wrapped the object in `json. Your parser either strips fences or it doesn't. If it does, that 6% costs you nothing at all. If it doesn't, it costs you 6% of requests, loudly, on day one, and you fix it before lunch.

In the second, the 6% is enum drift and numbers arriving as strings. Nothing throws. Every one of those responses is accepted by your code, written to your database, and rendered on someone's screen. You will find out when a customer does.

Same number. Completely different situation. One is a morning; the other is the six weeks at the top of this post.

The three buckets

So guard-lab sorts failures by what your code does about them rather than by what went wrong:

Crash — the output doesn't parse. JSON.parse throws, the request fails, and it appears in an error rate you're already watching. Genuinely fine. Loud failures get fixed.

Recoverable — the payload is correct but wrapped. A markdown fence, a sentence of preamble, a helpful trailing note. Free if you strip it, fatal if you assume the body is only JSON. Either way you find out immediately.

Silent — the output parses, your code accepts it, and the value is wrong.

The third bucket is the whole point, and it's bigger than it looks. It isn't only enum drift:

A missing field reads as undefined. Templates render blank, comparisons quietly go false, nothing throws.
"score": "85" as a string gives you NaN on arithmetic, or "8510" on + 10. No error either way.
An invented enum — "very high" — passes every check you wrote for the values you expected and hits the default branch as if it were routine.
"score": 150 on a 0–100 field is a valid number in an invalid position. It renders, it charts, it misleads.
null where a value was required passes an existence check and fails a type check somewhere far away, at a distance from the cause.

Every one of these is accepted by JavaScript without complaint. None of them will ever appear in your error rate. That's not a gap in your monitoring — it's a category your monitoring cannot see, because from the outside nothing went wrong.

Zero failures is not a zero failure rate

Here's the part I didn't expect to end up caring most about.

You run the tool, fifty times, and everything passes. Green across the board. That reads as proof, and it isn't one.

Fifty clean runs is a sample. The true failure rate is whatever it is, and your sample is consistent with a range of values — at n = 50 with zero observed failures, the 95% Wilson upper bound is 7.1%. At a thousand calls a day, that's up to seventy-one bad records daily that your test run had no power to detect.

So guard-lab never shows a bare zero. A clean sweep gets an interval and a sentence saying what it does and doesn't rule out, because the reassuring version of that screen is the single most misleading thing the tool could do. The whole reason someone runs it is to stop trusting a sample of one; handing them a sample of fifty dressed up as certainty would just move the same mistake somewhere more expensive.

The deflating corollary: ruling out a 1% failure rate takes 381 clean runs. Which is a real number that real teams will not run, and knowing that is still better than believing five runs settled it.

The strongest version of the counter-argument

Let me argue the other side properly, because there's a good objection here and the weak version isn't worth knocking down.

This is a solved problem. Use structured outputs. Tool use with a JSON schema, or a constrained-decoding mode, forces the model's output to conform at the token level. Fences, prose, unparseable output, missing fields, wrong types — most of the taxonomy above simply cannot happen.

That's correct, and it should be your first move. If you can use constrained decoding, use it; a validator downstream of an unconstrained prompt is a worse design than not needing one.

Two things survive it.

It constrains shape, not sense. A schema that says "risk" is one of three strings will get you one of three strings. It will not tell you whether the model picked the right one, and it won't catch a range violation you didn't encode — most people write "type": "number" and not the 0–100 bound, so 150 passes. Constrained decoding turns a syntax problem into a semantics problem. That's a large improvement and not a solution.

Plenty of production code isn't in a position to use it. A prompt template inside a vendor product. An internal gateway that normalises requests and drops the tool-use block. An older endpoint someone integrated in 2024 and nobody has budget to revisit. A model behind a proxy that only forwards messages. These are not exotic; they're most of what an engineer walking into someone else's environment actually finds.

And underneath both: constrained decoding doesn't give you a rate. It reduces the failure probability, plausibly by a lot, to some number you still don't know. If the answer to "how often does this break" is "less than before", you haven't measured anything — you've just moved the number somewhere you can't see it.

Two bugs I shipped into it

The useful part of building this was that I committed, twice, exactly the kind of error the tool is for.

The first was a silent failure inside the silent-failure detector. To be generous about what counts as recoverable, the parser salvages JSON out of surrounding prose by taking the widest span between the first { and the last } — which is roughly what defensive production code does. Then a test case handed it [{...}, {...}]: a model returning an array of results instead of one object.

The salvage worked. It pulled the first object out, reported a clean recovery, and silently discarded every element after it.

That is precisely the failure mode in the post you're reading — parses, is accepted, is wrong — committed inside the thing built to catch it. The fix is to parse the response as-is first, exactly as real code would, and only fall back to salvage when that fails. Valid JSON of the wrong shape now reaches the caller intact and gets judged as a crash, because that's what it is.

The second was a statistical self-contradiction. For "how many clean runs would it take to rule out a 1% failure rate" I used the rule of three: zero events in n trials bounds the rate near 3/n, so 300 runs. Textbook, correct, widely used.

Except everywhere else the tool reports a Wilson interval, and Wilson is slightly more conservative at zero. Run 300 clean and it would have told you 1.26% — after promising 300 would get you to 1%. Two individually defensible methods, disagreeing, in a tool whose entire pitch is that you should trust its numbers.

It now inverts the same interval it displays. The answer is 381 rather than 300: less satisfying, and consistent with what the tool will say to you afterwards.

Neither bug was caught by using the app. Both were caught by tests written against reference values — seventeen classifier cases, and Wilson checked against published intervals. Which is its own small argument for the thing I wrote about evals two posts ago: looking at the screen and seeing plausible numbers is not verification.

What it doesn't tell you

Anything about whether the answers are good.

A response can satisfy every field in the schema — correct enum, in-range score, all fields present — and be completely wrong about the claim. guard-lab measures whether the contract holds. Whether the judgement inside the contract is any good is the other kind of eval, the one that needs ground truth, and this doesn't touch it.

Contract compliance is the floor. It's just that most teams have never measured the floor, and it turns out not to be where they assumed it was.

guard-lab is free, open source, and browser-only — BYOK, and the key never leaves your tab because there's no backend to send it to. Source on GitHub.

It's the sixth in a set of tools built under one constraint: they have to work in a room where you can't install anything and the data can't leave. The rest are here.

The second customer tells you what you actually built

Ferhat Atagün — Tue, 01 Sep 2026 15:18:25 +0000

There's a sentence that gets repeated about forward-deployed work, usually attributed to Palantir's model, and it sounds like a slogan until you've been on the wrong side of it: one customer, many capabilities — as opposed to the SaaS shape, which is many customers, one capability.

The SaaS shape is well understood. Find one thing a lot of people need, build it once, sell it repeatedly, and every additional customer costs you almost nothing. The whole discipline of product management is built around identifying that one thing.

Forward-deployed work runs the other way. You embed with one organisation and build whatever that organisation actually needs, which turns out to be eleven things, most of which nobody outside the building would recognise as a product. And then somebody asks the question that decides whether you're a software company or a consultancy in a company's clothing: which of those eleven is a capability, and which is bespoke work you'll never sell twice?

Getting that wrong in either direction is fatal, and the two failures look nothing alike.

TL;DR

FDE work inverts the SaaS shape: depth with one customer instead of breadth across many. That produces capabilities faster than product discovery does, and no way to tell which ones generalise.
Generalise too early and you ship a configurable abstraction fitted to a sample size of one. It has settings for the axis that varied and hardcodes the one that didn't.
Generalise never and every deployment is bespoke. Margins go, the second engagement is as expensive as the first, and you're a consultancy that files its revenue under "platform".
Two customers asking for the same feature is not the signal. Two customers arriving at the same feature for the same underlying reason is. Same request, different causes, is a coincidence that will diverge the moment you build it.
The most reliable evidence isn't the request. It's the workaround: what both teams already built by hand before anyone asked you.

The two failures

Generalising too early is the one engineers do, because abstraction feels like craftsmanship. You've solved a problem for one customer, you can see the shape of it, and the shape looks reusable. So you pull the specifics out into configuration and ship a general version.

The thing you can't see from inside a single deployment is which axis actually varies. You had one data source, so you made the data source pluggable. You had one approval flow, so you left the approval flow hardcoded. Customer two arrives with the same data source and a completely different approval flow, and now you have an abstraction that is flexible in the direction nothing moves and rigid in the direction everything does. Both customers get a worse system than a bespoke build, and the abstraction is now load-bearing so removing it is a project.

Never generalising is quieter and takes longer to hurt. Every engagement is a fresh build. Each one goes fine. The engineers get very good and the delivery gets very reliable, and the cost of the tenth deployment is roughly the cost of the first. That's a viable business — it's just consultancy, and it's usually being valued and staffed as though it were a product, which is where it comes apart. The gap shows up as margin, then as hiring, then as an inability to explain to anyone what the company sells.

Both failures come from the same missing thing, which is a test for when a capability is real.

The request is not the signal

The intuitive test is repetition. Two customers ask for the same feature, so build it properly.

I don't think that holds, and the reason is the more interesting half of this.

Two organisations can ask for the same thing for entirely unrelated reasons. Both want an export to CSV: one because their analysts live in Excel, the other because their compliance team needs an immutable artifact for an audit trail. Same request, same three words in the ticket. Build the general version and you'll discover the first one wants live data and column reordering, and the second needs a frozen file with a checksum and a retention policy. The feature that satisfies both is two features with a shared name.

What actually predicts generalisation isn't the request converging. It's the constraint converging.

This is the same argument I made two posts ago about my own tools, from the other end. I built five things browser-only and BYOK out of what felt like personal preference, and later noticed those are precisely the constraints of working inside someone else's regulated environment — nothing to install, no data crossing the boundary, nothing for a security team to threat-model. I hadn't copied that from anywhere. Two problems with the same shape produce the same engineering decisions independently, without the people involved ever talking.

That independence is the whole signal. When two customers land on the same design because their situations share a structural constraint — a boundary the data can't cross, a regulator who needs the reasoning preserved, a workflow that has to survive its owner going on leave — the thing you build for both is genuinely one thing. When they land on the same request through different constraints, you're looking at a naming collision.

So the question to ask about a repeated request isn't how many customers want this. It's what makes them want it, and is it the same thing.

Look at the workaround, not the ask

The most useful evidence I've found isn't in what customers ask for at all. It's in what they've already built by hand.

Every organisation running a real workflow has accumulated a layer of manual compensation: a spreadsheet someone maintains, a Slack channel that functions as a queue, a person who checks a thing every morning at nine. Those exist because something in the system doesn't do a job that needs doing, and somebody cared enough to fill the gap with their own time.

Two customers who've independently built the same workaround are telling you something much stronger than two customers asking for the same feature. A request can be aspirational, or repeated back to you from a conference talk, or a stakeholder's preference. A workaround is paid for. Somebody spends an hour a day on it. Nobody sustains that for a nice-to-have.

And workarounds are legible about the underlying constraint in a way requests aren't. The morning check exists because the process is asynchronous and nobody trusts it to fail loudly. The spreadsheet exists because the system's model of the work is one field short of reality. That's the constraint, stated in a form you can compare across customers — and comparing constraints is the thing the feature-request list can't do for you.

The FDE is the sensor, and the reporting line is the problem

If this is right, the person embedded in the deployment is the highest-bandwidth product-discovery instrument the company has. They're not reading a survey. They're watching the workaround get used, at nine in the morning, by the person who built it.

Which raises a structural question that I think is underrated: who reads what that person learns?

In most arrangements, the forward-deployed engineer reports through delivery. Delivery is measured on shipping this deployment, on time. The observation that customer three has independently rebuilt the same spreadsheet as customer one is not a delivery artifact. There's usually no field for it, no forum where it lands, and no incentive for a busy engineer to write it up.

So it stays in one person's head, which — as the previous post argued about customer handover — is the same as it not existing. The failure mode is identical, just pointed inward: context accumulates in an individual, doesn't get bled out to the organisation, and leaves when they do. A company can lose its product roadmap this way without ever noticing it had one.

The fix isn't complicated, it's just nobody's job: a standing, low-ceremony way for embedded engineers to report constraints rather than requests. Not "customer wants export" but "third customer this year whose data can't leave the boundary, all three built a manual extract". Three of those in a row is a product decision that made itself.

What the series has actually been about

Five posts, and the pattern underneath them is more consistent than I expected when I started.

Pilots fail at the interface, not the model. The tools that survive are the ones with an articulated constraint. The deliverable is the eval, because the eval is what lets someone else change the thing. The handover works when a team can make a change and know it was safe. And the capability generalises when two customers reach it through the same constraint rather than the same words.

Every one of those is the same claim in a different setting: the hard part is the transfer, not the build. From a model to a person, from a prototype to a workflow, from you to the team that stays, from one customer to the next. The model was never the bottleneck, and it's less of one every quarter.

Which is, I think, why the role exists at all, and why it's being hired for eight times as often as it was — not because the technology got harder, but because the transfer never got easier, and it turns out that was always the expensive part.

Part five, and the last, of a series on the last mile of enterprise AI.
Part one: Nobody's model failed. The interface did. ·
Part two: I accidentally built a forward-deployed engineer's field kit ·
Part three: The deliverable isn't the prompt. It's the eval. ·
Part four: You're not done when it works. You're done when they can change it.

You're not done when it works. You're done when they can change it.

Ferhat Atagün — Tue, 01 Sep 2026 15:12:49 +0000

The deployment I think about most didn't fail. It passed UAT, it went live, the customer was pleased, and there was a genuinely nice email at the end of it.

Six months later it was still running and nobody had touched it. Not once. The prompt was byte-identical to the one I'd left. A model version had been deprecated underneath it and the team had pinned the old one rather than test the new. Someone had asked for a small change to the output format and been told it wasn't worth the risk.

Nothing broke. That's the part that took me a while to understand. The system didn't die when it stopped working. It died when it stopped being changed — and it had been dead for about five of those six months while continuing to return responses.

That's the failure mode nobody writes a post-mortem for, because there's no incident. There's just a thing in the corner that everyone routes around, until a reorg or a contract renewal quietly removes it.

TL;DR

A deployment's real end date is the last time someone modified it with confidence, not the last time it responded to a request.
Documentation describes a state. Ownership requires being able to make a change and know it was safe. Those are different deliverables and only one of them survives contact with the first edit.
Three questions decide whether the work outlives you: can they change it, can they tell if the change worked, and do they know why it's like that. Most handovers answer only the first.
The why decays fastest and is written down least. Code records what, git records when, and the reason a strange clause exists lives in one person's memory until it looks like noise and gets deleted.
As a forward-deployed engineer you are the bus factor by design. The job is to make yourself unnecessary on a schedule, and feeling indispensable is the signal that it's going wrong.

The documentation trap

The instinct at the end of an engagement is to write everything down. Architecture diagram, sequence of calls, deployment runbook, a wiki page per component. It feels like diligence, and it produces an artifact you can point at in the final meeting.

It also goes stale the first time someone makes a change — and if nobody makes a change, it didn't matter that you wrote it.

The uncomfortable version: a thorough document can make things worse, because it substitutes for the thing that would actually have helped. The team reads it, understands the shape of the system, and still doesn't touch it, because understanding the architecture was never what was stopping them. What was stopping them is that they had no way to make a change and find out whether they'd broken something.

Documentation answers "what is this". Ownership needs an answer to "what happens if I change this". No amount of the first produces the second.

The three questions

Whether a deployment survives handover comes down to three things, and they get progressively less likely to be addressed.

1. Can they change it?

The mechanical one. Do they have the credentials, the repo access, the deploy path, the ability to roll back. This is the question every handover checklist covers, and it's genuinely necessary.

It's also the easiest one, and clearing it feels like progress in a way that's slightly misleading — because a team with full production access and no confidence changes nothing, which is observationally identical to a team with no access at all.

2. Can they tell if the change worked?

This is the previous post's whole argument, and it's the hinge.

The frozen snapshot — a set of real inputs and the decisions the current system makes on them — isn't primarily a quality tool. Its real function is permission. It's the thing that converts "I think this prompt edit is fine" into "twelve of the fourteen cases are unchanged and here are the two that moved". The first sentence gets a change reverted in review. The second gets it merged.

A team without that will not edit the prompt. They'll be right not to. Editing a prompt with no way to check the blast radius is not caution, it's gambling with someone else's workflow, and sensible people decline.

3. Do they know why it's like that?

The one that almost never gets handled, and the one that quietly determines everything.

Every system that has met reality has clauses in it that look wrong. A sentence in the prompt insisting on a format that seems redundant. A retry with an oddly specific backoff. A field that gets stripped before it's shown. Each of those exists because something happened — a specific failure, on a specific day, with a specific customer record that broke a specific assumption.

Six months later, whoever inherits it sees an ugly clause with no explanation and does the reasonable thing: cleans it up. And the original failure comes back, except now nobody in the room remembers the first one, so it presents as a new bug.

Code records what. Git records when and, if you're lucky, a commit message records a compressed version of why. Nothing records the reasoning at the place where someone will encounter the decision.

Attach the why to the thing, not beside it

The practice that has actually worked for me is unglamorous: put the reason where the decision is, not in a document about the decision.

For prompts, that means a comment block in the prompt file itself, naming the failure that produced each non-obvious instruction. Not "be concise" — "be concise: the 2,400-word answers in week two were being pasted into a field with a 500-character limit downstream, which truncated mid-sentence and looked like a model failure."

For behaviour, it means a test named after the incident rather than the function. A case called keeps_the_account_number_when_the_name_is_missing is a sentence about the business. A case called test_parse_edge_case_3 is a sentence about nothing, and it's the one that gets deleted when it goes red.

This is the same move as the frozen eval set, applied to reasoning instead of behaviour. Both work because they're attached to something that runs. A wiki page can go stale silently; a named test goes red loudly, and the name tells the person staring at it what they're about to break.

The test is: someone who wasn't there reads the clause and can decide, on their own, whether the reason still applies. That's the whole bar. Not "they understand the system" — they can safely disagree with me.

You are the bus factor, on purpose

There's an awkward thing about this role that's worth saying plainly.

A forward-deployed engineer is, structurally, a single point of failure introduced deliberately. You're embedded because the customer's team can't yet do the thing. Every week you're there, you accumulate context that exists nowhere else — which workflow diverges from its documentation, which stakeholder's objection is real and which is positioning, why the second data source can't be trusted on Mondays.

That context is what makes you effective. It's also, if you don't actively bleed it out, the reason the deployment dies when you leave.

And the incentives run the wrong way. Being the person who understands the system feels like doing well. Getting the call because nobody else can debug it feels like value. It reads as indispensable, and indispensable feels like success right up until it's the postmortem.

The metric that actually matters is uncomfortable to optimise for: how quickly you become unnecessary. Not how much you shipped. How fast the people who stay stopped needing you in the room.

Which is a strange thing to be measured on, and I don't think most engagements measure it at all — they measure delivery, and delivery is the part that happens while you're still there.

What I'd actually leave

Concretely, stripped of the parts that sound good and don't survive:

The frozen set, in the repo, with a one-command way to run it. Not a notebook, not a process someone has to remember. npm run eval or the equivalent, output that a non-author can read.
A decisions file, but scoped hard: only decisions where the obvious choice was rejected, each one naming the thing that made it obvious-but-wrong. Ten entries that matter, not sixty that document the defaults.
The reasons inline — in the prompt, in the test names, next to the retry constant. Anywhere someone will be standing when they consider changing it.
One change they make and I don't. Sitting there while someone else edits the prompt, runs the eval, reads the diff, and ships it. Watching a team do it once is the only handover step I've never seen substituted successfully. Everything else can be faked with a good document; this one can't.

That last one is the whole thing, really. Everything before it is preparation for the moment where someone other than you changes the system and nothing bad happens. If that moment doesn't occur before you leave, you have no evidence it can.

And "no evidence it can" is, in practice, the same as "it won't" — which brings the six-month clock back around, and the system that's already dead while it's still answering.

The uncomfortable implication of all of this is that the work isn't really about the model, or the prompt, or even the interface — it's about whether one team can hand a capability to another. Which is a much older problem than anything in this field, and the subject of the last post in this series.

Part four of a series on the last mile of enterprise AI.
Part one: Nobody's model failed. The interface did. ·
Part two: I accidentally built a forward-deployed engineer's field kit ·
Part three: The deliverable isn't the prompt. It's the eval. ·
Part five: The second customer tells you what you actually built

"That library will hurt performance" is not a number

Ferhat Atagün — Tue, 01 Sep 2026 10:07:12 +0000

Somebody in the room says the marketing team's chat widget is making the site slow. Somebody else says it drives a third of the qualified leads. Both are probably right, neither has a number, and the decision gets made by whoever sounds more confident. Two sprints later the widget is still there and the page is still slow.

I've been on both sides of that conversation and I've never once seen it settled with evidence. Not because the evidence is hard to get — because the tool that would produce it doesn't exist. Every performance tool we have reports in the present tense. Lighthouse tells you what the page costs. WebPageTest tells you what the page costs, in more detail. CrUX and RUM tell you what the page costs, for real users. All of them answer "what is this page doing right now", and none of them answers the question the meeting is actually about, which is "what would it cost without that".

So I built perf-lab. Drop in a Lighthouse JSON report, switch resources off, and watch the score move. It turns "that library will hurt performance" into "that library costs 0.4 s of LCP and 7 points" — which is a sentence you can put in a ticket.

TL;DR

Every performance tool is a measurement tool. The decisions teams argue about are counterfactual, and nothing measures counterfactuals.
A projection is only worth arguing with if it lands on the scale the stakeholders already screenshot, so perf-lab reimplements Lighthouse's actual scoring — the log-normal curves, the published control points, the category weights. On a real report it recomputes 57 against Lighthouse's reported 57.
Ranking by bytes is actively misleading. A 410 KB image that finishes loading after LCP is worth zero points; a 124 KB render-blocking script is worth eleven.
Savings overlap. Four independent "+7 point" fixes do not add up to 28, and a tool that implies they do is setting up a disappointing retro.
Everything is a delta against measured values, never an invented absolute — and where a number is modelled rather than measured, the UI says so.

Why the score has to be Lighthouse's score

The first version of this idea I sketched had its own 0–100 index. It was useless within a day.

The whole point is to end an argument, and the argument is being had by people who look at Lighthouse. If my tool says "this saves you 12 points" on a scale I invented, the reply is "twelve of what", and now we're arguing about my scale instead of about the chat widget. The projection has to arrive in the same units as the screenshot in the ticket.

So scoring.ts is a reimplementation of the real thing. Lighthouse doesn't score metrics linearly — it maps each one through a log-normal curve calibrated to two control points, the 10th percentile of real sites and the median:

export const METRIC_WEIGHTS = {
  fcp: 0.1, si: 0.1, lcp: 0.25, tbt: 0.3, cls: 0.25
} as const;

export const CONTROL_POINTS = {
  fcp: { p10: 1800, median: 3000 },
  si:  { p10: 3387, median: 5800 },
  lcp: { p10: 2500, median: 4000 },
  tbt: { p10: 200,  median: 600  },
  cls: { p10: 0.1,  median: 0.25 }
};

Two things fall out of those numbers that are worth internalising even if you never open the tool.

TBT is 30% of your score and CLS is 25%. Together, two metrics that have nothing to do with how fast the page looks are 55% of the number. LCP — the one everybody talks about — is 25%.

The curve is brutal near the top and forgiving in the middle. Going from 4.0 s to 3.5 s of LCP buys you very little. Going from 2.8 s to 2.4 s buys you a lot. Which means "we improved LCP by half a second" is not a fact you can price without knowing where you started.

I did eventually verify the reimplementation rather than trusting it, which I should have done far earlier than I did. I ran real Lighthouse against my own site and fed the JSON straight into the parser:

reported: 57   recomputed: 57

That agreement is the entire credibility of the tool. There's a scripts/probe.ts in the repo whose first line of output is exactly that comparison, so if the implementation ever drifts from Lighthouse's, it fails loudly instead of quietly producing plausible numbers on the wrong scale.

The inversion that makes byte-sorted reports lie

Here's the demo page perf-lab ships with — a fairly ordinary marketing page, 1.1 MB, scoring 57. Sorted by what removing each resource is actually worth:

Resource	Size	Worth
`consent-banner.js`	124 KB	+11
`intercom.js`	168 KB	+11
`framework.bundle.js`	178 KB	+10
`gtm.js`	92 KB	+7
`below-fold-gallery.webp`	410 KB	0

The biggest file on the page is worth nothing. It's a gallery image below the fold that finishes loading after LCP has already fired, so it was never on the critical path — removing it makes the page lighter and doesn't make it faster. Meanwhile the third-smallest file in that list is worth eleven points, because it blocks rendering.

This is not an exotic edge case. It's the normal shape of a web page, and it's why the "largest assets" table in every performance report sends teams to optimise the wrong thing. The question was never what is heavy. It's what is heavy **and* on the critical path*.

The bug that proves the point

I want to be straight about this one, because it's the most useful thing in the post.

I shipped the tool, opened it, and the demo showed below-fold-gallery.webp at +12 points — the exact inversion the tool exists to correct, in the tool, on the first screen. Removing any resource shrank the total transfer, and I was applying the transfer saving to FCP and LCP without first checking whether the resource had anything to do with them.

I fixed that, and then found a second version of the same mistake hiding one layer down, in the parser:

beforeLcp: endTime <= metrics.lcp || startTime <= metrics.lcp

Almost every request on a page starts before LCP fires. That second clause marked nearly everything as critical-path, which quietly dismantles the tool's central claim on every real report.

It survived because the sample data sets beforeLcp by hand, so the demo — the thing I kept looking at — never went through that code path. The only way to catch it was to run a real Lighthouse report through the parser, which I hadn't done until the tool was already live. The demo data that makes a tool easy to show off is the same data that stops you testing the part that matters.

Savings don't add up, and pretending they do is worse than useless

First version ranked every resource against the untouched report. Which reads fine, until you notice it's promising each removal the full win independently — four "+7 point" fixes that sum to 28 on the slide and 11 in production, because they overlap. Once you've dropped two analytics tags, TBT has already come down, and the third tag has less left to give back.

So the ranking is now marginal: it re-simulates against whatever you've already switched off. In the demo, removing Intercom drops the hero image from +11 to +6 — same file, same bytes, worth half as much once LCP has already moved.

That behaviour turned out to be the most persuasive thing in the tool. Watching the list re-sort as you strip things out teaches the overlap in about fifteen seconds, and it's the sort of thing teams normally learn in a retro.

What's measured and what's modelled

A tool that projects the future can very easily become a tool that makes things up confidently. The rule I held to: never invent an absolute number when a measured one exists. Everything works on deltas applied to the metrics Lighthouse actually recorded, so the baseline is always ground truth and only the difference is modelled.

Taken straight from the report: transfer size and timing per request (network-requests), parse and execute time per script (bootup-time), render-blocking status and its measured saving (render-blocking-resources), third-party attribution (third-party-summary).

Modelled, and therefore approximate: transfer time recovered on a throttled connection, how much of a script's execution lands inside the TBT window, and whether removing something moves LCP. Even the TBT fraction is anchored to the report rather than to an industry average — it uses the ratio this page exhibited, measured TBT over total measured CPU.

Every projection carries a confidence label — measured, modelled, or speculative — that degrades as you move away from the conditions the report was captured under. Change the throttling preset and remove something Lighthouse never measured, and the tool tells you it's directional only, rather than handing you a number with false authority.

It found a bug on the site it's published from

The real Lighthouse run I'd done to verify the scoring was against my own /tools page. Having gone to the trouble, I read the rest of it.

Mobile score 57. LCP 7.2 s, which the curves price at roughly 23 of the 43 points I was losing. And the LCP element — named right there in the report — was the first tool card's screenshot.

Every card image on that page was marked loading="lazy".

The one image the score is measured on was the one I was deliberately deferring. Lazy-loading is good practice, I'd applied it uniformly, and applying it uniformly is exactly the mistake: the LCP element must never be lazy. First card is now eager with fetchpriority="high", the rest stay lazy.

The tool also flagged 238 KiB sitting in PNGs that should have been WebP. Converting them saved 620 KB.

I'd been working on that page for weeks, with SEO and performance explicitly in mind, and I'd have kept not seeing it. That's the argument for the whole category, really. Measurement tells you the page is slow. You already suspected that. What changes a decision is knowing what it would cost to be otherwise — and until you can put a number on the counterfactual, the loudest person in the room is going to keep being right.

perf-lab is free, open source, and browser-only — the report is parsed in your tab and nothing is uploaded, which matters when the report came from a client's staging environment. Source on GitHub.

Generate a report with:

npx lighthouse https://example.com --only-categories=performance \
  --output=json --output-path=./report.json

Or DevTools → Lighthouse → run → ⋮ → Save as JSON. No report handy? There's sample data one click in.

The deliverable isn't the prompt. It's the eval.

Ferhat Atagün — Wed, 12 Aug 2026 17:33:53 +0000

Every field guide to forward-deployed work says a version of the same thing:
define your evaluation framework before you build anything.

And every engineer who has actually landed in a customer's environment has had
the same reaction, which is: with what?

On day three you don't have labelled data. You don't have domain expertise —
the people who do are the ones you're building for, and they're busy. You have
five examples someone showed you on a shared screen, a workflow that diverges
from its documentation in ways nobody has written down, and a rough sense that
the current process takes too long. Building a golden dataset off that is not a
task. It's a fantasy.

So the advice gets skipped. Not out of laziness — out of practicality. And then
about ninety days later the system starts drifting, quietly, and nobody notices
until the customer does.

I think the advice is right and the framing is wrong. You're not being told to
skip evals. You're being told to build the wrong kind first.

TL;DR

Standard eval guidance — golden datasets, rubrics, labelled examples — is correct and close to unusable in week one, which is exactly when it's prescribed.
"Eval" is doing double duty for two different questions. Is this output good? needs ground truth. Did this change? needs only a snapshot.
The second one is buildable on day three, and it catches most of what actually kills deployments: silent drift after a prompt edit, a model version bump, a parameter change nobody logged.
Don't diff prose — LLM output is nondeterministic and you'll drown in noise. Diff the decision: the structured claim underneath the words.
The regression harness then writes your golden dataset for you. Every failure it catches is a labelled example you didn't have to imagine.
This is why the eval is the deliverable. When you leave, the prompt is a text file anyone can edit — and someone will. The eval is what tells them whether the edit was safe.

Two different questions wearing the same word

Here's the distinction that unlocked this for me.

	Quality eval	Regression eval
Question	Is this output correct?	Did this change from what we accepted?
Needs	Ground truth, domain expertise, rubric	A frozen snapshot
Buildable on day 3	No	Yes
Catches	Bad design	Silent drift
Cost to build	Weeks	An afternoon

We call both of them "evals," which is how the cheap one gets postponed
alongside the expensive one.

The expensive one is genuinely expensive and genuinely later. The cheap one has
no prerequisites at all, and skipping it is what turns a working deployment into
a mysteriously-degraded one.

What you can actually build on day three

Concretely, five steps, none of which require you to know what "good" means:

1. Collect real inputs. Not synthetic ones. Sit with whoever does the work
and take five to ten actual cases off their screen — including the two weird
ones they apologise for. The weird ones are the whole point; they're where the
system will break and where the documentation is silent.

2. Run them and print the output. No scoring, no rubric.

3. Get a binary from the person who owns the workflow. Not "rate this 1–5."
Just: would you have been comfortable if the system did this on its own? Yes
or no. People are fast and reliable at this and slow and unreliable at rubrics.

4. Freeze the yeses. That file is your eval. It is not a prototype of an
eval, it is the thing.

5. Re-run it on every change. Prompt edit, model bump, parameter tweak,
context restructure. Anything.

The frozen file is boring on purpose:

[
  {
    "id": "claim-2024-0871",
    "input": "…the actual case, verbatim…",
    "decision": { "risk": "high", "route": "manual-review" },
    "acceptedBy": "ops lead",
    "acceptedOn": "2026-03-04"
  }
]

And the runner is about twenty lines:

for (const c of frozen) {
  const output   = await run(currentPrompt, c.input);
  const decision = extractDecision(output);      // structured claim only

  if (!deepEqual(decision, c.decision)) {
    report({ id: c.id, was: c.decision, now: decision });
  }
}

That's it. There's no framework here, and there shouldn't be. It's a habit with
a file attached.

Diff decisions, not prose

The obvious objection: LLM output is nondeterministic. Diffing it produces
nothing but noise. Correct — if you diff the wrong thing.

Run the same prompt twice and the prose will differ. Word order, hedging, how
the explanation is structured. None of that is a regression. If you string-diff
the response body, every run fails and you'll turn the harness off inside a week.

So don't. Extract the decision and diff that.

Almost every useful LLM step in a production workflow reduces to a small
structured claim buried in the prose: a category, a number, a routing choice, a
yes/no. The prose around it is presentation. The claim is the behaviour.

// prose varies; this must not
type Decision = {
  risk: 'high' | 'medium' | 'low';
  route: 'auto' | 'manual-review';
};

If the model routes claim-0871 to manual-review on Monday and auto on
Friday, that is a regression regardless of how nicely it explained itself. If it
routes it to manual-review both times using completely different sentences,
nothing happened.

This also has a side benefit I didn't anticipate: forcing yourself to name the
decision type clarifies the design. If you can't write that type down, you don't
yet know what the model is for — and a step whose output can't be reduced to a
claim is usually a step that should have been two steps.

What this catches that nothing else does

Every one of these is real, silent, and invisible without a frozen set:

The prompt edit that fixes case A and breaks case D. The most common one. You improve something in response to a complaint, ship it, and quietly regress a case nobody complained about yet.
A model version bump. Behaviour shifts at the margins. Your happy path is fine. Your two weird cases aren't.
Parameter drift. Somebody changes temperature during debugging and doesn't change it back.
Length-induced behaviour change. Input grows over months; the model starts truncating or de-prioritising instructions that used to hold. I've written about seeing this before you ship — the frozen set is how you find out it already happened.
A caching boundary move. Someone interpolates a timestamp into what used to be a stable prefix. Cost triples. Nobody's watching that number either.

None of these announce themselves. All of them show up as a diff in a
twenty-line script.

Where this is weaker than it sounds

I'd rather say this than have you find out.

You can freeze a bug. If the day-three output was subtly wrong and got a
yes, you've now enshrined it and your harness will defend it. The mitigation is
unglamorous: revisit the frozen set once you know more, and treat early
acceptance as provisional rather than permanent.

Small N misses things. Ten cases will not cover a workflow with real
variety. It's a floor, not a ceiling — but a floor you have on day three beats a
ceiling you get in month three.

It says nothing about quality. A regression harness will happily confirm
that your mediocre system is still exactly as mediocre as it was. That's a real
limitation, and it's why this is the first eval rather than the only one.

The upgrade path is free

Here's the part I like.

Every failure the regression harness catches is a labelled example. Case D broke
when you fixed case A? You now have a case with a known-correct answer and a
known failure mode, discovered from real behaviour rather than imagined during a
planning meeting.

After a month of this you're not starting a golden dataset from zero. You're
curating one that assembled itself out of actual failures — which is a
substantially better dataset than one written in advance by someone guessing at
what might go wrong.

The cheap eval isn't a compromise you eventually replace. It's the collection
mechanism for the expensive one.

Why this is the deliverable

I've made a version of this argument before, for a solo developer choosing
between two prompts — that you don't remember your prompt being better, you
just remember it being better.
That post was about self-deception on a personal scale, and the stakes were an
afternoon.

The stakes change when you hand something over.

When you leave, the prompt is a text file. Someone will edit it — reasonably,
for a good reason, in response to a real complaint. That's not a failure mode,
that's the system working. The failure mode is that they have no way to know
what their edit cost somewhere else.

A prompt without an eval is a config file nobody is allowed to touch, which
means either it never improves or it degrades unpredictably. Both are how a
deployment dies quietly. A prompt with a frozen set is something a team can
actually own after you're gone — and whether they can own it is the thing that
decides if the work survives.

Which is the real subject of the next post: not what you build, but what you
leave behind.

Part three of a series on the last mile of enterprise AI.
Part one: Nobody's model failed. The interface did. ·
Part two: I accidentally built a forward-deployed engineer's field kit

This post is mirrored from ferhatatagun.com/blog/the-eval-is-the-deliverable — that's the canonical URL.

More from the same place:

Long-form blog — AI, LLM tooling, frontend at the model boundary
The five tools — browser-only, BYOK, open-source Claude API dev tools
GitHub · X · LinkedIn

Happy to discuss here or on the canonical post — both threads stay open.

I accidentally built a forward-deployed engineer's field kit

Ferhat Atagün — Mon, 10 Aug 2026 07:48:53 +0000

A year ago I deleted @anthropic-ai/sdk from a project and wrote about 150
lines of TypeScript to replace it.

The reason I gave myself was honest and boring. The SDK pulled node:fs/promises
in through an agent-toolset module, which broke the browser bundle. I could have
waited for a browser-clean entry point. Instead I hand-rolled an SSE parser and
wrote a post about why.

I filed that away as a bundler story.

Then I read a line in a piece about tooling for forward-deployed engineers — the
people who get sent into a customer's building to make an AI deployment actually
work. Roughly: every framework dependency is something the customer's security
team will ask about, and bare-metal code is easier to debug in an environment
that isn't yours.

Same decision. Same reasoning. Different discipline, different job title,
different problem entirely — except it wasn't a different problem.

I hadn't been solving a bundler issue. I'd been solving a deployment problem
without ever naming it, which is why I mistook it for taste.

This post is what happened when I went back through every architectural decision
in the five tools I've shipped and asked a question I'd never asked: what
constraint was actually operating here?

TL;DR

Five tools, all browser-only, all BYOK, none with a backend. I picked those for what felt like preference and mild laziness.
They are, almost exactly, the constraints of working inside someone else's regulated environment: nothing to install, no data leaving the boundary, nothing for a security team to threat-model, minimal dependency surface to defend.
Constraints converge. Two disciplines solving the same shape of problem land on the same engineering decisions without talking to each other.
Mapped against the four categories FDE tooling usually gets split into — observability, evaluation, orchestration, guardrails — I have three. The fourth is missing, and pretending otherwise would be the least interesting thing I could do.
The transferable part isn't my repos. It's that an articulated constraint is what separates a tool from a toy, and most side projects never articulate one.

The decision, and the reason underneath it

Here's the thing about the SDK call that I got wrong at the time.

I framed it as: the SDK doesn't work in my environment, so I'll write the
minimum that does. Reasonable. What I actually did was reduce the tool's
dependency surface to fetch and a TextDecoder — and the consequence of that,
which I did not think about even once, is that the entire runtime is
inspectable. There's no framework doing something clever between my code and the
wire. When a stream terminates weirdly, I read my own parser.

That property is worthless when you're debugging on your own laptop with a
debugger attached and all the time in the world.

It is the whole game when you're on a call with someone else's engineer, in
someone else's network, and they ask why the response cut off, and the honest
answer needs to arrive in the next ninety seconds.

I didn't build it for that. But that's what it is.

Four more decisions, same shape

Once I started looking, the pattern didn't stop.

BYOK — bring your own key. What I told myself: I don't want to run a proxy,
I don't want to hold anyone's credentials, and I really don't want to pay for
strangers' tokens. All true. What it actually is: the customer's key never
leaves their browser and their prompt never touches infrastructure I control.
There's no data-processing agreement to negotiate because there's no data
processing. The "I'm too lazy to run a backend" version and the "this passes a
compliance review" version are the same architecture.

No backend at all. What I told myself: static hosting is free and I don't
want to maintain servers for a side project. What it actually is: there is no
server to threat-model, no attack surface to document, no uptime story to tell,
and no vendor security questionnaire that takes six weeks to clear. The thing a
security team can approve fastest is the thing that doesn't exist.

Every tool is a URL. What I told myself: it's easier to share a link than to
tell someone to clone a repo and run npm install. What it actually is: zero
install. Nothing enters the customer's machine. The people who most need a
diagnostic tool are exactly the people who are least permitted to install one.

Cost rendered live, not logged. What I told myself: the number is
interesting and I wanted to see it move. What it actually is: the person who
approves the renewal can read it without asking anyone. I made an entire
argument about
prompt caching being the cheapest optimization nobody measures, and I still
framed it as an engineering-hygiene issue. It isn't. It's a
trust issue, and trust is
what decides whether a deployment survives.

Four decisions. Four reasons I gave at the time that were true but shallow. One
constraint underneath all of them that I never said out loud.

What browser-only actually costs

I don't want to make this sound cleaner than it is, so let me argue against
myself for a second.

Browser-only is a genuinely bad choice for a lot of software. You get no
server-side secret handling. You fight CORS constantly — and for some providers
you simply lose, because they don't send the headers and there's nothing you can
do about it from a tab. You have no durable storage worth the name, no scheduled
jobs, no background processing, no way to do anything computationally serious.
You can't build a product this way. I'm not going to pretend you can.

But none of those limitations bind on a diagnostic instrument.

A multimeter doesn't need a database. The tool that tells you why the system is
behaving strangely is not the system. It has to be trustworthy, portable, and
readable — and it has to work at the exact moment when everything heavier is
unavailable to you. Every constraint I listed as a cost is irrelevant to that
job, and two of them (no storage, no server) are the reason it can be used at
all in a place that would reject a real deployment.

The limitation and the qualification are the same fact viewed from two sides.

The map

FDE tooling gets described, fairly consistently, in four categories: agent
orchestration, evaluation, guardrails, and observability. I went and put my
things in the boxes.

Category	Tool	What it does
Observability	claudoscope	X-rays a live call — token composition, cache reads and writes, cost, as the response streams
Observability / debugging	agent-replay	Replays a finished agent trace as a timeline instead of a wall of nested JSON
Pre-flight	context-lens	Counts the prompt before you send it — window position, cost, caching boundaries
Evaluation	prompt-lab	Two prompts, one input, side by side on output, latency and cost
Orchestration	tool-lab	Sandboxes the tool-use loop — define tools, mock responses, drive it by hand
Guardrails	—	—

Three categories covered, one pre-flight bonus, one hole.

I want to be careful here, because there's a version of this post that's a
portfolio tour with an FDE hat on, and that version is worthless. So: I didn't
plan this map. It's a retrofit. The tools were built one at a time over a couple
of weekends each, and the only through-line I could have articulated at the time
was "make the Claude API
legible". The
categories came from somebody else's discipline. The fit is real, and it's also
an accident, and both of those things can be true.

The gap I can't paper over

Guardrails is missing, and it's the one that would hurt most in the field.

Guardrails is the layer that keeps model output inside a shape your code can
safely consume. You ask for {"risk": "high"|"medium"|"low", "score": 0-100}.
What actually comes back, across enough calls:

the JSON wrapped in a markdown fence
"HIGH" instead of "high" — enum drift
"very high" — an enum value that doesn't exist, invented on the spot
"score": "85" as a string, silently breaking arithmetic downstream
score missing entirely
a sentence of prose before the JSON starts

Every one of those either crashes your parser or, worse, doesn't — and quietly
puts a wrong category on someone's dashboard.

The reason this needs an instrument rather than a unit test is that the failures
are distributional. You run it once, it works, you ship. Then one call in
forty returns a hallucinated enum, and you never see it, because you looked at
one sample. What you need is the failure rate, broken down by failure type,
across fifty runs. That's a different question than "does it work."

So that's the sixth tool, and it's the honest one to build next: define a
schema, run a prompt against it N times, show the distribution of ways it
breaks. Same constraints as the rest — browser-only, BYOK, no backend, because
by now those aren't a preference, they're the spec.

Why any of this generalizes

Strip out my repos and there's one idea left worth keeping.

Most side projects have no constraint. That's why they read as toys — not
because they're small or unfinished, but because nothing about them was forced.
Any decision could have gone the other way and nothing would have broken. A
reader can feel that.

A tool with an articulated constraint reads completely differently, even when
it's fifty lines. "This runs in a tab because it has to work where nothing can
be installed" is a design brief. "This is a React app" is not. The first one
tells you what the author was up against; the second tells you what they typed.

The uncomfortable part, for me, is that I had the constraint the whole time and
couldn't name it. I shipped five things under a rule I was following
unconsciously, and because I never said it out loud, I also couldn't tell you
what the work was evidence of. It looked like five small tools. It was a
position.

If you're a frontend engineer looking at the current AI hiring market and
wondering how to become legible to it: you very likely already have
constraint-shaped work sitting in your repos. The move isn't to build something
new. It's to go back and figure out what you were actually solving for, and then
say it in the README.

Mine took a year and a stranger's sentence about security teams.

Part two of a series on the last mile of enterprise AI. Part one is
Nobody's model failed. The interface did.
Part three is about the thing I keep saying and haven't yet defended properly —
that on this kind of work, the deliverable isn't the prompt, it's the eval.

This post is mirrored from ferhatatagun.com/blog/accidental-fde-field-kit — that's the canonical URL.

More from the same place:

Long-form blog — AI, LLM tooling, frontend at the model boundary
The five tools — browser-only, BYOK, open-source Claude API dev tools
GitHub · X · LinkedIn

Happy to discuss here or on the canonical post — both threads stay open.

Nobody's model failed. The interface did.

Ferhat Atagün — Mon, 10 Aug 2026 07:41:06 +0000

Three numbers, and they don't sit comfortably next to each other.

Postings for one job title — forward deployed engineer — went up roughly
800% in 2025. Total comp at the frontier labs settled somewhere around
$350–550K for mid-to-senior. And enterprise AI pilots still fail at a rate
that lands somewhere between 70% and 90%, depending on whose survey you
read. One widely-cited figure has the share of companies abandoning AI
initiatives going from 17% to 42% in a single year.

If the models were the bottleneck, the fix would be a better model. Instead the
industry's answer has been to hire humans and put them inside the customer's
building. That is an unusual response to a technology problem. It's the response
you get when the thing that's broken isn't the technology.

So: what exactly keeps failing?

TL;DR

The canonical FDE success story — OpenAI at Morgan Stanley — is about 6–8 weeks of technical scaffolding followed by roughly four months of getting advisors to actually use the thing. The reported win condition is 98% adoption, not accuracy and not latency.
An adoption number is not a model metric. It's a measure of whether a human decided to trust what was on their screen. That decision is made at the interface.
The three places that trust reliably dies — hidden uncertainty, invisible cost, and a system that won't show its work — are all rendering decisions, not modelling decisions.
FDE roles are written and staffed as backend/ML/infra roles. Read a dozen job specs and count how many mention what the operator sees. The failure surface and the hiring surface don't line up.
This isn't an argument that the backend doesn't matter. It's an argument that the backend is the six weeks, and the last mile is the four months.

The number that doesn't fit

Every write-up of the forward-deployed model eventually reaches for the same
case study: OpenAI's engineers embedded at Morgan Stanley, building an assistant
for wealth advisors.

The part everyone quotes is the ending — 98% adoption. The part worth staring at
is the shape of the timeline that produced it. Roughly six to eight weeks of
technical scaffolding: integration, data plumbing, evals. Then about four months
of running pilots with actual advisors.

Four months. After the system worked.

Whatever was happening in those four months, it wasn't model training. The model
was done. The integration was done. What remained was a group of experienced
professionals deciding, one at a time, whether they were willing to put their
name on an answer a machine produced.

That is the last mile. And notice how it's measured: not "the model scored 0.91
on our eval set" but "98% of them use it." A deployment that nobody opens is
indistinguishable from a deployment that doesn't work.

Adoption is an interface number

Here's the reframe I'd argue for.

We treat adoption as a change-management problem — training sessions,
champions, executive mandates. Some of that is real. But most of what determines
whether a professional trusts a system is not a memo. It's the accumulated
experience of using it: what it showed them, what it hid, and whether it was
honest about the difference.

A wealth advisor doesn't trust a model because someone told them its F1 score.
They trust it because over three weeks it never confidently handed them
something wrong without warning them first — and when it was unsure, it said
so in a way they could act on.

That's not a model property. That's a rendering property. It's a decision
somebody made about what appears on screen when the confidence is 0.51 instead
of 0.98.

Three places trust dies — all of them on screen

I'd argue there are three recurring failure modes in AI-powered interfaces, and
none of them is a model defect.

1. Uncertainty is flattened. The model returns something it half-guessed,
formatted identically to something it's certain about. Same font, same
confidence, same tone. The operator has no way to distinguish. They find out the
hard way — once — and after that they double-check everything, which means the
system has stopped saving them time, which means they stop using it.

The fix is a design decision: give low-confidence output a different affordance.
Not a scary red banner. Something that says check this one in a way a busy
person will actually parse at a glance.

2. Cost is invisible until the invoice. The people who approve the renewal
are not the people using the tool. If the only time the budget owner sees a
number is on the monthly bill, every conversation about the system is a
conversation about a surprise. I've written about
why nobody measures prompt caching
and about
pre-flighting a prompt before you send it,
and both of those posts were, in retrospect, about this: cost is a first-class
piece of interface state, and treating it as an ops metric hides it from the
person who decides whether the project survives.

3. The system won't show its work. When output is wrong and the operator
can't see why, they don't file a bug — they lose confidence in the whole
thing, including the parts that were right. A system that can't be inspected
can only be trusted or abandoned, and people abandon.

This is why I ended up building
trace replay for agent runs.
Not because traces are interesting, but because "why did it do that" is the
question that decides whether a deployment survives its first bad week.

Every one of these three is fixed in the interface layer. None of them is fixed
by a better model.

The strongest version of the counter-argument

Let me argue the other side properly, because the weak version is easy to knock
down and I don't want to do that.

The serious objection goes: the last mile isn't a UI problem, it's a data
problem. The customer's schema is undocumented. Half the workflow lives in a
spreadsheet somebody's assistant maintains. The API you were promised doesn't
exist. Nothing renders correctly because nothing is correct upstream. The head
of OpenAI's FDE team has said more or less this — that what a customer describes
during scoping routinely fails to match the reality of the systems on the
ground.

That is true, and it's the hardest part of the job. I'm not disputing it.

But look at the timeline again. That work is the six weeks. It's necessary and
it is absolutely not sufficient, and the evidence is that the project didn't
succeed at week eight — it succeeded four months later, after a completely
different kind of work.

Both halves are real. My claim is narrower than "UI is what matters." It's:
the second half is the larger half, it's the half where projects actually die,
and it's the half almost nobody is being hired against.

The hiring gap

Go read a batch of forward-deployed engineer job specs. You'll see: production
LLM experience, advanced prompt engineering, agent frameworks — LangGraph,
LangChain, CrewAI, DSPy — evaluation frameworks, deployment at scale,
multi-step tool-use chains, sometimes air-gapped and bare-metal provisioning.

All of it legitimate. All of it necessary.

Now count the lines about what the operator sees. About how uncertainty is
surfaced. About whether the person doing the work can tell the difference
between a confident answer and a guess.

The role was defined by people solving the integration problem, and it was
staffed by people who are excellent at the integration problem. Meanwhile the
number everyone reports as the win condition — adoption — is decided somewhere
else entirely.

I don't think this is a conspiracy or an oversight by unserious people. I think
it's what happens when a role gets invented under time pressure by the
discipline that noticed the problem first. Palantir invented this model in the
2000s for intelligence customers who literally could not describe what they
needed. The problem then really was mostly data and access. The problem now has
a large human-facing component, and the job description hasn't caught up.

What I'd actually do this week

If you're shipping an AI feature into somebody else's workflow, three concrete
moves, in order of how cheap they are:

1. Give uncertainty a visual identity. Pick one signal your model already
emits — a confidence score, a refusal, a low-agreement result from two sampled
runs — and render it differently. One afternoon of work. It converts your
system from "trust it or don't" into "trust it here, check it there," and that
distinction is the entire difference between a tool people keep and a tool
people quietly stop opening.

2. Put cost on screen for whoever signs the renewal. Not in a dashboard they
have to remember to open. In the thing they already look at. A number that
updates is a number that never becomes a surprise, and surprises are what kill
renewals.

3. Make one thing inspectable. Pick the single most consequential decision
your system makes and give the operator a way to see how it got there. Not full
observability — one path, one explanation. The first time something goes wrong,
that path is the difference between a bug report and a lost account.

None of these require touching the model. All of them move the number the FDE
model says is the win condition.

Where this goes next

If the argument holds — that the last mile is substantially an interface
problem — then there's a follow-on question worth taking seriously: what does an
engineer working that mile actually carry?

Because the constraints are brutal and specific. You're in someone else's
environment. You can't install anything without a six-week security review. The
data can't leave their boundary. Every dependency you bring is a question their
security team gets to ask. And you still need to answer "why did it do that"
while a stakeholder watches over your shoulder.

I've spent a year building tools under exactly those constraints, and until
recently I thought I was doing it for aesthetic reasons.

That's the next post.

This is part one of a series on the last mile of enterprise AI. Part two —
"I accidentally built a forward-deployed engineer's field kit" — is about the
tools, and the constraint that turns out to define them.

This post is mirrored from ferhatatagun.com/blog/nobodys-model-failed — that's the canonical URL.

More from the same place:

Long-form blog — AI, LLM tooling, frontend at the model boundary
The five tools — browser-only, BYOK, open-source Claude API dev tools
GitHub · X · LinkedIn

Happy to discuss here or on the canonical post — both threads stay open.

How I shipped a blog Google couldn't see

Ferhat Atagün — Mon, 15 Jun 2026 06:57:08 +0000

Every blog post on my site looked fine. Open /blog/something, the
article was there — title, paragraphs, code blocks, the works.

Then I ran curl https://ferhatatagun.com/blog/four-tools-in-two-weekends
on a hunch, and the HTML had zero of the body text. Title in <head>,
layout chrome, a perfectly empty <div class="markdown-container" />, and
nothing else. The article only rendered after JavaScript loaded — meaning
the version Google indexed had no article in it.

This had been the case for every blog post on the site, for months.

TL;DR

The site used adapter-static with prerender = true, which suggests "all routes are rendered to HTML at build time." That's true for the page chrome — but not the body.
The Markdown component parsed content inside onMount, so the HTML on disk had a skeleton and nothing else. The article materialized only after hydration.
Two ways this hides from you: every browser you test in runs the JS, so the page looks fine; and the page reports a healthy 200 response, so monitoring stays green.
The fix is mechanical (parse markdown at module scope, inject via {@html}), but it cascaded: prerender OOMed on the worker heap, the prerender crawler followed embedded .md links and 404'd, and the static adapter's fallback was clobbering the prerendered home. Each problem appeared only because the previous one was fixed.

This is a write-up of the regression: what I missed, how it stayed hidden,
the actual code-level fix, and the three secondary failures that the fix
unblocked.

What the page was actually serving

The Svelte component looked harmless:

<script lang="ts">
    import { marked } from 'marked';
    import { gfmHeadingId } from 'marked-gfm-heading-id';
    import { mangle } from 'marked-mangle';
    import createSanitizer from 'dompurify';
    import Prism from 'prismjs';
    import { onMount } from 'svelte';

    let container: HTMLDivElement;
    export let content: string;

    onMount(() => {
        marked.use(gfmHeadingId());
        marked.use(mangle());
        const sanitizer = createSanitizer(window);
        const parsed = marked.parse(content);
        container.innerHTML = sanitizer.sanitize(parsed);
        Prism.highlightAllUnder(container);
    });
</script>

<div bind:this={container} class="markdown-container" />

The whole thing — parsing markdown, sanitising, highlighting — lives
inside onMount. That callback fires only in the browser, after
hydration. During SvelteKit's prerender pass, onMount never runs.
So the HTML on disk contains exactly what's in the template: an empty
<div class="markdown-container" />.

The article body was being added imperatively to the DOM at runtime.
That's invisible to Google. It's invisible to OG/Twitter card scrapers.
It's invisible to anyone who fetches the URL with curl.

Two checks that would have caught this and didn't:

My browser tabs were always rendering the right thing, because they ran the JS. Testing the live page by looking at it is testing the hydrated version, not the indexed version.
The page returned 200. Uptime monitors stayed green. Status pages stayed green. Lighthouse scored fine, because Lighthouse runs JS too.

The only way to see the regression is to bypass JS. curl does. So does
Googlebot's render preview. So does the View-source feature your browser
hides three menus deep. I'd been opening DevTools to inspect post-hydration
DOM for months, and never View Source on the raw response.

The numbers, once I looked:

$ curl -s https://ferhatatagun.com/blog/four-tools-in-two-weekends \
    | wc -c
32280
$ curl -s https://ferhatatagun.com/blog/four-tools-in-two-weekends \
    | grep -c "claudoscope"
0
$ curl -s https://ferhatatagun.com/blog/four-tools-in-two-weekends \
    | grep -c "TL;DR"
0

The page is 32 KB and contains no part of the article body. "claudoscope"
appears half a dozen times in the post; in the HTML, zero. Same for
"TL;DR". The HTML was 100% layout chrome.

Why `prerender = true` wasn't enough

The static adapter's prerender pass walks every route, calls the page's
load, and renders the resulting component tree to HTML. It runs all the
top-level Svelte component code. What it does not do is run lifecycle
hooks like onMount, because those are explicitly contracted to be
browser-only.

So prerender = true was doing exactly what it advertises. The bug was
that the data dependency lived behind a lifecycle that prerender skips.

The fix is to make markdown parsing module-level, not lifecycle-level:

<script lang="ts">
    import { marked } from 'marked';
    import { gfmHeadingId } from 'marked-gfm-heading-id';
    import { mangle } from 'marked-mangle';
    import { onMount } from 'svelte';
    import 'prismjs/themes/prism-tomorrow.css';

    marked.use(gfmHeadingId());
    marked.use(mangle());

    export let content: string;
    $: parsed = marked.parse(content) as string;

    let container: HTMLDivElement;

    onMount(async () => {
        const Prism = (await import('prismjs')).default;
        await import('prismjs/components/prism-typescript');
        if (container) Prism.highlightAllUnder(container);
    });
</script>

<div bind:this={container} class="markdown-container">{@html parsed}</div>

Three things changed:

marked.use(...) moved to module scope. It now runs both during prerender and during hydration, configuring the same extensions in both environments.
parsed = marked.parse(content) is a reactive top-level statement. It runs synchronously inside the component's render pass, so its output is in the HTML that goes to disk.
Prism syntax highlighting stays inside onMount, dynamic-imported. Prism touches self at import time, which is fine in the browser but not on the prerender worker. Highlighting is cosmetic — losing it on prerender is invisible until JS loads, which is acceptable.

I also dropped DOMPurify. The original code piped marked's output
through it before injecting. That was paying a sanitiser's bundle cost
plus a render-time cost, but the input was our own ?raw-imported
markdown files, not user content. Defending against ourselves was
defensive theatre. If a hostile actor can write to my markdown source,
sanitising the output is the wrong layer to do it at.

The result of the fix:

$ wc -c www/build/blog/four-tools-in-two-weekends.html
45292
$ grep -c "claudoscope" www/build/blog/four-tools-in-two-weekends.html
3
$ grep -c "TL;DR" www/build/blog/four-tools-in-two-weekends.html
1
$ grep -c "<h1 id=" www/build/blog/four-tools-in-two-weekends.html
1

45 KB instead of 32 KB. The 13 KB difference is the article body —
the part Google had been seeing as empty.

The home page was even worse

The home page had a different version of the same problem. SvelteKit's
adapter-static accepts a fallback option for SPA-style hosting; if
a path doesn't have a prerendered HTML file, the server can serve the
fallback and let the client-side router resolve it.

The config was:

fallback: 'index.html'

Which writes the fallback shell to index.html. The home route at /
also prerenders to index.html. So you have two operations writing to
the same path. The fallback wins, because the adapter writes it after
the prerender. The 40 KB prerendered home gets overwritten with the
13 KB SPA shell.

$ wc -c www/build/index.html
13096

That's the bare HTML that the bundler emits as the SPA's entry point —
just the imports for the JS bundles, no body content. Anyone hitting /
with a non-JS user agent was getting that.

The fix is one character of intent:

fallback: '200.html'

200.html is a convention some static hosts (Surge, Netlify with
configuration) use to mean "the SPA fallback." The static adapter
doesn't care about the name; it just writes the fallback to whatever
path you give it. Renaming to 200.html keeps the fallback for unknown
paths without colliding with the prerendered home.

$ wc -c www/build/index.html
40871

3.1× growth, all of it actual rendered home content.

The three problems the fix uncovered

Each of these only became visible because the previous one was fixed.

1. Prerender worker OOM

Once the markdown was actually being parsed during prerender, the
GitHub Actions build started failing with:

Error [ERR_WORKER_OUT_OF_MEMORY]: Worker terminated due to reaching
memory limit: JS heap out of memory

The default Node heap on ubuntu-latest is about 1.4 GB. The
prerender pass was now doing real work — marked.parse on every blog
post markdown source, each producing 10–15 KB of HTML. Across 14 blog
posts plus the markdown rendering inside other routes, that pushed the
worker over the line.

Two fixes:

env:
  NODE_OPTIONS: --max-old-space-size=4096

That alone unblocked it. Belt-and-suspenders, I also guarded the
marked.use(...) calls so they only configure once per Node process,
in case Vite's SSR ever re-imports the module across routes:

const __markedKey = '__omni_marked_configured__';
const __markedScope = globalThis as unknown as Record<string, boolean>;
if (!__markedScope[__markedKey]) {
    marked.use(gfmHeadingId());
    marked.use(mangle());
    __markedScope[__markedKey] = true;
}

2. The prerender crawler followed every link in the rendered HTML

Several blog posts embed relative links to translations or contributing.md
files that live in the source repos they reference — [/i18n/README.tr.md],
[/contributing.md], that kind of thing. When markdown rendering was
client-side, those got hydrated into <a> tags but the prerender crawler
never saw them.

Now the crawler sees them, follows them, and treats the 404s as build
errors:

Error: 404 /contributing.md (linked from /skills/nextjs)

These are not site routes I own — they're content links inside post bodies.
The fix is to demote prerender 404s from errors to warnings:

prerender: {
    handleHttpError: 'warn',
    handleMissingId: 'warn'
}

3. The shared host wasn't pulling from gh-pages

The repo's CI deploys to gh-pages. The ferhatatagun.com domain
points at a Spaceship shared host that I FTP-upload to. The two were
unrelated, which meant every CI deploy updated gh-pages and the live
site stayed exactly as it had been.

This wasn't a CI bug; it was a deployment-pipeline shape I'd let drift.
The fix isn't code — it's "manually FTP the gh-pages contents to the
shared host's public_html," or rebuild the deploy pipeline to push
directly. For one-shot remediation, I went with the FTP path.

How to catch the next one before it ships

The reason this regression survived for months is that none of my
verification ran with JS disabled. To prevent the next one:

Always view source on a critical route after a deploy. Not DevTools — that shows the hydrated DOM. The browser's "View Source" shows what arrived on the wire. The two should differ in trivial ways (hydration markers, attribute order); they should not differ in content.
curl | grep your most important sentinel. For a blog: a phrase you know is in the body. For a product page: the price. For a marketing page: the value prop. Make it a 10-second post-deploy check.
Test once with JavaScript disabled. It's a one-time check per major template change. The first time a critical body of text is missing from the JS-disabled page, you have the answer.
For static sites, diff the page sizes you ship over time. A 40 KB → 13 KB drop on a single route would have lit up. I had no alert because I'd never measured a baseline.

For SvelteKit specifically, the pattern is one rule: anything that
materializes data into the rendered DOM should be reactive or
top-level, not in onMount. onMount is for browser-only side
effects — DOM measurement, third-party widget init, anything that
needs window. As soon as you put content production in there, the
prerender stops seeing it. The same shape exists in React (useEffect),
Vue (onMounted), and every framework that distinguishes hydration
from render.

What this cost

Two evenings to find it. Forty-five minutes to fix it. Three
follow-up commits to deal with the secondary failures the fix exposed.

The harder cost is the months of indexing where every post's body was
empty. Google's view of those pages now has the title and the OG image
and an empty <div>. The dev.to mirrors of seven of the posts, which I
published with canonical_url pointing back to my site, were more
indexable than the originals.

Search engines will recrawl. The mirrors will eventually catch up. But
this is the kind of bug that doesn't reverse itself instantly — the
right move after the fix is to submit a fresh sitemap, request reindex
on the most important URLs, and wait.

The win-condition I'm watching for is the same curl | grep that
revealed the bug, run against the production URL:

$ curl -s https://ferhatatagun.com/blog/four-tools-in-two-weekends \
    | grep -c "TL;DR"
1

Zero is the bug. One is the fix.

This post is mirrored from ferhatatagun.com/blog/how-i-shipped-a-blog-google-couldnt-see — that's the canonical URL.

More from the same place:

Long-form blog — AI, LLM tooling, frontend at the model boundary
The five tools — browser-only, BYOK, open-source Claude API dev tools
GitHub · X · LinkedIn

Happy to discuss here or on the canonical post — both threads stay open.

See the prompt before you ship it

Ferhat Atagün — Mon, 08 Jun 2026 11:29:39 +0000

The way most teams find out their prompt is too long is in the bill. The way most teams find out their prompt is approaching the context window is when the model starts dropping the system instructions. The way most teams find out their prompt-caching boundary is in the wrong place is by graphing a hit ratio that won't climb above 30%.

All three of these are diagnosable in advance, in about four seconds, for free. The reason they keep happening is that the tools every Claude developer reaches for — chat playgrounds, IDE plugins, the official SDK — are post-hoc. They show you what just happened. None of them shows you what your prompt looks like before you press send.

The other four tools I've shipped in this suite are all post-hoc too. claudoscope x-rays a finished response. agent-replay scrubs a finished trace. prompt-lab compares two finished runs. tool-lab sandboxes the agent loop. They're all "look at what just happened" microscopes. None of them is a "look at what you're about to do" lens.

context-lens is. Paste a system prompt and a user message; see exactly how the API will count them, where in the 200K window you sit, where caching boundaries fall, and what each call will cost. The pre-flight check that turns a guess into a measurement.

TL;DR

Token cost, context-window position, and prompt-caching layout are all knowable from the prompt alone — you don't need to send the request.
Anthropic's count_tokens endpoint gives you the exact number; a ~3.7 chars/token heuristic gives you a good-enough number while you type.
The most useful single number is "tokens × calls/day × dollars/token" — once you can compute it before deploying, "ship this prompt" stops being an aesthetic call and becomes a budget call.
A 4× difference in input length between two equivalent prompts is normal. Catching it before it goes to production saves more than the tool costs to build.

What you can actually pre-flight

Three things, all derivable from the prompt text alone:

1. Exact token count. Not an estimate. Anthropic ships a /v1/messages/count_tokens endpoint that takes the exact same shape as /v1/messages (system, messages, tools) and returns just the input_tokens number. Same tokenization as the actual API call would use. No model invocation, no output, no cost beyond a single tiny request.

2. Position in the context window. Sonnet 4.5 has a 200K-token window. Going past it doesn't error; the model silently drops the oldest content, which usually means dropping your system instructions, which usually means the model stops doing what you asked. The math is (input + max_output) / 200_000. You should never see "78% of window" in production without knowing about it.

3. Cost per call. Multiply input tokens by input price ($3/M on Sonnet), output tokens by output price ($15/M), and you have one number for the cost of one call. Multiply by your traffic and you have the bill. The interesting move: do this before you commit to a prompt design, not after.

The fourth thing — where prompt-caching boundaries should sit — is harder to derive purely from text, but it's still pre-flight: you choose where to put cache_control based on which prefix is stable across your real traffic. context-lens won't choose for you, but it will show you the boundaries you've chosen so you can sanity-check them.

The four-fold cost difference no one was looking for

A real example, the worked-out kind. Two versions of the same agent system prompt:

Version	Approach	Input tokens (counted)
A	Markdown headings, examples, long taxonomy, JSON schema embedded	3,847
B	Single paragraph, schema implied by one example, no preamble	612

Same model (Sonnet 4.5). Same user inputs (a code review task). The output was substantively equivalent on five real traffic samples — both caught the same critical bugs, both produced valid JSON, both came in under 800 output tokens.

The cost differential is mechanical:

A: (3847 × 3 + 800 × 15) / 1_000_000 = $0.0235 per call
B: (612 × 3 + 800 × 15) / 1_000_000 = $0.0138 per call

At 10,000 calls per day, that's $97/day saved, or $3,000/month. For a single prompt rewrite that took two hours to test in context-lens.

The salient detail: I didn't intend version B to be cheaper. I intended it to be more readable. The cost reduction was a side-effect that I would not have noticed without the pre-flight number, because both prompts felt "about the same length" to me in an editor. context-lens told me one was 6.3× the length of the other, in the only metric that matters: the metric the API uses.

The lesson is that "feels about the same" is a uniformly bad estimator for token count, and you stop making the mistake the day you start measuring before you ship.

Why the heuristic mode exists

context-lens does two things:

Live as you type: a fast heuristic, roughly 3.7 chars/token for English-ish text, that updates with every keystroke. No API call, no key required, instant.
On demand: a real API call to count_tokens that gives you the exact number Anthropic will use.

The heuristic isn't quite right — Turkish, code, and JSON all tokenize differently than English prose, sometimes by 30%. But it's a real-time signal while you iterate, which is more useful than an accurate-but-asynchronous one while you write. When you're ready to commit, you click the button and get the exact number. The two modes are intentional: one for the iteration phase, one for the verification phase.

The pattern generalizes. Every place you have a fast-approximate metric and a slow-exact one, ship both, label them clearly, default to the fast one. The fast metric should never be wrong by more than ~30%; otherwise it's not a useful approximation. ~3.7 chars/token meets that bar for the languages context-lens has to handle.

What about prompt caching

Caching is the lever most teams underuse — and the one context-lens helps with most by surfacing where the boundaries are. Anthropic lets you mark any segment of your prompt as cacheable with cache_control: { type: "ephemeral" }. The next 5 minutes, requests that share that exact prefix get the cached portion at 10% of the input price. The math flips: a 4,000-token system prompt that costs $0.012 per cold call costs $0.0012 per warm call. That's 10×.

The catch: every byte before the cache_control boundary must be identical. If you interpolate the user's name into the system prompt — gone. If your tool list reorders between requests — gone. If you append a timestamp — gone.

context-lens shows you the structure you're sending. It doesn't auto-detect cacheability for you, but it does let you toggle "assume input is cache-read" and see what the cost would be if your caching worked. If $0.012 → $0.0012 is interesting at your traffic level, the path to verify it works is in claudoscope, which shows you the actual cache-read and cache-write breakdown on a live call. The two tools are complementary: context-lens predicts, claudoscope measures.

I wrote a longer piece on the caching observability case in Prompt caching is the cheapest Claude optimization. Nobody measures it. if you want the full argument.

What I'd recommend you do this week

Three escalating moves:

Today (5 minutes): Take whatever prompt your team is shipping right now. Paste it into context-lens with a representative user message. Note the token count. Now write a 1-paragraph version of the same prompt and paste that. If the count drops by 50% with no quality regression on three real inputs, you have a free production cost cut.
This sprint (an afternoon): Add a pre-merge step to your prompt-change workflow: every PR that touches a prompt must include the context-lens token counts (before / after) in the description. Same energy as showing test results. If a PR triples your input tokens, that should be a conversation, not a stealth deploy.
This quarter (a habit): Track the prompt-cost-per-feature number across your product as a real metric. If feature X costs $0.02/call and feature Y costs $0.20/call, that's information you should know about before the bill teaches you. context-lens is the cheapest place to start collecting it — count_tokens is free to call.

The economics of LLM apps in 2026 are not about model selection, mostly. They're about prompt design. Teams that can see their prompts before they ship them will out-compete teams that can't, on cost first and on quality second. The "see them" part is what's missing in most setups, and what context-lens is for.

I shipped this in context-lens — paste a Claude prompt, see what it costs before you ship. BYOK, no backend, runs in the browser. Source: github.com/ferhatatagun/context-lens.

The same protocol-level approach also powers four sibling tools — claudoscope, agent-replay, prompt-lab, tool-lab. All open source, all BYOK: ferhatatagun.com/tools.

This post is mirrored from ferhatatagun.com/blog/see-the-prompt-before-you-ship-it — that's the canonical URL.

More from the same place:

Long-form blog — AI, LLM tooling, frontend at the model boundary
The five tools the post discusses — all browser-only, BYOK, open-source
GitHub · X · LinkedIn

Happy to discuss here or on the canonical post — both threads stay open.

What I learned shipping four open-source Claude dev-tools in two weekends

Ferhat Atagün — Mon, 08 Jun 2026 11:29:08 +0000

About a month ago I tried to import the Anthropic SDK into a Next.js project and the bundler crashed. The fix was straightforward — talk to the Messages API directly, ~150 lines of TypeScript replacing the SDK — but the side-effect was that I now had a hand-rolled SSE client lying around, with all of Claude's streaming behaviour visible to me at the protocol level for the first time.

That client became the seed of four small open-source tools, shipped over two weekends. Each one points a different microscope at the same protocol:

claudoscope — live x-ray of token economics: input, cache write, cache read, output, all visible as the response streams.
agent-replay — paste a Claude agent trace, replay it step-by-step on a cinematic timeline.
prompt-lab — run two prompts (or models) on the same input, side by side, with output/cost/latency compared.
tool-lab — define Claude tools in a JSON editor, type the mock responses by hand, watch the agent loop play out.

All four run only in your browser, BYOK, no backend, MIT-licensed. Together they're around 400 KB gzipped; the shared SSE client is the same file in all four repos. Five long-form posts on ferhatatagun.com/blog and Medium document the engineering decisions behind each one.

The work is done — the more interesting question for me now is what shipping them in this shape, on this timeline, taught me about building developer tools in the AI-tooling era.

TL;DR

Resistance from the official SDK ended up being the most generative constraint. Without the crash, I would never have written the parser, and without the parser, I would never have noticed how much the SDK hides.
"One tool per insight" beats "one tool for everything." Each of the four tools makes exactly one thing visible. They compose because they don't try to.
BYOK + browser-only is a credibility multiplier. The threshold for "I'll try this" drops dramatically when there's no account to make and no server to trust.
A <150-line shared protocol client across four projects is a more interesting reuse pattern than "extract into a library." It travels by copy-paste, but with intent.
The articles are not promotion; they're scaffolding. Every tool needs a long-form artifact that explains why it exists, not what it does.

The constraint that made the work possible

If the Anthropic SDK had imported cleanly into my Next.js bundle, none of this exists. I would have used the SDK, never seen the SSE event stream, never realized that the four usage fields are sitting there in every response, and shipped some boring product feature instead.

What broke first was the bundler — node:fs/promises from inside an agent-toolset module, deep in the SDK's transitive imports. The fix wasn't subtle: don't use the SDK. Talk to api.anthropic.com directly with fetch. Add the anthropic-dangerous-direct-browser-access header. Parse the SSE stream by hand. About 150 lines.

The interesting part wasn't the parser — it was what I saw because of the parser. I'd been calling Claude for months without ever noticing that cache_creation_input_tokens and cache_read_input_tokens were distinct fields. I'd never looked at the granular order of content_block_delta events. I'd never noticed that tool_use inputs arrive as partial-JSON deltas you have to accumulate. The SDK had been doing me a favor by hiding this stuff, and I'd been doing my apps a disservice by letting it.

The lesson, restated: when an SDK fights you, the fight is the gift. The work to bypass it gives you ground-truth visibility you'd never have bought yourself.

One tool, one thing it makes visible

The temptation, once I had the SSE parser, was to build "a Claude developer dashboard" — one tool that did everything. I almost did. The reason I didn't is that the most useful diagnostic tools I've ever used (Wireshark, Chrome DevTools' specific panels, the React Profiler) all share a property: each panel makes exactly one thing visible in a way no other tool does.

So I broke the work into four:

Tool	Makes visible
claudoscope	The four `usage` fields, live, as cost in dollars
agent-replay	The decision sequence inside a `messages` array
prompt-lab	The latency/cost/output diff between two variants
tool-lab	What the model actually does with your tool schemas

Each is a small surface area. None of them does the other three's job. They're all the same shape — paste-some-JSON, watch-some-output, see-the-thing — but the "thing" is intentionally different in each.

This decomposition cost me something: I have four landing pages to maintain, four READMEs, four sets of cross-links. But it bought me an asymmetric thing: a clear pitch per tool. "X-ray a Claude API call" is easier to share than "an all-in-one Claude developer console." On a Show HN front page or a Twitter timeline, the small specific claim wins.

BYOK + browser-only as a trust multiplier

The first version of each tool, in my head, had a backend. A small Node service, an API key kept server-side, maybe a rate limiter. I started building the first one this way, then stopped at the deploy step and asked: why am I making the user trust me with their key?

There is no good answer. For a developer tool that the user is going to use for ten minutes to debug their own work, no backend is necessary. Their key, their requests, their data. The browser is the right runtime; localStorage is the right persistence layer; "nothing leaves your tab" is the right privacy guarantee.

What this changed: the "try it" threshold collapsed. No account creation. No OAuth dance. No "should I trust this site with my key?" hesitation. Open the URL, paste a key, hit send. The tool is yours in under thirty seconds. The Anthropic header named anthropic-dangerous-direct-browser-access was clearly built for exactly this kind of usage — a developer wants to look at the protocol directly, on their own machine, with their own credentials.

The flip side: this design only works for developer tools used by their own creator. A production app that ships keys to users would still need a backend. But for the diagnostic case, BYOK + browser-only is the right architecture.

A 150-line client, copied four times

The shared SSE streaming client is src/lib/anthropic.ts in all four repos. Same file. Same 150ish lines. I considered extracting it to an npm package — @ferhatatagun/claude-fetch or similar — and decided against it three times.

The case against extraction is intuitive once you've worked at scale: a shared library across four tools creates a fan-out problem. A breaking change in the library breaks all four; a non-breaking change requires version-pinning logic; a hotfix requires four PR's to deploy. Meanwhile the four tools are small enough that the file is reviewable in five minutes. There's nothing to abstract over.

What I do instead: the file at the top of src/lib/anthropic.ts in each repo says, in a comment, where it was last synced from. When I improve the parser in one tool, I diff the file across the four repos and reconcile. It takes minutes, not hours, and the four tools stay in sync without the ceremony of a published package.

This isn't a universal pattern — for ten projects it would break down, for a hundred it's clearly wrong. But for four tools shipped by one person on weekends, it's strictly better than the npm-and-versioning alternative.

The articles aren't marketing — they're scaffolding

Each of the four tools has a long-form post that explains why it exists. claudoscope has two (one on the streaming client itself, one on cache observability). prompt-lab, tool-lab, and agent-replay each have one. There are also five matching Turkish translations on ferhatatagun.com.

These posts are not promotion in the marketing sense. I'm not optimizing them for SEO and I'm not pumping them on LinkedIn for impressions. (OK, I'm pumping them on LinkedIn a little. But that's not the point.)

The point is: a tool that does one specific thing benefits massively from an artifact that explains why that specific thing is worth doing. "Here's a tool to A/B test Claude prompts" is a less convincing pitch than "you're choosing prompts by vibes; here's what side-by-side reveals that sequential doesn't, with a worked example, and a tool for it." The article does the persuasion; the tool catches the convinced reader.

Without the writing, the tools look like toys. With the writing, they look like the natural conclusion of an argument. The two work as a pair.

What I'd do differently

A handful of small things I'd front-load if I were starting over:

Demo mode from day one. I added ?demo=1 to three of the four tools as an afterthought. It's the single highest-conversion feature — users who land on a tool and don't have a key still need something to look at, or they bounce. Should have been there at first commit.
Per-tool OG cards. I shipped each tool with a generic OG image and went back two days later to make per-tool 1200×630 cards in the right brand color. The first two days of traffic that came in via shared links looked generic. Should have been there at launch.
Cross-linking inside the tools. Each tool's footer points to the other three. I added this in the second weekend. The first weekend, every tool was a silo, and visitors discovered them one at a time. Should have been baked into the template.
A "what's in this for me" line on the landing page. I had four hero descriptions like "see what Claude is doing." Better: "see prompt caching save you 90% of your bill, live, as you debug." Specific outcome > vague capability. I corrected this in the second pass.

None of these are large fixes. They're all things that, if you've ever shipped a small developer tool, you already know. Knowing and remembering at the moment of shipping are different things.

What's next

Whatever the API surface adds next, the same pattern applies: ship a small visualizer for it the day it lands. Anthropic shipped MCP, batch, files, computer-use, and citations over the last year, and most of them still don't have great developer-side observability tools. Each one is a 200-300 line tool waiting to be built.

For now, the four-tool suite is at a natural stopping point. The work I'm interested in now is around adoption — making it visible enough that the people who need these tools can find them. If you've read this far and one of the four sounds like it would have saved you time last week, take it for a spin and let me know what's missing.

All four tools: ferhatatagun.com/tools.

Source on github.com/ferhatatagun. MIT, BYOK, no backend.

Articles on each one: ferhatatagun.com/blog.

This post is mirrored from ferhatatagun.com/blog/four-tools-in-two-weekends — that's the canonical URL.

More from the same place:

Long-form blog — AI, LLM tooling, frontend at the model boundary
The five tools the post discusses — all browser-only, BYOK, open-source
GitHub · X · LinkedIn

Happy to discuss here or on the canonical post — both threads stay open.

How I debug Claude agents by replaying their trace

Ferhat Atagün — Mon, 08 Jun 2026 11:28:37 +0000

Your agent did something weird in production. A user reported it, you found the failed run in your logs, and you're now staring at a JSON file that's 400 messages long, half of them are tool_result blocks the size of small databases, and somewhere in there is the moment the agent decided to do the wrong thing.

You can't re-run the agent: the API state has moved on, the tool would behave differently now, the prompt has been updated three times since. You have only the trace.

The way most of us read agent traces is: open the JSON in an editor, ctrl+F for the tool name we suspect, scroll through walls of escaped strings, try to mentally reconstruct the sequence. It takes thirty minutes, by the end of which you have one of three answers — "yeah I see what went wrong," "I'm pretty sure I see what went wrong," or "I have no idea what went wrong." About a third of the time it's the third one, and you go ship a band-aid that may or may not fix the actual problem.

The thing nobody talks about is that this isn't a hard problem. The JSON contains all the information. The issue is purely presentational — it's nearly impossible to read.

TL;DR

Agent traces are a sequence of decisions but stored as a wall of nested JSON. The signal is there; the format is the problem.
The right primitive isn't a JSON viewer — it's a timeline. Each thought, tool call, tool result, and final answer becomes its own discrete, color-coded step.
Once you can scrub through the trace step by step, the failure point becomes visually obvious in seconds instead of minutes.
This is post-hoc, not interactive. You don't need to re-run the agent or hit the API — replay works on the raw trace alone.
A browser-only tool can do this in 4 seconds. No backend, no key, just paste the JSON.

What an agent trace actually contains

When you save a Claude agent run, you usually persist the messages array — the full conversation including the model's responses and the tool results you fed back. A six-step agent run looks roughly like:

[
  { "role": "user", "content": "Find me the cheapest flight from IST to LAX next Tuesday" },
  { "role": "assistant", "content": [
    { "type": "text", "text": "I'll search for flights and check prices." },
    { "type": "tool_use", "id": "tu_01", "name": "search_flights", "input": {...} }
  ]},
  { "role": "user", "content": [
    { "type": "tool_result", "tool_use_id": "tu_01", "content": "[<2KB of JSON>]" }
  ]},
  { "role": "assistant", "content": [
    { "type": "text", "text": "Looking at three of those..." },
    { "type": "tool_use", "id": "tu_02", "name": "get_price", "input": {...} }
  ]},
  // ...four more steps...
]

Every interesting moment of the agent's behaviour is in there: which tool it picked, what arguments it constructed, what it said about its own reasoning, how it interpreted the result. The structure is fundamentally a sequence of discrete events, not a "document."

But you read it as a document, because that's what an editor shows you. The brain has to do the work of converting "alternating role: assistant / role: user with tool_result content blocks" into "step 3 was a tool call to get_price with argument X, which returned Y, which the agent then interpreted as Z."

That conversion is what kills your debugging time. Doing it manually for a 12-step trace takes minutes. Doing it for a 60-step agent on a complex task takes hours.

The right primitive: a timeline of decisions

The reframe is: stop reading the trace as JSON, start watching it as a sequence of decisions. Each step is one of:

💭 Thought — the model wrote text (the part of its response that isn't a tool call)
🔧 Tool call — the model invoked a tool with specific arguments
📥 Tool result — what came back, fed into the next turn
✅ Final answer — the model's end_turn, no more tools

Color-code those four event types. Lay them out in order, one card per event. You now have a timeline you can scrub, step through, and play back. The information density per card is high enough that you can read the entire trace at a glance, and zoom in only on the cards that look suspicious.

The structural insight: agent debugging is closer to debugging a script with breakpoints than to reading source code. You want to step through, not skim. JSON gives you no steps; the timeline gives you nothing else.

The bugs that become obvious in this view

Three failure modes I see repeatedly when I drop a trace into the timeline:

1. The wrong tool, picked silently. The model called search_archive when it should have called search_recent. In JSON this is one line out of 200 that flies past your eye. In the timeline it's a card with a tool name you didn't expect, and you click on it.

2. Hallucinated arguments. The model called the right tool but with an argument shape that doesn't match the schema — usually because the schema is ambiguous. In JSON you see {"q": "foo", "limit": "10"} and don't notice that limit should have been an integer. In the timeline the tool result card right after shows a 400 error and you trace it back one step.

3. The infinite loop precursor. Some agents get stuck in a pattern where they keep calling the same tool with slightly different inputs, never reaching a conclusion. In JSON it's a wall of near-identical blocks. In the timeline it's a visual rhythm — five purple cards in a row with the same tool name — that you can see in your peripheral vision the moment you scroll.

In all three cases, the bug isn't subtle. It just looks subtle when it's hidden in JSON.

What replay gives you that re-running doesn't

The temptation when an agent fails is to re-run it with print statements, see what happens, iterate. Don't. Three reasons:

It costs API calls. A failed agent that called 15 tools costs you 15× input tokens to re-run. With caching maybe less; either way, the bill is real. Replay is free.

The API state has moved. The tool you call today might return different data than the tool returned during the original run. You're not debugging the original failure anymore; you're debugging whatever happens now, which might be a totally different bug.

The model is stochastic. Even at temperature 0, retries can produce different outputs. Re-running an agent and getting a different failure mode means you've now got two bugs to investigate. The trace is the canonical artifact of what actually happened.

Replay sidesteps all three. You're inspecting a frozen artifact, deterministically, at whatever speed you want. The bug doesn't move while you're looking at it.

What this looks like in agent-replay

agent-replay is the tool I built for this. Paste your trace into a JSON pane on the left. The right pane renders it as a cinematic timeline:

Each event is a card with an icon and color
You can press space to play through the trace at 1× speed (one event per second), or scrub manually
Click any card to see the full content — the thought text, the tool call's input JSON, the raw tool result, expanded
Filter by event type — "show me only the tool calls" or "show me only the assistant thoughts" — when you want to focus
The whole thing is in your browser; no key needed, no backend, your trace never leaves the tab

There's a sample trace seeded on ?demo=1 if you want to see what a 12-step agent looks like without copying your own data anywhere.

The thing I keep finding: the moment I'm debugging is no longer "where in the JSON did the agent screw up." It's "which card looks wrong, and what does the next card show as a consequence." A 30-minute investigation becomes a 30-second one. Not because the tool is doing anything clever — it's just showing the same data in the right shape.

What I'd recommend you do this week

Three escalating moves:

Today (5 minutes): Find the last weird agent run you have a trace for. Paste it into agent-replay. See how long it takes to find the failure point. If it's faster than your usual JSON-scrolling approach, you just changed your debugging workflow.
This week (an afternoon): Add a trace-export endpoint to your agent. Every agent run, finished or failed, dumps the messages array to S3 or a database row. You need the trace before you need to debug it, not after.
This quarter (a habit): When a user reports "the agent did something weird," your first move is to pull the trace and open it in a timeline view, before you read the user's report carefully. Most of the time you'll know what happened before you finish reading the bug report.

Agent debugging is presented as an emerging engineering discipline. It isn't — it's a tooling problem we've solved many times before for non-AI systems. We just haven't built the tools yet for this one. Once the trace is in the right shape, the bugs are obvious. The work is laying out the data, not interpreting it.

I shipped this in agent-replay — paste a trace, scrub the timeline. No key, no backend, runs in the browser. Source: github.com/ferhatatagun/agent-replay.

The same SSE client (for traces that include streaming events) also powers three sibling tools — claudoscope, prompt-lab, tool-lab. All open-source, all BYOK: ferhatatagun.com/tools.

This post is mirrored from ferhatatagun.com/blog/debug-claude-agents-by-replaying-traces — that's the canonical URL.

More from the same place:

Long-form blog — AI, LLM tooling, frontend at the model boundary
The five tools the post discusses — all browser-only, BYOK, open-source
GitHub · X · LinkedIn

Happy to discuss here or on the canonical post — both threads stay open.

Build the sandbox before you write a single tool

Ferhat Atagün — Mon, 08 Jun 2026 11:28:05 +0000

The first time you ship a Claude agent that uses tools you'll do it the obvious way: design the schema, write the actual tool function, hit the API, parse the tool_use block, run the function, feed the result back, loop. It works. It also has a fundamental ordering bug:

You wrote the tools before you knew if they were the right tools.

By the time you've stood up a database query function, two API calls, and a thing that hits the file system, you've sunk maybe a day. You run the agent. It calls a non-existent tool. It hallucinates an argument shape that doesn't match your schema. It picks the wrong tool when both would have worked. Now you're going to redesign the schema, and the four real tool implementations you wrote are going in the bin or being rewritten.

The thing that makes this worse is that the failure mode looks like an "agent quality" problem when it's actually a "premature implementation" problem. The model knew what it wanted; you'd just built the wrong scaffolding around it.

TL;DR

Tool implementations are the slowest part of agent development; tool design is the fastest part to get wrong.
Decouple them: write the tool schemas, run the agent loop with mocked responses, see how the model picks and uses the tools — then write the real implementations only for the tools that survived.
The right mental model is "you play the role of every tool, by hand" — slow for the agent, fast for you, brutal for bad designs.
This is a fifteen-minute exercise for a five-tool agent that would otherwise take a day, and it catches design mistakes before they touch your codebase.
The whole thing fits in a browser tool with no backend.

What "premature implementation" actually looks like

A worked example. I was building a code review agent. My first instinct was four tools:

const tools = [
  { name: "read_file", description: "read a file from the repo", ... },
  { name: "search_code", description: "grep across the repo", ... },
  { name: "get_diff", description: "show the diff for this PR", ... },
  { name: "post_comment", description: "leave a review comment", ... },
];

I implemented all four. Real filesystem access. Real git invocation. Real GitHub API call. Probably four hours total. Then I ran the agent on a real PR.

What happened: the agent called get_diff first (good), then called search_code for every single identifier in the diff (catastrophic — the diff had 200 lines, 50 unique identifiers, my rate limit ran out). It never called read_file because the diff already contained the context. It called post_comment once at the end with a 4,000-word essay instead of inline comments.

Three of my four "real" tools were either misused or unused. The agent design was wrong, not the implementations. If I'd run the loop with mocked responses first, I would have:

Noticed it called search_code 50 times → split the tool into search_code(query, limit=5) with an explicit budget
Noticed it never used read_file → deleted it, saved myself an hour
Noticed post_comment was being used as post_essay → split into post_inline_comment(line, body) and post_summary(body)

That intervention takes fifteen minutes when the tools are mocked. It takes a day when they're real.

The role-play pattern

The trick is shockingly simple: write your tool schemas, send a real user message to Claude, and when the model produces a tool_use block, you hand-type the result and feed it back. The loop runs end-to-end, but you're playing every tool.

In code, this is the same agent loop everyone writes:

while (true) {
  const res = await callClaude({ messages, tools });
  if (res.stop_reason === "end_turn") break;

  const toolUses = res.content.filter(b => b.type === "tool_use");
  const toolResults = toolUses.map(t => ({
    type: "tool_result",
    tool_use_id: t.id,
    content: PROMPT_USER_FOR_RESULT(t.name, t.input),  // <-- you fill this in
  }));

  messages.push({ role: "assistant", content: res.content });
  messages.push({ role: "user", content: toolResults });
}

The only difference between this and a "real" agent loop is the PROMPT_USER_FOR_RESULT call — instead of executing a function, it shows you what the model called and what arguments it used, and waits for you to type the answer.

What that produces is surprisingly information-dense:

Did the model pick the tool I expected? If it took a different path you didn't anticipate, your schema is signaling something other than what you meant.
Did the input shape match my JSON schema? If the model is straining to fit the schema, the schema is too rigid or too loose.
How many tools did it chain? A 12-step tool chain to answer one question is a sign you decomposed the toolset wrong.
Did it ask follow-up questions before tool use? That's good — it means the model is trying to disambiguate. If it doesn't, your prompt isn't asking it to.

You see all of this in a five-minute conversation, before you've written a single line of real implementation.

When you can stop role-playing

The sandbox isn't a permanent state. It's a phase. You run it until you've answered three questions:

Are these the right tools? — Some get deleted, some get split, some get merged. Usually 30-50% of your initial toolset doesn't survive contact with a real prompt.
Are the schemas tight enough? — You see the model picking awkward argument values; you constrain the schema (enum instead of string, required instead of optional).
Does the agent loop terminate? — Some agents will keep calling tools forever if their stopping criteria are vague. The mock-response loop surfaces this immediately because you're the one getting stuck typing responses.

When those three are stable on a handful of real prompts, you write the real implementations. The implementation work is now de-risked: you know which tools to actually build, and the schemas are settled.

The thing you save isn't the implementation time itself — it's the rework. Writing a tool from scratch is fast. Rewriting a tool because its schema was wrong, then updating the prompt because the new schema needs different framing, then re-running every regression input, is what eats days.

What this looks like in tool-lab

tool-lab is what I built to do this without setting up a project each time. Three panes:

┌─ Tools (JSON editor) ─────────┬─ Conversation ────────────────────┐
│ [                             │  user: review this PR             │
│   { "name": "read_file", ... },│  assistant: I'll get the diff.    │
│   { "name": "search_code"...},│    → tool_use: get_diff()         │
│   { "name": "get_diff", ... },│    ← tool_result: <YOU TYPE>      │
│   { "name": "post_comment"...}│  assistant: ...                   │
│ ]                             │                                    │
└───────────────────────────────┴───────────────────────────────────┘

You paste your tool schemas on the left. Type the user message. The model streams its response into the right pane. When it lands a tool_use block, the conversation pauses with a text field for the result. You type whatever the tool would have returned — JSON, a string, an error, whatever. Hit continue. The loop runs again with your fake result included.

It's about 12KB of relevant logic on top of the shared SSE client I wrote about here. BYOK, no backend, your tool schemas and conversations live in localStorage only. There's a demo conversation seeded on ?demo=1 so you can see the loop run without writing tools yourself.

The thing I keep noticing: the tool-lab session for any new agent takes ten to twenty minutes. The agent design that comes out of it is consistently 30-50% smaller than what I would have written from intuition. Smaller agents with fewer, more focused tools are also dramatically easier to reason about when they go wrong in production — which is the other dividend of doing the sandbox phase.

What I'd recommend you do this week

Three escalating moves:

Today (10 minutes): Pick an agent you're already building. Paste its tool schemas into tool-lab, send a real user message, see what happens. If the agent picks the wrong tools or uses the right ones in surprising ways, you've just learned something.
This sprint (an afternoon): Make "sandbox before implementation" the default for new agents on your team. Stand up the tool schemas first, role-play five representative prompts, then write the implementations only for tools that survived. Track the count: how many initial tools made it through.
This quarter (a habit): When something goes wrong with an agent in production — wrong tool picked, weird argument shape, infinite loop — drop the trace into the sandbox before debugging the implementation. The bug is often in the design, not the code.

Tool implementations are not the hard part of agent development. Tool design is. The thing that separates teams that ship reliable agents from teams that ship agents that "mostly work" isn't the quality of their tool functions; it's how many bad tool designs they killed before writing the function.

You don't need a framework for this. You don't need a vendor. You need fifteen minutes and a willingness to play the role of every tool, by hand, until you know which ones deserve to be real.

I shipped this in tool-lab — define tools, mock responses, watch the agent loop. BYOK, no backend, runs in the browser. Source: github.com/ferhatatagun/tool-lab.

The same SSE client also powers three sibling tools — claudoscope, agent-replay, prompt-lab. All open-source, all BYOK: ferhatatagun.com/tools.

This post is mirrored from ferhatatagun.com/blog/build-the-sandbox-first — that's the canonical URL.

More from the same place:

Long-form blog — AI, LLM tooling, frontend at the model boundary
The five tools the post discusses — all browser-only, BYOK, open-source
GitHub · X · LinkedIn

Happy to discuss here or on the canonical post — both threads stay open.

DEV Community: Ferhat Atagün

The failures that don't throw

Counting failures is the wrong instrument

The three buckets

Zero failures is not a zero failure rate

The strongest version of the counter-argument

Two bugs I shipped into it

What it doesn't tell you

The second customer tells you what you actually built

The two failures

The request is not the signal

Look at the workaround, not the ask

The FDE is the sensor, and the reporting line is the problem

What the series has actually been about

You're not done when it works. You're done when they can change it.

The documentation trap

The three questions

Attach the why to the thing, not beside it

You are the bus factor, on purpose

What I'd actually leave

"That library will hurt performance" is not a number

Why the score has to be Lighthouse's score

The inversion that makes byte-sorted reports lie

The bug that proves the point

Savings don't add up, and pretending they do is worse than useless

What's measured and what's modelled

It found a bug on the site it's published from

The deliverable isn't the prompt. It's the eval.

Two different questions wearing the same word

What you can actually build on day three

Diff decisions, not prose

What this catches that nothing else does

Where this is weaker than it sounds

The upgrade path is free

Why this is the deliverable

I accidentally built a forward-deployed engineer's field kit

The decision, and the reason underneath it

Four more decisions, same shape

What browser-only actually costs

The map

The gap I can't paper over

Why any of this generalizes

Nobody's model failed. The interface did.

The number that doesn't fit

Adoption is an interface number

Three places trust dies — all of them on screen

The strongest version of the counter-argument

The hiring gap

What I'd actually do this week

Where this goes next

How I shipped a blog Google couldn't see

What the page was actually serving

Why prerender = true wasn't enough

The home page was even worse

The three problems the fix uncovered

1. Prerender worker OOM

2. The prerender crawler followed every link in the rendered HTML

3. The shared host wasn't pulling from gh-pages

How to catch the next one before it ships

What this cost

See the prompt before you ship it

What you can actually pre-flight

The four-fold cost difference no one was looking for

Why the heuristic mode exists

What about prompt caching

What I'd recommend you do this week

What I learned shipping four open-source Claude dev-tools in two weekends

The constraint that made the work possible

One tool, one thing it makes visible

BYOK + browser-only as a trust multiplier

A 150-line client, copied four times

The articles aren't marketing — they're scaffolding

What I'd do differently

What's next

How I debug Claude agents by replaying their trace

What an agent trace actually contains

The right primitive: a timeline of decisions

The bugs that become obvious in this view

What replay gives you that re-running doesn't

What this looks like in agent-replay

Why `prerender = true` wasn't enough