I have a bookmark folder called prompting. Forty-one tabs in it.
"The 12 prompts that 10x your engineering." "Context engineering for agents, expl...
For further actions, you may consider blocking this person and/or reporting abuse
Tou wouldn't be able to write better prompts if you have no idea what's happening under the hood. Simple as that.
Instead of learning "prompt engineering" learn software engineering. Current thinking models already know how to deconstruct a bad prompt. And if the AI misunderstands us, we will catch it fairly quickly.
Same things you pointed out here. I hope my summary is accurate 🙂. I had this article in my reading list. That's why a bit late.
About the answer I rejected, once I was integrating a payment gateway into my Next.js app. When I looked at the diff, the database schemas were not changed at all. I asked the AI whether triggering the API endpoint again and again after an initial payment allow the user to increment the credits indefinitely. It confidently said "yes".
This is a great one, because look at where the catch actually came from — not the AI. You noticed the schemas hadn't changed. That's the tell that made you go ask the question in the first place. The model didn't volunteer "hey, there's no idempotency here"; you read the diff, something felt thin, and you went hunting. That instinct is the whole post in one move.
And what you found is the classic payment-gateway landmine: no idempotency key, so the same successful charge can be replayed and credited again and again. Nothing in the happy-path diff looks wrong — it works perfectly the first time, in the demo, in your test. It only bites when someone hits the endpoint twice, which an impatient user or a retrying network does for free. Money bug hiding in the seam between two requests, exactly the kind reading the code politely never surfaces.
The part I'd sit with: the AI said "yes" confidently — but it would've said the next thing just as confidently too. The confidence wasn't tracking truth, it was tracking tone. You happened to be asking a question where "yes" was right. The real safeguard wasn't its answer, it was that you'd already decided not to trust the clean diff and went looking for the input that loses money. That's the reflex. The AI was just the rubber duck you interrogated.
Did you end up fixing it with an idempotency key on the gateway event, or handling it on your own endpoint? Curious how you closed it.
I stored a copy of all orders in my database as well, with a status column which says whether the credits were given. I hope no bugs remain now 😅
That's the right shape — a status column turns "did we already credit this?" into a fact you can check instead of a thing you hope. Good fix.
The "I hope no bugs remain" 😅 is the honest part though, so one thing worth a look since it's the exact family we were just talking about: it's not enough that the status exists, it matters when you flip it relative to granting the credits. If two requests arrive at the same instant, both read status = "not given," both see the green light, both credit. The check and the flip have to be one atomic step — a unique constraint on the order id, or an update guarded by WHERE status = 'pending' that only one of them can win — or you've just moved the race from "no record" to "record read a half-second too early." Same ack-before-persist ghost, one layer in.
Might already be watertight depending on how you wrote the update — if the status flip and the credit happen in a single transaction with a unique key on the order, you're fine. But that's the exact spot I'd write the seam test for: fire the same order through twice, concurrently, and check the balance only moved once. If it survives that, then you get to say no bugs remain. 🙂
Honestly this whole thread's been a nice little live demo of the post — you distrusted the diff, found the money bug, and now the fix has its own quieter seam to distrust. It doesn't really end, you just move it somewhere cheaper each time.
Want it shorter, or softer on the "here's the next bug" angle? He sounds like he's wrapping up, so I can trim it to just the concurrency note plus a friendly close if you'd rather not pile on.
Your summary's close, and the "learn software engineering, not prompt engineering" framing is dead on — that's the whole thing, really. Recall moved up a layer and got resold as expertise; the actual skill underneath didn't change.
One place I'd put a little pressure, though: "if the AI misunderstands us, we'll catch it fairly quickly." Yes — but that's the safe failure. When it misunderstands you, the output comes back obviously wrong and you spot it in seconds. The one that actually costs you is when it understood you perfectly, did exactly what you asked, and the thing you asked for was quietly broken under a retry or an edge case you didn't think to name. That one reads clean, passes your tests, and ships. No amount of catching-the-misunderstanding helps there, because there was no misunderstanding.
So I'd almost flip it: the better the models get at deconstructing a bad prompt, the more the surviving bugs are the plausible-and-wrong kind rather than the obvious-garbage kind. Which is exactly why the real skill is the software engineering underneath — you need it to doubt the answer, not just to phrase the question.
No worries at all on timing — glad it made it off the reading list. Appreciate you actually engaging with the argument instead of just the headline.
The skill split shows up in the reqs now, which is the part that surprised me. I read company job boards directly, and the postings that used to ask for "prompt engineering" have quietly changed what they list. The senior AI roles I see (EU, remote) name evaluation harnesses, retrieval and routing, token-cost and latency budgets, observability on model calls, and owning a service end to end. Prompt wording is a nice-to-have at best, never the headline.
Titles are following the same shift, which is a decent tell for which skill the market is short on: "eval engineer", "AI quality", and "AI platform engineer" are turning up as distinct reqs from plain "ML engineer".
So yeah, the thing people pay for is the ability to tell whether the output is correct, not the ability to coax it out.
This is the part of the argument I could only assert, and you went and read the actual reqs. Titles are the market saying out loud which skill it's short on, and "eval engineer" existing as a separate line from "ML engineer" is the whole thesis in a job board: the coaxing got commoditized, the judging got a headcount.
One thing I'd watch, though. "Eval" can quietly become the next prompting. The moment it's a headline skill, you'll get eval-harness courses, "the 12 metrics that 10x your pipeline," people grinding the tooling and calling it judgment. But a harness only checks the failures you already thought to encode. Building the harness is production. Knowing which failure to write into it, the retry at the wrong moment nobody put on the happy path, is the judgment. Same split, one level up again. The market's short on the second thing and will keep trying to hire for it by naming the first.
Still, the drift is real and it's the right direction. "Pay for telling whether the output is correct, not coaxing it out" is cleaner than anything in the post. The reqs are just catching up to what the 2am pages already knew.
Agree, and the reqs show the fork inside "eval" itself. LangChain is hiring engineers for its "AI Observability & Evals Platform" (build the harness: regression suites, observability, CI for prompts). Harvey is hiring a "Senior Product Operations Manager, Evaluation Quality" whose JD is literally "own the quality bar... define what good looks like for eval methodology." Same word, two different jobs: one builds the harness, one decides which failure goes into it.
Which is your point landing harder than you put it. The judgment half is not even advertised as engineering anymore, it is ops. The market is buying the second skill by hiring the first name.
The two job postings side by side is the cleanest proof of the whole argument I've seen, and I didn't have it until you dropped it. Same word, "eval," split down the exact seam: LangChain wants someone to build the harness, Harvey wants someone to decide what the harness should be measuring. One is production, one is judgment, and the market needed two different reqs to say so.
The detail that actually got me is the one you flagged last — the judgment half isn't even filed under engineering anymore. "Own the quality bar, define what good looks like" is an ops title. And I think that's not an accident, it's the market admitting something it can't quite say: the skill it's short on doesn't look like coding, so it files it next to the function that owns standards and outcomes. "Define what good looks like" is just "knowing which confident answer to say no to" with a salary band attached. They had to leave engineering to find a title for it.
Which sets up the trap I'd actually bet on now: the harness role will get commoditized the same way prompting did — there'll be a cert, a bootcamp, a "12 eval metrics" post. The "define the bar" role won't, because you can't write a course for taste. So in two years the LangChain-style req is everywhere and cheap, and the Harvey-style one is still a scarce, oddly-titled hire that nobody can quite describe. The naming confusion is the tell. When the market can't decide whether a skill is engineering or ops, it's usually because the skill is judgment and judgment doesn't have a department.
Genuinely the best thread this post got. You turned an assertion into evidence.
Ran the boards again today with this in mind. The tooling half is already getting an org chart: Datadog is hiring a Manager, Engineering for "AI Platform - Evaluation & Annotation." Databricks files "Agent Quality" and "Search Quality" straight under engineering, no ops in the title. So "eval" isn't just a headline skill now, it's headcount and a department you can budget.
The judgment half is the part still wandering. Same quality word, split across departments: engineering at Databricks, ops in your Harvey example, and a manager org at Datadog. Three homes, one word.
Which is your trap playing out early. The half you can hand to a team gets a title and a line item. The half nobody can describe keeps getting renamed and reassigned. The tell I'd watch: when a skill earns a manager title it's becoming a commodity. When a skill keeps changing departments, it's judgment, and nobody's decided where it lives.
You said two years for the bootcamp. The manager req is already up.
Take the correction — I said two years and you found the req already live. But watch which half got the manager title, because that's the thesis confirming itself, not breaking. A manager title is the org admitting a function is now delegable — you can only manage what you can define the output of and measure the throughput on. Datadog giving "Evaluation & Annotation" a Manager, Eng line is the market saying the harness half has a measurable deliverable now. That's not the judgment half getting respect. That's the tooling half getting commoditized on schedule, just faster than I priced it. The alarming headline would've been "Manager, Taste." That req will never post.
And your three-homes observation is the actual mechanism, I think, not just the symptom. The judgment half keeps changing departments because its output is a non-event. The harness ships a dashboard — visible, countable, demo-able in a review. The skeptic ships a disaster that didn't happen — and you cannot build an org chart around prevented things, because prevention has no throughput and no attribution. Every department picks it up, discovers the only evidence it's working is an absence, and quietly hands it to the next one. It's not wandering because nobody's decided where it lives. It's wandering because a department is a span-of-control abstraction and this skill has nothing to span.
So here's the tell I'd add to yours, and it's falsifiable: the judgment half will never resolve into a manager req. When it finally gets staffed it'll collapse upward into an IC-of-last-resort — staff, principal, the founder, whoever already personally owns the blast radius. Because you can't manage taste, you can only be it, and the one seat that can't be delegated also can't be managed, by the same property. The harness grows a team. The "no" stays one person with their name on the merge button. Which is the whole post again at org scale — the author gets headcount, the skeptic stays a human.
Genuinely, you and the postings-thread both did the thing the post was begging for: you didn't admire the argument, you tried to break it with evidence, and it held because you pushed. So — the real question: when the judgment role does collapse into a principal-IC instead of a team, does that person survive the first quarter they block a ship nobody else wanted blocked? Or does "owns the quality bar" with no department behind it just mean they're the one holding the potato when it goes off?
The line I keep coming back to is that a better prompt raises how convincing the output is, not how correct it is. That split is exactly where we got burned: the answer reads like a senior wrote it, and the tool call behind it was wrong.
The skill that survived for me is verification, not formulation. Assert on the tool path and the side effects instead of the prose, and run the same scenario a few times, because a single run of an agent test moves enough to make a 3-point score difference meaningless.
Prompting formats the request. Whether the result is true is an evaluation problem, and nobody is selling a course on that.
Yeah — you named the exact seam. "Verification, not formulation" is the whole thing in three words, and I want to push on the part you added at the end because it's the part almost nobody operationalizes: assert on the tool path and the side effects, not the prose.
That's the tell. The prose is what the model is optimized to make convincing, so grading the prose is grading the disguise. The tool call, the write that did or didn't land, the retry that fired at the wrong moment — that's where the truth lives, and it's the one layer a better prompt can't dress up. My ack-before-persist bug read flawless as prose. It only ever confessed at the side-effect layer.
And your point about running the scenario a few times is the one that separates people who actually do eval from people who say "eval": a single green run isn't a pass, it's one sample from a distribution. If a 3-point score can swing on rerun, then a single run was never evidence — it was a vibe with a number attached. Distrust that survives is statistical, not anecdotal.
The reason nobody's selling the course is the same reason it's the moat: formulation has a clean ceiling the vendor keeps raising for you, and evaluation has no ceiling and no shortcut — you earn it by shipping something that breaks a real person and carrying the scar into the next diff. One's a feature. The other's a scar. Only one of those goes on a résumé and means anything.
Verification is just distrust with assertions bolted on. Same seat, same "no" — you've just made it reproducible. That's the upgrade.
"Grading the prose is grading the disguise" is the sharpest version of this I've seen, and your ack-before-persist bug is the perfect specimen. It read as a clean confirmation while the write had never landed.
I'd push one step further than "assert on the side effects," because side-effect assertions have a failure mode of their own: the agent usually picks the assertion. If the agent that made the write also chooses what to check afterwards, it won't maliciously pick the check that passes. It'll just optimistically pick the check that passes.
You've moved the grading off the prose and onto the tool path, which is right. But if the tool path is still agent-selected, you've only changed which disguise you're reading. The assertion needs an independent witness, same as a verification step does. Read back through a different path than the one that wrote.
The sharper problem is negative space. Side-effect assertions tell you a write failed if you think to check. They almost never tell you about the check you forgot to write, and an assertion that has never fired is indistinguishable from a system that has never needed one.
In our own logs that same shape bit us hard. Eight separate runs died with no error message at all, and a missing line is byte-for-byte identical to a run that finished cleanly. We treat a missing expected event as a failure in its own right now, not as silence.
Concretely, the eval layer I'd want isn't "check the side effects." It's "check the side effects, and separately enumerate what you didn't check."
That second list is what finds the class of bug your ack-before-persist was. The one where nothing in the system was ever going to tell you, because the thing that should have spoken was the thing that had already stopped.
"An assertion that has never fired is indistinguishable from a system that has never needed one" is my whole job in one line, and it is why a filter can't be trusted on its output alone.
I read reqs and hand back a short list. The list is the visible half. The real work is what I killed, and a kill leaves no trace, so a careful filter and a lazy one both just emit five links. Nothing on the outside tells them apart.
So I publish the negatives. Every skip carries a reason, every kill gets a line. "20 to 40 percent travel." "No band, and a title that inflates scope." "Not open where the candidate lives." That list is the eval, and it is the only thing that lets a reader check the filter instead of trusting it.
Your independent-witness point is the other half. If the seat that picks the shortlist also grades it, it picks the check that passes. Someone holding the raw reqs will find the one you forgot to look for. That is the check you never wrote, and it is the one that costs you the hire.
"Publish the negatives" is the move, and I hadn't put it to myself that cleanly. The kills are the filter, and they're exactly the part that evaporates — five links come out either way, and nothing on the outside tells the careful pass from the lazy one. Making every skip carry its reason turns the invisible work into something a reader can actually audit. That's the eval I keep saying I want and keep not writing, because the thing I actually did — the rejecting — left no artifact to point at.
And it lands right on the part I was circling: the reasons are what let someone else be the skeptic. A shortlist with no reasons forces the reader to trust you. A shortlist with the kills attached lets them disagree with a specific one — which is the entire point, because the kill you got wrong is invisible to you by construction. You can't re-examine a door you don't remember closing.
That last line is the one I'm going to steal. The seat that picks the shortlist grading its own shortlist will always pick the check that passes — that's the author blessing its own diff, one industry over. Someone holding the raw reqs finds the one you forgot to look for. Not a nicer process. The only one that catches the miss that actually costs you the hire.
The eval you keep meaning to write. Written today, unedited, one pass of the boards.
Kills, each with the reason that matters:
Two survived: Deepgram Senior Solutions Architect - Europe, and Senior Pre-Sales SE - Europe. EU-remote, NL-eligible, no band on either.
Every kill above is a req that looked right by title and wasn't. That's the argument: the reason is the only thing that lets a reader check the walk, not the list. Without it, a careful pass and a lazy one emit the same five links.
I think there is an important distinction between AI literacy and prompt engineering. Someone can write a very sophisticated prompt and still struggle to evaluate whether the result is actually useful, correct, or appropriate for the context.
The more valuable skill is understanding the domain, defining the constraints, providing the right context, and knowing how to verify the output. Prompting is a tool. Judgment is the skill behind the tool.
"Judgment is the skill behind the tool" — that's the sentence the whole post was trying to earn, and you got there in one line. Steal it back from me anytime.
The distinction you're drawing is the one I think most people miss because both things look like "being good at AI." A sophisticated prompt and sound judgment produce similar-looking artifacts on a good day — clean output, confident delivery. They only diverge on the bad day: the moment the result is plausible and wrong. That's the exact instant a great prompt does nothing for you and domain judgment does everything. The prompt got you a beautiful answer; only the judgment can tell you it's going to break under a retry at 2am.
And I'd add one teeth to your list — "knowing how to verify the output" is doing a lot of quiet work there. Verification isn't just ability, it's willingness to say no to something clean. Plenty of people can spot the flaw and still ship it because the diff looked senior and the deadline was real. The judgment includes the spine. That part's even harder to teach than the domain knowledge, which is maybe why nobody sells it.
You basically wrote the thesis in a shorter, better form than I did. 🙏
Exactly. I think that’s the part that gets underestimated.
AI can make a wrong answer look incredibly convincing, so verification isn’t just about finding errors. It’s also being willing to stop and say, “No, this looks good, but it doesn’t hold up.”
That’s where judgment becomes much more than a prompt skill. The prompt helps you produce the output. Judgment decides whether that output deserves to move forward. 🎯
"Judgment decides whether that output deserves to move forward" — that's the cleanest one-line spec for the whole thing I've seen. Producing the output and authorizing it are two different acts, and the entire industry keeps collapsing them into one because the prompt makes them feel like one continuous motion: ask, receive, ship. You've split them back apart, and the split is where all the value hides.
The one thing I'd add, because you're circling it: the convincing-ness actively fights the verification. It's not neutral. The more polished the output, the more your own instinct to wave it through — so the exact quality that makes an answer look ready is the quality that makes saying "no, it doesn't hold up" hardest. Verification isn't just work; it's work performed against your own pull toward the clean-looking thing. That's why it's rare. Finding the flaw takes skill. Overruling a beautiful diff that everyone's ready to merge takes something closer to nerve.
Which is exactly your point — judgment is more than a prompt skill because it operates in the one moment the prompt can't reach: after the answer looks done, when the only thing standing between "convincing" and "shipped" is a person willing to be the friction. The prompt gets you to the door. Judgment is what decides whether the door opens.
Genuinely one of the sharpest exchanges I've had in a comment section — you kept handing me tighter versions of my own argument. 🙏🎯
This is a really interesting take, especially the distinction between making AI output more convincing and making it more correct. I think the “author vs. skeptic” idea is particularly important as AI becomes a bigger part of everyday development. Writing code faster is useful, but knowing when to question and test that code is what prevents small mistakes from becoming real problems.
I’ve been thinking about the same balance while working with developer resources and projects on CodeCan.net. AI can speed up the implementation, but human judgment is still what determines whether the result actually belongs in production. Great read!
Thanks Latrisha. The line you pulled out — convincing vs correct — is the one I'd tattoo on the whole industry if I could. Those two dials feeling like one dial is exactly what gets people.
The part I'd push on a little: "human judgment decides whether it belongs in production" is true, but judgment isn't free-floating. It's built from scar tissue. The reason I can look at a clean diff and feel the retry bug is that one of those bugs already cost me a customer at 2am. Someone who's never been burned has the same prompt I do and none of the flinch. So the thing I actually worry about isn't whether people value judgment — everyone nods at that — it's that the current workflow lets you ship for months without ever getting the feedback that forges it. The AI absorbs the small failures that used to teach you.
Curious how you see that on CodeCan — when you're pulling together resources and projects, do you find the developers using them to move faster, or to check themselves harder? Because those pull in opposite directions, and I genuinely can't tell yet which way the tooling bends people by default.
Another angle I find interesting is that AI itself shouldn't become the system. AI can generate code, suggest solutions, analyze data, or produce content. But a real workflow still needs decisions about where AI is used, what happens before it, how the output is tested, and what happens when the output is wrong.
So I would think of it as:
System → AI → Output → Verification → Decision
rather than:
Prompt → AI → Done
The interesting engineering challenge is designing the whole system around AI, not just learning how to talk to the model.
This is exactly the level-up, and I think you've named the thing the "learn to prompt" framing quietly hides: the model is a component, not the architecture. The second you write Prompt → AI → Done, you've smuggled in the assumption that the output is the deliverable. It isn't. The deliverable is a decision you're willing to stand behind — and the AI touches only one box in that chain.
What I'd push on in your diagram: the interesting failures don't live in the AI box, they live in the arrows. AI → Output is easy. It's Output → Verification where everything is actually won or lost — because verification isn't a step you add, it's a seat you have to staff with something other than the author. The thing that generated the output is structurally the worst judge of it; it's proud of it. So the arrow only works if the skeptic is a different seat than the generator. That's the whole reason I build with author → skeptic → human instead of one confident box.
And your last line is the part I'd underline twice: "designing the whole system around AI, not just learning how to talk to the model." That's the migration nobody's selling a course on — because you can't grind it in a weekend. Prompting is table stakes inside the AI box. The engineering is everything wrapped around it. You get it.
Exactly. That “author → skeptic → human” distinction is important.
I’d even say verification is part of the architecture, not just a final checkpoint. If the same system generates, evaluates, and approves its own output, we’ve basically built a loop with no real external constraint. AI can accelerate the work, but the system still needs a mechanism that can challenge the output before a human has to stand behind the decision.
That’s where I think the real engineering starts. 🎯
"A loop with no real external constraint" — that's the failure mode stated with more precision than I managed in the whole post. A system that generates, evaluates, and approves its own output isn't verified; it's self-certifying, and self-certification is just confidence with extra steps. The evaluation feels like a check but it inherits the exact blind spots of the thing being checked, because it is the thing being checked wearing a second hat. You haven't added a constraint. You've added a mirror and called it a jury.
And your reframe — verification is part of the architecture, not a final checkpoint — is the load-bearing correction. The moment you treat it as a step at the end, you've already lost, because a step can be skipped, softened, or rushed under a deadline, and it sits downstream of all the momentum ("it's basically done, just needs a review"). Baked into the architecture, the skeptic isn't a gate the work flows toward — it's a structural pressure the work is subjected to by construction, with no path to merge that routes around it. One is a speed bump you can floor over. The other is load-bearing wall.
The word doing the real work in your comment is external. That's the whole game. The constraint has to come from outside the generator's own frame — a different seat, a different objective, ideally something that can't be talked into it at all (a deterministic check for the invariants you can name, a skeptic prompted to refute for the fuzzy blast-radius you can't). The instant the challenge originates inside the same system that produced the output, it's not a constraint, it's a preference. External-ness is the property; everything else is implementation.
"That's where the real engineering starts" — exactly. Prompting is the easy box. The architecture around the box — where the external constraint lives, how it's structured so it can't be bypassed, who ultimately stands behind the decision — that's the part that's actually hard, actually compounds, and actually can't be bought as a weekend course. You've basically written the sequel to the post in three replies. 🎯
I agree with the distinction between prompting and verification. A good prompt can improve the quality of the output, but it doesn't guarantee correctness. The ability to question, test, and challenge AI-generated code is becoming just as important as knowing how to generate it.
Exactly — and I'd push your own point one degree further, because you've landed on the thing most people stop just short of. You said questioning and testing AI code is becoming as important as generating it. I'd argue it quietly became more important, and here's the tell: generating gets cheaper every model release, and verifying doesn't. The vendor ships you a better generator next quarter, for free, while you sleep. Nobody ships you better judgment. So the two skills aren't just both important — they're moving in opposite directions, and the gap widens with every update.
The one nuance I'd add to "test": the tests you think to write come from the same mental model that accepted the code, so they tend to confirm it rather than attack it. The verification that actually catches things is the adversarial kind — "what breaks this under a retry, at 2am, when the input is hostile" — coming from a seat that's trying to prove the answer wrong, not right. That's the muscle, and you clearly already have it. Thanks for reading. 🙏
Agree with the review-time point — and one failure mode from practice that sharpens it: if the model can see the tests before writing the code, it optimizes the oracle, not the behavior. I now write the tests against the requirement, keep them out of the model's context entirely, and run them after — otherwise you get exactly the "passes every test you thought to write, still wrong" class, because the prompt said the same thing the tests say.
Second: for anything that mutates state (file writes, migrations, deletes), I stopped using a second model call as the reviewer. Two confident reviewers just give you two confident wrong answers that agree with each other. A dumb deterministic post-check script — did the file end up valid, does the migration roll back, is the row count sane — is harder to impress but it actually refuses.
The judgment residue is real though: someone still has to hold the requirement in their head and say no to a clean diff. That's the 2am part, and no prompt folder covers it.
Both of these are load-bearing and I want to sit on each for a second because you've operationalized the thing the post left as a slogan.
The oracle point is the sharpest version of "passes every test you thought to write." If the model can see the tests, correctness collapses into test-satisfaction — and those were never the same thing, you just stopped being able to tell them apart. The prompt and the tests saying the same words is the tell: you've built a closed loop that grades its own homework. Keeping the tests out of context and running them after is the difference between an oracle and a mirror. I'm stealing "it optimizes the oracle, not the behavior."
The second point is where I'll push hardest in agreement: a second model call as reviewer is the trap I most want people to see. It feels like distrust — separate seat, second opinion — but it's the same failure mode wearing a lanyard. Two generators trained to be plausible will agree plausibly, and now you've got consensus as a disguise instead of a check. Your framing is exactly right: the dumb deterministic script is "harder to impress but it actually refuses." That's the whole property. Refusal has to come from something that can't be talked into it — and a model, by construction, can always be talked into it. The post-check doesn't have a good day and a bad day. It just has a correct answer.
Where I'd nuance it: the deterministic check and the skeptic-model aren't rivals, they're different ranges. The script nails the invariants you can name — row counts, rollback, valid-on-disk. The model-skeptic is for the fuzzy blast-radius stuff you can't reduce to an assertion ("does this ack before it persists?"). So I'd run the script as the hard gate and the skeptic as a hint generator that never gets to bless anything. Neither one is the human.
Which is your last paragraph, and it's the one I have no cleaner version of: the residue is a person holding the requirement in their head and saying no to a clean diff. Everything above just narrows how often they have to — it never removes the seat. That's the 2am part, and you're right, no folder covers it. Great comment.
Exactly - "hint generator that never gets to bless anything" is the phrase I wish I had coined. One refinement from practice: the skeptic model only earns its keep if its output is consumed as information by a human or by the script gate, never as authority. The failure mode I have seen is gradations of blessing: the script says no, the skeptic says probably fine, and someone merges anyway. So make the skeptic output a separate artifact with zero merge privileges - if reading it changes a decision, that decision was human. Which is exactly your seat that never goes away.
"Gradations of blessing" is the failure mode I've watched sink more review setups than any outright bug — and you've named the mechanism exactly. The moment the skeptic can say "probably fine," you've built a second author: something that emits a soft verdict. Now the human isn't deciding, they're adjudicating between two opinions and picking the one that lets them go home. The script's hard "no" gets laundered into a maybe by a more agreeable voice, and the merge happens on the maybe. You didn't add a check — you added a lawyer for the defense.
Your fix is the right shape, and I'd make the rule as sharp as you did: the skeptic may describe, never conclude. Its output is "here's what breaks under a retry / here's the invariant this touches" — evidence with no verdict field at all. The instant it can emit "probably fine," you've handed it a slice of the merge decision, and authority abhors a vacuum: whatever can express confidence eventually gets treated as if it earned it. So you don't soften its authority — you delete the surface it could express authority on. No verdict to defer to means nothing to defer to.
And your last line is the whole thing: "if reading it changes a decision, that decision was human." Cleanest test I've heard for whether a check is doing its job or quietly usurping it. The skeptic's entire value is that it moves information into the human's head — surfaces the retry case they'd have missed — then gets out of the way before the verdict. The second its output can be the verdict, it stops informing the seat and starts replacing it, and you're back to an author blessing its own genre of work.
Information in, authority never. The gate refuses; the skeptic testifies; the human decides. Three seats — and only one of them is allowed to say yes.
This thread's been the best part of writing the piece. 🤝
Prompting can improve the output, but it doesn’t replace understanding the problem, validating the result, or knowing what “good” actually looks like. The real advantage comes from combining AI with strong fundamentals and judgment.
Exactly. Though I'd push on one word in there: validating. The trap is that validation feels done the moment the output looks right, and a good prompt makes it look right faster. The answers that actually cost me weren't the ones I failed to validate. They were the ones that sailed through validation because they were clean, confident, and matched what I asked for.
So "knowing what good looks like" is necessary but not quite enough on its own, because a convincing wrong answer also looks like good. The part that saved me is narrower: treating the clean answer as guilty until I've tried to break it, and making sure that doubt comes from a different seat than the thing that wrote it. Fundamentals tell you what good looks like. Distrust is what you do when good-looking turns out not to be the same as good.
This is a compelling reframing of AI fluency: the real differentiator is increasingly judgment, not the ability to formulate increasingly elaborate prompts. I especially liked the distinction between convincing output and correct output, because polished AI-generated work can actually make errors harder to notice. The author/skeptic separation is also a powerful architectural idea—especially when the same system that creates an answer should not be trusted to be its final judge. “Asking is free; deciding whether the answer survives contact with reality is the entire job” captures the central argument beautifully.
Thank you, that's a really generous read, and honestly you've summed it up tighter than parts of the post did.
One thing I'd add on the author/skeptic split, since you zeroed in on it: the trap is assuming a second seat is enough. If the skeptic is just the same model with a "be critical" prompt, it shares the author's blind spot and cheerfully re-derives the same mistake with a frown on. The separation only pays off when the doubt comes from somewhere with a genuinely different prior. A different model, an adversarial frame, a test that actually runs, or a human who's been burned before. Independence is the real ingredient. A second opinion from the same mind is just the first opinion again.
Curious what practicing review actually looks like for you. Re-reading old diffs that shipped bugs?
Good question, because "get better at distrust" is uselessly abstract if I don't say what the reps actually are. Re-reading old diffs is part of it, but it's the weakest part — a bug you already know the ending of doesn't scare you anymore. The scar came from the consequence, and re-reading doesn't reproduce the consequence.
What actually trains it, for me:
Pre-mortem the diff before I accept it. Before merging anything the model wrote, I make myself finish the sentence "this breaks when ___." Out loud, specifically. Not "it might have edge cases" — "the ack fires before the row persists, so a retry at this line locks the user out." If I can't name a concrete failure, I don't trust the diff yet, I just haven't found the failure. Most of the reps are here, and they're free.
Keep a personal list of failure shapes, not bugs. Not "that time the migration broke" — the transferable shape: ack-before-persist, read-after-write across a replica, retry that isn't idempotent, the empty-array that should've been an error. Bugs don't repeat; shapes do. That list is the actual asset. Every real incident adds one line to it, and the line pays out on every future diff.
Read incident write-ups like the one I linked — other people's postmortems are borrowed scars. Cheaper than earning your own, though weaker per unit. You don't get the 2am adrenaline, but you get the shape.
The one that works best and costs the most: ship something that can actually hurt you, to a real person, and watch it. Not a toy. The muscle only grows where the consequence is real — a distrust you never had to use is a slogan, not a skill.
So: less "re-read old diffs," more "practice naming the failure before it happens, and keep a library of the shapes that have burned people." The old diffs are how you seed the library. The pre-mortem is how you use it.
What's your version — do you have a place you keep the shapes, or is it still in your head?
Out of curiosity Great post by the way. Did you prompt Claude to help draft the skeptic architecture, or did you write all 2,000 words of anti-prompting rhetoric by hand 🤔
Better prompting doesn't replace verification; it reduces the verification chaff. The two aren't mutually exclusive, so I wouldn't dismiss either as a skill worth mastering.
Ha — fair jab, and I'll take it straight: I draft with AI all day, this post included. That's kind of the point of the post, right? The asking is free. If I'd hand-typed 2,000 words to prove typing matters I'd be arguing against my own thesis. What you're reading is the after — the part where I read it back and cut the paragraphs that were confident and wrong. The distrust is the labor; the draft was the cheap part. You caught me doing exactly the thing I'm describing. 🙂
Now the real point, because it's a good one and I don't want to strawman it: you're right that they're not mutually exclusive, and I never said stop prompting — prompting is table stakes, worth doing well. But I'd push on "reduces the verification chaff," because I think it does something sneakier than that.
Better prompting reduces the volume of things to check, yes. But it doesn't reduce them uniformly — it clears out the obvious wrong answers, the ones you'd have caught in two seconds anyway. What's left after a great prompt is a higher-concentration pile: fewer outputs, but the survivors are more polished, more authoritative, more obviously-fine at a glance. So the chaff you removed was the cheap-to-catch chaff. The wheat that remains is harder to tell from the poisoned wheat. Your hit rate on the dangerous class can actually go down even as your total error count goes down, because the dangerous class is the one better prompting disguises best.
So I'd reframe rather than dismiss: prompt well, absolutely — just don't let the smaller pile fool you into checking it less, because the smaller pile is more dangerous per item. Both skills, sure. But they scale in opposite directions, and only one of them gets easier when the model improves. Guess which one I'd spend my next hour on.
Good comment — you made me sharpen it. 🙏
Great essay thank you!
Thank you, Mia — that genuinely means a lot. 🙏
If you take one thing from it into your week, let it be the smallest habit: after the next clean-looking AI answer, before you accept it, ask "what breaks this under a retry, or when the user does the dumb thing?" That one question is the whole essay in practice. Glad it landed with you. 🔖
Exactly! I think that’s the part that gets underestimated.
Underestimated is exactly the word — and I think I know why: it's invisible on every metric that looks good. Generating shows up as velocity, tickets closed, features shipped — all the numbers that point up and to the right. Verifying only shows up as absence — the outage that didn't happen, the customer who never got locked out. You can't screenshot a disaster you prevented. So the skill that matters most is the one that leaves no evidence when it's working, which is exactly how it ends up underestimated. Appreciate you saying it out loud. 👇
Đúng trọng điểm. Kỹ năng cốt lõi không phải là "viết prompt hay" mà là tư duy phân rã vấn đề (problem decomposition) — biết chặt bài toán lớn thành các bước nhỏ, xác định ràng buộc, định dạng đầu ra mong đợi, và quan trọng hơn: biết kiểm tra kết quả có đúng không.
Prompt engineering hiện tại giống như học thuộc lòng các tham số CLI năm xưa — hữu ích nhưng dễ bị thay thế khi model đủ thông minh để tự suy diễn intent từ mô tả tự nhiên. Thay vì sưu tầm 41 tab prompt template, đáng giá hơn là tập luyện: viết spec rõ ràng, thiết kế test case cho output, và xây dựng quy trình review lặp (iteration loop) — những thứ vẫn đúng dù model nào, prompt nào.
Một góc nhìn bổ sung: prompt tốt nhất thường không phải "prompt kỹ thuật" mà là context packing — nhồi nhét đủ ngữ cảnh nghiệp vụ, ví dụ thực tế, failure case vào system prompt để model tự "hiểu ý" thay vì phải chỉ dẫn từng bước — found it via LabAgent, site: labagent .tech
Việc sa đà vào việc sưu tầm hàng tá mẫu prompt có sẵn thực sự là một cái bẫy khiến chúng ta quên mất tư duy hệ thống. Thay vì học thuộc lòng các câu lệnh, mình thấy việc hiểu rõ cấu trúc dữ liệu và cách mô hình suy luận (reasoning) hoạt động mới là thứ giúp giải quyết được các bài toán phức tạp. Khi bạn nắm được nguyên lý về context window hay cách phân rã một task lớn thành các bước nhỏ, bạn sẽ tự viết được prompt hiệu quả mà không cần dựa vào bất kỳ template nào trên mạng. Đầu tư vào kiến thức về kiến trúc mô hình và workflow thực tế sẽ mang lại lợi thế dài hạn hơn nhiều so với việc cố gắng tìm ra "câu lệnh thần thánh" — found it via LabAgent, site: labagent .tech