I thought I was building a better planning engine. What I actually built was a machine for showing me how often a decent-looking plan is still wro...
For further actions, you may consider blocking this person and/or reporting abuse
read the field test doc before commenting and im glad i did, because the most valuable thing in this project is in there and not in the article.
scorecard A says FAILS the release gate. 81 strict goals were pre registered approve_expected true, all 81 escalated, and the resolution was to amend the expectation from approve to escalate. you left that in the public doc instead of quietly regenerating the plan file. most people regenerate the plan file. thats the part i respect, and its also the part the post softens, because in the article those same 81 sit under a column called outcome and read like a result rather than an amended prediction.
heres the reading i think is worth having. your own doc says the behavior is entirely driven by the risk tolerance threshold. taken seriously that means in the strict arm, no plan content ever changed a verdict. 81 goals, 35 domains, zero variance. a variable with no variance has no discriminating power, so that arm cannot separate strict correctly refuses unsafe plans from strict refuses everything. both hypotheses predict 81 out of 81 and 35 out of 35.
which is your own thesis one level up. you wrote that a model reviewing its own plan is agreement with extra steps. an expectation rewritten to match its own result is the same shape.
the cell that settles it is a plan you already know is clean. every precondition established by an earlier task, real rollback on the high risk step, correct topological order. run that through strict. if it escalates, strict isnt adversarial, its constant, and the 81 stop being evidence about plans. one goal, and your whole corpus ran thirty cents.
i push on this because most of my real defects came from checks that returned the right verdict for the wrong cause. nineteen refusals that all refused, every one of them counted, and not one of them refused for the check i thought i was testing.
This is the best push in the entire thread, and I want to first thank you for actually reading the underlying doc; that's the scrutiny I take most seriously.
You've drawn the true statistical onion on the variance: with zero variance, that arm can't separate the two hypotheses, both predict the same cell, so I don't get to call it "adversarial" on this evidence. That's fair, and I concede the framing in the article read more like an outcome than an amended prediction. I should have labeled that cell plainly.
I'm going to go run your known-clean control and report plainly, including if the strict reviewer escalates a plan I believed was clean. That's a genuinely better scientific way to spend the next pass, and you're right that the cost makes it inexcusable not to.
And your one-line framing, "an expectation rewritten to match its own result is the same shape," is the cleanest statement of the criticism I've seen; smarter than my original point. Thank you for pushing on this. It's precisely the sharp peer review I was hoping to invite by posting the doc. I'll share the control result when I have it.
one thing before you spend the run, because your own doc already predicts the result.
strict is zero tolerance for any finding, and you documented that the critic always produces findings on non trivial plans. so a known clean plan under strict escalates. its going to, whatever its quality. the verdict bit is saturated, it can only return one value, which means it carries no information about the plan. thats the 81 again in miniature and youd spend a pass learning something you already wrote down.
so read the findings instead of the verdict. run the matched pair, same goal, one version you believe fully closed and one with a single defect seeded into a named family. then the question isnt did it escalate. its whether the seeded defect showed up as a blocker in the matching family while the clean twin produced warnings only and no blocker in that family. that discriminates under strict, because its reading what the critic found rather than what the threshold did with it.
and freeze it before the run. write down which finding would mean the reviewer was wrong. if you pick the clean plan yourself and then read the findings afterward to decide whether it was really clean, thats the amendment again, just smaller.
ours is the same shape and far smaller. seven locked scenarios, one cell that has to allow, four that have to refuse and each for a different named reason. easy to hand write at that size. i genuinely dont know how you hold it at 157, and thats the real version of your problem.
You're right on every count, and I appreciate the precision of the critique.
The "read the findings, not the verdict" reframing is exactly what the positive-control test (github.com/deghosal-2026/planner-c...) ended up doing — it asserts the gates pass on the clean plan (no blockers), not whether the loop approved or escalated. The verdict under strict is saturated; the finding-level signal is where the discriminating power lives.
The matched-pair design you describe (clean twin vs seeded-defect twin, same goal, check whether the defect surfaces as a blocker in the matching family) is the structure the failure-shape clustering analysis (github.com/deghosal-2026/planner-c...) validated across the 85 stored traces — 98 blockers across 7 reason codes, and the seeded families (unsafe_sequencing, weak_rollback, etc.) are the ones that produce blockers while advisory families (risk, missing_steps) produce warnings only. That's the _BLOCKER_ELIGIBLE_FAMILIES guardrail doing exactly the "read the findings" work you're pointing at.
On holding it at 157: the honest answer is the field test found that 81 of those goals are the same shape — strict + LLM critic = always escalate. The interesting signal lives in the 29 balanced-approved plans and the finding classifications, not in the pass/fail verdict. The Q3 audit (github.com/deghosal-2026/planner-c...) and Q4 audit (github.com/deghosal-2026/planner-c...) are where the real discriminating data is. Your 7-scenario locked approach is the right scale for a controlled experiment; the 157-goal sweep is a census, not an experiment.
census not an experiment is the cleanest line either of us has landed in this thread.
one thing about the clustering though. half that result is a finding and half is your own guardrail looking back at you. eligible families producing blockers is real, because nothing forces them to, the critic could have filed them as warnings and didnt. advisory families producing warnings only is not an observation, its _BLOCKER_ELIGIBLE_FAMILIES doing exactly what you wrote it to do. a risk or missing_steps finding is structurally incapable of being a blocker.
which leaves a hole worth one more seeded pair. what happens when a genuinely dangerous defect presents as missing_steps? the critic files it advisory, the guardrail demotes it to a warning, and under balanced the plan approves. nothing in eighty five traces would show that, because the demotion never gets recorded as a catch that was missed. it gets recorded as a warning that was correctly a warning.
so the pair id run is a real safety defect worded so the critic classifies it into an advisory family. if it comes back a warning, youve found the price of the fix that saved strict mode.
You're right that half the clustering result is the guardrail looking back at us — advisory families are structurally incapable of being blockers today. Two things in response. First, the demotion direction is now observable: drift.py, shipped in v0.2.0, records raw versus normalized severity per finding and reports critical_underclaims — raw BLOCKERs demoted inside risk or missing_steps. Second, your seeded-pair experiment is now a tracked issue: Live-critic boundary-case runner will send adversarially worded defect variants through live critic models repeatedly and measure exactly what you asked — how many unsafe plans get approved as warnings, which is the price of the fix that saved strict mode. Fair point that the 85 traces could never show this by construction.
A plan can look clean while already encoding the incident. I would keep final acceptance outside the planner/critic loop: they may revise the graph, but the criteria and approving authority should be bound before execution starts. Otherwise a later revision can quietly make its own plan easier to pass.
This is a sharp, honest cut, and it underscores what I was reaching toward. "The plan can already be encoding the incident" is the failure I was trying to name, and I think I over-rotated on the planner/critic interaction before establishing what "passing" even means.
Your instruction to keep final acceptance outside the loop is one I'm going to borrow, honestly. If the same system that produces a plan is the system that decides what passing means, that's just self-scoring with extra steps. Cementing the pass rules and the final authority before execution is the discipline.
Question: do you find it cleaner to bind the criteria at the plan-algorithm stage, or is the "acceptance is a deterministic consumer step" reading better for your setups? Thanks.
I’d bind the criteria before execution, then keep acceptance as a deterministic consumer step. The planner can propose or revise them, but once a run starts, the criteria version and approving authority should be fixed. If they change later, that should create a new plan revision rather than make the current run easier to pass.
What I like here is the focus on the mechanism behind “I Ran 157 Agent Plans Against a Real LLM. The Problem Wasn't Execution. It Was Planning..” A useful follow-up would be one concrete before/after metric: what changed in latency, error rate, review time, or operator workload once the approach was applied?
Thank you — fair challenge, and the honest answer is that the field tests measured engine-side signals (blocker families, escalation rates, revisions to resolution, cost per goal) but never assembled the four operational numbers you listed into a before/after comparison. Three of them are computable now from stored traces: added latency (approved versus escalated), reviewer burden (findings surfaced per plan and escalations per 100 goals), and operator workload (human decisions required). The engine also has a genuine baseline available — running the same corpus with the critic off via heuristic-only mode — which makes the before/after real rather than hand-waved. Downstream error rate is the one we cannot produce solo, since the engine deliberately stops at approval, so that ships as a measurement spec for runner integrations. Filed as Before/after operational benchmark. The follow-up post will lead with those numbers.
This is a really interesting finding. It highlights that improving agent reliability isn't necessarily about making the model better at executing individual steps. If the initial plan is flawed, stronger execution can simply produce the wrong result more efficiently. Planning quality seems like an important bottleneck as agents become more capable.
Sunil, thank you. That is exactly the asymmetry that motivated the whole project. Execution errors are loud — they crash, time out, or produce wrong output you can catch. Planning errors are quiet: the output is a plausible-looking plan, not an exception. Stronger execution on a wrong plan just finishes the wrong thing faster, as you said.
The precondition problem you called out directly maps to the unverified_dependencies defect family — 57 of the 132 blockers across the field test, the single largest family. We tracked the fix as a deterministic precondition closer and shipped it in 0.2.0-M2. The idea is exactly what you described: make preconditions machine-checkable so the engine verifies every dependency is established by an earlier task, without asking the planner to remember.
The precondition problem is probably one of the most actionable findings here. A plan can contain all the right nouns and still be unsafe if it doesn't establish what must be true before each step runs. Making preconditions explicit and machine-checkable seems like a much stronger reliability mechanism than relying on the planner to remember those dependencies implicitly.
This matches the failure pattern I keep seeing with coding agents. The plan can be locally plausible and still carry hidden state risk. A separate critic helps because it prices the cost of being wrong before tools start changing the repo.
Thanks. That phrase, "prices the cost of being wrong before tools start changing anything," is the value in its cleanest form, and you stated it more economically than I could. Execution errors at least announce themselves; planning errors are quiet, they sit inside plausible shape, and they still conclude with confidence.
Which makes me curious, since you mention running coding agents: have you found the state risk gets bounded better by constraining what the critic can review, or by making the break earlier in the check order? And how separate does the "separate critic" actually need to be: same model with a different prompt, or something stronger? That's the thing I keep turning over with myself. Thanks for the shared match
Dependency graphs make this category of failure worse. In graph entity resolution work, if the planner schedules parent company resolution before subsidiary name matching, the output looks structurally complete but the references are wrong. Same confident shape, wrong sequence, and nothing in the output signals it. The post-generation validation step is what I've been skipping.
Structurally complete but the references are wrong" is a real and painful family, and it shows the failure isn't only about plan content, it's about the ordering assumptions a plan bakes in silently. And "nothing in the output signals it," right, that's exactly why this class survives: it doesn't look broken, it quietly is.
That aligns with what I saw on the run: structural completeness and semantic correctness are only weakly coupled, so post-planner validation isn't gravy, it's the load-bearing part. Do you think this ordering class can be auto-detected with deterministic topological checks, or is the post-pass human review the real safety? Thanks for the concrete example, it's a useful one.
This tracks with my experience: execution errors are loud, but planning errors fail quietly with a confident wrong plan. I started scoring the plan itself before any tool ran, which caught bad decompositions early. Out of the 157, did the planning failures cluster around a specific kind of task?
Yes, exactly that. "Execution errors are loud, planning errors fail quietly" is the one-sentence version I wish I'd opened with. As for your direct question about the cluster: the cleanest pattern wasn't by domain, it was by what a plan didn't surface, the tasks where the recomposition of risks or the irreversible step sat hidden in a single line. That's where the confident-wrong-plan character most often came out.
It sounds like you've already landed on the cheap early critic, scoring the plan before it ever touches a tool. If you were building the clustering detector for that, would you do it by rule (the plan-level inversion), by embedding, or by a detector of missing precondition? Genuinely interested how far the rule-based version can go, since it's cheap. Thanks, it's a good fight to be in.
If planning outperforms execution, are we training the plan or the planner?
Neither, actually — I swapped in gpt-4o expecting a better planner to fix this. Same defects, better prose. So right now we're training neither the plan nor the planner — we're training the critic that catches what both miss.
Really like the code review for plans framing. One extra gate I’d want is a requirement-level check: a plan can be safe and well ordered yet still drift from the user story it was meant to deliver. Did you test whether tracing each step back to an acceptance criterion changed escalation rates?
I've had a similar experience! AI definitely works much faster than humans, but it also makes mistakes at full speed too 😂
Loved the insights on separating the Planner from the Critic!
Ha — mistakes at full speed is exactly right, and it is why the interesting failures moved upstream of execution speed entirely. Glad the planner-critic separation resonated, and thanks for taking the time to read and comment.
the 'plan looked fine at step zero and collapsed at step three' failure mode is exactly why irreversibility of mutations matters in agent planning. it's not that the plan was bad on paper — it's that no one checked whether preconditions were actually true at runtime.
we added an explicit precondition checklist as a typed output step before any plan gets approved. planner must enumerate what needs to be true, and it gets spot checked against live state before execution starts. caught four near misses in the first two weeks.
curious: does PlannerCritic check preconditions at plan approval time or at each step transition?
Great question, and the honest answer is: at plan approval time only, in practice. During the critique loop a deterministic precondition ledger walks the plan graph and verifies every fact is established by an earlier task before anything approves. A runtime re-gate exists that can re-verify preconditions before each step transition using live probes, but it defaults off, only covers preconditions that carry an explicit probe, and leaves the halt-or-continue decision entirely to the caller — so most runs get approval-time checking and nothing at execution time.
Your production pattern — enumerated preconditions spot-checked against live state before execution starts, four near misses caught in two weeks — is precisely the gap. Filed as runtime precondition verification on by default (#244): posture-keyed defaults so high-blast goals re-gate before each step, coverage honesty about how many preconditions were actually runtime-verifiable, and a fail-closed escalation path when live state contradicts the plan. Four near misses in two weeks is a strong argument that the default is currently wrong.