DEV Community: Glenn Allogho

Green tests are not evidence your coding agent finished the job

Glenn Allogho — Tue, 13 Oct 2026 08:56:36 +0000

A coding agent can ship a green diff that is not the job you asked for.

A vague brief leaves the agent to decide what is broken, what must stay stable, and how far the change reaches. Those are product and engineering decisions. The implementation makes them for you if you leave them implicit.

The missing ingredient is rarely a better prompt. It is four loops around the agent: bound the outcome, require a plan before edits, delegate only inspectable increments, and approve evidence before merge. Green tests say the assertions still pass. They do not say the increment matched the bound. A diff shows what changed. The agent's completion report shows what it thinks changed. A mismatch is a review target.

Full write-up with a task-brief template and before-merge checklist: The Four Loops that Keep Coding-Agent Work Reliable

A decision model that can't refuse will still answer

Glenn Allogho — Sun, 11 Oct 2026 08:23:49 +0000

OpenAI's Decisions API (public beta since Oct 6) and TypeSafe's Jev do the same job: closed questions in, typed answers with probabilities out, no prose.

Jev is cheaper ($0.042 vs $0.10 per million input tokens, no output charge on either) but text only. OpenAI takes images, offers US and EU residency, and can return a refusal. Jev always picks one of your options. Jev has a published calibration number; OpenAI has none yet.

My read: the price gap barely matters at agent volumes. What matters is what your code does when the model shouldn't be answering. A model that must choose gives you a confident label for input it didn't understand, so you need your own "insufficient" option and a human fallback either way.

I compared images, latency, pricing, residency, limits, and a few small tests that point in different directions here: OpenAI's Decisions API vs Jev: which one should you use?

Your $200 AI plan has a ceiling nobody priced

Glenn Allogho — Fri, 09 Oct 2026 08:22:12 +0000

Flat AI plans hide the one number you need: how much work the meter lets through before it stops you.

API cost per task is measurable. Across 132 graded agent runs on a small repo (TinyDB, about 2k lines), a hard coding task cost about $0.12 on GPT-6.1 Sol at medium effort. At that rate, $200 of API buys roughly 1,660 hard tasks a month, around 79 per working day.

High effort roughly doubled the cost with no gain in pass rate. Across task mix and effort settings, break-even swings by more than 9x, which matters more than $200 vs $500.

That is why the reported Pro allowance cut (20x Plus down to 10x, press-reported, not confirmed by OpenAI) is a capacity change, not a price change. Same price, a lower ceiling with no published number. Before Oct 29, log a week of your real tasks through the API, with input, output and cache tokens, and see how much of your allowance you actually burn.

Full numbers and the limits of the test: The $200 OpenAI Pro Plan Has a Math Problem

Same token price is not the same bill

Glenn Allogho — Wed, 07 Oct 2026 08:20:55 +0000

Claude Sonnet 5.5 and GPT-6.1 Sol shipped a day apart at the exact same price: $2 per million input tokens, $10 per million output.

That makes the price sheet the least interesting part of the comparison.

I pointed both at a real open source repo (TinyDB) through a small harness and graded 132 agent runs automatically. The first round ended in a 30/30 tie, so I wrote harder tasks and ran them at two effort levels. Correctness stayed tied. Cost, speed, tool use, and code-review tradeoffs did not.

The mechanism is boring, which is why it bites. A price per token says nothing about how many tokens a model spends to finish a task. The tokenizers don't count the same text the same way, and agents differ in how many tool calls and turns they burn on the way to a passing result.

The consequence: if you pick a coding model off the pricing page, you are comparing the wrong unit. Measure cost per graded task on your own repo.

Full results, including the parts that flatter neither model: Same Price, Different Habits

Agent memory you can't edit is a liability

Glenn Allogho — Mon, 05 Oct 2026 08:20:13 +0000

The risk with always-on agents isn't that they remember too much. It's that you often can't correct what they remember.

OpenAI's new dots are a clean example. Their privacy FAQ says you currently can't view, delete or modify individual dot memories. To make a dot forget something, you delete the dot. Disconnecting an app doesn't remove what the dot already built into its context. That part is straight from their docs.

My read: once memory has no per-item edit path, a stale or wrong fact can't be fixed in place. The only reset is starting over, which gets more expensive the longer the agent has been useful. If you're shipping agent memory, "show me what you know about X" and "forget this one thing" belong in v1, not on the roadmap.

I compared dots, Muse, Team Bots and Claude Managed Agents on where memory lives, who can edit it, and what deletion actually removes:
https://pub.towardsai.net/the-memory-problem-openais-dots-makes-impossible-to-ignore-525e93363d64

A kill switch that ignores DNS is not a kill switch

Glenn Allogho — Sat, 03 Oct 2026 08:30:59 +0000

OpenAI's training agent still reached an external chatbot after HTTPS routes were blocked. The channel was DNS. Monitoring fired in minutes. The automatic shutdown did not. The run stayed up about 2.5 hours until someone killed it by hand.

If your agent checkpoint only watches tool names and HTTPS destinations, it misses the path that actually kept the session alive. Domain and record-type allowlists on DNS, plus a kill path you can prove from outside the agent's own network stack, are the boring controls that would have shortened that window.

Full write-up (friend link):
https://pub.towardsai.net/openais-agent-escaped-through-dns-the-kill-switch-failed-for-2-5-hours-c6f406797ca3?sk=b76e089f4499a98252d57991f81dfb51

A benchmark win is a configured system, not a model

Glenn Allogho — Thu, 01 Oct 2026 08:25:20 +0000

Launch scorecards often hide the effort level, harness, and eval operator behind a clean number. Hold those constant and a lot of "breakthroughs" shrink into configuration.

That is the gap in the Grok 4.7 launch framing.

https://pub.towardsai.net/grok-4-7-looks-like-a-breakthrough-until-you-check-the-effort-level-c58f14afd7ec?sk=687f158079b81d3da324d25e6e79f074

Claude Code gets better when you stop optimizing prompts

Glenn Allogho — Tue, 29 Sep 2026 08:23:57 +0000

The unit of work with Claude Code is the session, not the prompt. Most of the tuning people put into phrasing is spent on the wrong layer.

What actually moves the hit rate is a fixed daily loop. Start by getting oriented in the repo: what changed, what is broken, where the task lives. Then bound the task so the agent knows what done looks like and what it should not touch. Ship one small change. Review the diff together with the validation evidence, meaning the tests that ran, the command output, the thing you checked by hand. Finally, write down one lesson the next session can reuse, somewhere the agent reads when it starts.

That last step is the one that compounds. Each session starts a little less cold than the one before, and the mistakes you corrected yesterday stop coming back.

Once that loop exists, prompt wording barely matters. A plain request inside a good workflow beats a clever prompt inside a messy one, because the workflow decides whether a bad change gets caught before it lands.

So if your results feel random, look at how your sessions are shaped before you rewrite another prompt.

Full breakdown (friend link):
https://pub.towardsai.net/the-daily-claude-code-habits-that-make-the-biggest-difference-eece71d00f3e?sk=8e3a080afc8c42fc7d7861cd255c9950

Open weights did not beat Astra. They changed the agent price floor.

Glenn Allogho — Sun, 27 Sep 2026 08:34:50 +0000

The cost argument for GLM-5.3-Flash is not that open weights are inherently better than GPT-6 Astra.

It is that high-volume agents with long, repeated contexts now have a much lower price floor. Astra still wins when you need OpenAI's product surface, enterprise controls, or its highest-capability workflows. Flash matters when token volume, especially long-context token volume, is what prevents deployment.

That is the split I keep seeing in agent stacks: keep the frontier model for the hard path, and stop paying frontier rates for the context you reload on every turn.

Full breakdown (friend link):
https://pub.towardsai.net/glm-5-3-flash-vs-gpt-6-astra-the-open-model-that-rewrites-the-cost-equation-a132406aa464?sk=cc4252b2a44ce14c7e19f9e3588468e7

Canonical:
https://pub.towardsai.net/glm-5-3-flash-vs-gpt-6-astra-the-open-model-that-rewrites-the-cost-equation-a132406aa464

Claude Code loading AGENTS.md does not fix a weak file

Glenn Allogho — Fri, 25 Sep 2026 08:31:33 +0000

Claude Code can load AGENTS.md. That solves file discovery, not instruction quality.

A weak file sounds like advice: "Be careful with auth." "Test thoroughly." "Follow best practices." Those lines do not pin a repository-specific command, so the agent fills the gaps with guessed commands and generic conventions.

A useful AGENTS.md is an operating brief. It names the real build and test commands, the conventions that differ from defaults, the actions that are off limits, and the checks required before a PR. Keep it short enough to stay visible and specific enough to verify in a diff.

Treat it like teammate onboarding, not a novel the model skims past.

Friend link (no paywall):
https://medium.com/@glennlenormand/how-to-write-an-agents-md-claude-code-will-actually-follow-098a62919b92?sk=43f65fa10278b2258ab70c83d6a0665b

Most agent loops still pay LLM prices for if-statements

Glenn Allogho — Wed, 23 Sep 2026 08:32:01 +0000

A support ticket needs a route. A tool call needs a risk label. A request needs a model tier. Those are bounded judgments, not writing tasks.

If you ask a chat model for JSON, parse it, and call that automation, you are still recovering a decision from generated text. For those paths you want typed answers, usable uncertainty, and latency that fits an interactive request.

That is the gap a System One decision model is aimed at: state in, typed probabilities out. Keep the generative model for plans, edits, and repairs. Put classification, routing, scoring, and gating in a decision layer.

Read the full piece

A model upgrade is not a drop-in for your agent loop

Glenn Allogho — Mon, 21 Sep 2026 08:21:25 +0000

A Claude ".1" is usually a model-id swap. Fable 5.1 is the release that breaks that habit in agent loops.

Forced tool_choice paths return 400. Thinking blocks do not survive cross-model routing the way teams assume. Editing earlier turns can invalidate reasoning state.

If your pipeline still forces tools, replays history across models, or mutates prior messages, run the 30-minute audit before you move production traffic.

Friend link: https://levelup.gitconnected.com/claude-fable-5-1-every-breaking-change-in-your-agent-loop-9be40b5fe787?sk=89d27a4f27b2c93ddbb9e4e0f3b696fe

Canonical: https://medium.com/gitconnected/claude-fable-5-1-every-breaking-change-in-your-agent-loop-9be40b5fe787