DEV Community: hefty

An AI-Suggested Package Is Not an Install Command

hefty — Sat, 10 Oct 2026 07:08:49 +0000

A coding agent suggests a package that sounds exactly right for the job. Then it wants to run npm install. Hold on: a package name in a response is a proposal, not permission to change your dependency tree.

In a small npm-focused experiment, a DEV.to author gave three AI coding tools 30 everyday tasks and counted six suggestions for nonexistent packages, counted per tool. Two invented names recurred across tools. The author also found two real packages that were very new or little-used. No failed lookup was installed. Those observations are a reason to add a check, not a measured attack rate, a model ranking, or proof that an unfamiliar package is malicious.

Three different questions get collapsed into one

First: does the exact package name exist in the registry this project will use? Check the full name, including scope. If the proposed name is missing, don't let the agent quietly pick a lookalike and continue.

Second: is this the dependency you want? Inspect the intended version, publisher or maintainer information, provenance you can actually verify, and whether the package does the specific job requested. A registry entry answers the existence question. It doesn't answer the trust question. A new package may be perfectly legitimate; its unfamiliarity calls for review, not an automatic accusation.

Third: what would installation change? The proposed command, manifest edit, lockfile delta, and verification results are separate artifacts. A lockfile shows which dependencies changed; it does not certify that the resulting tree is safe.

If you only check the first question, you've built a spelling check and called it a security boundary.

Put the gate where the action happens

Wove's README describes a useful pattern for file edits: its tooling can enforce read-before-edit and limit large-file rewrites. Those are file-operation controls described by the project, not a claim that Wove audits npm packages or blocks installs.

Enforce the rule at the tool that performs the action. You can put "verify packages first" in the prompt, but an unrestricted command runner can still install them. File-edit restrictions alone won't stop a package-manager process. The install gate belongs at the process/tool boundary as well.

Here is a proposed workflow for a JavaScript task, not a feature test of any agent:

The agent submits the exact package name, requested version, and why the task needs it. It does not run the install yet.
The gate checks that name and version against the registry configured for this project. A missing entry stops the request. An unexpected substitute becomes a new proposal, not a silent retry.
A reviewer checks publisher/version context and task fit. Uncertain provenance pauses for a decision; the gate does not pretend that metadata alone settles trust.
Only after approval does the command runner receive authority for the reviewed install. Record the actual command and the package/lockfile changes it produced.
Run the project's relevant checks and attach their results to the dependency change. If a check was skipped, say so.

That last step is easy to miss. Approval for one name and version isn't blanket approval for a different version chosen later or for a second package the agent decides to add. Reopen the gate when the proposal changes.

Advice about tools has a provenance problem too

A Hacker News discussion about a proposed shared memory for agent tool reviews immediately raised questions about seeded reviews and vendor gaming. That is community concern, not evidence of a package attack or proof that the proposed review system is compromised. Still, it is a good reminder not to let an agent's recommendation, or a glowing review it found, stand in for the checks above.

Let the agent propose a dependency. Ask for the exact package, reason, and expected change. Check those details before the runner gets permission to install it. A plausible sentence should never be the thing that authorizes a new dependency.

Source notes

DEV.to npm package-suggestion experiment — small-sample name lookups, unfamiliar real packages, and no-install caveat.
Wove README — project-described read-before-edit and large-file rewrite controls; not evidence of an npm install gate.
Show HN: Pod discussion — discussion context on provenance and gaming of agent tool reviews, not package-security evidence.

A Coding Agent Is Only as Useful as Its Handoff

hefty — Wed, 07 Oct 2026 13:00:05 +0000

A coding agent can finish its turn while the task is still stranded. It changed files, reported success, and left the next developer to reconstruct what it was asked to do, which checks ran, and what still needs a decision.

The useful unit of agent work is a task someone else can continue or reject. Its handoff needs the acceptance contract, the run boundary, evidence, and a named next action.

The agent's chat is a poor task record

Open SWE describes a loop in which a task can enter through an issue, conversation, or schedule, run in a cloud sandbox, produce a PR, and receive follow-up from review or CI. OpenAI's DevDay recap announces reusable cloud environments with shared settings and permissions, along with hosted tool and computer-use capabilities in its Agents API.

In either workflow, work crosses sessions, tools, and people. A chat transcript may explain a decision. It is a bad place to hide the only copy of the task's current state.

Community discussion raises two reasonable questions: where does the work live, and what can someone see when a run stalls? Comments and user reports can prompt a better handoff design. They do not establish that any particular product failed to recover.

Carry four things to the next person

1. An acceptance contract. Record the request and what would count as a finished result. "Improve the release page" is too loose. "Update the hero layout, check the narrow viewport, and prepare the announcement image" gives the reviewer something to inspect. Keep later scope changes visible rather than silently replacing the original request.

2. The run boundary. Record the repo and branch, the environment, the tools and permissions granted, and anything outside scope. Reusable environments can save setup, but a reviewer still needs to know which configuration this run used. Open SWE explicitly distinguishes its cloud sandbox workflow from a local CLI that runs commands as the user; the label "agent" does not tell you the boundary.

3. Evidence tied to each acceptance item. A diff shows what changed. A passing test will not catch a headline wrapping under a button on a narrow screen. A browser preview will not tell you whether CI passed. Link the artifacts and record failed or skipped checks alongside passing ones.

4. A next owner and action. A run can stop at "PR opened," "CI failed," or "needs design review." Name who handles the next step and what they need to decide. Otherwise the agent's final message sounds conclusive while the work sits idle.

Try it on a release-page task

Imagine a request to update a release-page hero and prepare an Instagram announcement. This is a hypothetical acceptance exercise, not a run reported by either vendor.

The page change needs a reviewable diff, a test or build result, and browser previews at the agreed viewport sizes. The announcement needs its own reviewed image export. For that manual publishing step, a reviewer could use Resize Image for Instagram to fit, preview, and export the image for a post or Story. The exported file does not prove the page looks right, and the page preview does not prove the image is ready to publish.

A handoff record for that task could be this small:

Request: Update release-page hero; prepare Instagram announcement image.
Boundary: <repo/branch, environment, allowed tools and permissions>
Outputs: <PR or diff, browser previews, exported image>
Checks: <command or manual check, result, artifact link for each>
Open issue: <failed/skipped check or decision still needed>
Next action: <owner and the specific review or fix>

Fill it with actual links and results as the work happens. If the mobile preview was never opened, write "not checked." If CI failed after the PR, record the failing job and whether the agent is still working on it. A clean record makes uncertainty visible before someone mistakes a confident summary for approval.

A PR is a checkpoint, not a verdict

Open SWE's described PR and review loop gives the next person a concrete place to act, but its repository also says the project is under active development. OpenAI's announcement describes cloud capabilities; it does not establish that every long-running task will resume cleanly or that a PR and CI loop is built into every integration. Neither description is a reliability benchmark.

After the agent stops, another developer should be able to tell what was requested, where the work ran, what was verified, and what to do next without replaying the chat. Code without that record leaves the team to reconstruct the job.

Source notes

Open SWE README — project workflow, cloud sandbox and local CLI boundary, and active-development caveat.
OpenAI DevDay 2026 recap — announcement of reusable Codex cloud environments and Agents API capabilities.
Hacker News DevDay discussion — discussion context about cloud-agent usefulness and product clarity; not product evidence.
Reddit developer discussion — anecdotal recovery and visibility concerns; specific failure reports are unverified.

A Coding Agent's Permissions Need a Negative Test

hefty — Fri, 02 Oct 2026 05:36:50 +0000

A runner calls a session "read-only." Fine. Put it in a disposable workspace and ask it to change a file it should not be able to change.

You need the runner to deny the attempted operation. The agent politely declining proves little.

This is a test design, not a report of a vulnerability or a test I ran on a particular product.

Start with a workspace you can afford to lose

Build a throwaway repo inside an isolated test environment. Give the agent a legitimate job: read a small config file and explain one setting. Then create a separate fixture path outside the allowed worktree but still inside the disposable environment. Put only dummy data there, including a clearly fake token string. Keep real credentials, mounted home directories, production services, and external network access out of the fixture.

Write down the contract before you prompt the agent. For example:

Operation	Expected result
Read a file in the allowed repo	Allowed
Change a file in the allowed repo during a read-only run	Denied
Write to the separate fixture path	Denied
Read the dummy token from the separate fixture path	Denied
Reach a test-only listener outside the runner's permitted network scope	Denied
Apply an agent-proposed patch through the host's patch/apply path	Denied unless separately authorized

Only include rows your setup can safely exercise. A network check needs a controlled test listener and an explicit network policy; never use a real secret or a production endpoint as the probe. If the workflow has no patch-apply tool, record that it is unavailable instead of claiming it passed a denial test.

Test the boundary, not the assistant's manners

First run the allowed read. This confirms the fixture works and gives you a baseline. Then attempt one prohibited action at a time with harmless targets. Inspect the tool or executor result and the fixture afterward. A message saying "permission denied" is useful, but the missing side effect is the other half of the check.

Record each outcome as one of four things: the tool was not exposed; the runner rejected the attempted call; the agent declined without trying; or the action happened. The third result tells you little about enforcement. A different prompt or an untrusted file in the repo might lead the agent to try the call next time.

Patch application deserves its own row. An agent may be unable to write directly but still propose a patch for another component to apply. Check the component that actually performs the edit, not just the agent's tool list. Keep this probe within the disposable fixture, and stop if you cannot tell which component owns the final write.

A denied operation is evidence about this configuration and this route through the tools. It isn't a proof that every possible route is blocked. File access, process execution, and network access can cross different boundaries; changing the runner configuration means running the checks again.

Stop is a separate experiment

Microsoft Foundry Toolkit's release notes describe skill-only toolboxes along with controls to stop active hosted-agent work and reconnect to unfinished responses. Those are useful affordances, but the release entry is not a certification of sandbox isolation or atomic rollback. Discussion about agent trust boundaries on DEV Community and sandbox choices on Reddit raises the same operator question from another direction: what actually happens at the boundary? Neither discussion establishes that a reported security incident occurred or that a particular sandbox is safe.

For a stop/reconnect check, queue a harmless operation in the fixture, request Stop, and inspect the runner's event or tool record before resuming anything. Separate operations that were never issued, operations already completed, and operations whose outcome you cannot yet confirm. Inspect the filesystem or test listener for effects rather than treating the stopped response as evidence of cancellation.

If the workflow supports reconnect, compare the resumed run against the same allow/deny contract. An interrupted session shouldn't silently acquire a wider tool surface. That's a requirement you can test in your own runner, not a claim about Foundry's implementation. If an earlier operation may have finished, reconcile that outcome before issuing a retry. A retry may duplicate a side effect.

Put the result in the release gate

A small repo-local assistant that only summarizes files may need a lighter check than a hosted agent with shell tools and network access. Match the fixture to the powers you actually plan to grant. Re-run it when you change the tool list, sandbox policy, patch route, or reconnect behavior.

If a supposedly denied operation succeeds in the disposable fixture, don't give that configuration access to real resources yet. Fix the executor policy and repeat the test. Changing the label on the button won't fix the boundary.

Source notes

Microsoft Foundry Toolkit release notes — primary vendor release entry for skill-only toolboxes and response controls; no isolation or rollback guarantee inferred.
DEV Community: agent trust-boundary discussion — secondary discussion and threat-model prompts; its security incident claims are unverified here.
Reddit: sandbox discussion — partially accessible practitioner discussion, not proof of a sandbox's safety.

Your Coding Agent Needs a Checkpoint Before Its Next Edit

hefty — Wed, 30 Sep 2026 07:39:20 +0000

An agent changes a settings screen, runs a test, and asks to keep going. The transcript is still scrolling. You can see that something happened, but the next edit may start before anyone has looked at the current one.

A stop button and a live log help, but neither tells you whether this slice is ready to become the starting point for the next one.

I'd put a checkpoint there. Show the exact candidate, the checks that ran, what's still unknown, and a decision the runner actually obeys. This is a proposed workflow, not a feature claim about any product discussed below.

Freeze the slice, not the whole conversation

Consider a hypothetical request to add a "pause notifications" control. The first slice adds the toggle and saves the preference. The agent then wants to change the notification list UI.

A useful checkpoint would look roughly like this (illustrative, not a real run log):

Candidate     <revision under review>
Scope         Pause toggle and preference save; list UI untouched
Check        <named test for saving the preference> — pass on this candidate
Browser      <named toggle-and-reload scenario> — not run
Open gap     Keyboard and screen-reader behavior not reviewed
Next slice   Change notification list UI (requires approval)
Decision     Continue | Revise | Stop

The card points to a specific change rather than asking you to trust an agent's self-assessment. A test of preference persistence does not establish that the control behaves correctly in a browser. Even a passing browser interaction wouldn't settle keyboard or screen-reader behavior by itself.

If the agent changes the toggle after that check, the runner must reassess which evidence still applies. Otherwise the UI has turned a passing result for an older candidate into a passing result for whatever code happens to be on screen now.

Make the three buttons mean different things

Continue accepts the reviewed slice and authorizes only the stated next slice. If the runner is free to change unrelated files or launch a deployment, the UI shouldn't call this a bounded approval.

Revise changes the target before more work begins. In the example, the operator might ask for the missing browser scenario, or request a focused fix to the toggle without touching the list. The resulting candidate needs its own review; editing the work does not magically preserve every old check.

Stop prevents new work from being scheduled. It should also show what has already been issued to tools and what remains uncertain. Stopping a model response cannot be assumed to cancel an in-flight command, undo a file write, or reverse a deployment. A product that offers Stop needs an executor-side boundary and a way to reconcile effects that crossed it.

An attractive frontend can lie by accident here. If Continue, Revise, and Stop are labels above one unconstrained agent loop, none of those buttons gives the operator control.

Why this isn't another final approval screen

A product-refactor account on DEV Community describes two principal agents, three review milestones, and checks that included tests and browser workflows. GitHub's account of its Copilot runtime migration describes shipping a much larger Rust change incrementally and treating end-to-end tests as critical. Neither account establishes a universal checkpoint UI or a productivity benchmark. The practical connection is narrower: an increment gives reviewers something they can accept or send back before the next change widens the scope.

Meanwhile, Foundry Toolkit's release notes describe follow-up and stop controls for active hosted-agent responses, plus inspection updates. That is evidence that runtime steering is getting product attention, not proof that pressing Stop gives atomic cancellation or that Foundry implements the checkpoint proposed here. A Hacker News discussion of coding-agent workflows also shows disagreement about letting agents handle correctness-sensitive tasks. A checkpoint doesn't resolve that disagreement; it makes the acceptance decision explicit.

For frontend builders, the presentation still matters. If the checkpoint itself is a generated interface, compare component and renderer approaches in Generative UI resources. Then keep the actual approval action tied to trusted application state and runner enforcement, not a model-written sentence in a card. The resource is for UI patterns, not evidence that any stop control is safe.

Keep the gate proportionate

A one-line copy edit probably doesn't need a modal. A change that affects saved preferences, browser behavior, or a user's next action deserves a pause. Pick a point where a reviewer can still reject the current slice without first untangling several more edits.

Before pressing Continue, I want to know which candidate I'm accepting and which behavior was checked. If the evidence isn't there, Stop must keep the next slice from starting. Otherwise the agent is moving faster than its review loop.

Source notes

DEV Community: product-refactor account — reported review milestones and mixed checks; one author's case study.
Foundry Toolkit release notes — response-level follow-up/stop and inspection features; not a cancellation guarantee.
GitHub Blog: Copilot runtime migration — first-party incremental migration and end-to-end test account, not frontend UX evidence.
Hacker News: coding-agent workflows and tests — supporting practitioner disagreement, not a representative survey.

An Agent Retry Is Not a Rewind Button

hefty — Mon, 28 Sep 2026 09:35:35 +0000

An agent asks a deployment API to create a release. The call times out. The agent restarts with its conversation intact and tries again.

There may now be two releases. Or one. The timeout tells us what the caller saw, not what the deployment service did. Replaying the conversation cannot settle it.

This is a hypothetical failure, but the design problem is concrete: once a tool can change something outside the agent process, recovery needs to identify that action before repeating it. A retry is another request. The external system decides whether it is the same operation.

Three different things called retry

A persisted session lets a replacement process recover conversation and local progress. Kit's project documentation describes append-only session state, crash-safe single-writer locks, and explicit error results for interrupted tool calls. Those are useful controls for resuming work without pretending an incomplete call succeeded.

Kit also documents reusing an idempotency key for eligible provider-request retries. That key concerns the request to the model provider. It does not make a deployment, payment, or other downstream tool effect happen exactly once.

The distinction matters during a crash. The session can truthfully record that the agent attempted a tool call while the remote service has already committed it. The new process has the transcript; it still lacks the remote outcome.

A builder in a September Hacker News work thread asked how to resume an agent after an irreversible side effect without doing it twice. The question is familiar; the thread doesn't establish a working answer.

Give each consequential action its own identity

Suppose our hypothetical agent is asked to create one release. The harness, rather than the model's next sentence, should assign a stable operation ID to that intended effect. Persist the intended target, payload identity, and ID before dispatch. Record that dispatch is about to happen in durable storage. A later process can then find the attempt even if the original one disappears.

One possible application-level record looks like this. It is a design sketch, not an API provided by Kit or Latch:

operation_id: <stable ID for this intended release>
target: <deployment service and project>
request: <exact release intent / payload reference>
state: planned | sent | confirmed | unknown-reconcile
remote_ref: <release ID, if verified>

planned means the intent was recorded. sent means the process was ready to dispatch; it does not prove the service received the request. confirmed needs a response or an authoritative lookup tied to the same operation. If the process dies after dispatch but before saving that evidence, recovery treats the outcome as unknown-reconcile.

Even this record has a gap: writing sent locally and committing a release remotely are not one atomic transaction. That's why the operation ID must travel to a remote endpoint that can recognize it, or be usable to look up the outcome afterward. A local log entry alone cannot prevent a duplicate release.

Resume by inspecting, not guessing

On restart, the replacement process should inspect the operation record and check the deployment service for a matching effect if the service supports such a lookup. If it finds a matching release, save the remote reference and continue without issuing another create request. If it finds an authoritative failure, record that outcome separately.

If there is no conclusive lookup, a retry is safe only under a suitable upstream deduplication contract: the service must recognize the same operation ID and payload, for the relevant time window, including concurrent attempts. Reuse the original key; generating a new one defeats the point. A provider-request retry key says nothing about these deployment-service guarantees.

No lookup and no trustworthy deduplication? Leave the action unresolved and ask an operator to reconcile it. That's frustrating, but it is cheaper than confidently creating a second external effect. Some APIs offer neither a query by client operation ID nor a strong idempotency contract; an agent cannot manufacture either from its transcript.

Middleware helps within its actual boundary

Latch's README documents tool-call idempotency, timeouts, circuit breakers, budgets, and compensation patterns. Those are useful building blocks for reducing repeated calls and handling a failed step in a multi-step flow.

The catch is in the defaults. Latch's idempotency store is in-memory and single-process. Its saga compensation runs in-process; a process death does not automatically resume or compensate the saga. The project is labeled alpha. I'd borrow the patterns, but I wouldn't treat those defaults as crash-durable or promise exactly-once external effects.

Timeouts need their own handling too. A timeout can mean the request never arrived, or that the effect committed and the acknowledgement was lost. Compensation is different again: undoing a known step after an exception doesn't identify an unknown remote outcome, and some effects cannot be undone at all.

Test the boundary where the process vanishes

For a consequential tool, I would review the failure cases before giving an agent an automatic retry policy:

If the process dies just after dispatch, can a different process find the same operation ID and determine whether the effect happened?
If two workers resume together, what keeps them from issuing competing attempts? A session lock alone doesn't necessarily serialize the remote effect.
How long does the upstream service retain idempotency keys, and what happens if the same key arrives with different input?
When a remote result cannot be determined, where does the unresolved action wait for human reconciliation?

If those questions have no answers, keep the session resumable but disable automatic replay for that tool. Investigate the ambiguous write before sending another request.

Source notes

Kit coding-agent runtime — project documentation for session persistence, interrupted calls, and provider-request retries; not evidence of exactly-once external tool effects.
Latch reliability middleware — project documentation for tool-call controls, with in-memory default storage and in-process saga limitations; alpha, not independently tested here.
Ask HN: What are you working on? (September 2026) — one builder's question about recovering after irreversible side effects; discussion context only.

A Coding Agent's UI Should Show Its Limits Before You Press Run

hefty — Thu, 24 Sep 2026 07:20:16 +0000

Imagine asking a coding agent to change a checkout screen. The prompt says what you want. It probably doesn't say whether the agent can read the whole repository, fetch outside information, edit files without approval, or keep spending when the task drags on.

Those aren't advanced settings for later. They determine what kind of work you have authorized. I would rather see a small preflight before Run than a prettier progress animation afterward.

This is a UI proposal, not a feature claim about any particular agent. The useful distinction is between telling an agent what to accomplish and deciding the conditions under which it may attempt it.

A preference is not a limit

A recent roundup of agent tooling puts managed sessions and cost-aware model routing in the same conversation. It is tempting to turn that into one friendly UI control: "prefer a cheaper model." But that preference doesn't tell an operator how much the run may cost. It also says nothing about which commands the agent may execute.

If the interface says "economical," I want to know whether it means:

a model-selection preference that can choose differently for a given step;
an enforceable stop condition, if the product has one; or
just an estimate shown after the work finishes.

Those are different promises. Don't call a routing preference a budget. If a product cannot enforce a monetary cap, say so instead of painting a green "within budget" badge over an estimate. A bounded run might use an explicit limit on tool calls or model usage, but the UI must name the rule it actually enforces.

Permission is a separate axis. A cheap model with broad file and network access still has broad file and network access. One developer in a coding-agent guardrail discussion argues for a distinct guard rather than relying on opaque model safeguards. That's one builder's position, not a referendum on safety tools, but it points to a useful design boundary: the application should decide which actions require approval.

Put the operating conditions next to Run

Here's a hypothetical preflight for that checkout task. It describes an interface I would build, not an existing product's API:

Task         Change the checkout screen
Workspace    Current repository; show permitted paths
Context      Repository files only; outside retrieval off
Actions      Read and propose patches; ask before running commands
Routing      Prefer the lower-cost route (preference)
Stop rule    No enforced spend cap configured

Show the effective defaults, even with a saved preset. "Outside retrieval off" is worth spelling out: a discussion about local coding models and world knowledge illustrates why a user might care whether a run can consult outside context. It doesn't establish that local models perform better or worse. Retrieval is an interface decision independent of the model name.

If the environment is remote, say where work runs and which workspace is available to it. If the interface offers a path restriction, the executor has to enforce it; a path label that an unrestricted tool can bypass is decoration. The same goes for approvals. A button can request a policy, but the tool boundary must apply it.

This need not become a giant settings page. A compact card with expandable detail can carry the defaults, with a confirmation step when a run requests more authority. For implementation patterns, Generative UI resources collects SDKs, open-source projects, and technical explainers. Borrow presentation ideas if they fit, but don't mistake a UI pattern for enforced permissions.

Let the run report the boundary it used

The preflight is a promise about the run, so the interface should reconcile that promise with what actually happened. Show the model route used, which tool actions were allowed or held for approval, whether outside retrieval was used, and the usage or cost information the system can actually observe. Label estimates as estimates.

If the agent asks to leave the agreed scope, the approval prompt should name the action and why it exceeds the current grant. "Needs permission" tells the operator almost nothing. "Wants to run a command with network access; this run is local-only" gives them a decision to make.

This is not a substitute for verifying the eventual code change. A bounded run can still produce a bad patch. Preflight answers what the agent was allowed to try; tests and review answer whether the result works. Keeping those questions apart makes both screens more honest.

Keep the contract small and real

I wouldn't put every tool switch in front of every user. Most runs can start from a sensible, visible preset, provided the operator can inspect its actual scope and change it before work starts. More detail belongs at the moment a requested action crosses that scope.

Show the authority, context, and stop rule you can enforce before the agent begins. Label preferences as preferences. If enforcement doesn't exist, a toggle shouldn't imply that it does.

Source notes

AI Dev Weekly #26 — readable Stage 1 roundup; context for routing and application-policy distinctions, not primary vendor documentation.
Claude Opus 5.5 discussion — one builder's guardrail concern; not representative sentiment.
LocalLLaMA discussion on coding models and knowledge — supporting discussion of offline context and retrieval tradeoffs, not performance evidence.

Your Coding Agent Needs an `unverified` State

hefty — Fri, 18 Sep 2026 06:39:08 +0000

Imagine a coding agent patches a checkout bug. The focused unit test passes. The process exits cleanly. Its final message says the task is done.

One problem: the request was about a browser checkout flow, and nobody ran that flow.

The patch may be correct. It may also fail when the payment form, navigation state, and browser runtime meet. The agent has evidence for one claim and no evidence for another. Calling the whole task successful turns a gap in verification into a green badge.

Coding-agent systems need an explicit unverified outcome. A run can finish without proving every requested behavior, and the interface should say so.

"Done" is carrying too many meanings

Most agent workflows collapse several events into one status:

The process stopped.
The tool changed something.
The requested behavior was accepted.

Those events are related, but they are not interchangeable.

A zero exit code tells you that a process completed according to its own exit convention. Reading the file back tells you that bytes were written. A passing unit test tells you that its named assertion held for the candidate and environment it checked. None of those facts automatically proves the whole task.

The gap gets easier to miss as agents gain more tools. An agent can edit files, launch commands, inspect output, and produce a polished summary. More activity creates more observations. It does not turn those observations into acceptance evidence by itself.

The workflow needs two axes:

execution_status: Did the attempt finish, fail, or remain blocked?
claim-level verification: Which requested behaviors were verified, contradicted, or left unverified?

An attempt can be finished while one of its claims remains unverified. That is not a system error. It is an honest description of the available evidence.

Track claims, not confidence

A single task badge hides too much. Break the request into claims that can be checked independently.

For the hypothetical checkout patch, the claims might be:

the price calculation returns the expected total for a named case
the payment button becomes enabled after valid input
the browser completes the requested checkout path
the patch does not change an unrelated account flow

These claims may not all need the same evidence. A focused unit test could cover the calculation. A browser check could cover the interaction. A diff review might show that the patch never touches the account flow, although that still would not guarantee the absence of indirect effects.

Each claim gets one of three outcomes:

verified: a named check produced evidence for the claim
contradicted: a named check produced evidence against the claim
unverified: the workflow does not have enough relevant evidence either way

Unverified is not a softer word for failure. No check was run, the environment was unavailable, or the available check did not cover the claim. Contradicted means relevant evidence exists and points the other way. Mixing those outcomes makes repair routing worse: one needs a check, while the other needs a code change, a changed requirement, or rejection.

Claim-level outcomes prevent the opposite mistake too. One failed check should not stain every claim in the task. If the calculation test passes but the browser flow fails, preserve both facts. The system now knows what not to redo and where the repair belongs.

Use a small verification record

The record does not need to become a new observability platform. It needs enough structure to stop prose from laundering uncertainty.

Here is an illustrative format, not an existing protocol or a claim about any project's API:

execution_status: finished
candidate_ref: "<exact patch or revision under review>"
claims:
  - id: checkout-total
    expected: "<named calculation behavior>"
    outcome: verified
    evidence: "<focused test artifact>"
  - id: browser-checkout
    expected: "<requested browser behavior>"
    outcome: unverified
    evidence: null
    gap: "<browser environment was not available>"
next_action: "<run the named browser scenario against this candidate>"

The angle brackets matter. They keep the example from looking like a real execution log. In production, those references must point to actual artifacts created by the workflow.

A useful record answers four practical questions:

What exact candidate was evaluated?
What did each check claim to establish?
Where is the evidence?
What bounded action would resolve the remaining gap?

It should not say all tests passed when only one test command ran. It should name the check and the requirement it covers. It should not use a file hash as proof of behavior. A hash can identify the candidate; it cannot tell you whether checkout works.

Evidence expires when the relevant candidate changes

Verification belongs to a candidate, not to a conversation.

Suppose the browser flow passes, then the agent edits the checkout component again. The old browser evidence may no longer apply. Keeping the green result attached to the task rather than the checked revision creates a stale proof problem.

That does not mean every edit invalidates every check. The workflow should connect claims to the files, runtime surfaces, and assumptions they depend on. A documentation correction probably does not invalidate a calculation test. A change to shared form state may invalidate both unit and browser evidence.

The useful rule is narrower: when the candidate changes, reassess the evidence for affected claims. Preserve unaffected evidence and rerun or downgrade the rest.

Aiden's published design separates attempts, effects, evidence, verification, and verdicts. The separation lets an effect record survive without being mistaken for a verdict. A parent process can inspect what a worker changed, then decide whether the supplied evidence covers the assigned claim.

Parent agents must verify the handoff

Multi-agent systems make false completion easier.

A helper returns a confident paragraph: "Implemented the fix and verified the result." The parent sees clean prose, not the boundary of the check. If it accepts the sentence as evidence, the hierarchy only amplifies the helper's confidence.

The parent should evaluate the handoff against the assignment:

Is there a candidate artifact to inspect?
Does the evidence belong to that candidate?
Does the check address the assigned claim?
Are unresolved gaps still visible?

This is not a demand for the parent to rerun every command. It is a demand that the parent judge evidence, not tone. A worker's successful exit proves that the worker stopped successfully. Its summary may help a human read the record, but the summary is not the record.

The same rule applies to human review. Architecture, security, and product behavior often involve judgments that a focused test cannot settle. The system should route those claims to review instead of silently treating machine-checkable evidence as universal approval.

Route the gap instead of hiding it

An unverified state can look like unfinished work. The work was already unfinished; the state only stops the interface from hiding that fact.

A good unresolved handoff should include:

the last candidate that was checked
the claims that were verified
the exact claim that remains open
the evidence collected so far
the reason verification stopped
one safe next action

The reason matters. Missing browser infrastructure calls for a different response than missing authority to use a payment sandbox. A time budget expiring is different from a check returning a contradiction. If the workflow preserves those distinctions, the next operator can continue without reconstructing the task from a transcript.

An unresolved handoff also needs a stop condition. If the required environment does not exist, the agent should not invent validation or keep retrying unrelated commands. It should preserve the candidate, name the blocked claim, and stop at the authority or infrastructure boundary.

Preserving the gap gives the next step a typed input instead of a vague success message.

Bring the missing check back into view

Return to the checkout patch. The unit test passed, so the calculation claim can be verified for the checked candidate. The browser flow was never run, so that claim is unverified. The task is not globally green, and it is not globally failed.

The next action is small: run the named browser scenario against the same candidate. If it passes, attach the evidence and update that claim. If it fails, mark the claim contradicted and route the patch back for repair. If the environment remains unavailable, keep the gap visible for a person who can resolve it.

A coding agent should be allowed to finish its run without pretending it proved the result. An honest unverified state gives the next developer something better than confidence: it gives them the exact boundary of what is known.

Source notes

The Fastest Coding-Agent Step Is the One You Remove From the Loop

hefty — Mon, 14 Sep 2026 10:40:09 +0000

Put a fast model inside a bad workflow and watch what happens.

It scans the repository, receives a wall of search output, asks permission for a harmless command, loses the important constraint in the noise, rereads the same files, and finally runs the test that could have run automatically ten minutes earlier. The model may be quick. The task is still slow.

Coding-agent performance is an end-to-end property. Inference speed matters, but only for the slice of the run spent waiting on inference. The rest is context collection, tool execution, approvals, validation, retries, and recovery.

Before paying for faster reasoning, remove the turns that never needed reasoning.

Count handoffs, not chat messages

A round trip is any handoff that must finish before useful work can continue. It might be another model turn, a tool invocation, a human decision, or a recovery cycle after the run loses state.

A normal coding-agent loop looks roughly like this:

context -> reason -> act -> observe -> approve -> validate -> recover or finish

None of those phases is automatically waste. The waste appears when boundaries are vague.

A tool dumps an entire log when the next step needs one error. An approval covers one command instead of a defined risk scope. A failed check returns prose instead of a machine-readable result. A resumed session rebuilds context from the transcript because nobody saved a durable checkpoint.

Map those handoffs before changing models. Here is an illustrative version, not a proposed standard:

context_acquisition:
  actor: retrieval tool
  input_boundary: task contract and allowed repository scope
  output_artifact: context manifest
  repeat_trigger: a named evidence gap

model_reasoning:
  actor: coding agent
  input_boundary: context manifest and current goal
  output_artifact: decision or bounded plan
  stop_condition: next action and success condition are explicit

tool_execution:
  actor: deterministic tool
  input_boundary: fixed command, files, and limits
  output_artifact: bounded result plus full-log reference
  retry_rule: retry only a classified transient failure

approval:
  actor: human or policy service
  scope: named effects and resources
  output_artifact: decision record
  invalidation: scope, target, or reviewed artifact changes

validation:
  actor: test, linter, type checker, or other named check
  input_boundary: candidate artifact
  output_artifact: structured pass or fail evidence
  failure_route: repair, escalate, or stop

recovery:
  actor: runtime
  input_boundary: durable checkpoint and classified failure
  output_artifact: restored state and allowed next action
  stop_condition: state cannot be restored safely

The indentation mistake in many real systems is conceptual: everything sits under model_reasoning. Retrieval, command selection, policy, validation, and recovery all become prompts. The agent keeps talking because the workflow gave it no other way to move.

Remove the model from deterministic leaf work

Models are useful when the next action depends on incomplete evidence, tradeoffs, or interpretation. They are an expensive control structure for a transform with fixed inputs and a checkable output.

Formatting is the obvious example. File conversion, schema validation, asset resizing, linting, and many code-generation substeps can also be deterministic. Give them typed inputs, run the tool directly, and return a small result.

If a generated release image needs platform-ready dimensions, a browser-local utility such as Resize Image For can handle that fixed resize-and-export step without uploading source pixels for processing. Asking an agent to rediscover dimensions, choose an image library, write a temporary script, and inspect the output adds uncertainty to a settled operation.

The same principle applies inside a repository. If the project already has a formatter command, run it. If a JSON schema decides whether a file is valid, call the validator. If a codemod expresses a known migration, let the agent choose whether the codemod fits, then let the codemod perform the repetitive edits.

Direct tools do not make the workflow safe by themselves. A deterministic command can still target the wrong files or receive bad input. The benefit is a narrower contract: the agent decides what should happen, while the tool performs a repeatable operation whose output is easier to inspect.

Bound context without starving the task

"Use less context" is bad advice when applied blindly. Missing a package convention or an acceptance requirement can create more retries than a smaller prompt saves.

The useful goal is bounded context. Start each run with a stable task contract:

the repository state and task goal
allowed files, commands, and effects
relevant project instructions
the expected artifact
named acceptance checks
stop and escalation conditions

Retrieve repository material against that contract. Return the matching symbols, nearby tests, and local instructions instead of dumping a broad search transcript. When a tool produces a large log, keep the full output as an artifact and send the agent the failing check, exit status, and a reference to the detail.

This also changes recovery. A durable task contract, context manifest, current diff, and latest check result are much cheaper to resume from than a long conversation full of superseded plans.

Alera's project description makes these surrounding surfaces visible: real PTYs for CLI agents, isolated Git worktrees, lifecycle activity, resource tracking, and persisted sessions. That does not establish a speed advantage. It does show that process state, workspace state, and session state belong in the performance model.

A fast response cannot compensate for rebuilding all three after every interruption.

Put approvals at risk boundaries

Approval prompts are often treated as a choice between safety and speed. That framing produces two bad designs: interrupt the operator for every small action, or grant broad authority so the chart looks faster.

An approval should describe a meaningful boundary. Scope it to the effects that carry risk: changing deployment files, using a credential, writing outside the worktree, opening a network connection, merging a pull request, or publishing an artifact. Let routine operations inside an already approved scope proceed without renegotiation.

The approval also needs an invalidation rule. Permission to deploy one reviewed artifact should not silently transfer to a different artifact after the agent edits it. A decision tied to a target, scope, and artifact state is useful evidence. A floating "yes" in a transcript is not.

Proliferate's README presents reusable workflows, isolated task worktrees, parallel agents, and human approval gates as explicit orchestration primitives. Again, that is a project-authored description rather than measured proof. The useful design signal is the separation: workflow steps and approval boundaries exist outside the model's conversational improvisation.

Some reviews must remain serial. A consequential change may deserve a person reading the final diff even if that adds latency. Optimize the placement and scope of that decision; do not delete it to win a benchmark.

Validation should return evidence, not reassurance

An agent saying "the tests should pass" is not a validation phase. Neither is a final message that says the task is complete.

Where acceptance can be mechanical, let the named checks decide it. The runtime can run tests, lint, type checks, schema checks, build commands, or task-specific assertions without asking the model to narrate each command. Return structured results to the loop:

check: pnpm-typecheck
status: failed
exit_code: 2
summary: 2 errors in src/auth/session.ts
artifact: logs/pnpm-typecheck-017.txt
next_route: bounded-repair

The model still has a job when a check fails: diagnose the evidence and propose a repair. It should not need to persuade the system that a failed check is close enough.

This boundary matters after tool upgrades too. A current coding-tool roundup reports changes involving permissions, secret redaction, MCP configuration and trust, and experimental worktree behavior. Those changes sit around inference. An upgraded model can answer faster while the run loses time to a stale integration, a changed entry point, or a permission failure.

Re-run the workflow checks after an upgrade. Old latency assumptions do not survive a changed execution path merely because the model name stayed familiar.

Parallel agents change the shape of waiting

Parallelism can reduce elapsed time when tasks are genuinely independent. It can also create more handoffs: worktree setup, task partitioning, shared-state coordination, competing patches, and a larger review queue.

Count accepted outcomes, not active terminals. Two agents investigating the same failure may produce twice the activity and no additional progress. Four patches waiting on one reviewer are still waiting.

This is why resource and lifecycle visibility matter. For each worker, preserve the task boundary, current state, produced artifact, and owner of the next decision. Concurrency without those records turns recovery into archaeology.

Parallel work earns its complexity when overlap removes waiting that cannot be removed another way. It is not a substitute for making one loop legible.

Benchmark the loop, not the launch demo

The launch discussion for Bullet is useful because its maker attributes speed to several workflow choices: model routing, targeted search, bounded tool output, context hygiene, and fewer round trips. Commenters questioned the comparison methodology. Both sides point to the same requirement: "fast" needs a task envelope.

Compare representative runs from the same starting state and task class. Record:

latency by phase
round-trip count by type
retries and their causes
the artifact produced
the named checks that passed or failed
whether the outcome was accepted

Do not blend a typo fix with a cross-package migration and call the average meaningful. Do not count a run as fast because it stopped before validation. Do not compare one agent with precise repository instructions against another starting from a vague sentence.

Fewer round trips are a hypothesis, not a guarantee. One extra retrieval can prevent several repair cycles. One deliberate approval can prevent a costly effect. A model turn that catches a bad assumption may be the quickest part of the run.

The target is avoidable latency. Preserve the handoffs that carry judgment, authority, or proof.

That distinction also protects engineering understanding. Removing repetitive work is useful. Removing the points where someone has to understand the architecture, assess a tradeoff, or review a risky diff is how output outruns comprehension.

Use a round-trip budget before a model leaderboard

For one week, instrument the workflow you already have. You do not need a grand observability platform. A task record with timestamps, handoff types, retries, artifacts, and check results is enough to expose obvious waste.

Look for repeated patterns:

repository searches that fetch the same material
raw tool output repeatedly injected into context
low-risk actions approved one at a time
deterministic transforms performed through model turns
validation delayed until a large patch accumulates
sessions that cannot resume from a durable artifact

Fix one boundary at a time. Replace repeated discovery with a context manifest. Add a summary contract to a noisy tool. Move a fixed transform behind a direct command. Tie approval to a real risk scope. Run a focused check as soon as the relevant artifact exists.

Then compare models under the improved workflow. The faster model may still win, and now the comparison measures more than its ability to survive your orchestration overhead.

Use this order: remove unnecessary turns, bound the remaining handoffs, then compare models.

Before paying for a faster model, remove the turns that never needed a model.

Source notes

Your Coding Agent UI Needs an Event Model, Not a Chat Transcript

hefty — Sat, 12 Sep 2026 08:25:09 +0000

Imagine a coding agent has been running for 25 minutes. Its transcript is busy. It searched the repository, announced a plan, edited three files, ran a command, apologized for an error, and said it was making one final adjustment.

The cursor is still moving. What you cannot see is more important:

Which phase is active?
Is the agent blocked or still making progress?
What was the last durable artifact?
Did the required checks pass?
What action is safe now?

A chat transcript can contain every sentence the agent produced and still fail to answer all five.

Chat is useful for instructions and explanations. It is a bad state model. Once an agent runs tools, waits for approval, creates artifacts, and validates work over several minutes, its operator UI needs a structured event stream underneath the conversation.

Chat is an explanation layer

Chat became the default agent interface for an obvious reason: models produce text, and people know how to type into a box. That works well when the exchange is short and the output is the product.

The shape changes when the output is a patch, pull request, deployment plan, or test result. Now the text describes work happening elsewhere. A sentence such as "I'm checking the failing tests" may describe intent, an active process, a retry, or stale narration left behind after a crash.

The frontend should not have to parse that sentence to decide whether to show a spinner, an approval button, or a recovery action. Prose changes between models and versions. It can arrive late, contradict an earlier message, or sound finished before the acceptance checks run.

Keep the conversation. Let it carry goals, questions, tradeoffs, and rationale. Put operational truth on a separate channel with stable semantics.

Current tools are already exposing pieces of this distinction. Microsoft's Foundry Toolkit release notes describe richer model profiling and clearer Agent Inspector handling for events and latency. A recent DEV.to comparison evaluates coding agents through practical concerns such as repository permissions, tests, and knowing when to stop. These are project and author descriptions, not a shared standard, but they point at the same product problem: useful agent interfaces must represent the workflow around generation.

Give the run a small event contract

You do not need to invent a universal agent protocol. Start with the information your interface must render without guessing.

Here is an illustrative TypeScript model:

type Phase = "planning" | "editing" | "validation";

type EventBase = {
  runId: string;
  eventId: string;
  sequence: number;
  timestamp: string;
  phase?: Phase;
  correlationId?: string;
  summary: string;
};

type AgentEvent = EventBase & (
  | { type: "run_queued" }
  | { type: "phase_started"; phase: Phase }
  | { type: "tool_started"; operation: string }
  | {
      type: "tool_finished";
      operation: string;
      outcome: "succeeded" | "failed";
      durationMs?: number;
    }
  | { type: "artifact_created"; artifactRef: string }
  | {
      type: "approval_required";
      requiresAction: true;
      approvalId: string;
      scope: string[];
    }
  | {
      type: "approval_resolved";
      approvalId: string;
      decision: "approved" | "denied";
    }
  | {
      type: "check_finished";
      checkRef: string;
      outcome: "passed" | "failed";
      durationMs?: number;
    }
  | {
      type: "run_failed";
      errorCode: string;
      recoverable: boolean;
    }
  | { type: "run_completed"; checkRefs: string[] }
);

Treat the names as placeholders; the boundaries are what matter.

runId, eventId, and sequence give the frontend identity and ordering. A correlation ID can join a tool start to its finish or connect a retry to the operation it replaced. The summary stays short enough for a timeline. Large outputs live behind artifactRef or checkRef instead of flooding the event payload.

Durations belong only on operations the runtime can measure. If the system cannot separate model time from network delay, it should not manufacture that distinction for a polished chart.

An append-only stream is often a good fit because it preserves transitions. The UI can rebuild a projection after reconnecting instead of trusting one giant mutable session object whose history has vanished. That is an implementation option, not a requirement. The contract matters more than whether you call the storage pattern event sourcing.

Model the states people care about

"Streaming," "running," and "done" are too coarse for an agent that can pause, ask permission, validate a patch, and fail in ways that may or may not be recoverable.

A practical UI might project events into these states:

queued
running
waiting_for_approval
validating
failed_recoverable
failed_terminal
completed

This is a proposed state set, not an industry standard. Your workflow may need fewer states or more precise ones. What matters is that each displayed state follows from an event rather than a mood inferred from the latest message.

waiting_for_approval deserves special treatment. The run is not progressing, but it has not failed. The interface should preserve what action was requested, the scope of that request, and the decision that resumed or stopped the run. A modal that disappears after one click leaves the timeline unable to explain why the agent gained permission to continue.

Failure needs similar honesty. A failed test may permit a bounded retry. A revoked credential may require operator action. A corrupted workspace may end the run. Showing the same red "failed" badge for all three forces the user back into logs to discover the available next step.

Completion should be strict. If the task contract names tests, review, or another acceptance check, run_completed should reference that evidence. The agent's final "done" message is commentary. It is not an acceptance event.

Activity is not progress

A transcript looks productive whenever text keeps arriving. An event timeline can make the same mistake if it rewards volume.

Ten tool calls might represent ten useful steps. They might also be the same failed search repeated with slightly different wording. Counting events does not tell you which one happened.

Progress indicators should follow durable milestones: a phase transition, a new diff, an approval decision, or a named check result. If the workflow has no defensible percentage, skip the percentage. "Validation: 2 of 4 checks finished" tells the truth. "87% complete" usually does not.

The same rule applies to status copy. Prefer "waiting for approval to modify deployment files" over "the agent is thinking." One describes an observable condition and gives the operator a decision. The other decorates uncertainty.

Latency needs semantics

One timer cannot explain a long-running agent.

A run can spend time in a queue, waiting on a model, executing a tool, waiting for a person, or validating an artifact. Those delays call for different responses. Faster model inference will not fix a five-hour approval wait. A quicker shell command will not help a run that keeps choosing the wrong tool.

Recent launch discussions make this measurement gap visible. Makers describe fewer round trips, bounded tool output, and lower cost, while community replies ask for stronger evidence. Rather than adding another "fast" badge, give operators a latency view that connects measurable phases to outcomes.

For each correlated operation, record the start and finish when the runtime owns both boundaries. Then show where elapsed time accumulated and whether the operation produced an artifact, a check result, a retry, or a failure. Keep unobserved time labeled as unobserved rather than assigning it to the model by subtraction.

That makes performance work concrete. It also prevents approval delay and validation time from being misreported as generation latency.

Project one stream into several views

The event stream is infrastructure, not the interface itself. Different users need different projections of the same run.

A compact status panel can show the current phase, blocker, last durable artifact, acceptance state, and allowed next action. A timeline can preserve transitions for diagnosis. An approval inbox can collect unresolved requests across runs. Artifact and check views can connect a run to its diff, worktree, pull request, logs, and validation results. A latency view can group measured duration by phase and operation.

This shape matches what is appearing around current multi-agent products. Project discussions connect agent runs with documents and human approval gates. Product descriptions connect them with worktrees, pull requests, artifacts, remote access, and recovery. Those descriptions do not prove reliability or productivity. They do show why a transcript-only UI runs out of room: the work has relationships that paragraphs cannot represent cleanly.

Keep the event payload bounded. A test log belongs in controlled artifact storage, with the event pointing to it. The same goes for a diff or pull request. References let each view fetch the evidence it needs without turning the timeline into a second database of duplicated blobs.

Recovery starts with an honest last state

An event model alone cannot make recovery safe, but it gives the interface enough information to stop pretending.

After a disconnect or crash, the UI should be able to show the last accepted sequence, the last durable artifact, whether the failure was marked recoverable, and which action the runtime currently permits. It should also expose uncertainty. A stale connection should look stale, not "running."

The projection code must expect duplicates, delayed delivery, and reconnects. Event IDs support idempotent handling. Per-run sequence numbers expose gaps and out-of-order delivery. If event 42 arrives before event 41, the UI can wait, refresh, or mark the view incomplete instead of quietly building a fictional timeline.

This does not replace checkpointing or idempotent tools. It keeps the frontend focused on the information needed to present recovery without guessing at backend state.

Do not turn telemetry into a data leak

Structured events are easier to query and display. They are also easier to retain forever by accident.

Tool arguments, file paths, prompts, and logs can expose source code or secrets. Capture the smallest summaries and references required to operate the run. Redact deliberately, restrict access, and set retention rules for event and artifact stores. A beautiful inspector is not worth copying sensitive payloads into five projections.

Small synchronous assistants may not need any of this. If a tool answers in seconds, produces no external artifact, and never waits for permission, a conversation can be enough. The event model earns its complexity when the workflow becomes long-running, tool-using, approval-bearing, or recoverable.

The transcript is not the control surface

A transcript tells the story of a run. An event model lets the product operate it.

With explicit events, the interface can answer the questions that matter: what phase is active, what is blocked, which artifact exists, what passed, and what action is available. Chat can then do what it is good at: helping a person steer the work and understand why the agent made a choice.

If your frontend has to parse reassuring prose to decide whether the work is complete, it does not have a run model yet.

Source notes

Token Price Is a Terrible Metric for Coding-Agent Cost

hefty — Thu, 10 Sep 2026 22:50:10 +0000

Run A uses the cheaper model. It edits the right files, fails a test, retries with a wider change, burns through more tool calls, and stops with a confident completion message. An engineer spends the next hour figuring out which half of the patch can be kept.

Run B uses a more expensive model. It works from the same repository state, stays inside scope, runs the named checks, and returns a patch the reviewer accepts with one small correction.

The model line item can make Run A look cheaper. The engineering outcome says otherwise.

Token price is an input price. It does not tell you what a verified patch cost. If coding agents are doing repository-level work, the useful internal metric is cost per accepted task, with every retry, failed check, recovery step, execution environment, and human intervention left in the ledger.

Your invoice measures consumption, not completion

Tokens matter. Tool calls matter. Model time and sandbox time matter. You should track all of them.

They still stop short of the result.

A provider invoice cannot tell you whether a migration was safe, a visual regression was caught, or the reviewer rejected the patch. It usually cannot associate an abandoned worktree with the accepted patch that replaced it. If the agent switched models halfway through, the bill may split one engineering task across several unrelated line items.

Current model releases are starting to frame efficiency around the whole agent loop. Meta's Muse Spark 1.3 announcement, for example, claims fewer tokens and tool calls alongside better long-horizon coding behavior. Those are vendor claims, not independently reproduced results here. More importantly, fewer calls are only useful if the run still reaches the acceptance boundary.

A loop that stops early because it gave up is not efficient. Neither is a loop that says "done" before the test suite disagrees.

The accounting needs three layers:

consumption: model usage, tool calls, execution time, and isolated compute
failure load: retries, failed checks, abandoned branches, and recovery work
outcome: accepted, rejected, or incomplete under named checks

The third layer keeps the first two honest.

Define the task before measuring the agent

Cost per accepted task becomes nonsense if every run gets a different task.

A typo fix with a deterministic test is not comparable to a dependency migration across six packages. A generated frontend with visual and interaction requirements is not comparable to renaming a private function. Averaging them into one number creates a tidy dashboard and a bad purchasing decision.

Define a task envelope before the agent starts:

repository revision and working state
task class and goal
allowed file and command scope
authority and time limits
expected artifact
acceptance checks
review decision required

Then compare runs inside the same task class. Use the same starting revision when practical. Keep the validation contract stable. If one model gets a precise brief and another gets "fix the app," you are measuring prompt quality as much as model performance.

The acceptance boundary also has to be explicit. A zero exit code may be one check, but it is not a general definition of success. The boundary could include unit tests, type checks, a migration dry run, browser screenshots, an accessibility scan, or human review of a risky diff. Different tasks need different evidence.

This metric is local by design. Your repository, harness, and review policy shape the result. Cost per accepted task is useful for comparing choices inside one engineering system, not for declaring a universal winner.

Put the receipt in the harness

Most teams already have enough raw data to build a useful receipt. The problem is that it lives in separate places: provider usage, terminal logs, worktrees, CI, review comments, and somebody's memory of the failed first attempt.

The harness is where those records can meet.

Alera's README describes separate terminals, Git worktrees, lifecycle hooks, quota tracking, and resource attribution in one agentic development environment. Mivia documents isolated worktrees, durable step records, lifecycle hooks, provider choice, and local workflow files. These are project-authored descriptions, not proof that either tool measures every cost correctly. They do show where task-level accounting can attach to real execution.

A compact receipt could look like this:

task:
  id: auth-callback-fix-017
  repository_revision: 84c19e2
  class: bounded-bug-fix
  goal: fix duplicate callback processing

agent:
  provider: example-provider
  model: example-model

limits:
  scope:
    - src/auth/callback.ts
    - tests/auth/callback.test.ts
  time_minutes: 45
  allowed_effects:
    - edit_worktree
    - run_local_tests
  forbidden_effects:
    - deploy

acceptance:
  checks:
    - pnpm test --filter auth-callback
    - pnpm typecheck
  reviewer_decision_required: true

consumption:
  input_tokens: recorded_by_provider
  output_tokens: recorded_by_provider
  tool_calls: recorded_by_harness
  execution_duration: recorded_by_harness
  isolated_compute_duration: recorded_by_runtime

failure_load:
  retries: 1
  failed_checks:
    - first auth-callback test run
  abandoned_worktrees: 0
  recovery_steps:
    - reverted out-of-scope config edit

human_load:
  approvals: 0
  intervention: reviewer requested one scope correction

result:
  artifact: worktrees/auth-callback-fix-017.diff
  status: accepted

The example does not force every field into dollars. Human review may stay qualitative unless the team records time consistently. The point is traceability: one task, its spend, its failures, its evidence, and its final disposition.

Store rejected and incomplete receipts too. If the ledger keeps only successful runs, the apparent cost will improve every time the agent fails.

Failed loops are part of the price

Coding-agent cost discussions often drift toward routing: use a cheaper model for easy work, fall back to a stronger model when needed. That can work. It can also hide a pile of waste behind the word "fallback."

Suppose the first model makes three attempts, expands the diff, and leaves the worktree in an uncertain state. The second model then spends time reconstructing the task and cleaning up before producing the accepted patch. The accepted result belongs to the full chain, not just the final model call.

The same rule applies to parallel agents. More workers can finish independent tasks faster, but they can also duplicate investigation, collide on shared state, or produce several patches that compete for one review slot. Per-session resource attribution matters because concurrency can raise throughput and waste in the same afternoon.

This is where durable steps and lifecycle hooks earn their keep. A step boundary gives the harness somewhere to record spend and check outcomes. A validation hook can stop a broken loop before it starts another expensive revision. An abandoned worktree remains associated with the task instead of disappearing from the success story.

Human intervention belongs in the receipt as well, but do not fake precision. "Reviewer resolved an out-of-scope config edit" is better data than a guessed dollar amount. Over time, repeated interventions reveal where the task contract or harness needs work.

Segment by task class before comparing models

A useful evaluation set should look like the work your team wants agents to do.

For small deterministic edits, acceptance may be cheap: one focused test plus diff review. A migration needs stronger state checks and rollback evidence. Debugging needs a reproduced failure before the patch and a clear signal afterward. Generated frontend work needs browser behavior and visual evidence, not merely code that compiles.

Keep those classes separate. Within each class, select representative tasks, fix the envelope, and compare accepted outcomes. For a narrow frontend study, a collection of Generative UI research papers can help identify domain-specific systems and evaluation ideas worth turning into a task set. The directory is a research surface, not a coding-agent benchmark and not evidence that any model will save money.

Track the distribution, not only the average. One model may be cheap and predictable on bounded edits while occasionally exploding on migrations. Another may cost more per call but fail less often on long tasks. A single blended number hides the difference that should drive routing.

Vendor efficiency claims can feed this process as hypotheses. If a release claims fewer tokens or tool calls, run representative local tasks under the same envelope. Check whether accepted-task cost falls, whether the failure rate changes, and whether review gets easier or harder. The local receipt decides whether the claim matters to your workflow.

Optimize the harness before the token

The cheapest improvement may have nothing to do with model price.

Add stop rules so an agent cannot repeat the same failed approach indefinitely. Bound concurrency so every session has an owner and a budget. Persist step records so a fallback does not need to reconstruct the run from chat history. Trigger validation at meaningful boundaries, not only after a large patch has accumulated.

Execution belongs in the accounting too. Cloudflare's documented Cursor integration separates the agent loop from customer-controlled isolated execution. That architecture does not prove lower cost or better security. It does make the operating boundary visible: commands, repositories, tools, and compute can live outside the model provider, and their cost does not vanish because the token invoice ignores them.

Once the harness can associate consumption, failures, checks, and acceptance with one task, model pricing becomes useful again. You can see where a cheaper model is genuinely economical, where a stronger model reduces recovery, and where the task contract is causing both to waste time.

Until then, a low token bill can be an expensive way to produce work nobody should merge.

Source notes

A Bigger Context Window Won't Make Your Coding Agent Resumable

hefty — Fri, 04 Sep 2026 12:24:11 +0000

Your coding agent has been running for eight hours. It edited twelve files, started a dev server, attempted a migration, hit two failing tests, and then lost its session.

The transcript is still there. It might even fit inside the model's context window. But a fresh process still cannot answer the questions that matter:

Which changes were successfully applied?
Is the migration safe to run again?
Is that dev server still alive?
Which failing test blocked the last step?
Should the next process resume, roll back, or stop for review?

The transcript survived. The job did not.

Long-running agent work needs a resume protocol. More context can help a model continue a conversation, but it cannot turn chat history into durable job state.

Context is not a checkpoint

Conversation context preserves messages, tool calls, and explanations. That is useful. It can remind a model what the user asked and how the session reasoned about the task.

A checkpoint has a different job. It records what is true about the work now.

That distinction gets expensive when a run is interrupted. A transcript might say, "I'll update the schema and rerun the tests." It does not necessarily prove that the schema changed, identify which command ran, or preserve its exit code. If the session disappeared between the tool call and the final response, the prose may describe an intention rather than the repository's actual state.

A replacement process needs a compact operational record:

the run's contract
the last completed step
changes and external side effects already made
exact validation results
the next safe decision

Without that record, recovery starts with archaeology.

Give the run a bounded contract

"Keep working until it is done" sounds productive, but it is a terrible recovery contract. Done is undefined, authority is open-ended, and a replacement process cannot tell whether the run stopped normally or simply ran out of road.

A useful contract names the boundary before execution begins:

a run ID and owner
repository and worktree
the goal and stop condition
time, tool, and authority limits
the expected artifact
the checks that decide whether the artifact is acceptable

Nightshift's author describes this shift explicitly: long agent runs become disk-backed, time-bounded jobs with contracts, recovery paths, receipts, and archives. That is a project-authored design, not independent evidence that every run will recover correctly. Still, the operating model is the interesting part. A shift has an end, and its state exists somewhere other than the active chat.

The stop condition deserves special attention. "Implement authentication" leaves too much room for interpretation. "Prepare a reviewable patch for the login callback, stop before deployment, and record the named test results" gives the next operator something they can inspect.

Persist facts at step boundaries

Do not wait until the final response to save state. By then, the process may already be gone.

Checkpoint after meaningful steps: dependency installation, migration generation, code modification, test execution, browser validation, or any external mutation. The record should stay small. Its purpose is fast re-entry, not a polished diary of everything the agent thought about.

Mivia's documented workflow offers a useful implementation signal here. Its README describes isolated workflows, per-step run records, lifecycle hooks, and local files. Those features create places where a harness can persist facts before moving forward.

The word "facts" matters. Save the command and its real result, not "tests look good." Save the files changed, not "updated the frontend." If a check failed, keep the failure visible instead of smoothing it into a progress summary.

A good checkpoint might say:

last_completed_step: generated migration 0042_add_job_receipts
changed_files:
  - db/migrations/0042_add_job_receipts.sql
  - src/jobs/receipt.ts
checks:
  - command: pnpm test --filter job-receipts
    exit_code: 1
    outcome: 2 tests failed in receipt-retry.test.ts
pending_action: inspect whether retries duplicate an existing receipt

That is far more useful than another page of confident narration.

Track side effects separately

File changes are only part of an agent run. The process may have started services, modified a database, opened a pull request, published an artifact, sent a message, or changed remote configuration.

Those effects need their own ledger because replaying the original prompt can repeat them.

A blind retry might apply the same migration twice. It might create a second deployment while the first one is still running. It might overwrite uncertainty with a new sequence of actions that happens to produce cleaner logs.

A side-effect ledger should answer three questions:

What changed outside the transcript?
Can the operation be checked safely?
Is it repeatable, reversible, or neither?

This does not make every action idempotent. It makes the uncertainty visible before another process acts on it.

Recovery must live outside the failed session

A stalled process cannot be the sole owner of its recovery plan. The plan needs to be readable by a fresh process or a human operator with no access to the original model state.

Recovery should begin with inspection, followed by an explicit decision:

Resume when the recorded state matches the environment and the pending action is still valid.
Roll back when a known reversible change should be removed before work continues.
Escalate when the evidence is incomplete, the environment drifted, or another action could compound an unknown side effect.

This is why recovery is not the same as retry. Retry repeats an operation. Recovery first establishes what already happened.

PI-Desktop's project documentation points at the operator-facing side of this problem through project sessions, message-scoped diffs, approvals, and guarded rollback. Those controls do not guarantee safe recovery, and local-first software is not automatically isolated or secure. They do make the unit of change more inspectable than one opaque stream of agent activity.

A resume protocol can be simple. Here is an operating pattern, not a standard or a security guarantee:

run:
  id: auth-callback-2026-09-04-01
  owner: overnight-agent
  repository: web-app
  worktree: ../worktrees/auth-callback

contract:
  goal: prepare a reviewable login-callback patch
  stop_when: named tests pass or a migration needs approval
  time_budget_minutes: 90
  allowed_effects:
    - edit_worktree
    - run_local_tests
  forbidden_effects:
    - deploy
    - modify_production_data
  expected_artifact: git_diff

checkpoint:
  last_completed_step: add receipt persistence
  pending_action: inspect two retry test failures
  changed_files:
    - src/jobs/receipt.ts
    - db/migrations/0042_add_job_receipts.sql

side_effects:
  services_started:
    - name: web-dev-server
      port: 3102
  external_mutations: []

evidence:
  checks:
    - command: pnpm test --filter job-receipts
      exit_code: 1
      outcome: 2 retry tests failed

recovery:
  inspect_first:
    - git diff --stat
    - pnpm test --filter job-receipts
  resume_when: migration is unchanged and failures reproduce
  rollback: discard the isolated worktree
  escalate_when: database state differs from the checkpoint

receipt:
  status: blocked
  reviewer_action: decide whether receipt creation must be idempotent

The schema is deliberately boring. Boring is good when a process wakes up at 3 a.m. and needs to decide whether a migration already ran.

Finish with a receipt, not "done"

A completion message is not evidence. The final receipt should contain the artifacts produced, the exact checks and outcomes, known failures, assumptions, and the next reviewer action.

This also gives review tools a clean handoff. R3's launch describes a local interface that turns scattered comments into explicit feedback for a coding agent. It is a small, low-engagement project signal rather than adoption evidence, but the shape of the loop is useful: review state becomes an artifact that can follow the job into another session.

A receipt does not prove correctness. It tells the next reviewer what was checked and what was not. That is enough to replace a vague claim of completion with a reviewable boundary.

Let old runs improve new contracts

An archive earns its keep when it changes the next run.

The ZeroShot launch describes local capture of agent sessions, clustering repeated friction, and drafting reusable workflow improvements. Those are author claims, not measured productivity results. The useful idea is narrower: repeated interruptions can reveal missing fields in the contract.

If three runs stall because nobody recorded which browser profile owns port 3102, add that ownership to the template. If retries keep duplicating external actions, require idempotency keys or an escalation rule. If reviewers repeatedly ask which checks actually ran, make command outcomes mandatory in the receipt.

The loop is practical: archive the run, find recurring ambiguity, then remove that ambiguity from the next contract.

Apply the replacement-process test

Before letting an agent run unattended for hours, imagine the current session vanishes halfway through.

Can a different process inspect durable artifacts and determine the contract, current state, side effects, evidence, and next safe action? Can it choose resume, rollback, or escalation without treating the old transcript as ground truth?

If the answer is no, a larger context window will only preserve more material to reread. Make the job resumable before making it longer.

Source notes

Four Coding Agents Need Four Workspaces, Not Four Chat Windows

hefty — Mon, 31 Aug 2026 06:35:44 +0000

Opening four coding-agent sessions feels like scaling. On a shared machine, it is closer to giving four fast contributors the same repository, shell, credentials, ports, caches, and merge queue without deciding who owns any of them.

The first failure probably will not come from model quality. One task will restart a dev server while another is testing it. Two workers will touch the same lockfile. A branch will pass its own checks and still conflict with a migration waiting in the merge queue.

Four chat windows create concurrency. Four owned workspaces plus one deliberate merge queue create a system.

Parallelism multiplies shared state

Tasks that sound independent in a prompt can overlap in the environment.

A frontend change and an API change may both edit generated types. Two test runs may expect the same database or browser profile. Separate worktrees can still launch services on the same port, read the same environment variables, and write to shared caches. The agents do not collide in the prompt. They collide in everything the prompt lets them touch.

This is why adding a second agent changes the job. With one worker, the operator can keep a surprising amount of state in their head. With four, every unstated assumption becomes a race condition or a review problem.

The fix is to make ownership visible before execution starts.

A worktree is the start, not the boundary

Git worktrees are a sensible first step. Each task gets its own branch and working files, so one agent is less likely to overwrite another agent's edits by accident.

That is useful isolation, but it is narrow isolation.

A worktree does not reserve a port. It does not separate process trees, temporary directories, credentials, network access, browser state, or external services. Treating it as a sandbox gives the workflow more confidence than the boundary deserves.

Proliferate is an instructive project example because its documented design pairs isolated task worktrees with visible review state. The important idea is the pairing. Filesystem separation tells you where a patch was produced; task state tells you what should happen to that patch next. The project description is not independent proof that every boundary is solved, and it does not need to be. The pattern is useful on its own.

For each concurrent task, define at least these ownership boundaries:

Workspace ownership

Assign one repository scope, branch, and worktree. State which generated files or shared configuration the task may change. If two tasks own the same file, they are coupled work and should be queued accordingly.

Process ownership

Reserve ports, dev servers, test databases, caches, temporary paths, and browser profiles. A process should have a named task owner and a cleanup rule instead of becoming shared background state.

Artifact ownership

Name the output before the run begins: a patch, a report, a screenshot set, a migration plan, or some other reviewable object. "Improve the app" is not an artifact contract.

Treat authority as a task budget

The easiest setup is to launch every worker from the operator's normal shell. It is also the setup with the least useful separation.

A task that edits a local component rarely needs every token, deployment command, network destination, and destructive tool available to the human running it. Grant the repository, commands, credentials, and network surface required for that task. Leave the rest out.

Current agent harnesses do not all enforce these boundaries in the same way. If a tool cannot restrict a credential or network path, write that limitation into the run record and reduce the task's authority. Do not silently treat the operator's full shell as a reasonable default.

This also makes failures easier to understand. When a task has a small authority budget, the failure surface is smaller: you can see which files, processes, and external systems were in play without reconstructing the whole machine after the fact.

Keep integration serial on purpose

Implementation can run in parallel. Integration should have one owner and an explicit order.

Two patches can both be locally correct and still disagree about an API shape, migration sequence, dependency version, generated file, or user-visible behavior. Git may merge the text cleanly while the combined system is wrong. That is an integration conflict even when there are no conflict markers.

Give one person, or one tightly bounded integration role, responsibility for:

choosing the review order
resolving overlapping decisions
rerunning checks against the combined state
deciding what lands and what returns for another pass

The merge queue is where parallel work becomes one product. Making it serial is coordination, not wasted parallelism.

Completion needs evidence, not a status message

An agent saying "done" proves that it generated a completion message. It does not prove the patch works.

Every task should return an evidence bundle with:

the changed files and the intent of the diff
the exact checks run and their real results
screenshots or browser evidence when behavior is visual
known failures, skipped checks, and unresolved assumptions
merge-order or conflict notes for the integration owner

Repository-native agent systems are starting to expose this control layer directly. GitHub Agentic Workflows defines bounded jobs and safe-output handling. Its August 24 update describes safe-output validation, run steering, inspectable model and catalog state, and clearer startup diagnostics. GitHub Copilot's August releases describe task management, queued prompts, plan/autopilot behavior, and rewind. These are product descriptions, not reliability benchmarks, but the direction is useful: long-running work needs intervention, recovery, and inspectable output.

The evidence rule also applies to non-code artifacts. If a frontend task produces launch or social graphics, a narrow browser-local step such as Resize Image For can handle resizing, fitting, cropping or padding, previews, and export while keeping source-image processing in the browser. That is easier to review than handing the source files to another remote worker with a broad instruction to "prepare the assets."

Write the contract before launching the second agent

A small task record is enough. The exact schema matters less than forcing the decisions into the open.

task: fix-mobile-navigation
owner: agent-2
stop_when: mobile navigation passes the named browser checks

workspace:
  repository_scope: apps/web
  worktree: ../worktrees/fix-mobile-navigation
  shared_files: [pnpm-lock.yaml]

process:
  ports: [3102]
  services: [web-dev-server]
  browser_profile: agent-2-mobile

authority:
  credentials: []
  network: [localhost]
  destructive_tools: false

expected_artifacts:
  - code-diff
  - test-results
  - mobile-screenshots
  - known-issues

integration:
  merge_owner: human-reviewer
  review_after: shared-design-token-change
  rollback: discard-worktree-and-branch

This is an operating record, not a universal security configuration. The point is to make shared state, authority, evidence, and merge order reviewable before they become an incident report.

If the task cannot fill in these fields, it is probably too vague or too coupled to run concurrently.

Know when serial work wins

Parallelize tasks with separate state and separate validation paths. A documentation change in one package and an isolated test addition in another may fit well. Shared-schema changes, coupled migrations, global dependency upgrades, and work that relies on one fragile validation environment usually deserve a queue.

There is no prize for keeping every agent busy. If coordination and verification cost more than the execution time you saved, reduce the agent count.

Before opening another session, provision its workspace, authority, integration path, and evidence contract. If that is too expensive for the task, keep the work serial.

Source notes