DEV Community: Zira

Confidence Is Not Authorization: A Safer GitHub Issue-Automation Policy

Zira — Tue, 28 Jul 2026 11:14:50 +0000

GitHub's new issue automations can now label, type, assign, or close issues with a confidence level and a rationale. That sounds like a small UX improvement. It is actually a policy-design problem: a model's confidence is a guess about its own output, while authorization is a rule about what the system is allowed to change.

If you let those concepts collapse into one threshold, you can build an automation that is easy to operate and hard to govern. This guide turns the preview into a reviewable policy: automate low-impact metadata only when the action is reversible, keep ambiguous changes in a suggestion queue, and put security-sensitive decisions behind a real control boundary.

Scope note: GitHub describes rationale, confidence, and approvals as a public preview. The examples below are a policy template, not a claim that the preview is a security product.

What GitHub actually added

GitHub's July 23 changelog describes three controls for supported issue actions:

Approvals: ask the automation to suggest a change instead of applying it immediately.
Confidence: the agent labels an action high, medium, or low confidence. High-confidence changes can apply automatically; medium and low-confidence changes wait as suggestions.
Rationale: each action records why it was proposed or applied.

The initial action set covers labels, fields, issue type, closing, and assignees. You can search for waiting items with has:suggestions, and repository administrators can configure the automation level that sets the confidence threshold.

The important limitation is in GitHub's own wording: approvals are a workflow convenience, not a security control. If an agent has permission to change issues, it can apply changes directly instead of suggesting them. A review panel improves the workflow, but it does not replace permissions, branch protection, environment gates, or server-side authorization.

The policy mistake to avoid

Here is the tempting rule:

if confidence == "high":
    apply_change()
else:
    ask_a_human()

That rule treats confidence as if it were permission. It is not. A high-confidence label can still be the wrong label, and a correctly classified issue can still be unsafe to close or assign. The risk is not only model error. It is also stale repository policy, ambiguous ownership, duplicate reports, and a permission that is broader than the workflow author intended.

A safer decision has at least four inputs:

Action impact: what happens if the change is wrong?
Evidence quality: what facts support the action, and are they still current?
Reversibility: can a maintainer undo it without losing information or hiding work?
Authority: is this actor and this workflow actually allowed to perform the action?

Confidence can help rank work inside that decision. It should not make the authority decision for you.

A practical triage matrix

Use a policy like this as a starting point, then tune it against false positives from your own issue tracker:

Action	Default mode	Minimum evidence	Human review
Add a non-sensitive topic label	Suggest, or auto-apply in a bounded repo	Explicit matching rule plus current issue text	Sample audits
Set issue type or project field	Suggest	Rationale plus field ownership rule	Yes for ambiguous cases
Assign to a team or agent	Suggest	Current ownership map and matching component	Yes unless ownership is deterministic
Close an issue	Suggest	Duplicate or resolution evidence, with a reopen path	Yes
Apply a security, legal, or escalation label	Suggest	Evidence from a trusted system, not prose alone	Yes

The matrix deliberately treats “close” and “assign” differently from a harmless topic label. The model may be very sure about its classification and still lack the authority or context to make the consequential change.

Configure the workflow around intent, not just output

For GitHub Agentic Workflows, the changelog documents issue-intents: true for supported safe outputs such as set-issue-type, set-issue-field, add-labels, close-issue, assign-to-agent, and assign-to-user.

A minimal policy sketch could look like this:

---
name: Triage incoming issues
on:
  issues:
    types: [opened, edited]
permissions:
  issues: write
issue-intents: true
---

# Keep the model's proposed action narrow.
# Require a rationale that names the evidence it used.
# Prefer a suggestion for assignment, closure, or escalation.

This is not a complete security boundary. The issues: write permission is still consequential, and the policy comments are not enforcement. Treat the workflow as an orchestrator whose permissions and repository settings need independent review.

Make suggestions auditable

For every applied or suggested action, retain a small record with:

issue number and repository;
action, old value, and proposed value;
confidence level;
rationale and the evidence fields used;
workflow version and model/tool version when available;
actor permissions at execution time;
reviewer decision, timestamp, and any correction.

Rationale is useful only if it can be compared with the action. “This looks like a bug” is not enough for an assignment or closure. A stronger record names the signal, for example: “The issue mentions payments/checkout, the component map maps that path to Payments, and the current owner is team-payments.” Even then, the record is evidence for review, not proof that the change was authorized.

Test the policy with a replay set

Before changing the confidence threshold, create a small labeled set from real issues. Keep the set representative rather than enormous:

issue_id,expected_action,expected_value,review_required
101,add-label,bug,false
102,assign-team,payments,true
103,no-change,,true
104,close-duplicate,101,true
105,add-label,security,true

Replay it after changing the prompt, workflow, model, label taxonomy, or ownership map. Track separate outcomes:

action precision: when the automation acted, how often was the action correct?
action recall: how often did it find actions that should have happened?
abstention quality: when it suggested or skipped, did it avoid a costly mistake?
review load: how many suggestions were actionable versus noise?
reversal rate: how often did maintainers undo an applied change?

Do not optimize only for the percentage of issues processed automatically. A lower automation rate can be the better result if it removes dangerous false positives and keeps review focused on high-impact work.

Failure modes worth adding to the test set

Include cases that exercise the policy, not only the happy path:

a stale component-to-owner mapping;
a duplicate issue whose “canonical” issue was already closed;
a prompt injection in an issue body or pasted log;
a high-confidence classification with a missing required field;
a low-confidence suggestion that would be harmless to apply;
a security label inferred from an untrusted user claim;
an issue that changes meaning after an edit;
a workflow with write permission that should have been read-only.

For high-impact actions, test the permission boundary separately. An approval panel is not a substitute for verifying that a workflow cannot perform an action outside its intended scope.

A short rollout plan

Start in suggestion mode. Use has:suggestions to build a review queue and inspect rationales.
Measure reversals and review cost. Do not raise the threshold because the queue feels busy; inspect which suggestions are actually safe.
Automate only reversible metadata first. Keep closure, assignment, escalation, and security labels in review until the evidence and authority rules are stable.
Separate policy from prompt text. Store ownership, allowed labels, and action boundaries in versioned configuration or code.
Re-test after every taxonomy or permission change. A label rename or team change can invalidate an apparently good automation.

Checklist

[ ] Confidence is treated as a ranking signal, not authorization.
[ ] Each action has an explicit impact and reversibility class.
[ ] The workflow permission is narrower than the broadest possible agent capability.
[ ] Suggestions and applied actions retain rationale and evidence.
[ ] Closure and assignment have a review path and a rollback path.
[ ] Prompt-injection, stale-data, and missing-context cases are in the replay set.
[ ] Threshold changes are evaluated against precision, recall, abstention, review load, and reversals.
[ ] A human reviews the policy itself, not only individual suggestions.

GitHub's preview makes issue automation easier to inspect, which is valuable. The safe interpretation is not “the agent is confident, so let it act.” It is “the agent produced a proposal with evidence; now apply the repository's policy for impact, authority, and review.”

Sources

How are you separating model confidence from actual authorization in your issue or repository automations?

The Security Fix Is Not Done When the Agent Writes the Patch

Zira — Mon, 27 Jul 2026 20:57:46 +0000

A security agent can close the scanner alert and still leave your security process weaker.

That is the uncomfortable shift in GitHub's new agentic autofix preview: the system does not only suggest a patch. It explores the repository, edits code, reruns the original analysis, iterates if needed, and opens a draft pull request.

That loop is useful. It is also where teams need to move their definition of “verified.” A clean CodeQL result answers one question: does the original query still report this alert? It does not prove that the behavior is correct, the exploit is impossible, or that the change did not break a neighboring path.

This is the review workflow I would put around it.

What the preview actually does

GitHub says agentic autofix works with CodeQL and third-party code-scanning alerts. Assigning an alert to Copilot starts an agent session that:

explores relevant files across the codebase;
proposes a fix;
reruns CodeQL to validate that the alert is closed;
iterates if necessary; and
opens a draft pull request with a summary and validation steps.

The feature is in public preview. It requires GitHub Code Security or Advanced Security plus a Copilot license with cloud agent enabled. A run consumes AI Credits and GitHub Actions minutes.

Those constraints matter. This is not a free “fix everything” button, and it is not a replacement for a security review.

Source: GitHub Changelog: Agentic autofix for code scanning alerts

Treat scanner revalidation as the first gate

The automatic CodeQL rerun is valuable because it closes the loop on the alert that triggered the work. Keep that result, but store it as evidence for one specific claim:

The original scanner condition was no longer detected at the time of validation.

Do not silently promote that into “the vulnerability is fixed.” A patch can suppress a data-flow path, change the input shape, or move the behavior somewhere the original query does not cover.

A useful draft PR should make the evidence visible:

alert rule and severity;
original location and changed files;
CodeQL result before and after;
tests added or changed;
dependencies and configuration touched;
remaining assumptions or unvalidated paths.

If the PR does not contain that context, the reviewer has to reconstruct the agent's reasoning from logs.

Add a behavioral test that the scanner cannot provide

Code scanning is a static signal. Pair it with a test of the security property.

For example, if an alert concerns path traversal, test the boundary rather than only the implementation detail:

from pathlib import Path


def safe_read(root: Path, user_path: str) -> bytes:
    base = root.resolve()
    candidate = (base / user_path).resolve()

    if base not in candidate.parents and candidate != base:
        raise ValueError("path escapes workspace")

    return candidate.read_bytes()

The important tests are not just “a normal file can be read.” They include:

../ traversal;
absolute paths;
symlink escapes;
encoded separators if input crosses a URL boundary;
a missing file and a permission failure.

The exact cases depend on the alert. The principle is stable: the agent should fix the reported condition, while your tests assert the intended behavior at the trust boundary.

Review the patch as an agent-generated change

A draft PR is a good human checkpoint, but only if the reviewer looks beyond the diff.

I would ask four questions:

1. Did the fix reduce capability or merely hide a path?

A security patch should usually narrow what an input, identity, or tool can do. If it only changes a query shape, adds a suppression, or catches an exception broadly, the review needs to stop.

2. Did the agent change authorization or destination logic?

Changes to permission checks, URL construction, filesystem roots, shell commands, package installation, and serialization deserve focused review. These are places where a “small” patch can alter the reachable behavior of the whole service.

3. What did the agent not test?

Read the validation summary as a boundary, not a guarantee. “CodeQL passed” is not the same as “integration tests passed,” and neither is the same as exploitability review.

4. Can the result be reproduced?

Keep the alert identifier, agent session or run link, commit, tool output, and test commands. If the patch cannot be explained and replayed, it is difficult to audit later.

A practical merge policy

For high-severity or externally reachable findings, I would use this sequence:

alert assigned
  -> agent opens draft PR
  -> scanner reruns successfully
  -> focused behavioral test passes
  -> dependency/config diff reviewed
  -> exploitability and regression review completed
  -> human approves
  -> merge with normal CI and deployment controls

For lower-risk findings, some steps can be lighter. The policy should still be explicit. “The agent verified it” is not a policy.

Also set a budget before experimenting. GitHub says agentic autofix consumes AI Credits when a fix runs, and the activity consumes Actions minutes. Batch only alerts that share a clear review strategy. A large pile of unrelated draft PRs can increase review debt faster than it reduces vulnerability debt.

The useful mental model

Agentic autofix is best understood as a patch-producing loop with one built-in oracle: the original scanner. That is a meaningful improvement over a one-shot suggestion, but it is still only one oracle.

The safest workflow combines three independent questions:

Question	Evidence
Did the original alert close?	Scanner rerun
Does the intended security property hold?	Focused behavioral tests
Is the change safe in this system?	Human review, integration tests, and threat context

If those answers disagree, the patch is not ready.

GitHub's preview makes security remediation more executable. The engineering challenge is making the evidence more executable too.

How are you reviewing AI-generated security fixes today: scanner reruns only, focused exploit tests, or a stricter draft-PR policy?

Grok Build Is Open Source. The Useful Part Is the Harness

Zira — Mon, 27 Jul 2026 11:54:52 +0000

xAI has open-sourced Grok Build, its coding agent and terminal UI. The interesting part is not that another coding tool appeared on GitHub.

It is that the release makes the harness inspectable: context assembly, tool dispatch, code editing, command execution, plan review, diffs, skills, plugins, hooks, MCP servers, and subagents.

That changes the question from “Which model does this agent use?” to:

“Can I inspect, constrain, and test the system that turns model output into repository changes?”

This is a more useful question for anyone running coding agents against real code.

What the release actually exposes

According to xAI’s announcement, the published source includes:

the agent loop and context assembly
parsing model responses and dispatching tool calls
tools for reading, editing, and searching code
command execution
a terminal UI with plan review and inline diffs
an extension system for skills, plugins, hooks, MCP servers, and subagents

xAI also says Grok Build can run local-first: compile it yourself, point it at local inference, and configure it through the config.toml file.

That does not automatically make a deployment private or safe. It does give engineers something valuable: a codebase they can inspect instead of a black-box permission surface.

Repository: https://github.com/xai-org/grok-build

Start with the harness, not the model

When evaluating an open coding agent, I would inspect these paths before comparing model quality:

1. Context assembly

What files, command output, history, and instructions enter the prompt? Look for:

repository-wide files that are always included
generated or untrusted text that is treated as instructions
secrets or environment values that can enter context
truncation and compaction behavior

A model can be excellent and still make poor changes if the context builder silently mixes source code with untrusted instructions.

2. Tool dispatch

Trace the boundary between a model-produced tool call and the actual process that runs it.

Questions worth answering from the source:

Are tool arguments schema-validated before execution?
Is the working directory explicit?
Are shell commands allowlisted, denied, or merely confirmed in the UI?
Does the agent distinguish read-only, mutating, and network-capable tools?
What happens when a tool times out or returns malformed output?

The important artifact is not the chat transcript. It is the tool-call ledger: tool name, normalized arguments, cwd, exit status, files changed, and approval decision.

3. Review and commit boundaries

A plan view and inline diff are useful, but they are not the same as a security boundary.

Before letting an agent work in a valuable repository, I would require:

a clean working tree or disposable branch
a visible plan
a diff review before commit
tests and static checks after the final tool call
an explicit human decision before push, merge, or deployment

The last step matters because a harmless-looking code edit can still alter CI, release scripts, credentials, or dependency resolution.

A small local-first preflight

This is the kind of preflight I would run around any source-available coding agent. It keeps the agent useful while making its blast radius observable:

# Start from a disposable checkout
worktree=$(git worktree add --detach ../agent-checkout HEAD | tail -1)
cd "$worktree"

# Record the baseline before the agent starts
git status --short
printf 'commit=%s\n' "$(git rev-parse HEAD)"

# Run the agent with the narrowest repository permissions available.
# Review the exact config.toml fields for your build and provider.
# Do not paste secrets into the repository or prompt.

# After the run, inspect every changed path
git diff --stat
git diff -- . ':!*.lock'

# Re-run the repository's checks before accepting anything
npm test  # replace with the project's actual checks

The command is intentionally boring. The point is to make isolation, baseline state, changed files, and verification routine rather than relying on a reassuring UI.

What local-first does and does not solve

Running the harness and inference locally can reduce the number of repository contents sent to a hosted service. It does not answer every data-handling question.

You still need to inspect:

MCP server endpoints and their authorization
plugins and hooks that make network requests
telemetry and error-reporting paths
shell commands with access outside the checkout
model files and runtime dependencies
logs that may contain source code or credentials

Local compilation is a deployment option, not a threat model.

There is also a practical tradeoff: source inspection and local inference can increase operational work. Teams now own provider configuration, upgrades, model serving, extension review, and incident debugging. That can be worthwhile for sensitive repositories, but it is not free.

A useful evaluation scorecard

Instead of asking whether Grok Build “feels smart,” measure the harness:

Dimension	Evidence to collect
Context quality	Files and instructions included per task
Tool safety	Allowed tools, approvals, cwd, network access
Change quality	Tests passed, files changed, review corrections
Recovery	Behavior after timeout, bad arguments, or failed tests
Reproducibility	Same task on a clean checkout with the same config
Cost and latency	Model calls, tool calls, wall-clock time

Keep model capability and harness behavior as separate columns. Otherwise, a strong model can hide weak controls, or a careful harness can be blamed for a task it was never given enough context to solve.

The takeaway

Open-sourcing a coding agent is most valuable when it lets engineers examine the path from context to action. Grok Build gives developers a concrete harness to study, extend, and run local-first.

I would not treat that as proof that it is production-ready. I would treat it as an invitation to perform the checks that black-box agents make difficult: inspect tool boundaries, test failure paths, review extensions, and measure the actual change surface.

If you could inspect one part of a coding-agent harness before trusting it with a production repository, would you choose context assembly, shell/tool permissions, extension loading, or telemetry? Why?

Sources:

xAI, “Grok Build Is Now Open Source”: https://x.ai/news/grok-build-open-source
Grok Build repository: https://github.com/xai-org/grok-build

Your Coding Agent Needs a Code Brain, Not More Context

Zira — Mon, 27 Jul 2026 05:47:19 +0000

Your Coding Agent Needs a Code Brain, Not More Context

Most coding-agent failures do not begin with a weak model. They begin with missing repository context.

The agent cannot find the architectural decision that explains a strange abstraction. It misses the convention hidden in an old package. It reads the same files again because nobody turned yesterday's discovery into reusable project knowledge.

LangChain's open-source OpenWiki takes a useful, deliberately boring approach: generate a local Markdown wiki for a repository, keep it updated, and teach coding agents to consult it before exploring the codebase from scratch.

That is not magic memory. It is a maintained context artifact. For many teams, that distinction is the point.

What OpenWiki actually changes

OpenWiki has a Code Brain mode for a repository. Its CLI can initialize and update documentation in an openwiki/ directory. The project also maintains AGENTS.md and CLAUDE.md at the repository root so compatible coding agents are instructed to use the wiki when looking for context.

The important design choice is that the memory is inspectable:

Markdown files live with the project rather than inside an opaque vector database.
Changes can be reviewed as a pull request.
The wiki can be regenerated when the repository changes.
The agent gets a stable map of the codebase before it starts making edits.

The tradeoff is equally important: generated documentation can become stale or confidently wrong. A code brain is only useful if its update path is part of the engineering workflow.

A small workflow that is safe to try

Start with a branch, not your main checkout:

git switch -c chore/openwiki-refresh
openwiki --init
openwiki --update
git diff -- openwiki AGENTS.md CLAUDE.md

Read the diff as if it came from a junior engineer. Look for:

Commands or deployment details that should not be documented publicly.
Claims that are not supported by the current source tree.
Missing boundaries between generated files and human-owned guidance.
Instructions that could cause an agent to run destructive commands.
Large or noisy pages that will cost more context than they save.

If the result is useful, commit it and add an update job. The repository README shows a CI workflow pattern that can open a pull request when the documentation changes. Keep that PR reviewable. Do not silently merge generated architectural claims.

Make the agent use the wiki intentionally

A generated file does not automatically improve every agent. Make the contract explicit in the repository guidance:

# Agent context

Before changing code:

1. Read the relevant page under openwiki/.
2. Verify its claims against the current source files.
3. Treat source code and tests as authoritative when they disagree.
4. Update the wiki in a separate change when the architecture changes.
5. Never copy secrets, tokens, or private customer data into documentation.

This turns the wiki into a starting map, not an oracle. The agent still has to inspect the implementation and run tests.

What to measure after adding a code brain

Do not measure success by the number of generated pages. Measure the workflow:

Signal	What it tells you
Time to first correct file change	Whether the map improves navigation
Repeated file reads per task	Whether the agent is rediscovering the same context
Invalid tool calls	Whether instructions are clear enough
Review corrections about architecture	Whether the wiki is misleading
Task success and repair count	Whether the workflow is actually more reliable
Documentation refresh diff size	Whether updates are stable or noisy

Use the same small task set before and after adoption. Keep model, prompt, repository revision, and acceptance checks as constant as practical. A lower token count or shorter trace is not proof of higher developer productivity; it may just mean the agent skipped necessary verification.

Where this pattern fits

A code brain is most promising when a repository has:

multiple services or packages;
conventions that are hard to infer from one file;
recurring onboarding or maintenance tasks;
an agent that repeatedly explores the same architecture;
a team willing to review generated documentation.

It is less attractive for a tiny project whose README and tests already explain the system. It also does not solve authorization, sandboxing, prompt injection, flaky tests, or poor task specifications. Those remain separate controls.

The practical checklist

Before putting generated agent context into a shared repository, ask:

Is every page traceable to files, tests, or approved human notes?
Who owns stale-page cleanup?
Does CI open a reviewable PR instead of changing main silently?
Are secrets and sensitive source excerpts excluded?
Does the agent have a fallback when the wiki conflicts with code?
Can a developer delete or correct a page without fighting the generator?
Are task outcomes measured separately from context retrieval?

OpenWiki's interesting bet is not that agents need an infinite memory. It is that they need a compact, local, reviewable map of the software they are about to change.

That is a much easier capability to govern than “remember everything.”

Would you trust a generated openwiki/ directory in a production repository, or would you keep agent context in a separate, human-maintained handbook? What review rule would you require?

Sources

MCP 2026-07-28: A Migration Checklist for Stateless Servers and Safer Agent Tools

Zira — Sat, 25 Jul 2026 14:41:24 +0000

The next MCP specification is not a routine version bump. The 2026-07-28 release candidate removes the protocol-level session, adds first-class extensions and tasks, formalizes caching and trace context, and tightens the lifecycle for deprecated features.

The final specification is scheduled for July 28, 2026. If you maintain an MCP client, server, gateway, or coding-agent integration, the useful question is not “when do I upgrade?” It is “which assumptions in my stack are about to become invalid?”

This is a practical migration checklist based on the official release candidate, not a claim that every SDK has already adopted it.

What changes in the release candidate

Area	Before	2026-07-28 direction	Why it matters
Protocol lifecycle	Initialize/initialized handshake and session ID	Stateless request model	Easier horizontal scaling, fewer sticky-session assumptions
HTTP routing	Gateways often need body-aware routing or session affinity	`Mcp-Method` and `Mcp-Name` headers	Route and observe operations at the edge
Results	Limited cache semantics	`ttlMs` and `cacheScope` for list/resource reads	Make freshness an explicit product decision
Observability	Trace propagation varied by implementation	W3C Trace Context in `_meta`	Correlate model, gateway, tool, and backend work
Composition	Extensions were mostly experimental	Negotiated, independently versioned extensions	Add capabilities without coupling every change to the core spec
Long-running work	Request/response was the dominant shape	Tasks and multi-round-trip patterns	Model asynchronous work explicitly

The candidate is still a release candidate. Treat the final specification and your SDK’s support matrix as the compatibility authority.

1. Remove accidental session state

The biggest operational change is the removal of the protocol-level session and the Mcp-Session-Id header. A remote server should no longer depend on a request returning to the same process because a prior MCP handshake happened there.

That does not mean your application cannot have state. It means application state must be explicit rather than hidden inside the protocol connection.

Audit for:

in-memory maps keyed only by session ID;
sticky-session requirements in the load balancer;
initialization code that populates process-local capabilities;
connection cleanup that is doing business-state cleanup;
tests that send a handshake once and reuse an implicit server session.

A safer shape is to make the request carry the identity your application actually needs:

from dataclasses import dataclass

@dataclass
class AgentContext:
    tenant_id: str
    run_id: str
    user_id: str


def authorize_tool_call(ctx: AgentContext, tool_name: str) -> None:
    policy.check(tenant=ctx.tenant_id, run=ctx.run_id, tool=tool_name)

Keep this context in your auth layer, task store, or trace system. Do not recreate protocol session affinity under a new name.

2. Make the gateway boring

Stateless MCP is valuable only if the edge can treat requests as ordinary routable traffic. Before changing the load balancer, test that every instance can handle:

a fresh request;
a request after another instance handled the previous request;
a retry after a timeout;
a request with no process-local warm-up;
a request whose backend operation is still running asynchronously.

Log the protocol method, MCP name, request ID, server instance, tenant, and trace ID. Never put API keys, tokens, or user content into routing headers. Headers are visible to proxies, access logs, and telemetry systems.

3. Add cache policy, not just a cache

The candidate’s ttlMs and cacheScope fields make caching inspectable, but they do not decide whether your data is safe to reuse.

For every list or resource read, answer:

Is the result public, tenant-scoped, user-scoped, or run-scoped?
What event invalidates it?
Is stale data acceptable for this tool?
Can a cache key include the full authorization scope?
Will the agent mistake a cached result for a live check?

A useful default is short-lived and scope-bound. Cache documentation and immutable metadata more aggressively than issue status, permissions, or deployment state.

4. Wire trace context end to end

Propagating traceparent, tracestate, and baggage in _meta is only useful if your system keeps one trace across the whole action:

agent run -> model call -> MCP gateway -> MCP server -> database/API -> tool result

At minimum, record latency, retry count, authorization decision, tool name, result size, and failure class. Redact secrets before exporting metadata. This is especially important for coding agents, where a “successful” tool call can still produce an unsafe edit or an incomplete migration.

Zira’s earlier guide on observability for coding agents covers the broader instrumentation trade-offs. MCP trace context gives that instrumentation a protocol-level place to travel.

5. Treat Tasks as a resource with limits

Tasks are a better fit for long-running work than holding an HTTP request open, but they introduce lifecycle and capacity questions. Define:

who may create a task;
maximum runtime and queue depth;
polling or notification policy;
cancellation semantics;
retention and deletion rules;
idempotency behavior after client retries;
what the agent is allowed to do while the task is pending.

Do not let “async” become “unbounded.” A task that invokes tools, starts builds, or touches a repository needs the same approval and sandbox policy as a synchronous call.

6. Validate extensions and deprecations explicitly

The release candidate makes extensions first-class and introduces a formal deprecation policy. That is healthier than silently depending on undocumented behavior, but it means capability negotiation belongs in tests.

Create a compatibility matrix for each client/server pair:

negotiated protocol version;
supported extension IDs and versions;
Tasks support;
Apps support, if applicable;
transport support;
deprecated feature usage;
fallback behavior when a capability is missing.

Run it in CI against the exact SDK versions you ship. The previous finalized specification may remain in use while SDKs adopt the candidate at different speeds.

A 72-hour migration plan

Today: inventory session, routing, cache, trace, task, and deprecated-feature assumptions. Pin current SDK versions and save representative request fixtures.

Next: run a two-instance server behind a non-sticky load balancer. Add negative tests for missing capabilities, expired tasks, unauthorized scopes, duplicate retries, and stale cache entries.

Before July 28: read the final specification, check your SDK release notes, and canary one low-risk server. Keep rollback support for the previous finalized version until your clients have converged.

The practical takeaway

MCP 2026-07-28 is a scaling opportunity, not a reason to delete every state store. Remove hidden protocol coupling, make application state explicit, bind caches to authorization scope, propagate traces, and put hard limits around asynchronous work.

The teams that migrate safely will be the ones that test the boundaries around the protocol, not just the happy-path tool call.

If you maintain an MCP server today, which assumption is hardest to remove: sticky sessions, cache scope, trace propagation, or long-running task control?

Sources

Gemini 3.6 Flash Is Built for Faster, Cheaper AI Agents

Zira — Wed, 22 Jul 2026 07:07:38 +0000

Google has released Gemini 3.6 Flash, a production-ready model focused on coding, multimodal processing, and multi-step agent workflows.

The new model is available through the Gemini API using:

gemini-3.6-flash

What Changed?

Gemini 3.6 Flash improves on Gemini 3.5 Flash in several practical areas:

17% fewer output tokens on the Artificial Analysis Index
Fewer reasoning steps, tool calls, and repeated execution loops
Better code generation with fewer unwanted file changes
Improved document, chart, image, audio, video, and PDF understanding
Native support for function calling, code execution, search grounding, file search, and computer use

Google reports that the model scored 49% on DeepSWE, compared with 37% for Gemini 3.5 Flash. It also improved on OSWorld-Verified from 78.4% to 83%.

Context Window and Pricing

Gemini 3.6 Flash supports:

1,048,576 input tokens
65,536 output tokens
$1.50 per million input tokens
$7.50 per million output tokens

The input price remains unchanged from Gemini 3.5 Flash, while the output price drops from $9 to $7.50 per million tokens.

Basic Python Example

from google import genai

client = genai.Client()

response = client.interactions.create(
    model="gemini-3.6-flash",
    input="Review this codebase and create a safe migration plan."
)

print(response.output_text)

One Migration Warning

When moving to Gemini 3.6 Flash, remove deprecated parameters such as:

temperature
top_p
top_k
candidate_count

Applications using thinking_budget should migrate to the newer thinking_level setting.

Final Take

Gemini 3.6 Flash is not simply a faster chatbot model. Its real target is production agent systems that repeatedly inspect files, generate code, call tools, and complete multi-step tasks.

The lower output price and reduced token usage could make it a practical upgrade for teams already running Gemini-based coding agents or automation workflows.

Official resources: Gemini 3.6 Flash documentation and Google’s release announcement.

Automate Bug Fixes From GitHub Issues With Claude Code

Zira — Tue, 21 Jul 2026 11:41:22 +0000

Small bug fixes often follow the same workflow:

GitHub issue → investigate → edit code → run tests → open pull request

Claude Code can automate most of this while keeping review and merging under human control.

1. Install Claude Code

npm install -g @anthropic-ai/claude-code
cd your-project
claude

Then connect it to GitHub:

/install-github-app

Follow the setup steps to install the GitHub app and add the required API key secret.

2. Add Repository Instructions

Create a CLAUDE.md file:

# Project Rules

- Run `npm test` before completing changes.
- Run `npm run lint`.
- Add a regression test for every bug fix.
- Do not modify unrelated files.
- Do not add dependencies without approval.
- Keep changes small and focused.

This prevents the agent from improvising its own development philosophy, which is rarely a gift.

3. Add CI Checks

Create .github/workflows/ci.yml:

name: CI

on:
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm

      - run: npm ci
      - run: npm run lint
      - run: npm test

Now every generated pull request is checked independently.

4. Trigger the Workflow

Create a detailed GitHub issue, then comment:

@claude investigate this bug and create a fix.

Requirements:
- Find the root cause
- Make the smallest safe change
- Add a regression test
- Run lint and tests
- Create a draft pull request
- Do not modify unrelated files

Claude Code can inspect the repository, implement the fix, and prepare the pull request.

5. Keep Humans in Control

Do not let the agent merge directly into main.

Require:

Passing CI checks
Human review
Branch protection
Draft pull requests
Manual approval for security, payments, or authentication changes

The useful goal is not fully autonomous development.

It is reducing repetitive work while humans still make the final decision.

Claude Code vs Codex vs Gemini CLI: The Harness Matters More Than the Model

Zira — Tue, 21 Jul 2026 08:42:02 +0000

Developers often compare AI coding agents by asking one question:

Which model is smartest?

That is increasingly the wrong question.

Claude Code, OpenAI Codex, and Gemini CLI can all inspect repositories, edit files, execute commands, run tests, and automate development tasks. Their real differences come from the systems surrounding the models:

How they collect context
Which tools they can access
How they request permission
How they recover from failure
How easily they fit into scripts and team workflows

The model matters. But the agent harness determines how that intelligence behaves inside a real codebase.

Quick comparison

Area	Claude Code	OpenAI Codex	Gemini CLI
Best fit	Deep repository work	Structured coding automation	Open and customizable CLI workflows
Interface	Terminal-first agent	CLI and broader Codex workflows	Open-source terminal agent
Project context	`CLAUDE.md`	Repository instructions and task context	`GEMINI.md`
Automation	Hooks, print mode and scripts	CLI, structured workflows and integrations	Non-interactive mode, JSON and stream output
Extensibility	MCP and hooks	MCP, skills and external commands	MCP, extensions and custom tools
Main advantage	Strong codebase reasoning	Controlled execution and workflow integration	Openness and provider ecosystem
Main tradeoff	Can become expensive on long sessions	Results depend heavily on task specification	Fast-moving releases may introduce regressions

There is no universal winner. The correct choice depends on the type of work being delegated.

Claude Code: best for deep repository work

Claude Code is designed to operate directly inside a project from the terminal. It can search files, understand relationships across a repository, edit code, run commands and iterate against test results.

Its biggest strength is not merely code generation. It is maintaining a useful mental model of a codebase while moving through a multi-step task.

That makes it particularly useful for:

Investigating unfamiliar repositories
Planning large refactors
Tracing bugs across multiple modules
Updating implementations and related tests together
Reviewing architectural decisions
Executing tasks that require repeated reasoning

Projects can provide persistent instructions through CLAUDE.md. Teams can use this file to describe architecture, commands, conventions, testing requirements and boundaries the agent should respect.

Claude Code also supports Model Context Protocol integrations, allowing it to interact with external tools and data sources instead of treating the repository as an isolated folder.

Its CLI also supports structured output, including JSON and streaming JSON, which makes it possible to use Claude Code inside scripts rather than only through interactive sessions.

Where Claude Code fits best

Use Claude Code when the task sounds like:

Understand how this subsystem works, identify the root cause, propose a fix, implement it and verify that nothing else broke.

That is different from asking an assistant to produce one isolated function.

Main limitation

Long exploratory sessions can consume substantial context and tokens. Without clear repository instructions, agents may repeatedly search the same areas or make technically valid changes that conflict with local conventions.

Claude Code performs better when the repository contains:

Clear project instructions
Reliable test commands
Small, reviewable task boundaries
Explicit rules for generated files and migrations
Permission limits for destructive operations

The better the environment, the less intelligence gets wasted rediscovering basic facts.

OpenAI Codex: best for structured execution

Codex is strongest when coding work needs to become a repeatable workflow rather than an extended conversation.

It is well suited to tasks such as:

Implementing clearly scoped changes
Updating documentation from source code
Creating internal developer commands
Running repeatable evaluation tasks
Automating maintenance work
Integrating coding actions with team systems

OpenAI’s official Codex examples increasingly extend beyond writing code. They include maintaining documentation, creating command-line tools, preparing product requirements and combining repository context with external organizational information.

This represents an important shift.

A coding agent is no longer only a developer sitting inside the terminal. It can become an execution layer between repositories, issue trackers, documentation systems, logs and internal tools.

Where Codex fits best

Use Codex when the task sounds like:

Take this defined issue, make the required changes, run the relevant checks and return a reviewable result.

Codex benefits from clear acceptance criteria. The more precisely the expected result can be verified, the easier it is to delegate safely.

Main limitation

Structured execution does not eliminate ambiguity.

If a task contains vague product decisions, hidden architectural assumptions or unclear ownership boundaries, the agent may confidently produce a complete solution to the wrong interpretation.

Codex should not be treated as a replacement for task definition. It amplifies the quality of the task it receives, including the bad parts.

Gemini CLI: best for open and customizable workflows

Gemini CLI is an open-source terminal agent released under the Apache 2.0 licence.

It supports:

Interactive terminal sessions
Non-interactive prompts
JSON output
Streaming JSON events
Multiple project directories
Persistent context through GEMINI.md
MCP servers
Extensions and custom commands
Built-in tools such as web search

This makes Gemini CLI particularly interesting for developers who want to inspect, modify or extend the agent environment instead of accepting a fully closed workflow.

Its non-interactive mode can be used in shell scripts:

gemini -p "Explain the architecture of this repository"

Structured output can be requested for automation:

gemini -p "Run the tests and summarize failures" \
  --output-format json

For long-running workflows, it can stream newline-delimited events:

gemini -p "Run tests, fix failures and report the result" \
  --output-format stream-json

Gemini CLI also supports MCP and custom tool discovery, providing several ways to connect the agent to external systems.

Where Gemini CLI fits best

Use Gemini CLI when the task sounds like:

Build a transparent terminal workflow that I can customize, script, inspect and connect to other tools.

It is especially relevant for:

Open-source toolchains
Experimental agent workflows
CI scripts
Model and tool research
Developers already working heavily with Google’s ecosystem

Main limitation

Gemini CLI publishes preview, stable and nightly release channels. That is useful for rapid experimentation, but preview and nightly builds may carry regressions or incomplete validation. Production workflows should pin versions and avoid casually tracking the newest build simply because it exists.

What empirical research says

Most coding-agent comparisons are based on demos, benchmarks or one developer’s preferred stack.

Recent research paints a less convenient picture.

A 2026 study analysed 7,156 pull requests created by Codex, GitHub Copilot, Devin, Cursor and Claude Code. It found that task type had a major effect on acceptance.

Documentation changes had an acceptance rate of 82.1%, while new-feature work reached 66.1%. That 16-point difference was larger than the usual performance gap between agents.

The study found:

Claude Code led on documentation and feature tasks
Cursor performed strongly on fix tasks
Codex maintained relatively strong results across task categories
No agent performed best on every type of work

The useful conclusion is not that one tool won.

It is that the task category may matter more than the brand of agent performing it.

Reliability is still the uncomfortable part

Another study examined more than 3,800 publicly reported bugs across Claude Code, Codex and Gemini CLI.

More than 67% were functionality-related. API, integration and configuration problems accounted for 36.9% of identified root causes.

Common symptoms included:

API failures
Terminal problems
Broken commands
Tool-invocation errors
Command-execution failures

The research found that tool invocation and command execution were among the most failure-prone parts of the agent workflow.

This matters because developers often evaluate agents using successful demonstrations.

Production systems must be evaluated using failure recovery.

A useful coding agent needs to do more than produce correct code when everything works. It must also behave predictably when:

A command exits unexpectedly
Authentication expires
A tool returns malformed data
Tests are flaky
Repository instructions conflict
A dependency has changed
The agent reaches a permission boundary

The polished demo is the least interesting part of the system.

Model versus harness

A coding agent can be simplified into five layers:

Model
  ↓
Context management
  ↓
Tool selection
  ↓
Execution environment
  ↓
Verification and recovery

Developers usually focus on the first layer.

Production failures frequently occur in the remaining four.

Two agents using similarly capable models can behave very differently because one:

Reads project instructions more reliably
Searches fewer irrelevant files
Uses tools more efficiently
Requests permission at better moments
Runs the correct tests
Recognizes failed execution
Preserves state between steps

This is why switching models does not automatically fix a weak agent workflow.

Sometimes the agent does not need a smarter model. It needs better context, narrower tools and a clearer definition of completion.

Which one should you choose?

Choose Claude Code when:

Repository understanding is the main challenge
Tasks regularly cross several files
You perform architectural investigation
You want strong interactive reasoning
Your team can maintain useful project instructions

Choose Codex when:

Tasks have clear acceptance criteria
You want repeatable coding automation
Work needs to connect with other systems
You are building delegated engineering workflows
Execution and review need clearer separation

Choose Gemini CLI when:

Open source matters
You want to inspect or extend the agent
CLI scripting is central to your workflow
You need JSON or streaming output
You are experimenting with MCP and custom tools

A better evaluation method

Do not evaluate coding agents using one prompt that generates a landing page.

That test mostly measures how effectively the agent can produce something visually convincing before anyone reads the code.

Instead, create a small internal evaluation set based on real work:

Fix a bug spanning multiple modules.
Add a feature with existing architectural constraints.
Refactor code without changing behaviour.
Update tests after an API change.
Explain an unfamiliar subsystem.
Diagnose a deliberately broken build.
Follow repository-specific security rules.
Stop safely when requirements are ambiguous.

For each task, record:

Completion rate
Human corrections required
Tests passed
Unnecessary files changed
Tokens or credits consumed
Time spent reviewing
Permission violations
Recovery after failed commands

The best tool is not the one that generates the most code.

It is the one that produces the highest percentage of changes you are willing to merge.

Final verdict

Claude Code, Codex and Gemini CLI are converging on a similar interface: an agent that can inspect a repository, call tools, execute commands and modify code.

Their meaningful differences now lie in workflow design.

Claude Code is strongest for deep, interactive repository reasoning.
Codex is strongest for defined tasks and structured execution.
Gemini CLI is strongest for open, customizable terminal automation.

None should be trusted without tests, permission boundaries and human review.

The next generation of coding-agent competition will not be decided only by which company releases the smartest model.

It will be decided by which harness turns intelligence into reliable engineering work.

Observability for Coding Agents: OpenAI Agents SDK vs LangChain vs Google ADK

Zira — Tue, 21 Jul 2026 06:49:53 +0000

Your coding agent can pass a demo and still be impossible to debug in production.

When a run edits the wrong file, burns through a retry loop, or silently loses a child-agent span, the useful question is not only “what did the model answer?” It is “what happened across the entire workflow?”

This comparison looks at observability paths in three popular Python stacks:

OpenAI Agents SDK
LangChain agents with LangSmith
Google Agent Development Kit (ADK)

The practical hook: choose the stack that gives your team the right evidence for a failed coding task without leaking repository content or locking every service to one telemetry destination.

Scope and evaluation criteria

This is an official-documentation comparison checked on July 21, 2026. I did not run a shared benchmark, measure export latency, or score the dashboards. Treat the “best fit” guidance as engineering judgment, not a universal ranking.

I compared five things:

Default capture: What does one agent run record without custom spans?
Control and export: Can you disable, extend, replace, or fan out telemetry?
Coding-agent usefulness: Can a trace explain model turns, tool calls, handoffs, and failures?
Privacy controls: How explicit is the path for excluding prompts, outputs, or message content?
Evaluation loop: How naturally can traces become datasets, feedback, or regression checks?

At a glance

Stack	Default observability path	Strongest debugging signal	Export/control shape	Main trade-off
OpenAI Agents SDK	Built-in traces and spans sent to the OpenAI Traces dashboard	A structured hierarchy for runs, model turns, tools, guardrails, and handoffs	Add processors or replace the default processors; community integrations are listed in the docs	The default backend is OpenAI, and tracing is unavailable for organizations using OpenAI API Zero Data Retention
LangChain + LangSmith	LangChain agents automatically support LangSmith tracing when enabled	Step-by-step agent traces with tools, model interactions, decisions, metadata, and tags	Native tracing plus OpenTelemetry export, alternate OTLP endpoints, and collector fan-out	Requires LangSmith setup for the easiest path; distributed traces can disappear if ancestors are never exported there
Google ADK	Built-in logging, metrics, and traces	Agent workflow telemetry plus events from tools and multi-agent execution	OpenTelemetry is a first-class route; integrations can add analysis and monitoring	You still need to choose a backend and decide how much message content to capture

The table describes documented capabilities, not equal implementations. “Built-in” does not mean identical span names, retention, sampling, or dashboard features.

1. OpenAI Agents SDK: fastest path to a workflow trace

The SDK traces a complete Runner.run() by default. Its documented hierarchy includes task and turn spans, agent spans, model generations, function tools, guardrails, handoffs, and custom spans. That is a good match for a coding agent where the failure may be in the transition between steps rather than in the final text.

A minimal run can therefore answer questions such as:

Did the agent call the repository search tool before editing?
Which model turn produced the risky tool arguments?
Did a handoff send the task to the reviewer agent?
Did a guardrail run, and where did it sit in the trace?

The SDK also exposes a useful control boundary. Add a custom trace processor to send a second copy elsewhere, or replace the default processors entirely. For long-running workers, the docs call out flush_traces() when you need an immediate delivery guarantee after a unit of work.

The privacy footgun is equally clear: generation and function spans may contain sensitive data, and trace_include_sensitive_data is true by default. A coding-agent team should decide whether prompts, diffs, file contents, and tool arguments belong in the trace before enabling this in a repository with secrets or customer data.

Choose this path when: you are already using the OpenAI Agents SDK, want rich workflow structure with little instrumentation code, and accept OpenAI Traces or are prepared to install a custom processor.

2. LangChain + LangSmith: the most complete trace-to-evaluation workflow

LangChain’s current agent docs say agents built with create_agent automatically support LangSmith tracing. Enabling it is primarily configuration: set LANGSMITH_TRACING=true and provide an API key. The resulting traces cover the input-to-output path, including tools, model interactions, and decision points. Tags and metadata can be attached per invocation, which is especially useful for coding-agent slices such as repository, branch, model route, or task type.

The differentiator is not merely “it has traces.” LangSmith’s observability docs connect traces to debugging, evaluation, monitoring, feedback, and datasets. That makes it a natural fit for a team asking:

Which tool-error pattern is recurring across repositories?
Did a prompt or model change increase failed patch attempts?
Can production traces seed a regression set before the next harness change?

It also has a vendor-neutral route. LangSmith documents OpenTelemetry tracing for LangChain and non-LangChain applications, including collector fan-out to multiple destinations. That is useful when your coding agent spans an API service, a sandbox worker, and a separate test runner.

There is a subtle distributed-tracing limitation worth testing: the docs state that a span whose parent is never sent to LangSmith is dropped. A successful OTLP response from the collector is not proof that the complete trace will appear as one connected run. Export the ancestors you need, or make the loss mode visible in your own checks.

Choose this path when: trace search, metadata, human feedback, evaluation, and production monitoring are part of the same improvement loop, or when you need OpenTelemetry fan-out.

3. Google ADK: telemetry as a framework capability, backend as a choice

Google ADK’s observability documentation treats logging, metrics, and traces as built-in capabilities rather than an afterthought. Its examples show OpenTelemetry for traces and a logging plugin for detailed output. The docs also warn that basic input/output monitoring is insufficient for complex agents because debugging needs visibility into reasoning traces, tool calls, and other internal activity represented in telemetry.

That model fits multi-agent coding workflows where the orchestration graph matters: a planner may delegate to a code-search agent, which invokes tools, which triggers a test runner. You want to correlate those events with service-level metrics, not only inspect the final response.

ADK leaves more backend choice to you. The official quick start configures an OTLP exporter, and the docs point to observability integrations for additional monitoring and analysis. This can be an advantage for teams that already run an OpenTelemetry Collector, but it is also setup work: you must choose the exporter, backend, retention policy, and content-capture policy.

The privacy control is explicit in the docs’ Kotlin example: capturing full message content is configurable and should be used with caution in production. The same operational rule applies to a coding agent in any language: structured metadata such as repository ID, task ID, model name, and exit status is often safer than storing full prompts, diffs, or tool payloads.

Choose this path when: you want framework-level logging, metrics, and traces, run multi-agent workflows, and already have an OpenTelemetry-oriented platform team.

A practical coding-agent trace contract

Whichever framework you choose, define a small contract before adding every possible payload to telemetry:

task_id: task_2026_07_21_0042
repo: payments-api
base_revision: abc123
agent: implementer
model: router-selected
tool: run_tests
side_effect: read-only
outcome: failed
error_class: test_failure
duration_ms: 1840

Store the full diff, command output, and prompt only where your security policy allows it. Keep the trace useful even when content capture is disabled. A task ID and stable error class are often enough to join a redacted trace to an internal artifact store.

This complements, rather than replaces, a regression strategy. A trace shows what happened; an evaluation or replay check decides whether the behavior is acceptable. For a concrete workflow, see Stop Replaying Coding-Agent Bugs by Hand: Turn Traces Into Regression Tests.

Decision checklist

Pick OpenAI Agents SDK if:

You want detailed agent-native spans immediately.
Your team is comfortable with OpenAI’s tracing destination or custom processors.
The agent’s main complexity is handoffs, guardrails, and tool execution.

Pick LangChain + LangSmith if:

You need traces connected to feedback, datasets, evaluations, and monitoring.
You want tags and metadata for slicing coding tasks.
You need documented OpenTelemetry export or multi-destination fan-out.

Pick Google ADK if:

You want logging, metrics, and traces as part of an ADK workflow.
You operate a multi-agent system and already have an OTLP backend or collector.
Backend portability matters more than a zero-setup hosted dashboard.

If your agent performs side effects, pair observability with explicit error classification and retry policy. A trace can reveal that a tool failed, but it should not silently turn a non-idempotent write into a retry. Tool Errors Are Not Retries covers that boundary.

What to do now

Pick one representative coding task: a failing test plus a small patch is enough.
Run it with content capture disabled first.
Verify that the trace still records agent, model, tool, handoff, duration, status, and task ID.
Export one trace to the backend your team actually operates.
Intentionally fail a tool and confirm the failure is visible without exposing secrets.
Promote the trace to a regression case only after you can state the expected outcome.

Candid limitations

This is a documentation-based comparison, not a common benchmark. I did not measure dashboard query speed, export overhead, retention, pricing, or trace completeness under process crashes. Framework versions and integrations change quickly. OpenTelemetry compatibility also does not guarantee identical semantics across backends: span names, attributes, sampling, and redaction behavior still need a small integration test.

The right observability stack is the one your team will actually inspect during a failed run and can operate within its data-retention and security constraints.

Sources

What is the first field or event you would refuse to omit from a coding-agent trace, and why?

Durable State for Coding Agents: LangGraph vs Google ADK vs OpenAI Agents SDK

Zira — Tue, 21 Jul 2026 03:49:23 +0000

Long-running coding agents fail in ways a chat demo hides: the worker restarts, a human approval arrives hours later, or a tool call is interrupted after changing the repository. “Memory” alone is not enough. You need to know what state is saved, what resumes, and what may run twice.

This is a documentation-based comparison of LangGraph, Google ADK, and the OpenAI Agents SDK. It is not a benchmark, and I did not run a common workload across the three. I reviewed the official documentation on July 21, 2026.

Evaluation criteria

I compared each option on:

State model: conversation history, workflow state, and cross-session data.
Durability boundary: what is persisted and when.
Resume primitive: how a paused or interrupted run continues.
Replay and side effects: whether tools or nodes can execute again.
Production path: storage backends and operational control.
Coding-agent fit: repository edits, tests, approvals, and long waits.

At a glance

Framework	Primary durable abstraction	Resume handle	Cross-session memory	Main sharp edge
LangGraph	Checkpointer for thread state; Store for shared data	thread_id plus graph input or Command	Store, separate from checkpoints	Resumed nodes can replay; side effects must be idempotent or isolated in tasks
Google ADK	Session events and state via a SessionService	session_id plus prior invocation_id when resumability is enabled	user: and app: state scopes with a persistent service	Tools are at-least-once on resume and may run more than once
OpenAI Agents SDK	Session history, RunState, or server-managed conversation	Same session plus resumed RunState	Pluggable sessions or OpenAI-hosted Conversations	Sessions preserve history, but durable workflow execution is an integration concern

These are not interchangeable terms. A conversation session can preserve messages while still leaving your workflow step, approval status, or file-write outcome undefined.

1. LangGraph: explicit checkpoints plus a separate Store

LangGraph splits persistence into two deliberate layers. A checkpointer saves graph state for one thread. A Store holds application-defined key-value data across threads. That maps cleanly to a coding agent:

checkpoint: the current repository task, plan, tool results, and approval pause;
store: reusable user preferences, team conventions, or a verified project index.

The thread ID is the durable cursor. For a human approval, interrupt() pauses execution, saves state, and later resumes with a Command using the same thread. This is a strong fit when you want the workflow graph and its control flow to be inspectable.

The important limitation is replay. LangGraph documents that execution resumes from a checkpoint boundary, not from the exact Python line. A node can run again. Wrap nondeterministic work and side effects in tasks or nodes, make writes idempotent, and use an idempotency key for operations such as opening a pull request or applying a migration.

Best fit: a coding workflow with explicit stages such as inspect → plan → approve → edit → test → review, especially when you need time travel, human-in-the-loop pauses, or recovery after failure.

2. Google ADK: event-backed sessions and resumable workflows

Google ADK’s Session is more than a message list. Its state is a serializable key-value scratchpad, and the SessionService determines whether it survives a restart. InMemorySessionService is not durable; DatabaseSessionService and VertexAiSessionService are the production-oriented persistent choices documented for state.

ADK also makes scope visible through prefixes:

no prefix: current session;
user: shared across that user’s sessions;
app: shared across the application;
temp: current invocation only and not persistent.

With resumability enabled, ADK records completed workflow tasks in Events. A stopped workflow can be resumed with the saved session and invocation IDs. Sequential and parallel workflows use recorded completion state to avoid re-running completed branches where possible.

But the safety warning matters for coding agents: the official resume documentation says tools are guaranteed to run at least once and may run more than once during resume. A tool that edits a file, pushes a branch, or creates an issue therefore needs duplicate protection and a post-run reality check.

Best fit: teams already using Google’s agent runtime and wanting session-scoped, user-scoped, or app-scoped state plus resumable sequential or parallel workflows.

3. OpenAI Agents SDK: flexible sessions, explicit run continuation

The OpenAI Agents SDK provides a pluggable Session interface. The runner loads session items before a run and persists new items afterward. The official Python docs list file-backed SQLite, Redis, SQLAlchemy, MongoDB, Dapr, encrypted wrappers, and OpenAI-hosted Conversations as options.

For a paused human approval, the SDK serializes a RunState. You approve or reject the interruption, then run again with that state and the same session. That is a useful distinction: the session stores the conversation history, while RunState represents the interrupted execution that must continue.

The SDK also lets you limit retrieved history or customize how history is merged before a model call. That helps control context growth, but it is not the same as checkpointing every arbitrary workflow step. For long waits, retries, or process restarts, the official running-agents guide points to durable-execution integrations such as Temporal, Dapr, Restate, and DBOS.

Best fit: an agent application that primarily needs durable conversation and approval continuation, or a team comfortable adding a workflow runtime when execution must survive long outages and waits.

The coding-agent decision rule

Choose based on the failure you must recover from:

“Resume this graph at a known stage and inspect prior state.” Start with LangGraph.
“Persist session, user, and app state inside Google’s agent runtime.” Start with Google ADK.
“Persist conversation history and resume an approval, then add a workflow engine when needed.” Start with OpenAI Agents SDK.

Do not select based only on the word “memory.” Ask where these facts live after a crash:

Which commit or worktree was active?
Which tools completed successfully?
Was the approval granted for this exact command and arguments?
Is the next step safe to replay?
Can an operator inspect and delete stale state?

A practical state contract

Regardless of framework, persist a small, explicit contract rather than dumping the entire prompt into memory:

{
  "task_id": "issue-1842",
  "repo_sha": "abc123",
  "worktree": "/worktrees/issue-1842",
  "phase": "tests",
  "approved_action_hash": "sha256:...",
  "completed_tools": ["read_diff", "run_unit_tests"],
  "next_action": "run_integration_tests",
  "idempotency_keys": ["issue-1842:patch:1"]
}

Keep secrets and untrusted tool output out of durable memory unless you have a retention, encryption, and retrieval policy. Persist identifiers and verified results, not credentials.

What to do now

Pick one failure scenario: worker restart during tests, approval wait, or duplicate patch application.
Draw the state boundary: conversation, workflow cursor, tool result, and cross-session memory are separate things.
Add a stable task ID and idempotency key to every side-effecting tool.
Run the same scenario twice: once normally and once after killing the worker at each tool boundary.
Verify the repository, branch, issue tracker, and CI system after resume. “The framework resumed” is not proof that the side effect happened exactly once.

Candid limitations

This comparison is based on official documentation, not a common benchmark. APIs and integrations change quickly, and “persistent” does not by itself specify transactionality, encryption, retention, multi-writer behavior, or exactly-once effects. Storage latency, deployment topology, and tool design can dominate the framework choice. Treat the documented resume semantics as design constraints, then test your own failure modes.

If you are working on observability, see how to turn agent traces into regression tests. For a related control-flow choice, see handoffs vs subagents across agent frameworks.

Sources

Discussion

What is the first side effect your coding agent must make safe to replay: a file edit, a commit, a pull request, or a production change?

Handoffs vs Subagents: OpenAI Agents SDK vs Google ADK vs LangGraph

Zira — Tue, 21 Jul 2026 00:53:48 +0000

“Multi-agent” is often used as if it describes one architecture. It does not.

For a coding workflow, there is a big difference between transferring ownership of the conversation to a specialist, calling a specialist and getting a result back, and running a fixed pipeline where the application controls the next step.

Those choices affect context visibility, testing, retries, approval boundaries, and how easy it is to explain a bad run.

This is a documentation-based comparison of three approaches: OpenAI Agents SDK, with handoffs and agents-as-tools; Google ADK, with parent/sub-agent transfer and workflow agents; and LangGraph, with subgraphs and explicit state.

I evaluated them on five criteria: who owns the next response, how state is passed, whether routing is model- or application-controlled, suitability for parallel or repeatable work, and what a developer can inspect after a failure. I did not run a common benchmark, so this is not a claim that one framework is faster or more reliable.

The key distinction: transfer or return?

Start with a coding task: “Inspect this pull request, run the tests, and suggest a patch.”

A handoff is appropriate when a specialist should take over. The original agent is no longer the owner of the next response. In the OpenAI Agents SDK, handoffs are exposed to the model as transfer tools, and the receiving agent continues the run. The SDK also supports typed handoff inputs and history filters.

A subagent call is appropriate when a manager should remain in control. The specialist does a bounded job, returns a result, and the manager decides what to do next. This is the safer default when a planner needs to combine several independent reviews before producing one answer.

A workflow graph is appropriate when the application should make control flow legible. A graph can say “run security review and test review in parallel, then require a synthesizer,” without asking a model to invent that structure at runtime.

The practical mistake is treating these as interchangeable function calls. They are different ownership models.

Comparison at a glance

Dimension	OpenAI Agents SDK	Google ADK	LangGraph
Primary control primitive	Model-selected handoffs or agents-as-tools	Parent/sub-agent transfer plus Sequential, Parallel, and Loop agents	Explicit graph nodes, edges, subgraphs, and state
Specialist ownership	Handoff transfers the next response; agent-as-tool returns to manager	Transfer can move the conversation; workflow agents keep orchestration explicit	Depends on graph design; nodes return state to the graph
Routing style	Usually model-selected, with application-configured destinations	Hierarchy or workflow configuration; custom agents can implement routing	Application-defined edges or router nodes
Context strategy	Full history by default, with handoff filters available	Session/state mechanisms and agent hierarchy	Shared or private schemas; subgraph persistence can be per-invocation or per-thread
Parallel work	Compose it yourself or use multiple calls	First-class ParallelAgent	Model parallel branches explicitly in the graph
Best fit	OpenAI-first apps needing lightweight delegation and tracing	Python workflows with clear parent/child or sequential/parallel stages	Stateful, inspectable workflows where control flow is part of the product
Main caution	A handoff can make ownership less obvious than a returned result	Transfer and workflow-agent semantics are easy to conflate	More explicit machinery means more design and state-management work

OpenAI Agents SDK: choose the ownership model deliberately

OpenAI’s orchestration guidance separates agents as tools from handoffs.

Use agents as tools when a manager should retain control. For example, a code-review manager can call a security specialist, a test specialist, and a documentation specialist, then reconcile their outputs. The manager owns the final answer and can enforce a common output contract.

Use a handoff when the specialist should own the next response. A triage agent can transfer a repository task to a migration specialist, which then asks the user for missing version details or continues with its own tools.

That distinction is useful for coding agents because “review this diff” and “continue this migration” are not the same job. The first often benefits from a bounded result. The second may require a specialist to take over a multi-turn interaction.

A production guardrail follows from the docs: do not assume an input or output guardrail automatically surrounds every agent in a handoff chain. Put checks at the tool or workflow boundary when every side effect must be reviewed.

Google ADK: hierarchy for delegation, workflow agents for pipelines

Google ADK exposes two related but different ideas.

A parent agent can transfer to a sub-agent when the sub-agent should handle the next interaction. That resembles a handoff: control moves down the hierarchy.

For automated multi-step work, ADK provides workflow agents:

SequentialAgent runs children in a defined order.
ParallelAgent runs independent children concurrently.
LoopAgent repeats children until a termination condition or limit is reached.

This makes ADK a natural fit for a coding pipeline: collect repository context, run static analysis and tests in parallel, synthesize findings, then loop on repair-and-verify with a hard iteration limit.

The important choice is whether a user-facing specialist should take over, or whether the application should complete a predictable pipeline. For the latter, a workflow agent is easier to reason about than an unconstrained chain of model-selected transfers.

LangGraph: make state and control flow explicit

LangGraph treats the workflow as a graph. A node can call an agent, a tool, or ordinary application code. Conditional edges decide what runs next, and subgraphs encapsulate a specialist’s internal workflow.

Its most useful distinction for multi-agent coding systems is state scope. A subgraph can share the parent’s message state, or use a different schema and keep a private history. The docs describe per-invocation persistence as the usual choice for independent subagent tasks, with per-thread persistence when a specialist needs memory across turns.

That is more configuration than a basic handoff, but it answers questions that matter in production:

Does the security reviewer remember the previous request?
Which state is allowed to cross the boundary?
Can a failed branch be inspected and resumed?
Is a second invocation a fresh review or a continuation?

For a coding agent with approvals, durable state, or multiple branches, those are part of the workflow contract.

A decision rule for developers

Choose the smallest control model that makes failure modes visible:

Use a handoff when one specialist should own the next user-facing turn.
Use an agent-as-tool or bounded subagent when a manager needs a typed result and must keep control.
Use a sequential workflow when order is fixed and each step depends on the previous output.
Use parallel branches when reviews are independent and can be reconciled later.
Use a loop only with an explicit exit condition, iteration cap, and verification step.
Use a graph or durable workflow when approvals, restarts, auditability, or state scope are core requirements.

Ask: “If this specialist produces a dangerous suggestion, who is still responsible for deciding whether it becomes an action?” If the answer is unclear, the boundary is probably too implicit.

What to do now

Draw the ownership boundary for every specialist: takes over or returns a result.
Give each specialist a narrow output contract with findings, evidence, confidence, and recommended action.
Keep side-effecting tools behind an approval or policy boundary. Do not let a reviewer silently become an executor.
Add one failure test for each branch: timeout, malformed output, stale repository state, and rejected approval.
Record the state crossing each boundary. A short trace is more useful than a vague “multi-agent” label.

If you are measuring the system, track branch-selection accuracy, useful-result rate, review-to-action rate, tool failures, approval interventions, and cost per completed repository task. Framework choice alone will not tell you whether the workflow works.

Candid limitations

The comparison reflects official documentation available on July 21, 2026. APIs and recommended patterns can change quickly. The frameworks do not expose identical abstractions, so row-by-row equivalence is approximate. I did not run the same repository task across all three, measure latency, or compare model quality. Real outcomes depend on model, prompts, tools, state store, concurrency limits, and evaluation quality.

This is not an argument for using multiple agents by default. A single well-scoped coding agent with explicit tools and verification is often easier to test than a team of agents.

Sources

For the next layer of production detail, see Zira’s context compaction comparison, tool-error handling comparison, and coding-agent score guide.

Discussion: In your coding-agent workflows, do you prefer specialists to take over the conversation, or return bounded results to a manager, and what failure made you choose that design?

SWE-bench vs Terminal-Bench vs SWE-Atlas: What Coding-Agent Scores Actually Mean

Zira — Mon, 20 Jul 2026 21:47:30 +0000

A coding agent can score well on one benchmark and still disappoint on your repository.

That is not necessarily a contradiction. SWE-bench, Terminal-Bench, and SWE-Atlas ask different questions about software engineering. One centers on resolving an issue in a real repository. Another tests end-to-end work in a terminal sandbox. The third splits the engineering loop into codebase understanding, test writing, and refactoring.

The practical lesson is simple: do not treat a benchmark score as a universal ranking. Match the benchmark to the failure mode you need to reduce.

The evaluation criteria

I compared the three benchmark families on five criteria:

Primary task: What is the agent actually asked to do?
Environment and tools: Does it operate in a repository, a general terminal, or both?
Grading signal: What counts as success: tests, a rubric, or another verifier?
What it reveals: Which part of a developer workflow does the score illuminate?
Where it can mislead: What should you not infer from the number?

This is a documentation-based comparison checked against the maintainers' public documentation and papers on July 21, 2026. I did not run a common model, prompt, or harness across all three, so this is not a new head-to-head benchmark.

Quick comparison

Benchmark	Core question	Typical success signal	Best fit for	Main blind spot
SWE-bench	Can an agent resolve a real GitHub issue?	A generated patch passes the task's tests	Issue fixing and patch generation	It does not fully measure investigation, test design, or refactor completeness
Terminal-Bench	Can an agent complete a difficult end-to-end terminal task?	A task-specific test script passes in a sandbox	CLI-heavy, system, and multi-step work	Broad terminal capability is not the same as repository maintenance
SWE-Atlas	Can an agent understand, validate, and maintain a real codebase?	Task-specific rubrics and resolve rates across three tracks	Code comprehension, test writing, and refactoring	Smaller, newer suite with rubric and judge-model dependencies

The columns describe intended measurement, not a leaderboard ranking.

1. SWE-bench: the issue-resolution test

SWE-bench starts with a real software issue collected from GitHub, a codebase, and an expected patch. The agent must produce a change that resolves the issue. The official project describes Docker-based reproducible evaluation and multiple datasets, including SWE-bench, Lite, Verified, and Multimodal.

That makes SWE-bench a useful first question for a coding agent:

Given an issue and a repository, can it make the existing tests pass with a plausible patch?

For an engineering team, this is closest to triaging a backlog of bug fixes or small feature requests. It is also a relatively concrete success condition: the patch is evaluated against tests rather than judged only by a prose answer.

What the score tells you

A strong result is evidence that the evaluated model-plus-agent setup can navigate enough repository context to produce issue-resolving patches on that dataset. It is useful for comparing systems under the same split, scaffold, budget, and evaluation protocol.

What it does not tell you

A passing patch does not prove that the agent:

investigated the system deeply before editing;
wrote the right new tests;
removed every stale call site after a refactor;
produced a maintainable change for your review standards; or
will behave the same way on a private, newer, or differently structured repository.

There is also a measurement-health caveat: benchmark contamination and memorization are active concerns for SWE-bench-style evaluations. Treat the dataset version and reporting date as part of the result, not as footnotes.

2. Terminal-Bench: the terminal-work test

Terminal-Bench evaluates agents in real terminal environments. Its task suite includes work such as compiling code, training models, setting up servers, system administration, and security-oriented tasks. The project has two separable pieces: a dataset of tasks and an execution harness that connects an agent to a sandboxed terminal.

Each task includes an English instruction, a test script, and a reference solution. That structure is valuable because the agent must perform a sequence of actions rather than emit a single patch. It can expose failures in command selection, environment setup, debugging loops, and operational persistence.

What the score tells you

Terminal-Bench is a better lens than an issue-only benchmark when your agent has to:

install or configure dependencies;
compile or run software;
manipulate files and processes;
diagnose failures from logs; or
complete a multi-step CLI workflow.

It is particularly relevant to agents that live in a terminal, CI runner, container, or remote development environment.

What it does not tell you

A terminal task can be completed successfully without demonstrating long-term codebase stewardship. Passing a server-setup or systems task does not imply the agent will understand a large application, write regression tests with strong assertions, or complete a broad refactor.

The environment also matters enormously. Agent adapters, tool permissions, time limits, concurrency, model version, and container images can change the result. Comparing two numbers without those conditions is usually comparing configurations, not just models.

3. SWE-Atlas: the engineering-loop test

SWE-Atlas is designed to measure complementary capabilities across the software development cycle rather than collapse everything into issue resolution. Its public repository and Scale's overview describe three tracks:

Codebase QnA: understand a complex repository through runtime analysis and multi-file reasoning;
Test Writing: identify the right tests for important behavior and follow repository conventions; and
Refactoring: restructure code while preserving behavior, updating consumers, cleaning up stale code, and keeping documentation aligned.

This framing makes SWE-Atlas useful for a question that patch benchmarks often leave implicit:

Can the agent do the investigative and maintenance work around a code change, not just produce a patch that passes existing tests?

The suite contains 284 tasks across its three tracks, according to Scale's overview. The Codebase QnA track uses real repositories and task-specific rubrics. The Test Writing track cares about whether tests catch meaningful behavior, not simply how many test files the agent adds. Refactoring evaluates multi-file changes and completeness in addition to test preservation.

What the score tells you

SWE-Atlas can reveal different weaknesses in agents that look similar on issue resolution:

shallow repository exploration;
answers that sound plausible but lack runtime evidence;
tests that pass while asserting too little; and
refactors that leave missed call sites or dead code behind.

That makes it a strong complement to SWE-bench for teams building review, migration, or maintenance workflows.

What it does not tell you

SWE-Atlas is not a complete production simulation. Its tasks, repositories, rubrics, judge model, and harness still define the measurement. A rubric-graded task can introduce evaluator variance, while a benchmark with a fixed task set cannot represent every language, architecture, security policy, or organizational workflow.

The suite is also newer and smaller than the most established issue-resolution benchmarks. Use it as a diagnostic signal, not a definitive industry-wide ranking.

The most important distinction: model versus agent setup

These benchmarks measure more than a base model. They measure a model operating through an agent scaffold, tools, prompts, permissions, context limits, and execution environment.

That distinction is easy to lose in leaderboard tables. The same model can explore more effectively with a native coding-agent scaffold than with a generic harness. Conversely, a benchmark result from a highly optimized scaffold may not predict what you get from a minimal internal wrapper.

When reading a result, record at least:

model and exact version;
agent or scaffold;
benchmark name, split, and version;
token, time, and step budgets;
tool permissions and network policy;
number of trials and pass definition; and
whether grading is test-based, rubric-based, or judge-model-based.

This is the benchmark equivalent of keeping a trace rather than trusting a final “done” message. I use the same principle in turning coding-agent traces into regression tests.

A practical decision rule

Choose the benchmark that matches the work you want to delegate:

Backlog issue resolution: start with SWE-bench-style evaluation, then inspect patch quality and hidden-test behavior.
Terminal and infrastructure automation: use Terminal-Bench-style tasks with your actual permission and network model.
Maintenance-heavy engineering: add SWE-Atlas-style codebase QnA, test-writing, and refactoring tasks.
Long-running agents: evaluate repeated trials, not just one successful run. Reliability matters as much as peak capability.
Production rollout: build a private holdout set from your repositories. Public benchmarks are a calibration tool, not a substitute for internal acceptance tests.

Do not ask one benchmark to answer all five questions. A compact internal eval can combine one issue fix, one debugging task, one codebase explanation, one test-writing task, and one refactor, each with a clear rubric and a reviewable trace.

What to do now

Pick one real workflow you want to delegate, such as bug fixes, dependency upgrades, or test maintenance.
Map it to the closest benchmark lens: issue resolution, terminal execution, or engineering-loop quality.
Re-run a small private holdout with the exact agent scaffold and permissions you plan to use.
Score more than completion: include test quality, diff scope, investigation evidence, cleanup, cost, latency, and reviewer rework.
Repeat each task at least three times. Track pass rate and consistency separately.
Keep benchmark version, model version, and harness configuration beside every result.

For long-running work, context strategy is another variable worth recording. My context-compaction comparison for coding agents covers why a long context window alone does not guarantee durable task state.

Candid limitations

This comparison uses public documentation and research descriptions, not a controlled run. Benchmark versions and leaderboards change. The tasks are not interchangeable, and scores should not be averaged into a single “best coding agent” number. Some evaluation details, especially around scaffolds and judge models, can materially affect outcomes. Finally, passing a benchmark is evidence of capability under a protocol, not proof of safe autonomous production behavior.

Sources

Discussion

Which benchmark signal has been most predictive in your own coding-agent workflow: issue resolution, terminal execution, or engineering-loop quality?