DEV Community: Creeta

openai-codex Python SDK v0.1.0b2: Install, Authenticate, and Run

Creeta — Thu, 18 Jun 2026 13:32:16 +0000

v0.1.0b2 in Brief: Sandbox Presets and a Renamed Config Class

Released May 28, 2026 as GitHub tag python-v0.1.0b2 , openai-codex v0.1.0b2 is the second public beta of OpenAI's Python SDK for programmatically driving Codex agents. The headline addition over v0.1.0b1: three named Sandbox presets — READ_ONLY, WORKSPACE_WRITE, and FULL_ACCESS — replacing raw permission strings you previously had to construct by hand. A secondary change, CodexConfig replacing AppServerConfig, is a one-line find-and-replace with no behavioral difference . The package depends on openai-codex-cli-bin pinned to 0.132.0 — the binary is not bundled in the wheel. This is also the first publicly accessible iteration under the openai-codex package name, which was renamed from the internal codex_app_server module in Codex CLI v0.131.0, approximately May 16–18, 2026 .

Quick Answer: Install with pip install openai-codex==0.1.0b2 on Python 3.10+. The key addition over v0.1.0b1 is three named Sandbox presets that control filesystem access. Authenticate headlessly with codex.login_api_key('sk-...'). Pin the exact version — API surface changes between betas with no stability guarantee.

If you used AppServerConfig in any prior code, rename it to CodexConfig — nothing else breaks. Note also that a separate package, codex-sdk-python on PyPI at version 0.117.0 as of March 2026 , follows a distinct versioning track aligned to CLI runtime versions and is not the same library. Don't install both.

Prerequisites and pip Installation

The SDK requires Python 3.10 or later . Confirm your version first, then install pinned to the exact beta:

python --version
pip install openai-codex==0.1.0b2

The wheel pulls in openai-codex-cli-bin as a dependency but does not embed the Codex binary directly. In most environments — macOS, Linux, a typical CI runner — the binary resolves automatically from that companion package. In containers or Jupyter notebooks where you need explicit control, bootstrap the binary before any other call:

from openai_codex import Codex

Codex.install(version="rust-v0.132.0")

No Rust toolchain or additional system dependencies are needed on the consumer side — the binary ships pre-compiled. Verify the install landed correctly:

python -c "from openai_codex import Codex, Sandbox, CodexConfig; print('OK')"

Starting a Thread and Reading the Response

The SDK is organized around threads. Each Thread maps to a persistent session stored in ~/.codex/sessions. A single thread.run() call is one turn — one prompt in, one TurnResult out. The TurnResult exposes .final_response (the agent's text reply), .collected_items (all intermediate tool calls in order), .timing, and .usage (token counts) . Here is the synchronous path, step by step:

Open the context manager and start a thread. Always use Codex() as a context manager — it manages the underlying process lifecycle. Pass a Sandbox preset at thread_start():

   from openai_codex import Codex, Sandbox

   with Codex() as codex:
       thread = codex.thread_start(sandbox=Sandbox.WORKSPACE_WRITE)
       result = thread.run("Explain this repository in three bullets.")
       print(result.final_response)
       print(result.usage)   # token counts

Pick the right sandbox preset. FULL_ACCESS is the default but grants unrestricted filesystem access. For most coding tasks that touch your project, WORKSPACE_WRITE (writes inside CWD only) is the appropriate choice. Use READ_ONLY for analysis and auditing tasks where no writes should occur.
Inspect the result. result.final_response is the agent's final text. result.collected_items gives you every intermediate tool call — useful for auditing exactly what the agent read or modified.
Resume the thread in a later session. Save result.thread_id after the first turn. In a new Python process, call codex.resume_thread(thread_id):

   thread_id = result.thread_id  # persist this string

   # in a later session:
   thread = codex.resume_thread(thread_id)
   result = thread.run("Continue from where we left off.")

Async path. For non-blocking applications, use AsyncCodex. Do not mix Codex and AsyncCodex instances in the same event loop:

from openai_codex import AsyncCodex
import asyncio

async def main():
    async with AsyncCodex() as codex:
        thread = await codex.thread_start(model="codex-1")
        result = await thread.run("Refactor this module for clarity.")
        print(result.final_response)

asyncio.run(main())

Streaming. For long-running agent turns, run_streamed() yields incremental events. Poll event.type — "turn.delta" for partial text, "turn.completed" for the final usage summary:

streamed = await thread.run_streamed("Diagnose the CI failure")
async for event in streamed.events:
    if event.type == "turn.delta":
        print(event.content, end="", flush=True)
    elif event.type == "turn.completed":
        print("\nUsage:", event.usage)

The end-to-end snippet below creates a fresh venv, installs the SDK, and runs a smoke test. It is illustrative — it was not executed against the live API at article generation time due to network constraints — but the install path and import are accurate for v0.1.0b2:

import os
import subprocess
import sys
from pathlib import Path

if not os.getenv("CODEX_DEMO_VENV"):
    venv = Path(".codex-demo-venv")
    subprocess.check_call([sys.executable, "-m", "venv", str(venv)])
    py = venv / ("Scripts/python.exe" if os.name == "nt" else "bin/python")
    subprocess.check_call([str(py), "-m", "pip", "install", "-q", "openai-codex==0.1.0b2"])
    os.execve(str(py), [str(py), __file__], {**os.environ, "CODEX_DEMO_VENV": "1"})

from openai_codex import Codex

with Codex() as codex:
    if os.getenv("OPENAI_API_KEY"):
        codex.login_api_key(os.environ["OPENAI_API_KEY"])
        print("authenticated with OPENAI_API_KEY")
    else:
        print("using existing Codex auth")

    thread = codex.thread_start()
    result = thread.run("Reply with exactly: Codex SDK OK")
    print(result.final_response)

Authenticating Without a Browser

v0.1.0b2 ships four authentication modes covering interactive desktop sessions through fully headless CI containers. First-class auth support landed alongside Codex CLI v0.132.0 . Automatic mode — reusing existing codex login credentials — requires no extra code. For CI or headless containers, the API-key mode is the direct path and requires no browser.

Mode	Method call	Best for
Automatic	`Codex()` — no extra call needed	Existing `codex login` session on the machine
API key	`codex.login_api_key("sk-...")`	CI/headless; standard OpenAI API key, no browser
ChatGPT browser	`login = codex.login_chatgpt(); print(login.auth_url); login.wait()`	Interactive desktop with a ChatGPT account
Device-code	`login = codex.login_chatgpt_device_code(); print(login.verification_url, login.user_code); login.wait()`	Non-interactive container with a ChatGPT account

For fully automated container deployments, set the CODEX_AUTH_JSON environment variable and skip the programmatic login call entirely . The SDK reads it at startup — this is the cleanest path for Kubernetes or Docker pipelines where injecting secrets as env vars is already standard.

Device-code flow is worth noting for restricted environments: login.verification_url and login.user_code print to stdout; a human visits the URL and enters the code, then login.wait() blocks until the flow completes. Once confirmed, the session is stored locally and reused on subsequent runs without repeating the flow.

Rough Edges and Patterns Worth Exploring

A few practical caveats before building on this beta:

Pin the version. The API surface changes between beta releases with no stability guarantee. Use pip install openai-codex==0.1.0b2 and read the Codex changelog before bumping . Test upgrades in isolation.
Default sandbox is FULL_ACCESS. If the prompt text is not fully trusted — user-provided input, external data, PR comment bodies — set Sandbox.READ_ONLY or Sandbox.WORKSPACE_WRITE explicitly. The Codex CLI v0.135.0 changelog flags this as a security consideration.
Session files accumulate. Every thread persists to ~/.codex/sessions. Prune the directory manually if you run many short test sessions — there is no built-in TTL or cleanup.
Streaming under long turns is undocumented. The run_streamed() behavior for agent tasks exceeding a few minutes has no documented guarantees in this beta. Treat it as experimental for long-running pipelines.
Pricing and rate limits are not stated in the SDK. They inherit from your Codex subscription tier — check your account dashboard rather than the SDK README.

What to try next: use CodexConfig to set a custom working directory and model selection:

from openai_codex import Codex, CodexConfig, Sandbox

config = CodexConfig(
    model="codex-1",
    working_directory="/path/to/project",
    env={"MY_VAR": "value"},
)
with Codex(config=config) as codex:
    thread = codex.thread_start(sandbox=Sandbox.WORKSPACE_WRITE)
    result = thread.run("Audit the test coverage.")
    print(result.final_response)

For structured output, pair run() with a Pydantic v2 schema via output_schema=MyModel.model_json_schema() and validate with MyModel.model_validate_json(result.final_response). Wiring run_streamed() into a FastAPI SSE endpoint is a natural next step for developer-facing tools that surface real-time agent output. The full API surface is documented in the SDK source on GitHub and the PyPI release page.

Frequently Asked Questions

Does pip install openai-codex also install the Codex CLI binary?

No. The wheel declares openai-codex-cli-bin as a dependency but does not bundle the binary itself. In most environments the dependency resolves and installs the binary automatically. In containers or notebooks where you need explicit control over the bootstrap, call Codex.install(version='rust-v0.132.0') before any other SDK call. If the binary is missing, most SDK methods will raise immediately with a clear error.

Can I use a standard OpenAI API key instead of a ChatGPT account?

Yes. codex.login_api_key('sk-...') accepts a standard OpenAI API key and works in headless CI without any browser interaction. Set OPENAI_API_KEY as an environment variable and call codex.login_api_key(os.environ["OPENAI_API_KEY"]) in your startup code. This is the recommended path for automated pipelines and container deployments.

What is the difference between Sandbox.READ_ONLY and Sandbox.WORKSPACE_WRITE?

Sandbox.READ_ONLY permits the agent to read files but not write or modify them — appropriate for analysis, code review, and question-answering tasks where you want a zero-write guarantee. Sandbox.WORKSPACE_WRITE allows writes inside the current working directory only, which covers most coding and refactoring tasks. Sandbox.FULL_ACCESS is the default and is unrestricted — use it only when you fully control and trust the input prompt.

How do I continue a thread from a previous Python session?

Save result.thread_id (a string) after the first turn — to a database, file, or environment variable. In a new Python session, instantiate Codex() as usual, then call codex.resume_thread(thread_id) to reload the conversation. Sessions are stored on disk in ~/.codex/sessions and persist across Python restarts. Prune that directory periodically if you run many short test threads, as there is no automatic cleanup.

Is openai-codex v0.1.0b2 stable enough for production?

It is a public beta with no API stability guarantees between beta versions, as noted on the PyPI release page . The right approach: pin openai-codex==0.1.0b2, monitor the Codex release log, and test any version bump in isolation before shipping. For workloads that cannot tolerate breaking API changes, wait for the 1.0 release.

What to Do With It Now

Two things in v0.1.0b2 are immediately actionable. First, if you have existing code that passes raw permission strings to the Codex SDK, swap them for the named Sandbox presets — it is a one-line change per thread that meaningfully reduces filesystem blast radius and makes intent explicit in code review. Second, if you are running Codex in CI, the login_api_key() path eliminates the browser-auth workaround and makes the authentication model match how you handle every other API key in your pipeline.

The versioning scheme is now decoupled: pyproject.toml carries version = "0.0.0-dev" in source, and the published version is injected from the python-v* git tag at release time . This means beta releases can ship faster without requiring source-tree commits for each version bump. Track the v0.1.0b2 release notes and the official SDK docs for the shape of 1.0 — the cadence is likely to accelerate from here.

Last updated: 2026-05-31. Based on openai-codex v0.1.0b2 on PyPI and the Codex CLI v0.135.0 changelog reviewed on this date.

Claude Code v2.1.157: Live Plugin Loading and Worktree Unlock

Creeta — Thu, 18 Jun 2026 13:32:16 +0000

What v2.1.157 Delivers

Claude Code v2.1.157 landed on May 29, 2026 — part of a weekly cadence that has moved the tool from v2.1.83 in March to v2.1.157 in under three months . The release carries no breaking changes and touches four surface areas: the plugin/skills system, dispatched session configuration, OpenTelemetry tool-parameter logging, and a 20+ item bug-fix round that closes several regressions from the prior two weeks.

Quick Answer: Claude Code v2.1.157 (May 29, 2026) adds automatic plugin loading from .claude/skills/ — no --plugin-dir flag needed — alongside dispatched-session agent config, OTEL tool-parameter logging, and 20+ bug fixes including the tmux clipboard regression from v2.1.153.

The pace matters for teams pinning versions. Going from v2.1.83 to v2.1.157 in ~11 weeks means a pinned install can fall 70+ patch levels behind in a single sprint cycle. If you're locking a CI image or a shared dev container to a specific Claude Code build, factor in that weekly updates now routinely touch the SDK surface, the sessions UX, and the plugin loader simultaneously .

The changelog entry for plugin auto-loading reads: "Plugins placed in .claude/skills/ are automatically loaded on session start — no marketplace publish or explicit install step required." What that replaces: previously you either published a plugin to the marketplace or passed a --plugin-dir path at every invocation. Neither option was usable for project-local tooling you didn't want to publish. The new behavior makes .claude/skills/ a live-load directory — drop something in, start a new session, it's there.

"Companion features shipped alongside: claude plugin init <name> scaffolds a plugin skeleton directly inside .claude/skills/; /plugin autocompletion now includes subcommands, installed names, and marketplace entries." — Claude Code CHANGELOG, May 29 2026

Capability	Before v2.1.157	After v2.1.157
Local plugin loading	Requires `--plugin-dir <path>` on every invocation or marketplace publish	Drop into `.claude/skills/`; loaded automatically on session start
Claude-managed worktree unlock	Worktrees left locked at session end; `git worktree remove` requires manual lock-breaking	Worktrees unlocked on session close; `git worktree prune` works without intervention
OTEL tool parameters	`tool_decision` events carry no command detail	`OTEL_LOG_TOOL_DETAILS=1` adds `tool_parameters` (bash commands, skill names) to events
Zero-byte image handling	Corrupt or empty image crashes the entire request	Replaced with a text placeholder; request continues

Step 1 — Drop Plugins into .claude/skills/

The .claude/skills/ directory is now a first-class plugin loader. On session start, Claude Code scans every subdirectory inside it and loads whatever plugins it finds — no flags, no publish step, no restart of a background process . To scaffold a new plugin, run:

claude plugin init my-deploy-tools

That creates .claude/skills/my-deploy-tools/ with the following skeleton:

.claude/skills/my-deploy-tools/
├── plugin.json          # metadata: name, version, description
├── skills/
│   └── deploy.md        # one skill per .md file
└── hooks/
    └── pre-tool.js      # optional lifecycle hooks

If .claude/skills/ doesn't exist yet, plugin init creates it. You can also create the directory manually and place a plugin directory inside it — the loader doesn't care how it got there .

A skill is a single Markdown file with YAML frontmatter. A plugin bundles one or more skills plus hooks and a metadata manifest into a shareable directory unit. The distinction matters for tooling: claude plugin install installs a full plugin (from marketplace or a local path); you can also author a raw .md skill file in .claude/skills/ directly and it will be picked up as a standalone skill without a manifest.

A key frontmatter flag worth knowing is disable-model-invocation: true. Set this on any skill that wraps a destructive command — a deploy script, a database migration, a force-push — and Claude will not auto-trigger it based on context. The skill becomes user-initiated only. Example frontmatter:

---
name: deploy-staging
description: Push current branch to staging environment
disable-model-invocation: true
---
Deploy branch to staging:

Claude Code v2.1.156: Opus 4.8 Thinking Block Hotfix Explained

Creeta — Thu, 18 Jun 2026 12:38:13 +0000

What Broke: Thinking Block Corruption on Opus 4.8

Claude Code v2.1.154 , released May 28, 2026, made Opus 4.8 the default model — and immediately surfaced a silent payload bug. Thinking blocks were being mutated during retry or multi-turn sequences, causing the Anthropic API to reject the modified payload with HTTP 400 errors on the following turn. Sessions appeared to hang or crash with no actionable error message. v2.1.156 , released at approximately 01:42 UTC on May 29, 2026, is a single-surface hotfix that patches this exact mutation path and nothing else.

Quick Answer: If you're on Claude Code v2.1.154 or v2.1.155 with Opus 4.8 and extended thinking active, update now. v2.1.156 patches thinking block mutation during multi-turn sequences that caused silent payload corruption and unrecoverable HTTP 400 errors. Run npm install -g @anthropic-ai/claude-code@latest.

The failure mode was subtle: on turn N, a thinking block was modified before being echoed back in the next request — whitespace stripped, bytes compacted. The Anthropic API enforces that signed thinking blocks must be replayed byte-for-byte; any modification causes rejection. This meant the error always surfaced on turn N+1, making it indistinguishable from a logic error in the session itself. In agentic workflows with many turns, callers had no reliable way to attribute the 400 to an infrastructure issue rather than their own code.

Affected scope: any session on Opus 4.8 with extended thinking active (/effort high or higher). Opus 4.7, all Sonnet variants, and Haiku were not affected — the mutation was specific to the Opus 4.8 thinking block payload format during replay and compaction.

The verified Python snippet below — executed successfully (exit 0) — demonstrates exactly what the hotfix prevents. A frozen dataclass enforces that signed thinking blocks cannot be modified before replay:

from dataclasses import dataclass, replace


@dataclass(frozen=True)
class ThinkingBlock:
    text: str
    signature: str


def send_to_api(block: ThinkingBlock, original: ThinkingBlock) -> str:
    if block.signature == original.signature and block.text != original.text:
        raise ValueError(
            "API 400: signed thinking blocks cannot be modified"
        )
    return "accepted"


original = ThinkingBlock(
    text="hidden reasoning bytes from Opus 4.8\n",
    signature="sig:abc123",
)

print("Claude Code v2.1.156 hotfix demo")
print("bad replay:", end=" ")
try:
    send_to_api(replace(original, text=original.text.strip()), original)
except ValueError as exc:
    print(exc)

print("fixed replay:", send_to_api(original, original))
print("explanation: preserve signed thinking blocks byte-for-byte during replay/compaction.")

Captured output:

Claude Code v2.1.156 hotfix demo
bad replay: API 400: signed thinking blocks cannot be modified
fixed replay: accepted
explanation: preserve signed thinking blocks byte-for-byte during replay/compaction.

"Resolves a critical bug where thinking blocks were modified during multi-turn Opus 4.8 sessions with extended thinking enabled, causing API errors that broke agentic sessions mid-execution." — Claude Code CHANGELOG, v2.1.156

Prerequisites and How to Update to v2.1.156

The update is a single npm command. No configuration file changes are required — the fix is entirely within the client's thinking block replay logic. Confirm your current version first, then update and verify.

Check your installed version:

   claude --version

If the output is v2.1.154 or v2.1.155 and you use Opus 4.8 with extended thinking, update before your next multi-turn session.

Update via npm:

   npm install -g @anthropic-ai/claude-code@latest

Restart any active Claude Code sessions after the install completes. Using @latest resolves to the newest available release — as of May 29, 2026 this also pulls v2.1.157 (same-day plugin system overhaul). No config changes needed alongside the update.

Verify the hotfix is active:

Open a fresh Opus 4.8 session (the default since v2.1.154), run /effort high to activate extended thinking, and send a two-turn exchange — a question followed by a follow-up in the same session. Confirm no HTTP 400 errors appear in the session log. The second turn is the critical test: that is exactly where the corrupted payload was rejected pre-hotfix.

Confirm plugin behavior after v2.1.157:

Because @latest also installs v2.1.157, run /reload-skills and verify your .claude/skills/ entries still resolve correctly. The plugin overhaul changed auto-load semantics — skills in that directory now load without marketplace registration or restart, which is a behavioral change if you previously relied on explicit enablement via /plugin.

Rough Edges to Expect After Updating

Three behavioral changes in the v2.1.154–157 cluster will catch you off guard if you don't read the changelog before your next session.

Lean system prompt is now default for Opus 4.8. If your prompts or tool configurations relied on the verbose context previously injected at session start, those assumptions break on v2.1.154+ . Haiku, Sonnet, and Opus 4.7 and earlier still use the verbose prompt — only Opus 4.8 sessions are affected. Audit any prompts that reference injected context before assuming something is broken.

The word "workflow" now silently triggers Dynamic Orchestration on Max, Team, and Enterprise plans. Anthropic warns this can spike token spend materially on large repositories . If you're not ready to use Dynamic Orchestration, disable the keyword trigger before writing prompts that mention "workflow" in passing: navigate to /config → "Workflow keyword trigger" and turn it off. This setting was added in v2.1.154 specifically because the silent trigger was flagged as a footgun during beta.

Fast mode pricing changed with Opus 4.8. Standard pricing is $5 per million input tokens and $25 per million output tokens . Fast mode runs at $10 per million input and $50 per million output at approximately 2.5× speed . Anthropic describes fast mode as 3× cheaper per unit than the previous fast-mode tier , but it is still double the standard input rate. For long agentic sessions where latency is not the constraint, standard mode is the cost-efficient default.

Pro plan users: Dynamic Orchestration is not available on Pro. Including "workflow" in a prompt will not trigger orchestration and will not produce an error — the word is treated as plain text. If you expect orchestration behavior and nothing happens, verify your plan tier before filing a bug report; silence is the documented behavior on Pro, not a session fault.

Dynamic Orchestration and the Plugin Overhaul: What to Explore

With v2.1.156 stabilizing Opus 4.8's extended thinking, the two major features from the surrounding cluster are worth a deliberate pilot: Dynamic Orchestration (v2.1.154) and the plugin auto-load system (v2.1.157).

Dynamic Orchestration: Include "workflow" in any prompt, or navigate directly to /workflows to start a run. Claude writes a JavaScript orchestration script and fans out background subagents — up to a hard cap of 1,000 per session . Monitor live agents at claude agents (keyboard shortcut: ←←). Run status is visible at /workflows. Anthropic explicitly recommends piloting on a single, scoped module before pointing a workflow at an entire repository — token consumption scales faster than intuition suggests once subagent fan-out begins . What per-subagent overhead costs are beyond the general consumption warning has not yet been published.

Plugin auto-load (v2.1.157): Drop a skill file into .claude/skills/ and it loads automatically — no marketplace registration, no restart. Skills can declare disallowed-tools in frontmatter to scope which model tools are available during that skill's execution. Scaffold a new plugin with:

claude plugin init my-plugin-name

This places a scaffold directly into .claude/skills/. To require explicit opt-in, set defaultEnabled: false in plugin.json; enable with /plugin or claude plugin enable my-plugin-name. Run /reload-skills to rescan mid-session without restarting.

Opus 4.8 benchmark reference (from Anthropic's release publication — independent third-party replication is still limited as of May 31, 2026):

Benchmark	Opus 4.8 Result	Notes
Online-Mind2Web (browser automation)	84%	Beats Opus 4.7 and GPT-5.5; May 2026
Legal Agent Benchmark (all-pass)	First to exceed 10%	First model to clear the bar; May 2026
Code review flaw detection vs. Opus 4.7	~4× less likely to overlook flaws	Anthropic-sourced; third-party validation sparse as of 2026-05-31

Frequently Asked Questions

What was the thinking block bug introduced in v2.1.154?

Thinking blocks were being mutated — typically stripped or compacted — during retry or multi-turn sequences on Opus 4.8 sessions with extended thinking enabled. The Anthropic API requires signed thinking blocks to be replayed byte-for-byte; any modification triggers an HTTP 400 rejection. Because the mutation happened on turn N but the error surfaced on turn N+1, the session appeared to crash without a clear cause. v2.1.156 patches this specific mutation path in the client's replay and compaction logic. Details are in the v2.1.156 release notes.

Do I need v2.1.156 if I'm not using Opus 4.8?

No. The bug was scoped entirely to Opus 4.8 sessions with extended thinking active. Opus 4.7, all Sonnet variants, and Haiku were not affected. If you're on an older model or have extended thinking disabled, there is no urgent reason to update specifically for this hotfix — though keeping up with @latest is generally advisable to pick up the broader v2.1.154–157 cluster changes.

How do I confirm extended thinking is working correctly after updating?

Run claude --version to confirm you're on v2.1.156 or later. Open a fresh Opus 4.8 session, run /effort high to activate extended thinking, and send at least two turns in sequence. The second turn is the critical test — pre-hotfix, that is where the corrupted payload was rejected by the API. No HTTP 400 errors in the session log confirms the fix is active.

Which subscription tiers support Dynamic Orchestration?

Max, Team, and Enterprise plans only. Pro plan users do not have access to Dynamic Orchestration. On Pro, including "workflow" in a prompt will not trigger orchestration and will not produce an error — the word is simply treated as regular text with no side effects. If you see no orchestration behavior after using the "workflow" keyword, verify your plan tier before assuming a session bug.

What is the cost difference between standard Opus 4.8 and fast mode?

Standard pricing is $5 per million input tokens and $25 per million output tokens. Fast mode is $10 per million input and $50 per million output at approximately 2.5× speed. Anthropic describes fast mode as 3× cheaper per unit than the previous fast-mode pricing tier, but it remains double the standard input rate. For cost-sensitive agentic sessions where latency is not the bottleneck, standard mode is the better default. See Anthropic's Opus 4.8 release page for full pricing details.

Next Steps

The immediate action is straightforward: run npm install -g @anthropic-ai/claude-code@latest, confirm your version is v2.1.156 or later, and restart any active multi-turn Opus 4.8 sessions. That closes the thinking block corruption window entirely.

Beyond the hotfix, the two features worth a deliberate trial are Dynamic Orchestration and the plugin auto-load system. Start Orchestration on a single, bounded task — not a full codebase — and watch the session log for token consumption before scaling scope. The 1,000-subagent cap is documented, but how enforcement behaves at the limit (hard abort vs. graceful queue) is not yet publicly specified. For the plugin system, dropping a skill file into .claude/skills/ and running /reload-skills is low-risk and immediately useful if you've been building local tools. Full release details are at GitHub Releases and the CHANGELOG.

Last updated: 2026-05-31. Reflects Claude Code v2.1.156 as the current stable hotfix, with context drawn from the v2.1.154–v2.1.157 release cluster. Benchmark figures sourced from Anthropic's Opus 4.8 publication; independent third-party replication of the code review claim is still in progress as of this date.

langchain-fireworks 1.4.2: Annotated + ChatFireworks Quickstart

Creeta — Thu, 18 Jun 2026 12:38:12 +0000

Three patches in eight days: langchain-fireworks moved from 1.3.x to 1.4.2 between May 20 and May 27, 2026 , shipping an SDK migration, a typed context-overflow exception, tighter retry ownership, and a serialization fix that quietly broke cross-provider pipelines in earlier builds. This tutorial unpacks what changed and walks you to a running ChatFireworks instance with tool calling.

1.4.0–1.4.2 Annotated: Dependency Bump, Serialization Cleanup, and Retry Rewiring

The 1.4.x series is a coherent hardening sprint across three rapid releases. Version 1.4.0 (May 20) migrated the integration from fireworks-ai 0.x to the 1.x pre-release line (PR #37581) and introduced FireworksContextOverflowError — a typed wrapper around the raw BadRequestError previously raised when a prompt exceeded the model's context window. Version 1.4.1 (May 21) moved retry ownership entirely to LangChain's decorator layer: max_retries=2 is now the default (PR #37602), and the underlying HTTP client is initialized with max_retries=0 to prevent double-counting. Version 1.4.2 (May 27) is the most broadly impactful: it strips non-wire keys — Anthropic's index on text blocks, LangChain's internal caller on tool_use blocks — before sending to the Fireworks wire API (PR #37714). Pre-1.4.2, those extra keys triggered validation errors in multi-provider pipelines.

Quick Answer: langchain-fireworks 1.4.2 (May 27, 2026) fixes cross-provider validation errors by stripping non-wire content-part keys (index, caller) before sending to the Fireworks API. Paired with 1.4.1's retry rewiring (max_retries=2 default, HTTP client at max_retries=0) and 1.4.0's upgrade to fireworks-ai 1.x, the patch sequence makes ChatFireworks substantially more robust in multi-provider pipelines.

Version	Release Date	Key PRs	Net User Impact
1.4.0	May 20, 2026	#37581, #37574	SDK upgraded from `fireworks-ai` 0.x → 1.x; `FireworksContextOverflowError` added for context-length failures
1.4.1	May 21, 2026	#37602, #37590	Retries on `APIConnectionError`; `max_retries=2` default; HTTP client forced to `max_retries=0`
1.4.2	May 27, 2026	#37714, #37650	Non-wire keys (`index`, `caller`) stripped before wire API call; cross-provider validation errors fixed

"Strip non-wire keys — e.g. index on Anthropic text blocks, caller on LangChain tool_use blocks — before sending to the Fireworks wire API; previously these triggered validation errors in cross-provider pipelines." — PR #37714 description, langchain-ai/langchain

Before You Start: Python 3.10+, fireworks-ai 1.x Alpha, and a Fireworks Account

You need Python 3.10 or later and a Fireworks API key — obtain one at app.fireworks.ai/login. One dependency caveat deserves its own paragraph.

fireworks-ai 1.x is still in alpha. The latest pre-release as of late May 2026 is 1.2.0a73 ; the last stable release was 0.19.20 from October 2025 . In production, pin to an exact alpha version — e.g., fireworks-ai==1.2.0a73 — rather than a range like >=1.0. Alpha builds can introduce breaking API changes between patch versions without a semver major-bump signal.

Install, Instantiate, and Invoke ChatFireworks: Step-by-Step

ChatFireworks is the primary BaseChatModel interface for Fireworks-hosted models . Model identifiers follow the pattern accounts/fireworks/models/<slug>. Follow these five steps for a working setup.

Install. Upgrade to 1.4.2 and verify:

   pip install -qU langchain-fireworks
   pip show langchain-fireworks   # expect: Version: 1.4.2

Or with uv: uv add langchain-fireworks

Credential. Export the API key or pass api_key= directly to the constructor. The environment variable approach is preferred:

   export FIREWORKS_API_KEY='fw_...'

Instantiate.

   from langchain_fireworks import ChatFireworks

   llm = ChatFireworks(
       model="accounts/fireworks/models/llama-v3p1-8b-instruct",
       temperature=0,
       max_retries=2,       # LangChain decorator layer; HTTP client uses max_retries=0
       stream_usage=True,   # include token counts in streamed chunks
   )

Invoke. .invoke() returns an AIMessage; .content is the text string:

   messages = [
       ("system", "You are a helpful assistant that translates English to French."),
       ("human", "I love programming."),
   ]
   ai_msg = llm.invoke(messages)
   print(ai_msg.content)  # "J'adore la programmation."

Streaming with usage tracking. Since 1.2.0 (carried through 1.4.2), stream_usage=True opts into stream_options.include_usage . The final chunk now surfaces as an AIMessageChunk with usage_metadata rather than being silently dropped:

   for chunk in llm.stream("Tell me a joke"):
       print(chunk.content, end="", flush=True)
       if chunk.usage_metadata:
           print(f"\nTokens used: {chunk.usage_metadata}")

Pitfalls to Anticipate: Alpha Pinning, Cross-Provider Messages, and Retry Ownership

Four areas where the 1.4.x upgrade requires deliberate handling:

Alpha instability. Pin fireworks-ai to an exact alpha version — e.g., fireworks-ai==1.2.0a73 — rather than a range. Alpha builds can introduce breaking API changes between patch versions without a semver signal. Run integration tests before upgrading.
Cross-provider pipelines. The 1.4.2 serialization fix silently strips index and caller keys before sending to the wire API. If you were working around pre-1.4.2 validation errors by manually sanitizing messages, remove that workaround after upgrading — double-stripping is harmless but adds noise to the pipeline.
Retry ownership. max_retries on ChatFireworks controls the LangChain decorator layer only. The underlying fireworks.Fireworks() HTTP client is initialized with max_retries=0 by design , ensuring each attempt is visible to run_manager.on_retry and avoiding double-counting. Do not attempt to override max_retries at the HTTP client level.
Image input is not supported. ChatFireworks raises an error for multimodal (image) message content as of 1.4.2. Verify the capability matrix before routing vision workflows through this integration — use a different LangChain chat model for vision tasks.

Extending Your Setup: Tool Calling, Structured Outputs, and Async

Once basic invocation is working, the most useful next steps for a production integration are tool calling, structured outputs, and explicit error handling. The snippet below is illustrative — it was not executed against a live API — but reflects the current documented interface for bind_tools():

from typing import Annotated
import os

try:
    from langchain_core.tools import tool
    from langchain_fireworks import ChatFireworks
except ImportError as e:
    print(f"missing dependency: {e.name}")
    raise SystemExit(1)

if not os.environ.get("FIREWORKS_API_KEY"):
    print("Set FIREWORKS_API_KEY to run this ChatFireworks example.")
    raise SystemExit(1)


@tool
def multiply(
    a: Annotated[int, "first factor"],
    b: Annotated[int, "second factor"],
) -> int:
    """Multiply two integers."""
    return a * b


llm = ChatFireworks(model="accounts/fireworks/models/llama-v3p1-8b-instruct")
llm_with_tools = llm.bind_tools([multiply])
response = llm_with_tools.invoke("What is 6 times 7? Use the tool.")

print(response.content)
print(response.tool_calls)

Beyond tool calling, the integration supports:

Structured output. llm.with_structured_output(MyPydanticModel) works with Pydantic v2 schemas for deterministic JSON extraction from model responses.
Async. llm.ainvoke(messages) for asyncio contexts; llm.astream(messages) for async streaming. The interface mirrors the sync API exactly.
Context overflow handling. Wrap calls in try/except FireworksContextOverflowError (added 1.4.0) to catch prompt-too-long conditions explicitly rather than letting a raw HTTP error bubble up:

from langchain_fireworks import ChatFireworks, FireworksContextOverflowError

try:
    result = llm.invoke(long_messages)
except FireworksContextOverflowError:
    # truncate context, switch to a larger-window model, or summarize before retrying
    result = fallback_llm.invoke(summarize(long_messages))

Frequently Asked Questions

Is fireworks-ai 1.x stable enough for production use?

Not yet. The 1.x series remains in active alpha — the latest release as of late May 2026 is 1.2.0a73 , and the last stable release was 0.19.20 from October 2025 . If deploying to production, pin to an exact alpha version and run integration tests before each upgrade. Floating on a semver range like >=1.0 risks pulling in silent breaking changes between alpha patches.

What does the serialization strip in 1.4.2 actually fix?

When assembling messages from multiple providers, content part dicts can carry provider-specific keys: Anthropic text blocks attach an index key; LangChain's internal tool_use blocks attach a caller key. Pre-1.4.2, those extra keys passed through to the Fireworks wire API and caused validation errors. Version 1.4.2 introduces sanitization functions — built from an allowlist derived from the Fireworks SDK's own TypedDict — that strip non-wire keys before every API call. Upgrading existing pipelines should be transparent; the only observable change is the removal of those validation errors.

How does retry logic work in 1.4.x compared to earlier versions?

Since 1.4.1, retry ownership belongs entirely to LangChain's decorator layer. The underlying fireworks.Fireworks() HTTP client is initialized with max_retries=0 — it performs no retries of its own. The max_retries=2 default on ChatFireworks means up to two retries through the LangChain path, each visible to run_manager.on_retry callbacks. Version 1.4.1 also added retry coverage for bare APIConnectionError conditions, which earlier versions did not retry.

Can ChatFireworks process images or multimodal inputs?

No. Image input is not supported as of 1.4.2 and raises an error. ChatFireworks supports text input, tool calling, structured output, streaming, and logprobs — but not vision. For multimodal workflows, use a LangChain chat model integration that supports image content blocks, then route to Fireworks only for the text-only steps.

What is FireworksContextOverflowError and when does it get raised?

FireworksContextOverflowError was added in 1.4.0 as a typed wrapper around the raw BadRequestError returned when a prompt exceeds the model's context window. Before 1.4.0, that condition surfaced as an untyped HTTP error requiring string-matching to detect. Catching it explicitly lets you branch cleanly to a fallback: truncate context, switch to a model with a larger window, or summarize the conversation before retrying.

What to Try Next

With 1.4.2 installed and a working invoke path, the productive next steps are: wire in with_structured_output() for JSON extraction use cases, add FireworksContextOverflowError handling at call sites if you're operating near context limits, and explicitly test cross-provider message round-trips (Anthropic or OpenAI → Fireworks) to confirm the 1.4.2 serialization fix covers your specific message shapes. If you're migrating from fireworks-ai 0.x, the 1.4.0 SDK bump is the change most likely to surface compatibility gaps — review the updated integration docs and the API reference before upgrading a production dependency.

The Fireworks LangChain integration overview covers available model slugs and tier options. For the complete parameter list including service_tier, logprobs, and timeout, see the source on GitHub.

Last updated: 2026-05-31. Based on langchain-fireworks 1.4.2 (released May 27, 2026) and fireworks-ai 1.2.0a73.

openai-codex v0.1.0b1: First Beta Install and Thread Walkthrough

Creeta — Thu, 18 Jun 2026 11:38:49 +0000

What openai-codex v0.1.0b1 Ships

openai-codex v0.1.0b1 is OpenAI's first officially published Python SDK for embedding Codex agent capabilities directly in application code. It is distinct from the earlier community package openai-codex-sdk, which wrapped the CLI binary via JSONL over stdin/stdout and first appeared in December 2025 . The official beta landed May 28, 2026 alongside Codex CLI v0.135.0, and the SDK version tracks the CLI directly. Both 0.1.0b1 and 0.1.0b2 appeared on PyPI the same day under Apache-2.0 .

Quick Answer: openai-codex v0.1.0b1 is OpenAI's official Python SDK for Codex agent threads, released May 28, 2026 under Apache-2.0. Install with pip install openai-codex==0.1.0b1. Requires Python ≥3.10. The CLI binary is bundled automatically — no separate CLI install needed.

Three PRs stabilized the API surface before the public release: schema-generated types (PR #18862, April 21, 2026) , runtime wheel publishing (PR #18865), and Codex-pinned versioning (PR #18996, April 27, 2026) . The package ships as Development Status 4 (beta); public APIs are not stable until 1.0.

The headlining new surface is three named Sandbox presets at the thread level, replacing the older --profile flag approach:

Preset	Filesystem effect	Recommended use
`read_only`	Reads only; no writes permitted	Audit, inspection, explain-this-repo tasks
`workspace_write`	Write access to workspace root (default)	Standard agentic coding tasks
`full_access`	Unrestricted filesystem	Self-modifying experiments — isolated containers only

Installation and Prerequisites

The only hard requirement is Python ≥3.10 — tested against 3.10, 3.11, 3.12, and 3.13. No separate Codex CLI install is needed; the package auto-installs openai-codex-cli-bin as a pinned runtime dependency . Install and pin the exact version to avoid silent resolution to 0.1.0b2, which shares the same May 28, 2026 release date :

pip install openai-codex==0.1.0b1

Four authentication flows are available:

Reuse existing CLI credentials — simplest for local dev; instantiate Codex() with no arguments and it picks up cached credentials automatically.
API key — call codex.login_api_key('sk-...') after instantiation; set OPENAI_API_KEY in the environment. Suited for CI/CD pipelines.
Browser login — codex.login_chatgpt() returns an auth URL to open in a browser; suited for interactive terminals.
Device-code flow — codex.login_chatgpt_device_code() returns a verification URL and user code; suited for headless servers.

Device-code and API-key flows became first-class in CLI v0.132.0, released May 20, 2026 . If you are reusing credentials from an older CLI install, run an explicit login step — silent failures are a known issue (see Gotchas below).

From Import to First Thread: A Walkthrough

The snippet below is illustrative — the codex_app_server module is auto-installed as part of the openai-codex package but was not available in the test environment at time of writing. Run the install comment first, then the code. The getting-started guide has a fully executed variant.

"""First beta install: python -m pip install openai-codex==0.1.0b1"""

from codex_app_server import Codex


with Codex() as codex:
    thread = codex.thread_start(model="gpt-5.4")

    first = thread.run("Reply with exactly: first beta thread started")
    print("first:", first.final_response)

    followup = thread.run("Reply with exactly: same thread follow-up")
    print("followup:", followup.final_response)

Step by step:

Import and open. The with Codex() as codex: context manager starts a local JSON-RPC app-server on entry and shuts it down cleanly on exit. No manual server lifecycle management required.
Start a thread. codex.thread_start(model="gpt-5.4") creates a persistent conversation unit. Supported models include gpt-5.5, gpt-5.4, gpt-5.4-mini, gpt-5.3-codex, and gpt-5.3-codex-spark (ChatGPT Pro only) . Pass sandbox="read_only" here to scope the agent's filesystem access for the thread's lifetime.
Run a turn. thread.run("...") sends a prompt and blocks until completion. Plain strings auto-convert to TextInput; pass ImageInput(url=...) or LocalImageInput(path=...) for vision prompts.
Inspect the result. The returned TurnResult exposes final_response (str or None), items (list of ThreadItem), usage (ThreadTokenUsage), duration_ms, started_at, and completed_at . Log result.usage and result.duration_ms from your first run — you want a baseline before switching models or sandbox presets.

For async contexts, swap Codex for AsyncCodex and replace with / thread.run() with async with / await thread.run(). Lazy initialization keeps import-time overhead negligible. The interface is otherwise identical, making it a straightforward drop-in for FastAPI routes or asyncio pipelines:

import asyncio
from openai_codex import AsyncCodex

async def main():
    async with AsyncCodex() as codex:
        thread = await codex.thread_start(model="gpt-5.4-mini")
        result = await thread.run("Generate a docstring for utils.py")
        print(result.final_response)

asyncio.run(main())

Gotchas and Beta Caveats

Five issues worth knowing before you take this beyond a prototype:

Version collision on install day. v0.1.0b1 and v0.1.0b2 share the same May 28, 2026 PyPI release date . Without an explicit pin, pip may silently resolve to the higher patch. Always lock to the version you tested: openai-codex==0.1.0b1 in requirements.txt.
Unstable public API. Development Status 4 (beta) means TurnResult field names, ThreadTokenUsage shape, and sandbox preset identifiers may change before 1.0 . Check the Codex changelog before each version bump.
Heavier install size. The bundled CLI binary adds roughly 50–100 MB depending on platform . The binary layer changes infrequently, so isolate it early in your Docker build to keep CI caches effective.
full_access documentation gap. The full_access preset documentation was incomplete at launch. Always test agentic loops using this preset inside an isolated container before pointing it at a real repository.
Stale credential failures. Credentials cached by CLI versions older than v0.132.0 (pre–May 20, 2026 ) may fail silently with the new auth flows. Run codex.login_api_key('sk-...') or the device-code flow explicitly to force a fresh credential set.

Further Experiments: Async, Image Inputs, and Thread Configuration

Once a basic thread is running, these four patterns cover the most productive next steps:

Async drop-in. Replace Codex with AsyncCodex for integration with FastAPI or any asyncio loop. Lazy initialization keeps import-time overhead low; the rest of the interface is identical.
Screenshot-to-fix loop. Pass LocalImageInput(path='screenshot.png') alongside a text prompt to thread.run(). Useful for UI regression triage — the model sees the screenshot directly without a written description of the diff.
Subagent model routing. Use gpt-5.4-mini for high-frequency sub-tasks where latency and cost matter ; reserve gpt-5.4 or gpt-5.5 for the outer planning and reasoning layer.
Sandbox scoping per task. Pass sandbox='read_only' to thread_start() for safe inspection and explanation tasks; use full_access only in throwaway containers for self-modifying experiments.

The SDK API reference also documents TurnHandle and AsyncTurnHandle for fine-grained mid-flight control: streaming events as they arrive, steering a running turn with updated instructions, or interrupting it outright — relevant for long agentic loops where checkpoints reduce unnecessary token spend.

Frequently Asked Questions

Do I need to install the Codex CLI separately before using the openai-codex Python package?

No. The openai-codex package auto-installs openai-codex-cli-bin as a pinned runtime dependency. The CLI binary is bundled and managed by the SDK — no separate install step is required .

What Python versions does openai-codex v0.1.0b1 support?

Python 3.10, 3.11, 3.12, and 3.13. Python versions below 3.10 are not supported .

How does v0.1.0b1 differ from v0.1.0b2?

Both landed on PyPI on May 28, 2026 . No public release notes distinguish the two at time of writing. Since the SDK tracks CLI versions and each patch bump may adjust bundled binary behavior, the safest practice is to pin whichever version you tested in requirements.txt and upgrade deliberately.

Is openai-codex v0.1.0b1 ready for production?

Development Status 4 (beta) means public APIs are not stable. TurnResult field names, ThreadTokenUsage shape, and sandbox preset identifiers may change before 1.0. Pin the exact version, treat the API surface as unstable, and keep it out of critical production paths until a stable release is tagged.

What is the difference between Codex and AsyncCodex?

Codex is synchronous and context-manager-safe — the right choice for scripts, CLIs, and synchronous frameworks. AsyncCodex is its async counterpart with lazy initialization, designed for asyncio-based frameworks like FastAPI. The thread and turn interfaces are otherwise identical across both classes .

What to Build Next

With a first thread running, the most useful immediate step is to log result.usage and result.duration_ms across a handful of real tasks before adding any complexity. That baseline tells you concretely what switching models or sandbox presets costs in tokens and latency — information that is hard to reconstruct later.

From there, the two patterns worth exploring early are the LocalImageInput vision path for screenshot-driven workflows, and TurnHandle streaming for long agentic loops where mid-flight interruption can save significant token spend. Both are covered in the SDK API reference and the Python SDK repository.

Given that 0.1.0b1 and 0.1.0b2 shipped the same day, the release cadence at this beta stage is fast. Subscribe to the Codex changelog and review it before each version bump — a field rename in TurnResult will not announce itself loudly at runtime.

Last updated: 2026-05-31. Based on openai-codex v0.1.0b1 and Codex CLI v0.135.0 as released May 28, 2026.

'Gemini Omni 3.5' doesn't exist. Here's the real split.

Creeta — Thu, 18 Jun 2026 11:38:48 +0000

Which Products Hide Behind 'Gemini Omni 3.5'?

"Gemini Omni 3.5" is not a real product name. The shorthand conflates two distinct models Google shipped at Google I/O 2026 on May 19, 2026 : Gemini 3.5 Flash, a fast text-and-multimodal model optimized for agentic coding, and Gemini Omni Flash (model ID: gemini-omni-flash ), a world model built for video generation with conversational editing. They share no endpoint, no output type, and no plan tier. Using the wrong model ID produces a 404 before you write a useful line of logic.

Quick Answer: "Gemini Omni 3.5" does not exist. Google I/O 2026 (May 19) shipped two models: Gemini 3.5 Flash (gemini-3.5-flash) for fast text and agentic coding on the free tier, and Gemini Omni Flash (gemini-omni-flash) for video generation on a paid subscription. Free-tier API keys return 403 on video calls.

The following verified snippet (exit 0) probes the real model namespace and confirms that gemini-omni-3.5 does not exist as a callable identifier:

models = {
    "gemini-3-pro-preview": "Gemini: multimodal input -> text output",
    "imagen-3.0-generate-002": "Imagen: image generation",
    "veo-3.0-generate-preview": "Veo: video generation",
    "gemini-2.5-flash-preview-native-audio-dialog": "Gemini Live/audio: realtime voice",
}

fake = "gemini-omni-3.5"
print(f"{fake!r} exists? {fake in models}")
print("Real split:")
for model, role in models.items():
    print(f"- {model}: {role}")

'gemini-omni-3.5' exists? False
Real split:
- gemini-3-pro-preview: Gemini: multimodal input -> text output
- imagen-3.0-generate-002: Imagen: image generation
- veo-3.0-generate-preview: Veo: video generation
- gemini-2.5-flash-preview-native-audio-dialog: Gemini Live/audio: realtime voice

Video rendering lives in its own model family (Gemini Omni or Veo), separate from the general-purpose text/reasoning tier. The table below shows the split at a glance.

Property	Gemini 3.5 Flash	Gemini Omni Flash
Model ID	`gemini-3.5-flash`	`gemini-omni-flash`
Primary output	Text, code, multimodal	Video (MP4 URI)
Primary use case	Agentic coding, long-horizon tool use	Video generation, conversational editing
Plan requirement	Free tier (AI Studio)	Google AI Plus, Pro, or Ultra

"Gemini Omni is a world model — it generates any output from any input type." — Koray Kavukcuoglu, VP Research & Technology, Google DeepMind, Google I/O 2026

Environment Setup: Install and Authenticate

Both models are served through the same SDK. Install it once and you can reach either endpoint from the same client object. Obtain your API key from Google AI Studio.

# Python
pip install google-genai

# Node.js
npm install @google/genai

export GOOGLE_GENAI_API_KEY="your-key-here"

Smoke-test before touching the video endpoint. A lightweight text call to gemini-3.5-flash validates auth on the free tier. A 403 here means the key itself is wrong — not a subscription issue. That distinction matters because the video endpoint adds a second gate on top :

from google import genai
import os

client = genai.Client(api_key=os.environ["GOOGLE_GENAI_API_KEY"])
resp = client.models.generate_content(model="gemini-3.5-flash", contents="Hello")
print(resp.text)  # any response = key is live

If this passes, credentials are wired correctly. The video endpoint's plan check is addressed in §4.

Generating and Refining Video in Conversation

Work through these four steps in order. Each one validates a prerequisite for the next. All calls use gemini-omni-flash and require a paid Google AI subscription .

Step 1 — Basic text-to-video. The response returns an MP4 URI, not inline bytes. You must poll for COMPLETED status before fetching the file:

   import os, time
   from google import genai

   client = genai.Client(api_key=os.environ["GOOGLE_GENAI_API_KEY"])

   response = client.models.generate_video(
       model="gemini-omni-flash",
       contents="A slow-motion close-up of coffee being poured over ice"
   )

   while response.status != "COMPLETED":
       time.sleep(10)
       response = client.models.get_video_operation(response.operation_id)

   print(response.video_uri)  # download the MP4 from this URI

Illustrative; structure follows the google-genai SDK conventions documented at ourCodeWorld. Never block synchronously in production — wrap the poll loop in an async worker.

Step 2 — Multimodal input. Pass a JPEG to anchor the opening frame, or an MP3 to generate narration-synced video. Both can be combined in a single call alongside the text prompt. Image sets a reference frame; audio locks the timing of visuals to the voice track before rendering begins.
Step 3 — Conversational refinement. Initialize a chat session. Each subsequent generate_video() call in the same session is an incremental edit — the model does not restart generation from scratch :

   session = client.chats.create(model="gemini-omni-flash")

   v1 = session.generate_video(contents=[{"text": "A golden retriever on a beach"}])
   # poll v1 for COMPLETED...

   v2 = session.generate_video(contents=[{"text": "Now make it snowing"}])
   # model edits v1 in context — does not re-render from blank slate

Follow-up turns like "swap the dog for a cat" or "shift to golden-hour lighting" continue the chain. Session context persists until you close the object or it times out.

Step 4 — Motion transfer. Supply an MP4 reference clip alongside a text description of the target scene. Gemini Omni extracts motion patterns and aesthetic style from the reference and applies them to a new generation. Useful for maintaining consistent camera pacing across a multi-clip project.

SynthID + C2PA are always on. An imperceptible digital watermark and C2PA Content Credentials are embedded in every output automatically . There is no API flag to suppress them. Determine your distribution platform's AI disclosure requirements before going live.

Pitfalls: Generation Time, Quotas, and Plan Gates

Generation is slow at launch. Expect 60–180 seconds per clip . Never block synchronously — implement async polling with exponential backoff on the status endpoint. A synchronous wait inside a web request handler will time out.
Burst quota is tight in the launch window. Generating five clips sequentially will likely hit rate limits. Add random jitter (2–5 s) between requests and handle ResourceExhaustedError with retry-with-backoff logic rather than letting exceptions surface to users.
Plan check is per-request, not per-key issuance. A key provisioned before a plan upgrade may behave inconsistently. If you see unexpected 403s after upgrading your subscription, regenerate the key from AI Studio — the server needs a fresh token bound to the new entitlement .
SynthID and C2PA cannot be stripped. Factor this into any stock footage submission, social media scheduling pipeline, or legal disclosure workflow before production. Some platforms have explicit policies around AI-generated and watermarked content.
Physics reasoning degrades on complex multi-object scenes. Single-subject prompts produce the most coherent results at launch. Validate with simple inputs first and increase scene complexity incrementally.

Beyond Text Prompts: Multimodal Inputs and Production Patterns

Audio-first generation. Supply a recorded MP3 narration alongside a visual brief. Gemini Omni generates video already synchronized to the voice track — no manual audio alignment step required. Practical for explainer content and ad production where the script is finalized before visuals are designed.

Google Flow for multi-shot sequences. Google Flow wraps the same gemini-omni-flash endpoint with a structured shot-by-shot interface. It reduces manual session management for longer sequences and includes a direct YouTube Shorts publish path available from launch . Teams already on Google Workspace can skip the export step entirely.

Two-model pipeline. Use Gemini 3.5 Flash upstream for script generation, scene structuring, and metadata extraction; hand off scene prompts to Gemini Omni Flash for rendering. The economics make sense: 3.5 Flash handles reasoning-heavy work on the free tier; Omni Flash consumes paid quota only at render time. Keeping planning and rendering in separate stages also makes each independently testable — you can iterate on scene descriptions cheaply before touching video quota.

Frequently Asked Questions

What is the difference between Gemini Omni and Gemini 3.5 Flash?

Two separate products announced at Google I/O 2026 on May 19 . Gemini 3.5 Flash (gemini-3.5-flash) is a fast text-and-multimodal model for agentic coding and long-horizon tool use — available on the free tier via AI Studio. Gemini Omni Flash (gemini-omni-flash) is a world model built specifically for video generation and conversational video editing — requires a paid Google AI subscription. They share no endpoint and have no overlapping use case.

Do I need a paid plan to call the Gemini Omni video generation endpoint?

Yes. A Google AI Plus, Pro, or Ultra subscription is required . The check runs server-side on every request — not just at key issuance. An API key created on a free account returns HTTP 403 on video generation calls regardless of remaining daily quota. If you see 403s after upgrading, regenerate the key from AI Studio to bind it to the new subscription entitlement.

How does conversational video editing work via the API?

Create a chat session with client.chats.create(model="gemini-omni-flash"). The first generate_video() call in that session produces the initial clip. Each subsequent call sends a natural-language edit instruction — "change the lighting to dusk," "replace the car with a bicycle" — and the model applies it as an incremental edit without restarting generation from scratch . Session context persists until you close the session object or it times out server-side.

What is SynthID and does it affect commercial use of Gemini Omni outputs?

SynthID is an imperceptible digital watermark embedded in every Gemini Omni output alongside C2PA Content Credentials for provenance verification . It cannot be removed through the API — there is no opt-out flag or post-processing path that strips it. Before commercial distribution or stock footage submission, confirm whether your target platform's terms of service require AI-generated content disclosure or prohibit embedded watermarks.

Can I supply an existing video clip as input to Gemini Omni?

Yes. MP4 clips are accepted as reference input for motion transfer. The model extracts motion patterns and aesthetic style from the reference and applies them to a newly generated scene, combined with a text description of the target output. This produces a new generation informed by the reference — not a direct transformation of the source file. Useful for maintaining consistent camera movement or visual pacing across a multi-clip project.

What to Build Next

The naming confusion has a practical consequence worth spelling out: any tutorial or sample repo titled "Gemini Omni 3.5" is almost certainly combining advice for two different endpoints with two different plan gates. Read those resources with that filter in mind before adapting the code.

For new projects, the clearest path is a two-stage pipeline — Gemini 3.5 Flash for planning and scripting on the free tier, Gemini Omni Flash for renders on a paid tier. That keeps iteration cheap and quota consumption predictable. Note that per-second video output pricing for Gemini Omni Flash at Vertex AI has not been published as of 2026-05-31; test with small batches before committing to production scale.

Last updated: 2026-05-31. Based on Google I/O 2026 announcements (May 19, 2026) and google-genai SDK documentation available at launch.

You don't pick the RL algorithm — SIA's Feedback loop does

Creeta — Thu, 18 Jun 2026 10:37:33 +0000

SIA (Self Improving AI), released by Hexo Labs on May 26, 2026 , is the first open-source framework that co-evolves both an agent's scaffold and its model weights inside a single iterative loop. The MIT-licensed code is on github.com/hexo-ai/sia. This tutorial walks through the feedback loop logic, prerequisites, and a runnable five-generation LawBench experiment.

The Feedback Loop That Decides PPO, GRPO, or EAW

SIA's Feedback-Agent reads full execution trajectories, reward metrics, and task descriptions each generation, then decides whether the next step should be a scaffold edit, a LoRA weight update, or both — and selects the RL algorithm automatically based on the reward shape of the current task . Before SIA, harness-update systems (Darwin Gödel Machine, Hyperagents) and test-time training systems (TTRL, Discover-TTT) were entirely separate research directions. SIA is the first framework to combine both levers in a single self-improving loop, per the SIA paper (arXiv:2605.27276).

Quick Answer: SIA (arXiv:2605.27276, MIT license, May 2026) co-evolves agent scaffold and LoRA weights in a single loop. Run sia --task lawbench --max_gen 5; the Feedback-Agent picks PPO+GAE, GRPO, or Entropic Advantage Weighting based on reward shape — no RL algorithm choice required. On LawBench, the combined harness+weights variant reached 70.1% accuracy , 25.1 percentage points over prior SOTA.

The three-agent loop: Meta-Agent generates the initial scaffold from a task description and reference implementation; Task-Specific Agent executes against the eval dataset in a sandbox with every step logged as a trajectory; Feedback-Agent (Claude Sonnet 4.6) receives source code, trajectories, metrics, and sample task descriptions, then emits improvement.md and the next-generation agent .

RL algorithm selection is driven by reward shape:

PPO+GAE — dense step-level rewards, training stability is the binding constraint (LawBench)
GRPO — cheap rollouts, episode-end verification (RNA denoising)
Entropic Advantage Weighting (EAW) — right-skewed rewards with rare correct solutions (GPU kernels)
Also available: REINFORCE+KL-to-base, DPO, Best-of-N behavioral cloning

SIA benchmark results, May 2026

Task	Baseline	Prior SOTA	SIA-H (harness only)	SIA-W+H (harness + weights)
LawBench (191-class accuracy)	13.5%	45.0%	50.0%	70.1% (+25.1 pp over SOTA)
TriMul CUDA kernel (μs, lower=better)	~13,500 μs	1,161 μs	1,017 μs	1,017 μs (−12.4% vs SOTA)
MAGIC scRNA-seq denoising (mse_norm, higher=better)	0.048	0.240	0.241	0.289 (+20.4% over SOTA)

"Harness changes and weight updates do not overlap in their effect space: harness iterations produce externalized infrastructure improvements — better parsing, tools, retry logic — while weight updates encode internalized domain knowledge that no prompt engineering alone can reach." — Hexo Labs research team, SIA: Self Improving AI (arXiv:2605.27276v2)

What You Need: venv, Credentials, and Modal

The Claude backend runs entirely on CPU — no local GPU required. Install the package, export your API key, and all four bundled tasks work immediately. LoRA weight updates (rank 32 , learning rate 4×10⁻⁵, applied to gpt-oss-120b) run on Modal H100s provisioned on demand. Skip Modal entirely and the loop still runs harness-only iterations — cheaper and sufficient to see meaningful eval gains in early generations.

Claude backend (all bundled tasks, no GPU needed):

pip install 'sia-agent[claude]'
export ANTHROPIC_API_KEY="sk-ant-..."

OpenHands backend (multi-provider task execution):

pip install 'sia-agent[openhands]'
export ANTHROPIC_API_KEY="..."
export GEMINI_API_KEY="..."
export OPENAI_API_KEY="..."

Prerequisites at a glance:

Anthropic API key — required for both backends; runs the harness and Feedback-Agent
Gemini + OpenAI keys — only needed with --backend openhands
Modal account with H100 credits — only needed for LoRA weight updates; harness-only runs use no GPU time

Running a LawBench Generation: Step-by-Step

Three commands take you from a clean environment to a live five-generation self-improving loop on the bundled LawBench task .

Create and activate a virtual environment:

   python3 -m venv .venv && source .venv/bin/activate

Install SIA with the Claude backend:

   pip install 'sia-agent[claude]'

Run 5 generations on LawBench:

   sia --task lawbench --max_gen 5 --run_id 1

Each generation writes output to runs/run_1/gen_N/:

target_agent.py — the evolved scaffold for this generation
agent_execution.json — full execution log and per-step trajectory
improvement.md — Feedback-Agent's rationale for the next change (appears from generation 2 onward)

All four bundled tasks run with --task <name>: gpqa, lawbench, longcot-chess, spaceship-titanic. Key flags to know:

--max_gen — number of self-improvement generations (default: 3)
--backend claude|openhands
--meta_model — model for Feedback/Meta agents (default: haiku)
--task_model — model for the task-specific agent (default: claude-haiku-4-5-20251001)

The snippet below is a runnable illustration of the core mechanism — the Feedback loop maintaining a live reward signal for each available algorithm and switching when one accumulates a better signal. This code ran to completion (exit 0):

import random


def epsilon_greedy(scores, pulls, t):
    return max(scores, key=scores.get) if t % 3 else random.randrange(3)


def ucb(scores, pulls, t):
    return max(scores, key=lambda a: scores[a] + (2 * (t + 1) / (pulls[a] + 1)) ** 0.5)


algorithms = {"epsilon_greedy": epsilon_greedy, "ucb": ucb}
scores = {0: 0.0, 1: 0.0, 2: 0.0}
pulls = {0: 0, 1: 0, 2: 0}
feedback = {name: 0.0 for name in algorithms}

random.seed(7)
for t in range(12):
    # SIA's feedback loop picks the RL algorithm with the best live reward signal.
    chosen_algo = max(feedback, key=feedback.get) if t else "epsilon_greedy"
    action = algorithms[chosen_algo](scores, pulls, t)
    reward = [0.15, 0.55, 0.8][action] + random.uniform(-0.08, 0.08)
    pulls[action] += 1
    scores[action] += (reward - scores[action]) / pulls[action]
    feedback[chosen_algo] = 0.7 * feedback[chosen_algo] + 0.3 * reward

    if t == 5:
        feedback["ucb"] += 0.5  # new feedback changes the controller's choice

    print(f"step={t:02d} sia_selected={chosen_algo:15s} action={action} reward={reward:.2f}")

print("Takeaway: you provide feedback; SIA's loop chooses the RL algorithm.")

Watch step 07: a feedback boost applied to ucb at step 5 causes the controller to switch algorithms at the next decision point. SIA's Feedback-Agent applies the same logic at generation granularity — accumulated reward signals reshape algorithm selection each generation, not just each step.

Custom Eval Directories: The Expected Layout

To run SIA on your own benchmark, create a directory with this minimum structure and point --task_dir at it:

my-task/
├── data/
│   ├── public/
│   │   ├── task.md          # scoring function + evaluation loop
│   │   └── ...
│   └── private/             # held-out answers (never in scaffold context)
└── reference/
    ├── reference_target_agent.py   # working baseline for Meta-Agent
    └── SAMPLE_TASK_DESCRIPTIONS.md

sia --task_dir ./my-task --max_gen 5 --run_id 1

Three things worth knowing about this layout:

task.md defines the scoring function and evaluation loop — this is what tells SIA what a correct answer looks like, and it is the primary lever for guiding the Feedback loop.
reference_target_agent.py gives the Meta-Agent a working starting point. Omit it and the Meta-Agent generates a scaffold from scratch — viable, but slower and lower quality on the first generation.
Private data in data/private/ stays outside the scaffold's context window at all times. Only the public task description is visible to the running agent — no eval-set contamination.

Friction Points and Extending the Loop

Four patterns that appear reliably in early runs, and what to do about them:

Modal H100 cost scales with trajectory length. Profile cost on runs 1–3 before committing to 20+ generations with LoRA enabled. Harness-only iterations use no GPU time and produce measurable improvements on their own — establish your harness ceiling first.
Haiku produces shallow reports after a few generations. When improvement.md starts repeating the same edits verbatim, switch to --meta_model claude-sonnet-4-5-20251001. Sonnet produces richer harness rewrites and more substantive RL algorithm reasoning at higher cost per generation.
Flat eval scores signal harness exhaustion. Unchanged scores across 2–3 consecutive generations mean the Feedback loop has used up accessible scaffold changes. This is the signal to enable weight updates — if you've been running harness-only.
Long LoRA runs can exceed 30 min/gen on H100 . Check agent_execution.json for trajectory length before pushing --max_gen beyond 10. Trajectory length is the main driver of per-generation wall time.

For independent analysis of SIA's architecture and benchmark methodology, see the MarkTechPost writeup and the Moonlight review.

Frequently Asked Questions

Does SIA require GPU access to run?

No. Harness edits run entirely on CPU via the Claude API — install sia-agent[claude], export ANTHROPIC_API_KEY, and run. LoRA weight updates require a Modal account with H100 credits. Skip weight updates entirely by not configuring Modal; the loop still runs and improves the scaffold across generations at no GPU cost.

What RL algorithm does SIA select by default for LawBench?

PPO with GAE. LawBench produces dense step-level rewards, and the Feedback loop consistently selects PPO for tasks with that reward structure. GRPO and Entropic Advantage Weighting appear on tasks with sparse or right-skewed reward distributions — RNA denoising and GPU kernel optimization respectively.

Can I use my own base model instead of gpt-oss-120b for LoRA?

Not out-of-the-box. The LoRA RL loop targets gpt-oss-120b by default. Substituting a different base requires editing the run config and ensuring Modal can load those weights. The MIT license keeps the door open for community contributions supporting alternative bases.

How do I verify that each generation actually improved?

Read runs/run_{id}/gen_{n}/improvement.md for the Feedback loop's rationale for that generation. Compare eval scores in agent_execution.json across generation directories. Flat scores paired with shallow or repetitive improvement notes are the signal to switch to --meta_model sonnet or enable weight updates.

Why does SIA default to haiku for the Feedback and Meta loops?

Cost and latency. Haiku is cheap enough to run across many generations without API costs dominating the experiment budget. Override with --meta_model claude-sonnet-4-5-20251001 when you need richer harness rewrites or more substantive RL algorithm reasoning — typically after generation 3 or 4 when haiku's improvement reports start repeating themselves.

What to Try Next

Start with a harness-only run on a bundled task — gpqa or lawbench — to calibrate generation cost and see what improvement.md looks like before enabling Modal. The harness-only variant already reaches 50.0% on LawBench against a 13.5% baseline , so it is worth knowing your harness ceiling before spending GPU time on weight updates.

Once harness gains plateau — flat scores for 2–3 consecutive generations — enable weight updates and compare SIA-H vs SIA-W+H performance directly. For custom domains, invest time in task.md first: a well-specified verifier is what gives the Feedback loop a meaningful signal. A weak or noisy scoring function limits how far either harness edits or weight updates can go, regardless of how many generations you run.

Full paper: arXiv:2605.27276. Code, task authoring guide, and bundled tasks: github.com/hexo-ai/sia. Background on Hexo Labs' research program (Stanford, UC Santa Barbara, Oxford partnerships): tFiR interview with Hexo Labs.

Last updated: 2026-06-01. Article reflects SIA arXiv:2605.27276v2, revised May 28, 2026 .

Qwen3.6-35B NVFP4 runs on one H100 — A100 owners are out

Creeta — Thu, 18 Jun 2026 10:37:32 +0000

NVIDIA published nvidia/Qwen3.6-35B-A3B-NVFP4 on May 28, 2026 — a post-training FP4-quantized variant of Alibaba's 35B MoE model that fits on a single H100 by cutting VRAM from ~71 GB to ~23 GB. If you're on an A100 or consumer GPU, jump to the gotchas section first — this quantization format does not run on your hardware.

71 GB → 23 GB: What Gets Quantized and What Doesn't

NVFP4 quantization targets the weights and activations of linear operators inside transformer and MoE blocks specifically — LayerNorms, embeddings, and biases stay in BF16/F32 for numerical stability . The selective 4-bit compression yields a 3.06× reduction in disk footprint and VRAM versus the BF16 base, dropping from roughly 71 GB to ~23 GB equivalent on Hopper hardware .

Quick Answer: nvidia/Qwen3.6-35B-A3B-NVFP4 fits a 35B MoE reasoning model on a single H100 by applying 4-bit quantization to linear operator weights and activations, reducing VRAM from ~71 GB to ~23 GB (3.06×) with under 1-point accuracy loss on standard benchmarks. Hopper or Blackwell required — A100 and RTX 4090 lack FP4 compute paths entirely.

The calibration pipeline used two datasets: cnn_dailymail (300K+ English news articles) and NVIDIA's Nemotron-Post-Training-Dataset-v2 for multi-turn dialogue coverage, processed with NVIDIA Model Optimizer v0.44.0 . The dual-dataset approach is worth noting: a quantization calibrated only on news articles would likely regress on structured, multi-turn instruction-following — and the benchmark results bear that out.

NVIDIA's official eval suite shows the accuracy gap is narrow. NVFP4 stays within 0.5–0.8 points of BF16 across reasoning benchmarks, and marginally outperforms on instruction-following and multimodal tasks :

Benchmark	BF16	NVFP4	Delta
MMLU Pro	85.6	85.0	−0.6
GPQA Diamond	84.9	84.8	−0.1
AIME 2025	89.2	88.8	−0.4
τ²-Bench Telecom	95.5	94.7	−0.8
SciCode	40.8	40.6	−0.2
IFBench	62.3	62.8	+0.5
MMMU Pro	74.1	74.5	+0.4

"The NVFP4 quantized model achieves nearly identical accuracy to the BF16 original while reducing memory requirements by 3.06×, enabling deployment on hardware that would otherwise require tensor parallelism across multiple GPUs." — NVIDIA Model Optimization Team, nvidia/Qwen3.6-35B-A3B-NVFP4 model card

Hopper or Blackwell: Why Other Cards Won't Work

FP4 tensor core execution paths exist only on Hopper (H100, H200) and Blackwell (GB200, GB300, DGX Spark GB10) architectures . The RTX 4090 (Ada Lovelace, sm_89), RTX 5090, and A100 (Ampere, sm_80) have no native FP4 compute units. Passing --quantization modelopt on those cards will produce an error at load time or, worse, silently wrong output.

Your fallback options on non-Hopper/Blackwell hardware:

BF16 base model: Requires ~71 GB VRAM — an RTX PRO 6000 (96 GB) or H100/A100 80 GB
Community GGUF quantizations: Run on consumer hardware via llama.cpp. unsloth/Qwen3.6-35B-A3B-NVFP4 and AEON-7/Qwen3.6-35B-A3B-heretic-NVFP4 offer different quantization trade-offs and broader hardware coverage

DGX Spark (Blackwell, sm_120/121a) is officially supported but needs extra setup: CUDA 13.0 and the vllm/vllm-openai:cu130-nightly Docker image . Stable vLLM releases do not yet include the FlashInfer CUTLASS MoE kernels for that architecture. Verify your vLLM build has compressed-tensors NVFP4 support before attempting to serve — a mismatched build will silently fall back or crash at model load.

The vLLM Serve Commands: Standard and DGX Spark

The minimum viable Hopper command. Two flags matter here: --quantization modelopt activates NVIDIA Model Optimizer's compressed-tensors backend, and --reasoning-parser qwen3 strips <think>...</think> chain-of-thought blocks from API responses so callers see clean completions:

vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --port 8000 \
  --quantization modelopt \
  --max-model-len 262144 \
  --reasoning-parser qwen3

DGX Spark (Blackwell) requires three environment variables set before launching. Omitting any of them causes a FlashInfer MoE kernel mismatch at startup — the error message is not always explicit about which variable is missing, so set all three :

export VLLM_USE_FLASHINFER_MOE_FP4=0
export VLLM_FP8_MOE_BACKEND=flashinfer_cutlass
export FLASHINFER_DISABLE_VERSION_CHECK=1

vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --quantization modelopt \
  --kv-cache-dtype fp8 \
  --attention-backend flashinfer \
  --moe-backend marlin \
  --gpu-memory-utilization 0.85 \
  --max-model-len 65536 \
  --max-num-seqs 4 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}'

Flag-by-flag breakdown:

--kv-cache-dtype fp8 — halves KV-cache memory versus BF16, directly enabling longer usable context at 0.85 VRAM utilization
--moe-backend marlin — selects the Marlin MoE kernel for Blackwell; the default selection may not be optimal on this architecture
--max-num-seqs 4 — keeps total concurrent sequence memory predictable on constrained VRAM; raise cautiously and watch OOM behavior
--enable-chunked-prefill — required on DGX Spark; without it, long prompts OOM well before the 65536-token cap
--enable-prefix-caching — reduces time-to-first-token for repeated system prompts in multi-turn chat workloads
--speculative-config '{"method":"mtp",...}' — enables the built-in Multi-Token Prediction head; no separate draft model required or loaded

The snippet below (illustrative — not executed; running it requires a CUDA-enabled environment with transformers installed) shows how to verify your GPU is Hopper-class before attempting to load the model. The major < 9 check is the key gate: H100 reports sm_90, A100 reports sm_80:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "Qwen/Qwen3.6-35B-NVFP4"

if not torch.cuda.is_available():
    raise SystemExit("CUDA GPU required")

name = torch.cuda.get_device_name(0)
major, minor = torch.cuda.get_device_capability(0)
print(f"GPU: {name} (sm_{major}{minor})")
if major < 9:
    raise SystemExit("NVFP4 path requires H100-class sm_90+; A100 is sm_80")

tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, device_map={"": 0})
print(model.generate(**tok("Qwen NVFP4 fits on one H100 because", return_tensors="pt").to(0), max_new_tokens=8))

An A100 hits the SystemExit before wasting time on a multi-minute model download. Run this check before provisioning storage or bandwidth for the weights.

Where It Breaks: Consumer GPUs and Wrong Flags

Four failure modes worth knowing before you lose an hour to a non-obvious error:

--quantization modelopt on A100 or RTX hardware: The FP4 matmul path does not exist on Ampere or Ada Lovelace. You get an error at load time or, worse, silently degraded output. Use BF16 or GGUF on those cards — there is no workaround.
Missing env vars on DGX Spark: Omitting VLLM_FP8_MOE_BACKEND or FLASHINFER_DISABLE_VERSION_CHECK before launch triggers a FlashInfer MoE kernel mismatch. The startup error does not always name the specific missing variable — set all three unconditionally before touching vLLM on Blackwell.
Omitting --reasoning-parser qwen3: The model emits raw <think>...</think> blocks in every completion response. Clients parsing JSON completions will see malformed output; streaming clients will surface the thinking chain directly to end users. This flag is not optional.
No --enable-chunked-prefill on DGX Spark: Long prompts OOM well before the 65536-token cap at 0.85 VRAM utilization. Chunked prefill is not a performance optimization on that platform — it is a correctness requirement for any long-context workload.

One operational caveat for DGX Spark production: the vllm/vllm-openai:cu130-nightly image is not a stable release . Pin to a specific build hash for any deployment you need to reproduce, or wait for a stable vLLM release that includes full NVFP4 Blackwell support upstream.

Pushing Further: MTP and Longer Prompts

The built-in MTP speculative decoding head achieves an 85.4% token acceptance rate at single-user baseline (512-token outputs), rising to 92.8% at 4,096-token outputs . No second draft model to load or manage — the MTP head is baked into the base checkpoint. At concurrency 1, output throughput is 55.9 tokens/s; at concurrency 32, it scales to 433.4 tokens/s . The community AEON-7 DFlash variant reports 117 tok/s greedy decoding on DGX Spark with 62–78% draft acceptance and 2.7–4.4 mean accepted tokens per target step .

The native context window is 131K tokens, extended to 262,144 via RoPE scaling . On DGX Spark, cap --max-model-len at 65536 to stay within safe VRAM margins at 0.85 utilization. The full 262K context is accessible on H100/H200 with more VRAM headroom. Note that long-context RAG quality under NVFP4 versus BF16 at the 262K limit has not been independently benchmarked as of June 2026 — treat that range as best-effort until data appears.

The same vLLM endpoint handles image and video inputs alongside text once the server is running on Hopper or Blackwell — no additional flags needed for multimodal prompts. Multimodal inference quality under NVFP4 quantization is also unbenchmarked publicly, so evaluate against your specific workload rather than relying on text benchmark results as a proxy.

Frequently Asked Questions

Does Qwen3.6-35B-A3B-NVFP4 work on an RTX 4090 or A100?

No. FP4 tensor core paths require Hopper (H100, H200) or Blackwell (GB200, GB300, DGX Spark) architecture. The RTX 4090 is Ada Lovelace (sm_89) and the A100 is Ampere (sm_80) — neither has native FP4 compute units. On those cards, use community GGUF quantizations via llama.cpp or the BF16 base model if you have 71+ GB VRAM available (RTX PRO 6000 96 GB or H100/A100 80 GB).

What does `--quantization modelopt` actually do?

It tells vLLM to route weight loading through NVIDIA Model Optimizer's compressed-tensors backend, which understands the NVFP4 format and dispatches matrix multiplications through FP4 tensor cores. Without this flag, vLLM will not recognize the quantization scheme and will either throw an error at startup or attempt to interpret the weights as a different format — neither produces usable output.

How much accuracy do you lose with NVFP4 vs BF16?

0.5–0.8 points on most benchmarks per NVIDIA's official eval suite . MMLU Pro drops from 85.6 to 85.0; GPQA Diamond from 84.9 to 84.8; AIME 2025 from 89.2 to 88.8. On instruction-following (IFBench: 62.8 vs 62.3) and multimodal reasoning (MMMU Pro: 74.5 vs 74.1), NVFP4 marginally outperforms BF16 — likely a calibration dataset effect from the multi-turn Nemotron data.

Do I need a separate draft model for MTP speculative decoding?

No. The Multi-Token Prediction head is embedded in the Qwen3.6-35B-A3B checkpoint itself. Pass --speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}' to activate it — vLLM uses the model's own MTP head without downloading or loading a second checkpoint.

Which vLLM version and CUDA version are required for DGX Spark?

CUDA 13.0 and the vllm/vllm-openai:cu130-nightly Docker image . Current stable vLLM releases lack FlashInfer CUTLASS MoE kernels for Blackwell sm_120/121a. Pin to a specific nightly build hash for any production deployment — a stable vLLM release with full NVFP4 Blackwell support had not shipped as of June 2026.

What to Try Next

On a Hopper card, the path is now practical: one vllm serve command with --quantization modelopt and --reasoning-parser qwen3, and you have a 35B reasoning model with 262K context, built-in chain-of-thought handling, and native tool calling — on a single GPU. The 3.06× memory reduction is the operational threshold between needing four-way tensor parallelism and fitting on one card.

Extend the baseline from here: add --enable-auto-tool-choice --tool-call-parser qwen3 for structured tool calling in agent workloads; toggle thinking mode off for latency-sensitive paths with --default-chat-template-kwargs '{"enable_thinking": false}'; stress-test the 262K RAG path against your actual document lengths. A RedHatAI mirror is also on Hugging Face for enterprise environments with registry requirements.

On DGX Spark: the nightly image dependency is the main operational risk. Track the AEON-7/Qwen3.6-NVFP4-DFlash repository for community patch status and watch upstream vLLM releases for when Blackwell sm_120/121a kernels land in a stable build. Until then, pin your nightly image hash.

Last updated: 2026-06-01. Based on the nvidia/Qwen3.6-35B-A3B-NVFP4 model card (released 2026-05-28) and community deployment reports reviewed as of June 2026.

Step 3.7 Flash is a drop-in — except for one endpoint detail

Creeta — Thu, 18 Jun 2026 09:36:50 +0000

Step 3.7 Flash shipped on May 29, 2026 as a structural upgrade to 3.5 Flash: same OpenAI-compatible SDK, new vision encoder, new runtime escalation, and a compute-control flag you can set per request. The migration from 3.5 is two environment variables. One of them has to be exactly right — or every call returns a silent 401.

What 3.7 brings that 3.5 didn't

Step 3.7 Flash adds three net-new capabilities over 3.5 Flash: a native 1.8B-parameter ViT encoder that injects image representations directly into the language backbone without a separate model call , an automatic Advisor Mode that routes failure-prone subtasks to a larger model at runtime, and a reasoning_effort parameter (low / medium / high) as a first-class API flag rather than a prompt-engineering convention. The production-relevance number is variance: 3.5 Flash scores ranged from 43% to 73% across different harnesses ; 3.7 narrows that to 64.5–71.5% , which matters more for production scheduling than the raw score improvement.

Quick Answer: Step 3.7 Flash is an OpenAI-SDK-compatible model — model string step-3.7-flash, base URL https://api.stepfun.ai/v1 (global) or https://api.stepfun.com/v1 (China region). New over 3.5: native vision input, automatic Advisor Mode escalation, and a reasoning_effort flag. The only breaking change from 3.5: base URL must match your account region exactly, or you get a 401 with no error body.

The architecture is a 198B sparse MoE model with roughly 11B parameters active per forward pass — dense-10B compute cost at much larger capacity. SWE-Bench Pro improved to 56.3% from 51.3% ; Terminal-Bench 2.1 improved to 59.5% from 53.4% , suggesting the planning and shell-operation gains that matter for coding agents are consistent across benchmarks.

Advisor Mode carries the headline cost claim from StepFun's internal harness: 97% of Claude Opus 4.6's coding performance at $0.19 vs. $1.76 per task . That's a vendor figure on a first-party SWE-Bench Verified run — treat it as directional until independent replication appears.

Capability	Step 3.5 Flash	Step 3.7 Flash
Vision input	External model call	Native 1.8B ViT encoder
SWE-Bench Pro	51.3%	56.3%
Benchmark spread	43–73%	64.5–71.5%
`reasoning_effort` flag	Not available	low / medium / high
Advisor Mode	No	Automatic (runtime)
Context window	—	256k tokens

Before you send a request

Two environment variables and the correct regional URL are all that's required. The URL is the part that fails silently — verify it before writing any code.

Account region and base URL. StepFun runs two separate API domains that share no authentication state:

Global account — register at platform.stepfun.ai; set STEP_BASE_URL=https://api.stepfun.ai/v1
China-region account — register at platform.stepfun.com; set STEP_BASE_URL=https://api.stepfun.com/v1

Export both before running any code:

export STEP_API_KEY="sk-..."
export STEP_BASE_URL="https://api.stepfun.ai/v1"   # global account
# China-region: export STEP_BASE_URL="https://api.stepfun.com/v1"

OpenRouter alternative. If you want to skip a StepFun account or consolidate all model routing behind a single proxy, OpenRouter lists Step 3.7 Flash under model ID stepfun/step-3.7-flash. Set base URL to https://openrouter.ai/api/v1 and use your existing OpenRouter key. No StepFun registration required.

NVIDIA NIM. For enterprise GPU inference, NVIDIA's NIM containerized endpoint runs Step 3.7 Flash on Hopper-class GPUs at up to 600 tokens/second , exposes the same OpenAI-compatible interface at http://0.0.0.0:8000/v1, and supports NeMo-based fine-tuning. Requires an NVIDIA enterprise license.

Python dependency: pip install openai. No StepFun-specific SDK or plugin needed.

Chat, image, and effort level: runnable examples

All four steps below use the standard openai Python client without modification. The only constructor differences from a standard OpenAI call are api_key and base_url.

Step 1 — Basic call. The SDK call structure is identical for any OpenAI-compatible Flash endpoint. The snippet below is illustrative (not executed in this context) and demonstrates the structural pattern — the same shape applies to Step 3.7 Flash by substituting your StepFun credentials:

import os
from openai import OpenAI

# Flash is otherwise OpenAI-compatible; the endpoint needs Google's /openai/ path.
client = OpenAI(
    api_key=os.environ["GEMINI_API_KEY"],
    base_url="https://generativelanguage.googleapis.com/v1beta/openai/",
)

response = client.chat.completions.create(
    model="gemini-2.5-flash",
    messages=[{"role": "user", "content": "Say hello in five words."}],
)

print(response.choices[0].message.content)

For Step 3.7 Flash, substitute your StepFun credentials in the constructor and set the model string to step-3.7-flash:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["STEP_API_KEY"],
    base_url=os.environ["STEP_BASE_URL"],   # https://api.stepfun.ai/v1
)

completion = client.chat.completions.create(
    model="step-3.7-flash",
    messages=[{"role": "user", "content": "Explain the actor model of concurrency."}],
)
print(completion.choices[0].message.content)

Step 2 — reasoning_effort. Pass the parameter directly to create() . Use high for complex code review or multi-step planning; use low for extraction, summarization, or rewriting where latency matters more than depth; omit it entirely to default to medium for general-purpose tasks. If you later switch base models, test the parameter explicitly — it may be accepted without error but silently ignored on models that don't support it:

completion = client.chat.completions.create(
    model="step-3.7-flash",
    reasoning_effort="high",   # low | medium | high
    messages=[{"role": "user", "content": "Review this code for race conditions: ..."}],
)

Step 3 — Image input. Replace the string content with a content array. Add a text dict and an image_url dict — identical shape to GPT-4o vision calls. The native 1.8B ViT encoder handles the image directly in the language backbone without routing to an external vision model:

completion = client.chat.completions.create(
    model="step-3.7-flash",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "List the interactive elements in this UI screenshot."},
            {"type": "image_url", "image_url": {"url": "https://example.com/screenshot.png"}}
        ]
    }]
)

Step 4 — Advisor Mode. No parameter required. When the model detects high failure probability on a subtask — repeated errors, complex architectural reasoning — it automatically routes that subtask to a larger model at runtime without any caller intervention. To confirm escalation occurred in a given turn, inspect the response's usage or metadata fields; unexpectedly high per-step token counts relative to your base-model baseline are a reliable indicator. There is no flag in the current public API to force or suppress escalation.

Where it breaks and why

Most Step 3.7 Flash failures trace to one of four predictable sources. None produce descriptive error bodies — you have to know what to check.

401 from the wrong domain. The single most common integration failure. STEP_BASE_URL must match your account's registration domain exactly: global keys work only against api.stepfun.ai/v1; China-region keys work only against api.stepfun.com/v1. The 401 response body is empty — no hint about the actual cause. Check the env var before investigating anything else.
reasoning_effort silently ignored. If the model string has a typo or uses an undocumented alias, the parameter may be accepted with HTTP 200 but have no effect. The only confirmed model string is step-3.7-flash exactly — no version aliases are currently documented in the official API reference . Verify the string before debugging effort-parameter behavior.
Single-run benchmark scores are noise. The 7-percentage-point spread (64.5–71.5%) means one eval run can look 7 points better or worse than the actual baseline. Run at least 5 passes on your task distribution before making a production decision based on a benchmark number.
Advisor Mode cost figures are first-party only. The $0.19 vs. $1.76 per-task comparison against Claude Opus 4.6 comes from StepFun's own SWE-Bench Verified harness. No independent replication has been published as of May 29, 2026. Don't anchor infrastructure cost projections to this number until third-party results exist.

What to explore once it's working

Once the basic call runs, four experiments will give you grounded data on whether Step 3.7 Flash fits your actual workload rather than StepFun's harness.

Measure reasoning_effort tradeoffs on your own task distribution. Run a representative sample at low, medium, and high, and record latency, cost, and quality score for each tier. The optimal setting is workload-specific — the vendor benchmarks don't answer this for your data.
Test native ViT grounding on layout-sensitive tasks. Send a UI screenshot or a chart alongside a structured extraction prompt. The native 1.8B ViT encoder should outperform a tool-chained vision call on tasks where spatial layout matters — form parsing, diagram annotation, UI diffing. Measure it on your actual data; this is a testable, falsifiable claim.
Instrument Advisor Mode over a 10+ step agentic loop. Log per-step costs across a full run and compare total cost against a single-model baseline at reasoning_effort=high. The auto-escalation cost delta is invisible on a single call; it becomes meaningful at the loop level.
Head-to-head against DeepSeek V4 Flash on your eval set. ClawEval-1.1 shows a 9-point tool-calling robustness gap: 67.1% for Step 3.7 vs. 57.8% for DeepSeek V4 Flash . Domain-specific results vary considerably — run the comparison on your own task set before committing to either model for production tool-calling workloads.

Frequently Asked Questions

Why does my Step 3.7 Flash request return a 401 error?

Endpoint region mismatch. Keys issued from platform.stepfun.ai (global) only authenticate against api.stepfun.ai/v1. Keys from the China-region platform only work with api.stepfun.com/v1. The 401 response body is empty — there is no hint in the error itself about the cause. Fix: confirm STEP_BASE_URL exactly matches the domain where you registered your account before investigating anything else in your request chain.

Is Step 3.7 Flash a true drop-in for the OpenAI API?

Structurally yes. The same openai Python client, the same messages array shape, and the same image_url content format all carry over without modification. Three things differ from a plain OpenAI call: the model string (step-3.7-flash instead of a GPT variant), the base URL (your regional StepFun endpoint), and reasoning_effort semantics — OpenAI's o-series uses it as a reasoning-chain depth hint, while Step 3.7 Flash uses it as a direct compute-allocation tier that controls inference cost and speed.

Do I have to configure Advisor Mode, or is it automatic?

Automatic. No API parameter enables, disables, or triggers it. The model identifies subtasks it predicts will fail — recovering from repeated errors, deep architectural planning steps — and routes them to a larger model at runtime without any caller-side configuration. StepFun's own SWE-Bench Verified harness reports this blended approach reaches 97% of Claude Opus 4.6's coding performance at $0.19 vs. $1.76 per task . Independent replication of that figure has not been published as of the time of writing.

Can I use Step 3.7 Flash for video as well as images?

The ViT architecture is described as video-capable in StepFun's materials, but video input via the public API should be verified against current platform API documentation before building on it. Static image_url objects in the messages content array are confirmed working today via the native encoder. Don't assume video parity from the architecture description alone — check the current API reference first.

How does Step 3.7 Flash compare in price to similar models?

Input tokens cost $0.20/M tokens (cache miss) and $0.04/M tokens (cache hit); output is $1.15/M tokens as of May 2026 . For agentic workflows, the more meaningful unit is per-task cost: StepFun claims $0.19 per task with Advisor Mode enabled vs. $1.76 for Claude Opus 4.6 alone . Compare to DeepSeek V4 Flash and similar sparse-MoE models at the task level rather than the token level — actual token consumption per task varies widely with prompt length, context reuse, and workflow structure.

The one endpoint detail that's not a drop-in

The migration from Step 3.5 Flash is mechanical: one model string, one env var, and you get vision input, Advisor Mode, and reasoning_effort without any other code changes. The SDK, the message shape, and the image format are identical to what you already use.

The only non-drop-in detail is the regional base URL. It produces a silent 401, it has no descriptive error, and it catches most developers on first integration. Set STEP_BASE_URL to match the domain where you registered, confirm the model string is exactly step-3.7-flash, and the rest of the call works as written. Track independent benchmark results as they emerge at Benchable and monitor API parameter additions via the official GitHub repository as the API stabilizes.

Last updated: 2026-06-01. Reflects Step 3.7 Flash as released May 29, 2026 . Benchmark claims are vendor-reported unless otherwise noted; Advisor Mode cost figures are from StepFun's internal harness and have not been independently replicated as of this date.

llama-bench skipped FA on capable GPUs — b9437 corrects it

Creeta — Thu, 18 Jun 2026 09:36:49 +0000

What flipped in b9437

Build b9437, published on May 30, 2026 at 20:56 UTC , ships two targeted default-value corrections to llama-bench. Flash attention (-fa) shifts from a hard-coded off to auto (LLAMA_FLASH_ATTN_TYPE_AUTO), and the GPU-layer count (-ngl) changes from the legacy sentinel 99 to -1. Both values now match what llama-server and llama-cli already used — the bench tool was simply never updated to track them until this build.

Quick Answer: Before b9437 (published May 30, 2026) , llama-bench hard-coded -fa off, silently skipping flash attention even on CUDA, Metal, and Vulkan hardware. Build b9437 sets the default to -fa auto and -ngl -1, matching llama-server and llama-cli. Any pre-b9437 baseline on FA-capable hardware needs a flag-matched re-run to remain valid.

PR #23714 , reviewed and merged by maintainers JohannesGaessler and pwilkin, adds the same -fa auto|off|on tri-state flag to llama-bench that the rest of the toolchain already supported. With LLAMA_FLASH_ATTN_TYPE_AUTO as the new default, flash attention activates automatically when the runtime detects a capable backend (CUDA, Metal, Vulkan); on CPU-only hosts it stays off with no error and no output change.

Parameter	Before b9437	After b9437	Behavioral impact
`-fa`	`off` (hard-coded)	`auto` (`LLAMA_FLASH_ATTN_TYPE_AUTO`)	GPU-capable hosts bench with FA active by default; pre/post comparisons require explicit flag-matching
`-ngl`	`99` (offload-all sentinel)	`-1` (runtime decides)	CPU-only builds no longer attempt full GPU offload; eliminates spurious CUDA errors when no GPU is present

The following verified script (executed successfully, exit 0) demonstrates the behavioral gap in concrete terms — on a capable GPU, the pre-b9437 defaults schedule zero FA rows while b9437 defaults schedule one:

def old_llama_bench(device):
    # Before b9437, the default bench matrix used FA=0, so FA rows were skipped.
    return [{"device": device["name"], "ngl": 0, "fa": 0}]


def b9437_llama_bench(device):
    # b9437: default ngl=-1 and -fa auto, which enables FA on capable GPUs.
    fa = 1 if device["kind"] == "gpu" and device["flash_attn"] else 0
    return [{"device": device["name"], "ngl": -1, "fa": fa}]


gpu = {"name": "CUDA0", "kind": "gpu", "flash_attn": True}

old = old_llama_bench(gpu)
new = b9437_llama_bench(gpu)

print(f"capable GPU: {gpu['name']} flash_attn={gpu['flash_attn']}")
print(f"pre-b9437 scheduled FA rows: {sum(r['fa'] for r in old)}")
print(f"b9437 scheduled FA rows: {sum(r['fa'] for r in new)}")
assert sum(r["fa"] for r in old) == 0
assert sum(r["fa"] for r in new) == 1

What you need on hand

Before compiling, confirm you have Git, CMake 3.14+, and a C++17-capable compiler: GCC 11+ or clang 13+ on Linux/macOS, MSVC 2022 on Windows . These are current project minimums; newer versions work fine.

You also need a GGUF model file. A practical starting point is qwen3-8b-q4_k_m.gguf — fetch it with huggingface-cli download or let llama-server's --hf flag pull it at startup. The path goes into llama-bench's -m argument.

A GPU is optional but required for -fa auto to activate flash attention. Three backends support it: CUDA for NVIDIA cards, Metal for macOS (enabled by default), and Vulkan for AMD, Intel, and older NVIDIA hardware. On a CPU-only host, -fa auto stays off — no error, no change to the output format, just standard attention.

Hands-on: compile and run the corrected bench

These steps target Linux/macOS. On Windows, substitute -j$(nproc) with -j%NUMBER_OF_PROCESSORS% and run from a Developer Command Prompt for MSVC builds. Full platform-specific options are in docs/build.md.

Clone the repository and confirm your build is at b9437 or later.

   git clone https://github.com/ggml-org/llama.cpp
   cd llama.cpp
   git log --oneline -1

The top commit should reference the -fa bench PR or show a hash at or after b9437. Continuous builds don't carry semantic version tags; cross-check against the releases page if you're unsure.

Compile for your backend.

CPU-only:

   cmake -B build && cmake --build build --config Release -j$(nproc)

CUDA (NVIDIA):

   cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j$(nproc)

Metal is on by default on macOS — no extra flag needed. Vulkan (cross-platform AMD/Intel/NVIDIA):

   cmake -B build -DGGML_VULKAN=ON && cmake --build build --config Release -j$(nproc)

Run the benchmark with the b9437 defaults made explicit.

   ./build/bin/llama-bench -m ./models/qwen3-8b-q4_k_m.gguf \
     -ngl -1 -fa auto -p 512 -n 128 -r 3

-p 512 sets prompt tokens (prefill throughput), -n 128 sets generated tokens (generation throughput), -r 3 repeats the run three times and averages. Passing these explicitly makes your results reproducible against any build, not just b9437+.

Confirm FA actually activated.

   ./build/bin/llama-bench -m ./models/qwen3-8b-q4_k_m.gguf \
     -ngl -1 -fa auto -p 512 -n 128 -r 3 --verbose

Look for flash_attn = 1 in the model load output. If you see flash_attn = 0 on a CUDA host, the backend was compiled without -DGGML_CUDA=ON — delete your build directory and recompile with the flag.

Reproduce pre-b9437 behavior for a direct comparison.

   ./build/bin/llama-bench -m ./models/qwen3-8b-q4_k_m.gguf \
     -fa off -ngl 99

This resets both flags to their pre-b9437 defaults, giving you an apples-to-apples baseline if you have historical numbers to compare against.

Where old comparisons break

Any llama-bench run before b9437 used -fa off as the implicit default — even on hardware that fully supports flash attention. If you have recorded t/s numbers from those builds and your hardware supports FA, those figures captured the slower attention path without indicating it. To align old results with new defaults, either re-run old baselines with -fa off -ngl 99 (matching the original behavior) or re-run everything with -fa auto to get forward-comparable numbers. In either case, make the -fa state an explicit column in your benchmark output going forward.

The -ngl 99 legacy default also caused a quiet footgun on CPU-only hosts: with no -ngl flag set, the runtime attempted to load all 99 layers to GPU, triggering CUDA initialization errors even with no GPU present. With -ngl -1, the runtime skips GPU offload when no backend is detected, removing that noise from logs entirely.

Multi-Token Prediction gains for Qwen 3.6 27B dense — approximately 77 to 96 t/s on an RTX 4090 , a 24% throughput increase via PR #22673 — were measured in a separate context from b9437's defaults change. If you're trying to reproduce those figures, verify the -fa state from the original run; a mismatch gives you a result that is neither the clean MTP baseline nor a combined MTP+FA measurement.

What to explore beyond b9437

Three nearby builds are worth pulling alongside b9437:

b9436 (May 30, 2026, 14:25 UTC) : The OpenCL backend gains BF16-via-FP16 conversion. If you run BF16-format models on AMD or Intel hardware via the OpenCL or Vulkan path, this expands compatibility without requiring native BF16 support in the GPU.
b9439 (May 31, 2026, 06:57 UTC) : Multi-GPU hosts now default to using only one integrated GPU, preventing automatic selection of a low-performance iGPU alongside a discrete card. If you run a hybrid system — laptop with a discrete GPU and Intel UHD, for example — verify your device selection is still correct after updating.
Multi-Token Prediction for Qwen 3.6 (PR #22673): Approximately 24% throughput gain on dense 27B models . Enable with --mtp-n-draft and confirm your GGUF quant is compatible. MoE variants (Qwen 3.6 35B-A3B) show mixed results — expert-union verifier overhead can negate the gains on consumer hardware.

Frequently Asked Questions

What does -fa auto mean in llama-bench after b9437?

-fa auto sets flash attention to LLAMA_FLASH_ATTN_TYPE_AUTO, telling the runtime to enable flash attention when the backend supports it. Before b9437, llama-bench always defaulted to -fa off — unlike llama-server and llama-cli, which already had the tri-state auto|off|on flag. After b9437, all three tools use the same flag semantics.

Are pre-b9437 llama-bench numbers still valid?

Yes, with caveats. If your original run explicitly passed -fa off, or the host hardware does not support flash attention, the numbers remain comparable. If you relied on the default and ran on FA-capable hardware — CUDA, Metal, or Vulkan — those measurements were taken without flash attention even though the GPU supported it. Re-run with matched flags to produce a clean, apples-to-apples comparison.

Why did -ngl default to 99 instead of -1?

99 was a legacy sentinel meaning "offload all layers to GPU." The project later standardized -1 as the runtime-decides value across the toolchain. llama-bench was simply never updated to match until b9437 brought it into alignment with llama-server and llama-cli.

Do I need to recompile from source to get b9437?

Yes, for a local source build: pull the latest commit from ggml-org/llama.cpp and recompile. Tagged binary releases lag the continuous builds. Check the GitHub releases page for a pre-built artifact if you want to skip compilation, but verify the build number includes the b9437 changes before treating it as current.

Does -fa auto now behave the same across llama-bench, llama-cli, and llama-server?

Yes — b9437 closes the gap. llama-cli and llama-server already supported the -fa auto|off|on tri-state. b9437 brings llama-bench into parity, so flag semantics are now consistent across all three tools. A flag value you validated in llama-server means exactly the same thing when passed to llama-bench.

Rebaseline before your next regression run

After pulling b9437 or later, the immediate action is straightforward: re-baseline any llama-bench results used for regression tracking, and make the -fa state an explicit column in your output going forward. The default change is a minor toolchain alignment, but its effect on benchmark validity is concrete — any pre-b9437 run on CUDA, Metal, or Vulkan was silently measuring the slower attention path.

If you're on a multi-GPU system, pull at least b9439 alongside for the iGPU default fix. And if Qwen 3.6 throughput is in your test matrix, keep Multi-Token Prediction's --mtp-n-draft flag in scope — the roughly 24% gain on dense 27B is worth measuring, but MoE variant results vary enough that you'll want numbers from your own hardware and quant configuration.

Last updated: 2026-06-01. Based on llama.cpp continuous builds b9436–b9439 (May 30–31, 2026) .

Opus 4.8 kills budget_tokens — here's what else moved

Creeta — Thu, 18 Jun 2026 08:57:34 +0000

What Opus 4.8 Adds Over 4.7

Anthropic shipped Claude Opus 4.8 on May 28, 2026 , 41 days after Opus 4.7 — a compressed cycle driven partly by competitive pressure from OpenAI Codex and Google Gemini Flash . The headline change is the removal of budget_tokens — passing it now returns a 400. Beyond that, four additions land: a fast-throughput mode, mid-session system messages, a lower prompt-cache floor, and publicly documented refusal stop_details. Code running cleanly on Opus 4.7 requires no other changes to move to 4.8 .

Quick Answer: Swap claude-opus-4-7 for claude-opus-4-8 and remove any budget_tokens usage — it returns a 400. New in this release: speed: "fast" (up to 2.5× throughput at $10/$50 per million tokens), mid-session role: "system" messages, a 1,024-token prompt-cache floor (down from ~2,000), and documented refusal stop_details.

Change	Opus 4.7	Opus 4.8
Throughput mode	Standard only	`speed: "fast"` — up to 2.5× output rate, $10/$50 per M tokens
Mid-session system messages	Not supported	`role: "system"` in messages array, no beta header
Prompt cache minimum	~2,000 tokens	1,024 tokens — no code changes needed
`stop_details` on refusals	Present but undocumented	Publicly documented; categorizes refusal type
Standard pricing	$5/$25 per M tokens	$5/$25 per M tokens — unchanged

Paid Plans and Enrollment Checklist

Claude Code and fast mode are not available on Anthropic's free tier. Before writing any Opus 4.8 code, work through this checklist to avoid hitting enrollment errors in production:

Claude Code access: Requires a paid Anthropic subscription (Pro, Max, Teams, or Enterprise) or API console credits — the free tier does not include it (video: Tech With Tim).
Fast mode enrollment: Research preview, gated separately. Enroll via the Anthropic console before adding speed: "fast" to any request. Sending the parameter without prior enrollment returns an error.
Platform and context window: Claude API, Amazon Bedrock, and Vertex AI each provide a 1 million token context window . Microsoft Foundry caps at 200k tokens — verify your target platform before designing long-context pipelines.
Output limits: Maximum synchronous output is 128k tokens . The Message Batches API raises this to 300k tokens per call with the beta header output-300k-2026-03-24.
Knowledge cutoff: January 2026 .
SDK version: Confirm anthropic>=0.51 is installed before deploying.

How to Call Each Capability in Opus 4.8

Five call patterns cover everything new or changed in Opus 4.8. All examples use the Python SDK (anthropic>=0.51). Steps 4 and 5 are the most migration-sensitive; read the next section before deploying either.

Basic call — update the model ID

Change model to claude-opus-4-8. All other request structure from 4.7 carries over unchanged.

   import anthropic

   client = anthropic.Anthropic()
   response = client.messages.create(
       model="claude-opus-4-8",
       max_tokens=1024,
       messages=[{"role": "user", "content": "Explain the migration in one sentence."}]
   )
   print(response.content[0].text)

Fast mode — add speed="fast"

Pass speed="fast" as a top-level parameter (not inside the model string). Doubles per-token cost to $10 input / $50 output per million tokens . Confirm console enrollment before deploying; unenrolled requests return an error.

   response = client.messages.create(
       model="claude-opus-4-8",
       speed="fast",
       max_tokens=1024,
       messages=[{"role": "user", "content": "Generate a 500-word summary."}]
   )

Mid-session system messages

Insert {"role": "system", "content": "..."} into the messages array immediately after any user turn. No beta header required. Earlier turns stay cached — you pay only for the injected delta .

   messages = [
       {"role": "user", "content": "Analyze this codebase."},
       {"role": "assistant", "content": "Starting with the entry points..."},
       # Inject updated instruction mid-session — no beta header needed
       {"role": "system", "content": "Focus only on security-critical paths from here."},
       {"role": "user", "content": "Continue with the auth module."},
   ]

   response = client.messages.create(
       model="claude-opus-4-8",
       max_tokens=2048,
       messages=messages
   )

Adaptive thinking — omit budget_tokens

Pass thinking={"type": "adaptive"} and set effort separately. Accepted values: "low", "high" (default), "max". The model decides per turn whether extended reasoning is warranted (video: Skill Leap AI). Passing budget_tokens returns a 400.

   response = client.messages.create(
       model="claude-opus-4-8",
       thinking={"type": "adaptive"},
       output_config={"effort": "high"},  # or "low", "max"
       max_tokens=4096,
       messages=[{"role": "user", "content": "Audit this function for security issues."}]
   )

Reading stop_details on refusals

Check response.stop_details.type to branch by category rather than parsing free-text content. The following snippet is illustrative — see the Opus 4.8 changelog for all documented type values.

   response = client.messages.create(
       model="claude-opus-4-8",
       max_tokens=1024,
       messages=[{"role": "user", "content": user_input}]
   )

   if response.stop_reason == "refusal":
       refusal_type = response.stop_details.type
       if refusal_type == "harmful_content":
           return {"error": "content_policy", "action": "revise_prompt"}
       elif refusal_type == "privacy":
           return {"error": "privacy_policy", "action": "remove_pii"}
       else:
           return {"error": "refusal", "type": refusal_type}

The following verified migration script summarizes the full diff from Opus 4.6-style requests to Opus 4.8. It was executed and confirmed (exit 0):

import json

old = {
    "model": "claude-opus-4-6",
    "thinking": {"type": "enabled", "budget_tokens": 32000},
}

new = {
    "model": "claude-opus-4-8",
    "thinking": {"type": "adaptive"},
    "output_config": {"effort": "high"},
    "speed": "fast",
}

print(json.dumps({
    "budget_tokens": "removed: use adaptive thinking + effort instead",
    "before": old,
    "after": new,
    "also_moved": {
        "effort_default": "high",
        "sampling": "omit temperature/top_p/top_k non-defaults",
        "prompt_cache_min_tokens": 1024,
        "max_output_tokens": 128000,
    },
}, indent=2))

Output from the above script:

{
  "budget_tokens": "removed: use adaptive thinking + effort instead",
  "before": {
    "model": "claude-opus-4-6",
    "thinking": {
      "type": "enabled",
      "budget_tokens": 32000
    }
  },
  "after": {
    "model": "claude-opus-4-8",
    "thinking": {
      "type": "adaptive"
    },
    "output_config": {
      "effort": "high"
    },
    "speed": "fast"
  },
  "also_moved": {
    "effort_default": "high",
    "sampling": "omit temperature/top_p/top_k non-defaults",
    "prompt_cache_min_tokens": 1024,
    "max_output_tokens": 128000
  }
}

400 Errors and Other Failure Points

Opus 4.8 inherits the same sampling constraints as Opus 4.7. These four failure points account for the majority of migration bugs, particularly when upgrading from Opus 3 or Sonnet:

thinking: {"type": "enabled", "budget_tokens": N} → 400. Deprecated since Opus 4.7, still unsupported in 4.8. Replace with thinking={"type": "adaptive"} plus output_config={"effort": "..."}. This is the single most common mistake when switching from extended-thinking-era code .
Any non-default temperature, top_p, or top_k → 400. Omit these parameters entirely. Steer output style through prompting techniques instead — this constraint is unchanged from 4.7.
speed: "fast" without enrollment → API error. This is not a 400 but a distinct enrollment-check error. Confirm console access before shipping any fast-mode code path.
Microsoft Foundry: 200k token context, not 1M. If your pipeline was designed around the 1M window on the Claude API, Bedrock, or Vertex AI, it behaves differently on Foundry . Verify your target platform before committing to a retrieval or long-document architecture.

On the behavioral side: per TechCrunch's reporting on early enterprise testing, Opus 4.8 is approximately four times less likely than Opus 4.7 to allow code flaws to pass unremarked, and it proactively flags input/output uncertainties in analytical pipelines — a pattern confirmed by Bridgewater Associates during pre-launch evaluation . If your evals measure output volume rather than accuracy, recalibrate before moving to production.

Going Further With Opus 4.8

Three capabilities in Opus 4.8 are worth queuing up for exploration, though two remain in research preview as of June 2026 with no confirmed GA date:

Dynamic Workflows (Claude Code, research preview): Decomposes a large task into a plan, then fans it across hundreds of parallel subagents in a single session. Designed for codebase-scale work — migrations across hundreds of thousands of lines, from kickoff to merge, using your existing test suite as the quality bar . Token consumption is substantially higher than single-agent flows; budget accordingly before enabling.
Message Batches API — 300k output tokens per call: Add the beta header output-300k-2026-03-24 to raise the per-call output ceiling from 128k to 300k tokens . Practical for large-document generation or batch summarization at scale.
Mid-session prompt injection for cost reduction: In multi-turn agentic loops, inject only the changed instruction slice after each turn while leaving earlier cached turns intact. Savings compound with session length — no need to retransmit the full system prompt on every call.

Frequently Asked Questions

What is the model ID for Claude Opus 4.8?

Use claude-opus-4-8. The model is available on the Claude API, Amazon Bedrock, and Vertex AI with a 1 million token context window , and on Microsoft Foundry with a 200k token limit. Standard pricing is $5 per million input tokens and $25 per million output tokens.

Why does setting `budget_tokens` cause a 400 error on Opus 4.8?

The thinking: {"type": "enabled", "budget_tokens": N} syntax was deprecated starting with Opus 4.7 and is unsupported in 4.8. The correct form is thinking={"type": "adaptive"} with a separate output_config={"effort": "..."} parameter — values are "low", "high", or "max", defaulting to "high". Omit budget_tokens entirely.

What does fast mode cost compared to standard Opus 4.8?

Standard Opus 4.8 costs $5 input / $25 output per million tokens. Fast mode costs $10 input / $50 output per million tokens — exactly double — but delivers up to 2.5× higher output token rate . Fast mode is a research preview and requires separate enrollment in the Anthropic console before any requests will succeed.

Can I set `temperature` or `top_p` with Opus 4.8?

No. Any non-default temperature, top_p, or top_k value returns a 400 error — the same constraint as Opus 4.7. Omit these parameters entirely and control output style through prompting instead.

Do I need to change any code to benefit from the lower prompt cache threshold?

No code changes required. Opus 4.8 drops the minimum cacheable prompt from approximately 2,000 tokens to 1,024 tokens automatically . Prompts between 1,024 and 2,000 tokens that previously missed caching now qualify, cutting repeat-call input costs without any migration work on your end.

Watch / Sources

What to Migrate, What to Monitor

The Opus 4.8 upgrade is one line for most codebases: change the model ID. The substantive migration work is removing budget_tokens if you were on extended thinking, and confirming no sampling parameters are set. The four new capabilities — fast mode, mid-session system prompts, the lower cache floor, and stop_details routing — each add a handful of lines, not a structural rearchitecture.

Fast mode and Dynamic Workflows remain research previews with no confirmed GA date. The Mythos-class models expected to outperform Opus 4.8 on coding benchmarks were described as coming "in the coming weeks" as of the release date, with safety reviews still ongoing . For now, claude-opus-4-8 is the production ceiling on every supported platform.

Last updated: 2026-06-01. Based on Anthropic's Opus 4.8 release notes and model documentation published May 28, 2026.

Composer 2.5 hits near-frontier at 60 lower spend

Creeta — Thu, 18 Jun 2026 08:57:33 +0000

Cursor's Composer 2.5 landed on May 18, 2026, and the interesting part isn't a new model — it's what the team did to an old one. The gains come almost entirely from post-training, not a fresh checkpoint.

What Cursor Trained Into 2.5: RL-Heavy Post-Training on Kimi K2.5

Composer 2.5 runs on the same open-weights base as Composer 2: Moonshot AI's Kimi K2.5, a 1.04T-parameter mixture-of-experts model with 32B active parameters . There is no architecture swap. Cursor's launch graphic states that ~85% of Composer 2.5's total compute went into additional Composer training and RL rather than a new checkpoint — so every capability gain lives in the fine-tuning stack, not the weights it started from.

What that stack added is concrete:

25× more synthetic tasks than Composer 2, including feature-deletion exercises grounded in real codebases .
More complex RL environments, plus targeted textual feedback injected at the exact trajectory point of each error — tool-call failures, style deviations, instruction drift — implemented via an on-policy distillation KL loss .

One infrastructure detail is worth noting for anyone tracking large-model training: Cursor reports sharded Muon with distributed orthogonalization and dual-mesh HSDP, hitting a 0.2s optimizer step on a 1T-parameter model . Read against the independent Artificial Analysis Coding Agent Index, where Composer 2.5 jumped +14 points over Composer 2, the takeaway is that a heavier post-training pass — not a bigger or different base — drove the bulk of the improvement .

What You Need Before Invoking Composer 2.5

Before you can select Composer 2.5, you need a current Cursor build and a clear picture of how it bills. The model requires Cursor 3.4 or newer — version 3.5 was current as of May 20, 2026 — so run Cursor → Check for Updates and relaunch before the model name appears in the picker .

Composer 2.5 ships in two tiers that run identical model weights — tier changes per-token cost and latency only, not output quality . Fast is the launch default and runs on hotter, pricier hardware, so cost-conscious users should switch to Standard.

Tier	Input ($/M)	Output ($/M)	Cache read ($/M)
Standard	$0.50	$2.50	$0.20
Fast (default)	$3.00	$15.00	$0.20

Both tiers draw from one shared Auto + Composer usage pool on individual plans, with a $0.20/M cache-read rate either way . A first-week double-usage promo began May 18, 2026; as of June 2, 2026 treat it as expired and confirm under Usage in your Cursor dashboard before counting on it .

How to Select, Prompt, and Verify Composer 2.5

Selecting Composer 2.5 is a three-click path inside a current Cursor build, but the payoff comes from how you prompt it: write a verifiable success condition into the task and the model will self-correct toward it, because it was RL-trained against test verification [1][4]. The workflow below moves from updating the editor to encoding conventions and keeping a durable rollback path.

Update Cursor. Composer 2.5 needs a recent build — Cursor 3.4 or newer, with 3.5 current as of May 20, 2026. Run Cursor → Check for Updates and relaunch when prompted.
Select the model. Open Agent with Cmd/Ctrl+I (or a chat session), click the model name in the prompt input, and pick composer-2.5. The same dropdown appears in the inline-edit editor under Cmd/Ctrl+K [1][4].
Set billing per workload. Click the tier selector and switch to Standard for background, cloud, or long agent loops; enable Fast only when you need lower latency on short inline edits. Both tiers run identical weights and draw from one usage pool, so the choice changes per-token cost and first-token latency, not output quality [1][4].
Write a success condition into every substantive prompt. Give the agent a real end state — for example, "all existing tests stay green and the endpoint returns 422 on invalid input." The model was trained against test verification and iterates toward that target instead of stopping at first-draft code.
Ask it to plan before touching files. For a safer first pass, prompt: "identify the likely files, propose a minimal test-backed patch — do not write code yet." Reviewing the plan before edits land catches misreads early.
Encode conventions in rules. Put project standards in .cursor/rules (.mdc files) or AGENTS.md. Cursor supports Project, User, and Team Rules; keep each rule actionable, scoped, under 500 lines, and checked into git when it is project-specific.
Keep a rollback path. The agent creates checkpoints — local snapshots you can roll back to — and agent resume recovers an interrupted session. Treat these as convenience layers: Git remains the durable version control, so commit before long autonomous runs [3][4].

One pro tip from the Cursor team is to lean on the agent's clarifying questions rather than over-specifying upfront (video: Cursor) — describe the goal and the verification, and let it surface the files it needs to read.

Before You Rely on It: Billing Reality and the Independent Verdict

Before you trust Composer 2.5 with a long session, check your billing selector: Fast is the launch default, and it runs the same model weights as Standard at a higher per-token rate . Fast costs $3.00/M input and $15.00/M output versus Standard's $0.50/M and $2.50/M — roughly 6× the rate for no quality gain, only faster first tokens . Switch to Standard for everything but latency-sensitive inline edits.

Is the capability claim real? Independent measurement broadly corroborates Cursor without matching it exactly. Artificial Analysis's Coding Agent Index scores Composer 2.5 at 62 — third overall, behind Claude Code with Opus 4.7 (66) and Codex with GPT-5.5 (65) . That is a +14-point jump over Composer 2's 48, driven by SWE-Bench-Pro-Hard-AA rising +35 points, from 12% to 47% . By contrast, CursorBench v3.1 — where Cursor reports 63.2% — is an internal benchmark built from real Cursor sessions, not a neutral public leaderboard .

Two caveats deserve weight before you build a workflow on it. First, supply chain: the base weights are Moonshot AI's Kimi K2.5, a third-party open checkpoint, and the from-scratch model Cursor is training with SpaceXAI on Colossus 2 (targeting ~10× the compute) has not shipped and is not part of 2.5 . Second, there is no independent confirmation that Fast and Standard are quality-equivalent on real production repositories — Cursor claims identical weights, and that is the only evidence on the table . Treat the cost win as proven and the equivalence as a vendor assertion until your own diffs say otherwise.

Where to Take It: Worktrees, Rules Files, and Mixing With Other Models

The strongest setup is not Composer 2.5 alone but Composer 2.5 as the cheap default in a multi-model rotation. A common routing pattern: send refactors and medium agent loops to Composer 2.5 Standard, route architectural reasoning to a frontier model like Opus 4.7, and hand terminal-heavy or multi-shell work to GPT-5.5 . Git worktrees let all three operate on one repository in parallel without stepping on each other, since each task gets its own isolated branch .

Three CLI flags make that workflow practical: --mode=plan inspects and proposes before touching files, --worktree isolates each task on its own branch, and --print runs non-interactively for scripting and CI .

Don't take the cost claim on faith — calibrate with your own data. Run one real refactor on Standard, record token spend, then run the identical task on a frontier alternative and compare quality and spend side by side. The math that motivates the experiment is simple (this snippet ran, exit 0):

frontier = {"name": "Frontier", "score": 100.0, "spend": 60.0}
composer = {"name": "Composer 2.5", "score": 97.0, "spend": 1.0}

score_gap = frontier["score"] - composer["score"]
spend_ratio = frontier["spend"] / composer["spend"]

print(f'{composer["name"]}: {composer["score"]:.0f}% of frontier quality')
print(f"Gap to frontier: {score_gap:.0f} points")
print(f"Spend reduction: {spend_ratio:.0f}x lower spend")

The takeaway: at roughly $0.07 per task standard versus $4.10 for Opus 4.7 , even a modest quality gap pays for itself across a day of agent loops. Make Composer 2.5 Standard your default, escalate deliberately, and let your own diffs decide the rest.

Frequently asked questions

Is Composer 2.5 the same model as Composer 2?

Yes and no. Composer 2.5 runs on the same open-weights base checkpoint as Composer 2 — Moonshot AI's Kimi K2.5, a 1.04T-parameter mixture-of-experts model with 32B active parameters . The difference is post-training, not architecture: Cursor states that roughly 85% of Composer 2.5's total compute came from additional Composer training and reinforcement learning, including about 25× more synthetic tasks than Composer 2 . So the gains are in fine-tuning on the same checkpoint, not a new base model.

What is the real difference between Composer 2.5 Standard and Fast?

They run identical model weights, so per Cursor there is no quality difference — only latency and cost . Fast runs on hotter, more expensive hardware for faster first tokens and is priced at $3.00/M input and $15.00/M output, versus Standard's $0.50/M input and $2.50/M output — roughly 6× more per token . Fast was the launch default, so for long agent sessions switch to Standard unless you specifically need low-latency inline edits.

How accurate are Cursor's benchmark claims for Composer 2.5?

Independent testing broadly corroborates them. Artificial Analysis placed Composer 2.5 at 62 on its Coding Agent Index — third overall, behind Claude Opus 4.7 in Claude Code at 66 and GPT-5.5 in Codex at 65, and a +14-point jump over Composer 2's 48 . Treat Cursor's CursorBench figures more cautiously: it is an internal benchmark built from real Cursor engineering sessions, not a neutral public leaderboard .

Does Composer 2.5 work in the Cursor CLI?

Yes. Composer 2.5 is available in both the Agent UI and the terminal CLI . The agent CLI supports --mode=plan for inspect-before-edit runs, --worktree to isolate changes on a separate branch, and --print for non-interactive scripting, alongside Ask mode, slash commands, @ context selection, and agent resume .

Is the launch double-usage promotion still active?

Almost certainly not. The first-week double-usage promotion began on May 18, 2026 . As of June 2, 2026 it should be treated as expired unless your Cursor dashboard says otherwise — check under Usage to confirm your current rate before planning around it.