DEV Community: matsumotory

How to use Claude Desktop Switcher for separate workspaces on one Mac

matsumotory — Fri, 16 Oct 2026 06:50:16 +0000

Using Claude for work, personal projects, and client work can mean switching more than an account. You may also want separate chat, Projects, Cowork, Artifacts, and Claude Design contexts, with Claude Code following the same environment when you need it.

Claude Desktop Switcher is a macOS menu bar utility for managing those environments. The desktop workflow is entirely graphical. Developers can optionally connect a terminal to an environment with the separate csw command.

In this guide, I’ll walk through installation, the three separation modes, everyday switching, and the details worth checking before sharing or deleting data. It follows the current English user guide and v0.24.7 release.

Product screenshot from the project website.

1. Understand what an environment means

CSW calls your existing desktop and CLI setup Existing Claude. New environments have dedicated data directories. You decide which supported components should remain separate and which should carry over from Existing Claude.

There are three behaviors:

Shared: the environment uses the existing component through a link. Changes to shared data can affect both environments.
Isolated: the environment starts with its own independent component.
Copied: the component is duplicated once, then each copy evolves independently.

This distinction matters. Sharing rules or history is an intentional connection between environments. Choose separation based on the data you actually want to keep apart.

CSW provides application-data separation. It should not be treated as an operating-system sandbox or a guarantee against every form of data exposure. Keep appropriate backups and follow your organization’s rules for accounts, connectors, and files.

2. Install the desktop app

Start with the official Claude Desktop app available on your Mac. Then:

Open the v0.24.7 release page.
Download Claude-Desktop-Switcher_0.24.7_universal.dmg. This build supports both Apple Silicon and Intel Macs.
Open the disk image and drag Claude-Desktop-Switcher.app into Applications.
Launch CSW. Its settings window opens, and a menu bar icon appears.

The first-run cards explain Existing Claude and the environment workflow. If the menu bar is crowded, you can reopen settings from the Dock icon. If the installer remains mounted, the settings window offers an Eject action.

The DMG contains the desktop utility. It does not install the csw CLI. You can skip the terminal section below if you only want the graphical workflow.

3. Create an environment and choose its separation level

Click New environment, give it a name such as Work, and optionally choose an icon and add a note. A note identifying its purpose and intended account helps when your list grows.

Separate everything

This is the recommended default. Supported components start independently, with nothing carried over from Existing Claude.

Choose it for work and personal accounts, separate clients, or projects where you want a clean starting point. It also allows the environment to run alongside another Claude instance.

Separate conversations and memory too

This mode reuses supported global configuration while giving the environment its own project conversation history, auto-memory, and input history.

The carried-over CLI components include global rules (CLAUDE.md), rule files (rules/), skills, plugins, tool permissions and hooks, and worktree configuration. This suits a separate account or purpose where you want familiar rules without carrying over previous conversations.

Separate the account only

This mode also carries over CLI project conversations, auto-memory, and input history. It suits a continuous workspace used with different accounts, for example where billing is separated by purpose.

The label needs one qualification: desktop connector and app settings, device identifiers, and session state still remain separate. It does not make every desktop setting shared.

For any mode, open Configure in detail, item by item to inspect the individual choices. Then click Create this environment.

The guide’s component-level explanation is especially useful here: most carry-over options concern Claude Code configuration. Desktop MCP connectors live in the independently managed desktop configuration, so plan to configure them for each environment.

4. Launch it and sign in

Select the environment and press Launch this environment. On its first launch, sign in through the official Claude app with the account intended for that environment. Subsequent launches normally retain that sign-in.

There are two launch patterns:

Environments with shared settings run one at a time. Quit the running Claude before opening another shared environment.
Fully isolated environments offer Launch alongside, allowing you to open them without first quitting a running Claude.

For example, you can create independent Work and Research environments and keep both open. If Research instead shares configuration with Existing Claude, use the single-instance workflow to avoid competing writes to shared files.

To identify a window, use Check the current environment in the menu bar menu. To find a particular environment’s window, select it in CSW and press Bring to front. Multiple Claude windows can share one Dock icon, so the Dock alone is not a reliable identifier.

To return to your original setup, select Existing Claude and use Launch Existing Claude. v0.24.7 can reopen it while recognized, fully isolated environments remain running. It refuses to proceed if a running process cannot be verified as a known fully isolated environment. Follow the prompt to quit other instances when necessary.

5. Optional: connect Claude Code

This section assumes Claude Code is already installed and the claude command is available. The separate csw tool manages environments; it does not install Claude Code.

The terminal inside a Claude Desktop app launched through CSW inherits that environment. You can run:

claude

A terminal opened separately, such as Terminal or iTerm2, needs an explicit environment selection.

First, download the standalone csw asset from the release page. From the directory containing the downloaded file, the documented installation example is:

chmod +x csw
mv csw /usr/local/bin/
csw --version

Use a directory on your PATH that you can write to; /usr/local/bin/ is the example rather than a requirement. Updating the desktop app does not update this separate binary.

Then, for an environment named Work, run the documented command in your external terminal:

eval $(csw env Work)
claude

This points that shell’s CLAUDE_CONFIG_DIR at the environment. It does not rewrite your shell startup files or switch other terminal tabs.

Desktop and CLI authentication are separate. The first time you run Claude Code in an environment, complete its sign-in as well. For subscription usage, choose your Claude subscription account rather than an API-billed account. The built-in terminal and an external terminal pointed at the same environment use that environment’s CLI sign-in.

Check the current tab with:

echo $CLAUDE_CONFIG_DIR

A path ending in .context-switcher-claude/profiles/Work/cli-data identifies the Work environment. csw status reports CSW’s selected environment, which can differ from the environment of the terminal you are using. Launch alongside does not change that selection.

If you already customize CLAUDE_CONFIG_DIR, note that CSW looks for Existing Claude’s CLI data at the standard ~/.claude location. It does not automatically adopt a custom location or rewrite your existing setup.

6. Check sharing, and handle duplication carefully

In the environment detail screen, use Isolation check → Check now to verify whether links and isolated components match the configuration. The check is read-only. Its CLI equivalent is:

csw doctor

If it reports broken or missing share links, the documented repair command is:

csw doctor --fix

This can create or correct links when their declared source exists. Recent versions can also replace an empty placeholder directory with the expected link; they do not replace a nonempty directory. Read the reported result rather than assuming every issue is automatically repairable.

One Cowork caveat: the guide notes that Claude Code in desktop Cowork sessions does not load a linked ~/.claude/CLAUDE.md or ~/.claude/rules/. If your workflow depends on these shared rules, check that behavior before relying on it.

Duplicate is useful for copying an environment, but use New environment for a different account. Duplication copies Desktop cookies, so the duplicate can open with the original account’s sign-in. Logging out in either can log out the other. CLI Keychain sign-in is normally not copied, although a fallback .credentials.json file is copied if present. Plugins also have special handling and may need reinstalling in an isolated duplicate.

7. Updates, recovery, and deletion

Claude Desktop’s own update relaunch can bypass CSW and open Existing Claude. If that happens, check CSW’s environment marker, quit the relaunched instance, and launch your intended environment through CSW again.

When you quit all Claude instances for an update, CSW remembers the environments that were open together. The Reopen banner restores them on request, skipping those already running. Shared environments still obey the single-instance restriction. Nothing reopens merely because the banner appears.

For backups, expand Location of this environment’s data and use Show in Finder. The detail view includes paths, approximate sizes, and shared-link targets.

Normal deletion moves an environment into macOS Trash. Before emptying Trash, you can restore its folder to:

~/.context-switcher-claude/profiles/

Return to CSW and it should reappear. Delete permanently skips Trash and cannot be undone. In v0.24.7, both deletion paths refuse to remove a running environment, and deletion also stops if the running state cannot be checked. Quit the environment before deleting it.

Privacy and next steps

The Privacy and Transparency document explains the files CSW accesses and provides verification commands. The project documents no network communication or telemetry from CSW itself; opening documentation or release links delegates to your browser. The official Claude applications retain their own network behavior.

CSW is an unofficial, MIT-licensed project, with no warranty. Use accounts in accordance with Anthropic’s current terms and your organization’s policies.

Start with one fully isolated environment, sign in, and check which window and account you are using. Once that workflow is clear, decide whether sharing selected configuration would help.

This article was written with AI assistance and reviewed by me.

A message between AI agents was shown as sent, but it was not in the inbox

matsumotory — Tue, 13 Oct 2026 09:39:53 +0000

Summary

On 2026-08-19, in work on an initiative I am pursuing (research that rebuilds the development environment for AI agents from OSS components only), an instruction was given to an AI agent to send a message. The conversation took place on the screen of a desktop GUI app. The instruction asked the agent to send the message to the other agent with the message-sending tool (send_message) of an MCP server built in this research. This MCP server provides message exchange between sessions, and it writes a JSONL file for each destination agent name. This article calls this file the inbox file. The agent in the conversation was one connected to Qwen Code (version 0.21.13), one of three public OSS agent runtimes. According to the records of this research, Qwen Code called a built-in send tool with a similar name. The screen showed that the send succeeded, but the message had not reached any inbox file. This discrepancy was found by looking at the contents of the inbox files during manual work.

The core of the body is how a check script that judges message exchange between AI agents is built. The check script registers this MCP server with three agent runtimes, namely OpenCode, Qwen Code, and OpenHands CLI (versions 1.18.18, 0.21.13, and 1.16.0, respectively). It then judges whether message exchange succeeded with three checks: a send check, a receive check, and a control check. In none of the checks does the check script decide pass or fail from the AI agent's own report. The result of the send check is decided by the lines in the other agent's inbox file. The result of the receive check is decided by whether a marker string, generated at random for each run, appears in the output, and this string is not included in the prompt. The control check gives the same instruction as the receive check in a state where no message has been sent, and checks that the marker string does not appear in the output. On 2026-08-27, a tool call log was added. This log is a file to which the MCP server appends one line for each tool call. The check script also made it a pass condition that the IDs in the log match the IDs of the lines in the inbox file. The purpose is to distinguish cases in which the AI agent read or wrote the inbox file directly without calling the tool. In measurements with the CLI on 2026-08-19, 2026-08-27, and 2026-09-02, all three checks passed in all nine runs.

The body first describes the event in which the screen showed a successful send and the message had not arrived. Next, it describes how the check script that decides pass or fail by the lines of the inbox file and the marker string is built. It then describes how the judgment was strengthened with the tool call log, and the results of the measurements on the three agent runtimes. After that, it describes the conditions of the conversation in which the mix-up occurred, and an example from the previous day in which a report that child agents had completed disagreed with the records. It also describes the item added to the rule documents (instruction documents that AI agents read) after the observation. Finally, it describes the grounds for judging by records in a separate place rather than by output, together with where external research and official guidance stand on this.

What you can take away

For people who test collaboration between AI agents, I explain the following three things.

The send check decides pass or fail by the lines in the other agent's inbox file, and the receive check decides pass or fail by a marker string that is not in the prompt. The receive check is paired with a control check run in a state where no message has been sent. In the body, I show in a table the instructions and the pass conditions of the three checks
Matching the IDs in the tool call log against the IDs of the lines in the inbox file lets you distinguish cases in which the AI agent read or wrote the inbox file directly without calling the tool. In the body, I also explain why this distinction is not complete
All nine passes were measured with the CLI, and in the conversation in the desktop GUI app where the mix-up occurred, the tools of the MCP server did not appear. In the body, I describe a procedure to follow when registering an MCP server and confirming that it works: before giving an instruction, look at the configuration file and at whether the MCP server process started, and confirm that the tools are present in the conversation

The body is a discussion based on the records of work from 2026-08-18 to 2026-09-02 on the initiative I am pursuing, and on external materials checked on 2026-10-11.

The body names the products of the three agent runtimes. The content does not represent the views of any of their providers.

The send shown as successful, and the message that did not arrive

On 2026-08-19, in the work of this research, an instruction to send a message was given to an AI agent in a conversation in the desktop GUI app. The screen showed that the send succeeded, but the message had not reached any inbox file. An inbox file is a JSONL file that the MCP server built in this research writes for each destination agent name.

This MCP server provides message exchange between sessions. This article calls this MCP server the message-sending MCP server. The message-sending MCP server has the following three tools.

list_agents returns a list of the other agents
send_message sends a message
check_inbox retrieves messages from the inbox file

Each time send_message sends one message, one line with the sender, the body, and an ID is added to the destination's inbox file.

The agent runtimes used in this research are three public OSS ones, namely OpenCode, Qwen Code, and OpenHands CLI (versions 1.18.18, 0.21.13, and 1.16.0, respectively). ACP is a public standard for running agents from GUIs and editors. The name ACP is made of the initials of three words: Agent, Client, and Protocol. The agent in the conversation was an agent connected to Qwen Code through ACP.

The instruction was to send a message with send_message of the message-sending MCP server. Among the built-in tools that Qwen Code registers by default, there is a tool that sends messages. This article calls this tool the built-in send tool. This research has a document that records trial and error and a document that explains the integration with the desktop GUI app. The two documents say that Qwen Code called the built-in send tool, which has a similar name. The conversation records do not contain the name string of the tool that was called.

In addition to the success shown on screen and the message that did not arrive, the document that records trial and error also describes how this discrepancy was found. This discrepancy was found not by automatic judgment but by looking at the contents of the inbox files during manual work.

A check script that decides pass or fail by the lines of the inbox file and a marker string

The check script of this research decides pass or fail for message exchange by the lines of the inbox file and a marker string, not by the AI agent's report. The check script is a Python script that registers the message-sending MCP server with the three agent runtimes and judges whether message exchange succeeded. This script was newly created in the work of this research on 2026-08-19. The check script runs three checks, the send check, the receive check, and the control check, in this order.

The check script generates a random string for each run. This article calls this string the marker string. The result of the send check is decided by the lines in the other agent's inbox file. The results of the receive check and the control check are decided by whether the marker string, which is not included in the prompt, appears in the output.

The integration test from before 2026-08-19 checked only sending. This test was deleted in the PR that added the check script. The format for registering the MCP server was confirmed for each agent runtime with the output that its official registration command writes. The key in the configuration file is mcpServers for Qwen Code and OpenHands CLI, and mcp for OpenCode.

The check script launches a separate process as the other party in the checks. This process runs the same message-sending MCP server program as the AI agent uses, directly over JSON-RPC on standard input and output, without going through a model. This article calls this process the counterpart process.

The following table lists, for the three checks, the instruction to the AI agent and the pass condition.

Check	Instruction to the AI agent	Pass condition
Send check	Find the other party with list_agents, and send the marker string with send_message	The other party's inbox file has a line whose sender is the AI agent's name and that contains the marker string
Receive check	Read the message that the counterpart process sent beforehand from the inbox file with check_inbox, and output its body. This marker string is not included in the prompt	The marker string appears in the AI agent's output
Control check	In a state where not a single message has been sent, give the same instruction as in the receive check	The marker string does not appear in the AI agent's output

The description at the top of the check script states the purpose of the control check. The purpose is to avoid passing the receive check when the marker string appears in the output without the inbox file being read. As such a case, the description gives the case in which the marker string output in the receive check leaked into memory or storage and remained there.

The check script does not pass states that it cannot judge. The states that it cannot judge are the following four.

The agent runtime fails to start
The run times out
The inbox file does not exist
The output is empty

If the destination is not registered, a call to the message-sending tool results in an error. For this reason, in the receive check, the AI agent's name is registered as a destination of the message-sending MCP server only while the counterpart process sends. The process that registers this name is a different process from the counterpart process.

Strengthening the judgment with the tool call log

On 2026-08-27, pass conditions that match against the tool call log were added to the send check and the receive check of the check script. The tool call log is a JSONL file to which the message-sending MCP server appends one line each time a tool is called. Each line has the name of the agent that called the tool and the message ID, and it has no body. The code that writes this log was added to the message-sending MCP server on the same day, in the work of this research.

The following table lists the pass conditions added to the send check and the receive check on 2026-08-27.

Check	Pass condition added on 2026-08-27
Send check	The ID of the line in the inbox file matches the ID of the line that recorded the message-sending tool (send_message) under the AI agent's name
Receive check	The line that recorded the tool that retrieves messages from the inbox file (check_inbox) under the AI agent's name has the ID of the sent message

The purpose of this strengthening is to distinguish message exchange through the tool from cases in which the AI agent reads or writes the inbox file directly without calling the tool. The check script also does not pass the check when there is no tool call log or when the tool call log contains a broken line.

The description of the check script says that this distinction is not complete. The reason is that the AI agent can read the location and the format of the tool call log with the same user's permissions. An AI agent that can read the location and the format can also write both the inbox file and the tool call log itself, in the matching format.

Results of the measurements on the three agent runtimes

In the measurements on 2026-08-19, 2026-08-27, and 2026-09-02, all three agent runtimes passed all three checks. Each agent runtime was run once on each day, so there were nine runs in total.

The following table lists, for each measurement day, the judgment conditions and the number of agent runtimes that passed.

Measurement day	Judgment conditions	Number of agent runtimes that passed
2026-08-19	Lines of the inbox file and the marker string	All three
2026-08-27	Lines of the inbox file and the marker string, plus matching IDs in the tool call log	All three
2026-09-02	Lines of the inbox file and the marker string, plus matching IDs in the tool call log	All three

The three agent runtimes all used the same open-weight model from the same inference service.

All nine passes were measured with the CLI. In these measurements, the tools of the message-sending MCP server were available in each of the three agent runtimes. The mix-up on 2026-08-19 occurred in a conversation in the desktop GUI app. The tools of the message-sending MCP server did not appear in this conversation.

The document that explains the integration with the desktop GUI app contains a statement to the following effect. Even if the screen shows success, the send check does not pass when the message is not in the inbox file. The reason is that whether the send passes is decided by the contents of the inbox file. This statement is derived from how the checks are built, and there is no record of using the check script to reproduce the mix-up that occurred in the conversation.

Conditions of the conversation in which the mix-up occurred

In the conversation where the mix-up occurred, the tools of the message-sending MCP server did not appear in the conversation with Qwen Code connected through ACP. The two documents of this research say that Qwen Code called the built-in send tool. However, there is no statement that confirms whether the model chose the built-in send tool or Qwen Code resolved the tool by name. The scope checked was the document that records trial and error, the bodies of the two PRs from 2026-08-19, and the document that explains the integration.

This MCP server had been registered with the feature that manages MCP settings in bulk, on the settings screen of the desktop GUI app. With this registration, the connection check passed. However, no tools appeared in the Qwen Code conversation, and the MCP server process did not start either.

The two documents of this research describe Qwen Code's built-in send tool as a tool with a similar name. The version of Qwen Code on the day of the observation was 0.21.13. According to a check on 2026-10-11, in this version's tool name definition file and registration code, the built-in tools registered by default included send_message and list_agents. These two have names that are the same strings as the names of the tools of the message-sending MCP server. Qwen Code has a setting for a team feature, and these two built-in tools are always registered regardless of that setting.

The built-in list_agents returns a list of background agents. The built-in send_message has two modes: one that sends to team members and one that sends to background tasks. The words agent and task are the words used in the descriptions of the respective tools.

That the names matched is the result of checking the public source. It is not the result of confirming, in the conversation records, the name of the tool that was called in the conversation.

The Tools page of the Model Context Protocol specification, version 2025-11-25, has no provision for cases where a name overlaps with a tool built into the client. It also has no provision for cases where names overlap between different servers. This page says that each tool is uniquely identified by its name. It also says that tool names should be unique within a single server (SHOULD).

On the same day, message exchange from the desktop GUI app succeeded when the conversation was connected to OpenCode. In this conversation, the registration of the message-sending MCP server had been added to OpenCode's configuration file. The agent in this conversation found a CLI session with list_agents and sent the marker string. A line whose sender was the agent in this conversation arrived in the CLI session's inbox file. In the opposite direction, a Qwen Code CLI session sent a message to this conversation. The agent in this conversation read that message from the inbox file with check_inbox and output it on the screen.

An example from the previous day in which a report that child agents had completed disagreed with the records

On 2026-08-18, with an experimental parallel execution feature of Qwen Code, a report that the child agents had completed disagreed with the JSON record file that this feature writes. This feature launches child agents in parallel according to an orchestration script written by the model, and it writes a JSON record file for each orchestration run.

In the work of this research, this feature was measured in non-interactive runs. In seven trials there were 13 child agent launches, and none of the 13 completed. The child agents consumed 0 tokens, and they never called the model. In a re-measurement in interactive mode on the same day, two child agents completed. The run records of this research state the conclusion that the phenomenon of child agents not completing occurs only in non-interactive runs.

In the last trial, the model wrote the answer directly without launching any child agents, and in its final response it reported that three child agents had finished. In this trial's JSON record file, the field for the number of attempted child launches was 0. The value of this field showed mechanically that the report was wrong.

The program built in this research that runs child agents in parallel always writes two counts in the run records of this research. The two counts are the number of attempted child launches and the number of child agents that returned a response body. The README of this program gives the false completion report of 2026-08-18 as the reason for making the two counts mandatory.

The document of this research that records trial and error classifies, as one type, failures in which a report of completion or success disagrees with a count field or with the contents of the inbox file. In this document, the successful send shown on screen on 2026-08-19 is the second example, following the completion report of 2026-08-18.

Articles from the same research that deal with the parallel-execution MCP tool and child agents include Once I wrote in the MCP tool description when to use the tool, AI agents called it and A failure that looked like a capability gap between coding agents was caused by how the API key was passed.

The item added to the rule documents after the observation

In a PR merged on 2026-08-20, one item with the same content was added to each of two rule documents. A rule document is an instruction document that AI agents read. The two are the default instruction document that AI agents always read and the skill document that they read when they use a feature that spans sessions. An AI agent created this PR.

The added item is an instruction for environments where the MCP server is not configured. The item instructs the agent not to substitute another tool with a similar name, to first check whether the tool exists, and, if it does not, to report that configuration is needed. As the reason, the item also states that a failure occurred in which a tool with a similar name was used and nothing arrived even though the screen showed success.

Grounds for judging by records in a separate place rather than by output

Both the report on 2026-08-18 that child agents had completed and the successful send shown on screen on 2026-08-19 disagreed with data that remains in a place separate from the report. The error in the 2026-08-18 report was confirmed with the field for the number of attempted child launches in the JSON record file that the experimental feature writes. The error in the 2026-08-19 display was confirmed with the inbox file. These facts show that it is better to decide the result of a check on agent collaboration by data that remains in a place separate from the report, rather than by the AI agent's output.

The document of this research that records trial and error states what is judged by what, as follows.

Whether sending a message succeeded is judged by the lines of the inbox file
Whether something was executed, such as the launch of a child agent, is judged by counts written in the records, such as the field for the number of attempted child launches
Whether something completed is judged by the deliverables
The success shown on screen and what the AI agent reports in its text are not used for judgment

The receive check confirms that the output came from the inbox file by checking that a marker string that is not in the prompt appears in the output. The control check confirms that the same output does not appear when no message has been sent. When the two checks are paired, the control check can fail a case in which the marker string leaked into memory or storage, remained there, and appeared in the output without the inbox file being read. This property is derived from how the checks are built, and in the nine runs there was no case in which the control check failed.

The document of this research that records trial and error also describes a procedure to follow when registering an MCP server and confirming that it works. Before giving an instruction, look at whether the MCP server is written in the configuration file and whether the MCP server process started, and confirm that the tools are present in the conversation.

The idea of judging success or failure by the state and records left in the environment, rather than by the AI agent's report, also appears in external research and official guidance.

According to the abstract of a study that measured false success (Advani, 2026), the failure in which an LLM agent claims to have completed a task even though the state of the environment does not show completion is called false success. One of the measured targets is 9,876 trajectories in which AI agents worked on tasks in tau2-bench, spanning eight model families. Whether each trajectory succeeded was decided without depending on the agent's text. In the domains of tau2-bench that the study calls single-control, false success is reported to account for 45% to 48% of failures. The abstract does not define single-control. Judgment by an LLM (LLM judge) is reported not to have detected this failure well enough. AUROC is a metric that expresses detection performance from 0 to 1, and 0.5 is the same as chance. The judging configurations given in the abstract are five models, five prompts, and the full text of the task specification. In every configuration, AUROC on tau2-bench did not exceed 0.65. The abstract says that LLM judges rely on confident closing phrases and the volume of the action sequence, and that they do not look at verified changes in state.

Demystifying evals for AI agents gives an example of a flight booking agent. In this example, the outcome is judged not by whether the agent said it made the booking but by whether the booking exists in the environment's SQL database.

The abstract of a study that detects hallucinations with tool execution receipts (Basu, 2026) proposes a framework for detecting hallucinations. In this framework, tool execution receipts are matched against the claims in a response. The receipts are signed with HMAC so that the LLM cannot forge them.

The limit of the check script was that, because the AI agent can read the tool call log with the same user's permissions, there remains room for it to write the log itself in the matching format. These facts show that this limit is the same kind of problem as the forgery of records that the receipt study prevents with signatures. They also show that the approach of deciding pass or fail by data that remains in a place separate from the report is not a conclusion drawn only from the work records of this research.

Studies I referred to

The studies referred to in the body are the following two, and I checked their arXiv abstract pages on 2026-10-11.

Laksh Advani, "From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents" arXiv:2606.09863
Abhinaba Basu, "Tool Receipts, Not Zero-Knowledge Proofs: Practical Hallucination Detection for AI Agents" arXiv:2603.10060

Materials I referred to

The public materials referred to in the body are the following three, and I checked their original texts on 2026-10-11.

Tools (the 2025-11-25 version of the Model Context Protocol specification) https://modelcontextprotocol.io/specification/2025-11-25/server/tools
Demystifying evals for AI agents (Anthropic Engineering, 2026-01-09) https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
tool-names.ts (the tool name definition file of Qwen Code v0.21.13) https://github.com/QwenLM/qwen-code/blob/v0.21.13/packages/core/src/tools/tool-names.ts

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

Rejecting unsupported MCP versions left an AI client unable to connect

matsumotory — Sat, 10 Oct 2026 20:59:09 +0000

Summary

The MCP server for a service I develop at my own company rejected requests whose version header held an unsupported version, and it did so with HTTP 400 before authentication. MCP is an open standard for connecting AI applications to external tools and data sources, and the name is short for Model Context Protocol. The version header is a header defined by the specification that HTTP clients attach to MCP requests, and it indicates the MCP version with a date string such as 2025-11-25. The server decided whether it supported a version by checking the list of supported versions. The list of supported versions enumerates the versions that the MCP server declares it supports, and it is written in the server's code. Because of this rejection, the major AI client could not connect to this MCP server on 2026-09-24. The major AI client is one of the AI chat products in which users can add an MCP server as a connector.

In its first fix, the AI agent only added the observed 2025-11-25 to the list of supported versions. The major AI client is updated frequently, so in response to this fix I pointed out that rejecting a connection over a small difference in version was too strict. In its second fix, the AI agent changed the list of supported versions to two entries, 2025-11-25 and 2025-06-18. In the same fix, it made the server accept values in date form that are smaller than 2026-07-28, even when they are not in the list. With the 2026-07-28 version, the MCP specification changed to an approach that does not use the initialize request, the request a client sends first to decide the version. The AI agent left in place the handling that returns 400 for values dated 2026-07-28 or later and for values not in date form. The MCP specification has a MUST requirement, that is, an obligation that must always be met: a server that receives a version header with an unsupported version responds with 400. Based on my decision, the AI agent intentionally did not implement this requirement for versions earlier than 2026-07-28 that are not in the list. On the next day, 2026-09-25, the AI agent changed the MCP server so that it could also respond to requests that use the 2026-07-28 version.

In the first section, I explain the version header check that stopped the major AI client from connecting, and I show in a table the results of the check before the fix and after the second fix on 2026-09-24. In the next section, I explain how the server identifies legacy versions and how it handles modern versions. Legacy versions are the versions up to and including 2025-11-25, which establish a connection with the initialize request, and modern versions are the versions from 2026-07-28 onward. That section also gives the reason the MCP server was made to support both kinds of versions the next day. The section that follows compares, in a table, the two MUST requirements of the specification with how the MCP server handles them after the second fix. The section after that explains the first fix, which added one observed version, and the point I raised about it. The last section explains the robustness principle of being liberal in what you accept, and the limits that RFC 9413 placed on this principle. Building on that, Building on that, I classify the MCP server with the second fix as follows. It is neither an application of this principle's aspect of tolerating unexpected inputs nor an extension under clear rules as described in Section 2.2 of RFC 9413, and it is an implementation that intentionally chose handling different from the handling the specification defines.

What you can take away

For those who develop MCP servers, I explain the following three things.

When an MCP server checks the version header against a list of supported versions, a fix that only adds the one observed version to the list leads to the same 400 the next time a client raises its version. In the body, I explain why from the rules of the check before the fix.
The MCP server with the second fix treats a header value as a legacy version if the value is in date form and smaller than 2026-07-28. In the body, I use tables to show this rule and, of the two MUST requirements of the specification, the one this MCP server meets and the one it intentionally does not meet because of my decision.
The second fix, which accepts legacy versions not in the list, is neither an application of the aspect of tolerating unexpected inputs within the robustness principle of being liberal in what you accept, nor an extension under clear rules as described in Section 2.2 of RFC 9413. In the body, I classify the MCP server with this fix as an implementation that intentionally chose, for inputs whose handling the specification defines, handling different from the specification's.

The body is based on the records of fixing the MCP server of my own company's service from 2026-09-24 to 2026-09-25, and on the original texts of the MCP specification and the RFCs, which I checked on 2026-10-04.

The version header check that stopped the major AI client from connecting

I develop a service at my own company. The MCP server of that service rejected, with HTTP 400 and before authentication, any version header whose version was not in the list of supported versions. As a result, the major AI client could not connect on 2026-09-24.

MCP is an open standard for connecting AI applications to external tools and data sources, and the name is short for Model Context Protocol. The major AI client is one of the AI chat products in which users can add an MCP server as a connector. The version header is a header defined by the specification that HTTP clients attach to MCP requests. Its value is a date string such as 2025-11-25, and it indicates the MCP version. The list of supported versions enumerates the versions that the MCP server declares it supports, and it is written in the server's code.

In versions up to and including 2025-11-25, the client first sends an initialize request and decides with the server which version they will use. Versions that establish a connection in this way are called legacy versions. This name is a term from the 2026-07-28 version of the specification.

Before 2026-09-24, the MCP server's list of supported versions had only one entry, 2025-06-18. A note in the server's code gave the reason for not declaring the 2025-03-26 version. That version has a requirement that implementations accept JSON-RPC batches, and the MCP server does not support batches. The MCP server accepted requests that had no version header.

On 2026-09-24, I added the MCP server in the test environment to the major AI client as a connector. The major AI client's screen stopped while still showing that it was connecting. The screen showed an error saying, in effect, that something was not found, along with HTTP 400. In the observation that day, the major AI client put 2025-11-25 in the version header starting from the first initialize request.

On the same day, the AI agent fixed the MCP server twice. In the second fix, it changed the list of supported versions to two entries, 2025-11-25 and 2025-06-18. If a client asks for a version in the list in the initialize request, the MCP server responds with that version. If the version is not in the list, the server responds with 2025-11-25. When the MCP server rejects a request with 400, it logs the rejected version value and the User-Agent of the connecting client as a warning.

Versions from 2026-07-28 onward have no initialize request, and the client declares the version in _meta in the request body of every request. These versions are called modern versions. The following table lists the results of the check for each header value.

Value of the version header	Before the fix	After the second fix on 2026-09-24
2025-06-18	Accepted	Accepted
2025-11-25	Rejected with 400	Accepted
Other date values smaller than 2026-07-28. An example is 2025-03-26	Rejected with 400	Accepted
Values of modern versions. Date values of 2026-07-28 or later	Rejected with 400	Rejected with 400
Values not in date form	Rejected with 400	Rejected with 400

Identifying legacy versions and handling modern versions

The MCP server with the second fix treats a header value as a legacy version if the value is in YYYY-MM-DD form and smaller than 2026-07-28. Because which value is smaller is decided by string comparison, this check depends on the value being in YYYY-MM-DD form. Dates between 2025-11-25 and 2026-07-28 do not correspond to any version of the specification, but this check treats them as legacy versions.

The AI agent left in place the handling that returns 400 when the value of the version header is a modern version. An implementation that supports both legacy and modern versions is called dual-era. This name is also in the 2026-07-28 version of the specification. The reason the AI agent wrote in its description of the fix is that a dual-era client looks at the body of the 400 response and switches to a legacy version.

On 2026-09-25, I approved the work to make the MCP server dual-era. The AI agent wrote the reason for this work as follows. As of 2026-09-24, the major AI client was communicating with a legacy version. However, client updates come faster than server fixes, so it would make the MCP server dual-era before the major AI client moved to a modern version. The AI agent merged the fix for this work on 2026-09-25. The MCP server then became able to respond to requests with modern versions as well.

The specification's obligations and the MCP server after the second fix

The MCP server with the second fix meets the obligation to respond with a different version when a client, in the initialize request, asks for a version not in the list. On the other hand, for legacy versions, it intentionally does not meet the obligation to return 400 for a version header with an unsupported version.

The Lifecycle documents of the 2025-06-18 and 2025-11-25 versions contain a MUST requirement on the version the server responds with. A server that supports the requested version responds with the same version, and a server that does not support it responds with a different version. The specification defines, as a SHOULD requirement, that is, a recommendation, that this different version be the server's latest version.

The Transports documents of the 2025-06-18 and 2025-11-25 versions contain a MUST requirement on the header. A server that receives a version header with an invalid or unsupported version responds with 400 Bad Request. The AI agent wrote this obligation in the server's documentation as an obligation that the MCP server does not meet.

MUST requirement in the specification	Versions that have the requirement	MCP server with the second fix
A server asked in the initialize request for a version it does not support responds with a different version that it supports	2025-06-18, 2025-11-25	Responds with 2025-11-25
A server that receives a version header with an unsupported version responds with 400	2025-06-18, 2025-11-25	Intentionally does not return 400 for legacy versions not in the list

Intentionally not implementing the obligation to return 400 is based on my decision not to reject connections over a difference in version alone. The values that the second fix newly accepts are 2025-03-26 and 2024-11-05, which are versions of the specification, and date values that do not correspond to any version of the specification. The AI agent explained the reasons for this decision as follows.

Rejecting connections over a difference between legacy versions brings no benefit.
If updates to the list of supported versions fall behind, the MCP server ends up rejecting clients' connections. This harm is greater than the benefit of rejecting connections over a difference between legacy versions.
Even if a client uses a feature that the server does not implement, the connection is kept. For example, for JSON-RPC batches, which are in the 2025-03-26 version, the MCP server only returns a standard error.

The first fix, which added one observed version

In its first fix, the AI agent only added the observed 2025-11-25 to the list of supported versions. Before the fix, the MCP server had been rejecting with 400 the value 2025-11-25 that the major AI client sent.

In response to the first fix, I pointed out that rejecting a connection over a small difference in version was too strict. This is because the major AI client is updated frequently.

The check before the fix rejected every value not in the list with 400. These facts show that with a fix that adds one observed version to the list, the same 400 occurs the next time a client raises its version.

The robustness principle and the limits set by RFC 9413

The robustness principle is the implementation principle of being liberal in what you accept and conservative in what you send. RFC 1122 lists this principle and says that it is especially important at the Internet layer.

RFC 9413 says that the robustness principle has three aspects. The three aspects are robustness to software defects, robustness to attacks, and robustness to unexpected inputs. Section 1 of RFC 9413 says that cases where the specification explicitly defines how a faulty message is handled are excluded from the aspect of robustness to unexpected inputs. RFC 9413 also says that the first two aspects are necessary guiding principles. On the other hand, it says that the interpretation of tolerantly accepting unexpected inputs can no longer be called the best approach in every situation.

Section 2.2 of RFC 9413 says that extensibility is sometimes mistaken for an application of the robustness principle. It also says that in a well-designed extensibility mechanism, the handling of new messages and parameters is decided by clear rules, so they become expected inputs.

The MCP specification has a requirement to respond with 400 to a version header with an unsupported version. Therefore, the MCP server with the second fix is an implementation that, for inputs whose handling the specification defines, handles them differently from the specification. The aspect of robustness to unexpected inputs does not include inputs whose handling the specification makes explicit. The extension under clear rules in Section 2.2 is a mechanism in which the protocol defines the rules and every implementation behaves according to those rules. These facts show that the MCP server with the second fix is neither an application of the robustness principle's aspect of tolerating unexpected inputs nor an extension as described in Section 2.2. They also show that it is an implementation that intentionally chose handling different from the handling the specification defines.

Materials I referred to

The public materials I referred to in the body are the following six items, which cover eight documents. I checked the original text of each of them on 2026-10-04.

Lifecycle (the 2025-06-18 version of the MCP specification) https://modelcontextprotocol.io/specification/2025-06-18/basic/lifecycle
Lifecycle (the 2025-11-25 version of the MCP specification) https://modelcontextprotocol.io/specification/2025-11-25/basic/lifecycle
Transports (the 2025-06-18 version of the MCP specification) https://modelcontextprotocol.io/specification/2025-06-18/basic/transports
Transports (the 2025-11-25 version of the MCP specification) https://modelcontextprotocol.io/specification/2025-11-25/basic/transports
Versioning and Compatibility (the 2026-07-28 version of the MCP specification) https://modelcontextprotocol.io/specification/2026-07-28/basic/versioning
Base Protocol (the 2025-03-26 version of the MCP specification) https://modelcontextprotocol.io/specification/2025-03-26/basic
RFC 1122 Requirements for Internet Hosts -- Communication Layers https://www.rfc-editor.org/rfc/rfc1122.txt
RFC 9413 Maintaining Robust Protocols https://www.rfc-editor.org/rfc/rfc9413.txt

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

A failure that looked like a capability gap between coding agents was caused by how the API key was passed

matsumotory — Sat, 10 Oct 2026 20:57:54 +0000

Summary

An AI agent compared three coding agents in a research repository that is scheduled to be made public. The research rebuilds an integrated agent development environment with OSS. The model and the tasks were the same for all three, and there were eight tasks. The source material consists of the records in this repository. In this comparison, a parent agent calls a parallel-execution MCP server. The parallel-execution MCP server is a program built as part of the research, and when it is called, it launches child agents in parallel. Only in the three runs with OpenHands CLI as the parent did none of the 32 child agents launched in total complete. Both the status field in the run record files and the result field of the measurement stayed normal, and for about a day this outcome was treated as a result of the comparison.

The cause was not a difference in capability between the coding agents but the way the measurement script was built. The measurement script left the delivery of the API key that the child agents use to two things: inheritance of the parent process's environment variables and loading of a .env file. Only when the parent was OpenHands CLI and the run took place in an isolated working directory did the API key fail to arrive by either route.

The AI agent fixed the measurement script so that it writes the API key in the env field of the MCP server configuration and passes it that way. After the fix, the AI agent measured the condition with OpenHands CLI as the parent again, once each in two cases: with the set of instruction files in place and without it. The set of instruction files is a group of files, such as AGENTS.md, placed in the working directory. Under both conditions, the API key no longer caused every child agent to fail. With the set of instruction files in place, all eight child agents completed. Without it, 15 of the 16 child agents launched did not complete, for reasons other than the API key.

The first section of the body gives the overall picture of the cause and the fix. The sections after it are arranged in the following order.

Three sections explain how the comparison results were skewed, in this order: the measurement conditions and results, why the failure did not show up in the results, and how the cause was isolated
Whether the parent process's environment variables reach the MCP server is checked for each implementation in the public source code
Three sections explain the fix, in this order: the fix to how the API key is passed and the re-measurement, the mechanism that does not hide failures and the unit tests, and the protection of the API key
Before the section on the materials referred to, two rules for designing comparison measurements are presented

The body names the products of the three coding agents compared. The content does not represent the views of any of their providers.

What you can take away

If you launch other agents from a coding agent through an MCP server, you will be able to do the following three things.

Check in the source code which of the parent process's environment variables your coding agent passes to a stdio MCP server. The range it passes differs by implementation
Fix your setup so that the API key the child agents use is written in the env field of the MCP server configuration and passed that way
Add to your own measurements a step that does not count failures caused by defects on the measurement side as results of a coding agent's capability

This article is a discussion based on the run records in the research repository scheduled to be made public and on the public source code checked at the time of writing.

Two implicit routes that the API key delivery depended on

This section first explains why every child agent failed in the comparison measurements and what the setup looks like after the fix.

The comparison measurements are a series of runs that measured three coding agents in turn, with the same model and the same tasks, while changing the conditions. A coding agent is a CLI program that is started from a terminal and that edits files and runs commands while calling a model. In the comparison measurements, the parent agent calls the parallel-execution MCP server. The parent agent is the coding agent process that receives the tasks and is started first.

The parallel-execution MCP server is an MCP server built as part of the research, and when it is called, it launches child agents in parallel. An MCP server is a program that provides tools for a coding agent to call through a common protocol called MCP. The MCP servers in this article are of the stdio type: the agent starts them as child processes and communicates with them over standard input and output. A child agent is a coding agent process launched by the parallel-execution MCP server.

The measurement script is a Python script that passes the tasks to the parent agent and starts it, then judges the results from the actual files and records them. Before the fix, the child agents' API key arrived by one of the following two implicit routes. The API key is the credential for calling the inference service. The inference service is a service that makes models available to call through an API.

Environment variable inheritance: the environment variables held by a parent process are passed as they are to the child processes it starts
Loading the .env file: the route by which the parallel-execution MCP server reads a .env file. A .env file holds configuration values such as API keys as lines of the form "name=value", and it is kept out of git

Only when the parent was OpenHands CLI and the run also took place in an isolated working directory did the API key fail to reach the child agents by either of the two routes. An isolated working directory is a separate working directory of the repository, created with git worktree. This summary combines three conditions found in the research records, and the research records contain no statement in this form. The three conditions are explained in the section "Isolating the cause by rerunning with the run records kept".

After the fix, the measurement script writes the API key in the env field of the MCP server configuration and passes it explicitly. The env field of the MCP server configuration is a field in the coding agent's configuration file. This field lists, for each MCP server, the environment variables to pass to the process that is started. The field is named environment in OpenCode and env in the other two.

When the API key is needed but missing, the parallel-execution MCP server stops before launching child agents and marks the run record file as failed. A run record file is a JSON file that the parallel-execution MCP server writes for each call. It has fields for the number of child agents launched, the number of child agents completed, and the status.

The following diagram shows the routes before and after the fix.

Before the fix
  The shell that started the measurement script (with the API key loaded)
    └ Parent agent
        └ Parallel-execution MCP server
            │  Route 1: environment variable inheritance
            │           Did not arrive when the parent was OpenHands CLI
            │  Route 2: read the .env file at the top level of the repository
            │           The isolated working directory has no .env file
            └ Child agent (the API key value is an empty string)

After the fix
  Measurement script
    └ Writes the API key in the env field of the MCP server configuration
        └ Parallel-execution MCP server
            ├ API key present: launches child agents
            └ API key needed but missing: stops before launching and marks the run record file as failed

Conditions of the comparison measurements and the number of completed child agents per parent agent

This section shows the conditions of the comparison measurements and the results for each parent agent in a table.

The conditions of the comparison measurements were as follows.

Versions: OpenHands CLI version 1.16.0, Qwen Code version 0.21.13, and OpenCode version 1.18.18 were used
Model: one model was fixed for all runs and called through the OpenAI-compatible endpoint of the inference service
Tasks: eight independent programming tasks. Each task has a task statement and a test script
Instructions to the parent agent: two, namely to finish all eight without leaving any out and not to rewrite the test files. The instructions did not mention the name of the tool
How the parallel-execution MCP server was provided: it was registered in each coding agent's configuration file. Where it was registered differs among the three. For Qwen Code, it was registered in a configuration file placed in the working directory. For OpenCode, it was also registered in a configuration file placed in the working directory. For OpenHands CLI, it was registered in mcp.json placed in a dedicated storage directory
Child agents: for every parent agent, they were started in Qwen Code's non-interactive mode

The following table summarizes, for each parent agent, the runs in which the parallel-execution MCP server was called.

Parent agent	Runs in which a call occurred	Run record files	Child agents launched	Child agents completed
Qwen Code as parent	3 runs	1, 1, and 2 per run	8 per run	8, 8, and 7 per run
OpenCode as parent	1 run	3	16 in total	8 in total
OpenHands CLI as parent	3 runs	4 in total	32 in total	0 in total

In the three runs with OpenHands CLI as the parent, 0 of the 32 child agents launched completed. In the run with OpenCode as the parent, the parent agent fixed an error in the orchestration script by itself and ran it again. The orchestration script is a script, run by the parallel-execution MCP server, that describes the steps for launching child agents.

Even in the three runs with OpenHands CLI as the parent, all eight tasks passed, and there were 0 rewrites of the test files. The records can be read as showing that the parent agent solved by itself the work that the child agents did not complete. However, the research records contain no breakdown of the parent agent's work.

A judgment design in which the failure of every child agent did not show up in the results

This section explains how a result in which every child agent failed continued to be treated as a result of the comparison.

The measurement script judged whether the parallel-execution MCP server had been called by whether a run record file existed. Only the parallel-execution MCP server process writes run record files, so none is created if there is no call.

The measurement script recorded the numbers of child agents launched and completed by summing the fields of the run record files. For pass or fail on the tasks, it restored the test files, ran the tests for each task, and recorded the number of tasks that passed.

Even though every child agent failed, the failure did not show up in the results, for the following three reasons.

If a run record file existed, the measurement script treated the run as having a call and set the result field to success. Even if not a single child agent completed, the result field became success
The run record files written by the parallel-execution MCP server also all had ok, which indicates normal, in the status field
All eight tasks had passed, so the failure was not visible from the task results either

The result in which OpenHands CLI was the parent and 0 child agents completed was treated as a result of the comparison measurements for about a day.

Isolating the cause by rerunning with the run records kept

This section explains how the cause was isolated, in the order of the work. According to the research records, the AI agent did the implementation, ran the measurements, and handled the PRs. This AI agent works separately from the reviewing AI agent that examines the diffs of PRs.

The AI agent reran the condition with OpenHands CLI as the parent just once. This time, it kept the directory that holds the run record files instead of deleting it after the run. In the records it retrieved, all eight child agents had exited about 1.2 seconds after starting. The reason for exiting was an error saying that no authentication method had been selected.

The text of this error is in the file in Qwen Code version 0.21.13 that checks authentication in non-interactive mode. When Qwen Code starts in non-interactive mode, it determines the authentication method from the command arguments, the environment variables, and the configuration files. If none of them determines it, Qwen Code stops with this error.

The cause was the combination of the following three conditions.

When the parent was OpenHands CLI, the parent process's environment variables did not reach the parallel-execution MCP server process. When the parent was Qwen Code or OpenCode, they did reach it, so the API key loaded into the shell made it all the way to the child agents
When the API key was not in the environment variables, the parallel-execution MCP server was built to read the .env file in the top-level directory of its own repository. The comparison measurements ran in an isolated working directory, and that directory had no .env file
The measurement script did not write the API key in the env field of the MCP server configuration

When launching child agents, the parallel-execution MCP server put an empty string into the environment variable if it could not get the API key value.

Before that, there had also been a separate measurement with OpenHands CLI as the parent. In that measurement, the parent agent was explicitly made to call the parallel-execution MCP server, and one child agent had completed. The research records say that the explanation that fits best is that this run took place in the main working directory and the .env file was read. There is no record of which working directory it ran in, so this explanation has not been confirmed.

Differences between implementations in whether the parent process's environment variables reach the MCP server

This section shows the results of checking, for each implementation in the public source code, whether the parent process's environment variables reach a stdio MCP server. The checks were made on 2026-09-29.

The following table summarizes the environment variables that each implementation passes to the MCP server process when it starts a stdio MCP server.

Implementation	Environment variables passed to the MCP server process	Handling of the env field in the configuration	Source
The official MCP Python SDK	By default, only a list of environment variables that are safe to inherit. On POSIX, these are six: HOME, LOGNAME, PATH, SHELL, TERM, and USER	Passed layered on top of the six	Source code of the stdio client
The official MCP TypeScript SDK	On POSIX, the same six as the Python SDK	Passed layered on top of the list	Source code of the stdio client
Qwen Code version 0.21.13	All of the parent process's environment variables, excluding only three environment variables used to authenticate this product's daemon and its child processes	Passed layered on top of all of them	Source code of the MCP client
OpenCode version 1.18.18	The whole of the parent process's environment variables	The environment field is passed layered on top of the whole	Source code of the MCP handling
OpenHands CLI version 1.16.0	Reading the dependency path, the six defaults of the official MCP Python SDK	The env field of the configuration is passed to the startup arguments of the official MCP Python SDK	Dependency definition, MCP handling in openhands-sdk, stdio handling in fastmcp

Notes on each row are as follows.

The same default of the official MCP Python SDK is also in the source code of the stdio client in the 1.x series. The latest release is v2.2.0, published on 2026-09-07
Qwen Code has a file with a function that removes values that are not to be passed (sanitize-child-env.ts). The comments in this file give the reason for limiting what is excluded to internal values. The reason is that commands users run in the shell depend on inheriting third-party credentials
Qwen Code 0.24.6 was published on 2026-09-26. On the main branch after that as well, the Qwen Code CLI is still built to inherit the whole of the parent process's environment variables
On OpenCode's dev branch as well, the design is the same. The latest release is v1.18.33, published on 2026-09-28
OpenHands CLI version 1.16.0 pins openhands-sdk to 1.21.0 in its dependency definition. The MCP handling in openhands-sdk passes the configuration to fastmcp's MCP configuration type. The current source code of fastmcp passes the env field of a stdio MCP server's configuration as it is to the startup arguments. The startup arguments are passed to the official MCP Python SDK. The latest release of OpenHands CLI is 1.16.0, published on 2026-05-08, the same version used in the measurements

The OpenHands CLI row is an explanation obtained by reading the source code of the MCP part. This explanation fits best with the observation that the environment variables did not arrive when the parent was OpenHands CLI. However, the exact versions of fastmcp and the official MCP Python SDK installed at the time of the measurements have not been checked. So this explanation is not the result of checking the whole route in the measurement environment. The observed fact is limited to the way these measurements were started. When the parent was OpenHands CLI version 1.16.0, the parent process's environment variables did not reach the MCP server.

The official documentation of the three coding agents describes the field for passing environment variables. The pages checked are the following three.

On the other hand, none of the three pages says whether a started MCP server receives the whole of the parent's environment. The inheritance behavior is determined only by the implementation in the source code, and it can change between versions.

Both designs have reasons behind them. The default in the official SDKs narrows the inherited environment variables to avoid leaking secret values. Qwen Code limits what it excludes to values for the product's internal use. The comments in the source code give the reason that commands users run in the shell depend on inheriting third-party credentials. Qwen Code version 0.21.13 passes the parent's environment variables only through that same function when it starts an MCP server as well. The measurement script after the fix does not depend on either design. This is because the measurement script writes the API key in the env field of the MCP server configuration and passes it explicitly.

The fix that passes the API key explicitly in the env field of the MCP server configuration, and the re-measurement

This section shows the fix to how the API key is passed and the results of the re-measurement after the fix.

The AI agent fixed the measurement script. After the fix, the measurement script writes the API key that the child agents use in the env field of the MCP server configuration and passes it explicitly. The AI agent also fixed, in the same way, a separate measurement script that explicitly makes the parent agent call the parallel-execution MCP server. That values written in the env field reach the MCP server in all three coding agents had already been confirmed earlier, by passing a different value.

The AI agent set up a unit test to confirm that the API key is written in OpenHands CLI's mcp.json. After that, it extended the unit test to all three coding agents.

The following examples show the form for writing the API key in the env field of the MCP server configuration, written from scratch to match the form in the official documentation. The server name, the environment variable name, and the value are placeholders. The first is the Qwen Code form, and OpenHands CLI's mcp.json uses the same form. The second is the OpenCode form.

{
  "mcpServers": {
    "parallel-runner": {
      "command": "python",
      "args": ["-m", "example_parallel_runner"],
      "env": {
        "EXAMPLE_API_KEY": "replace-with-your-api-key"
      }
    }
  }
}

{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "parallel-runner": {
      "type": "local",
      "command": ["python", "-m", "example_parallel_runner"],
      "environment": {
        "EXAMPLE_API_KEY": "replace-with-your-api-key"
      }
    }
  }
}

After the fix, the AI agent measured two conditions with OpenHands CLI as the parent again, once each. The two conditions differ in whether the set of instruction files for agents is in place. The set of instruction files for agents is the group of AGENTS.md, skill files, and memory files placed in the working directory. The following table shows the results of the re-measurement under the two conditions.

Condition	Child agents launched	Child agents completed	Tasks passed	Time taken
Set of instruction files in place	8	8	8 of 8	556.1 seconds
Set of instruction files not in place	16 in total	1 in total	8 of 8	761.1 seconds

The time limit was 900 seconds. Under the condition with the set of instruction files in place, the parent agent called the parallel-execution MCP server on its own. All eight child agents completed. This was the first run, under a condition with OpenHands CLI as the parent, in which the parent agent called the parallel-execution MCP server on its own and every child agent launched completed.

Under the condition without the set of instruction files, a call also occurred, and there were three run record files. Of the 16 child agents launched under this condition, 15 did not complete, for reasons other than the API key. The research records say that these 15 include the pattern of fixing an error in the orchestration script and running it again. The research records also say that observation of the causes of these 15 will continue.

From these results, the AI agent concluded as follows. The cause of 0 completed child agents was not a difference in capability between the coding agents but the way the measurement script was built. In response to feedback from the reviewing AI agent, the AI agent rewrote the scope of the conclusion. The reviewing AI agent is an AI agent that is started after a PR is created, separately from the agent that did the implementation. It examines the diff read-only. In the rewritten conclusion, the AI agent stated definitively that the problem of every child agent failing because of the API key had been resolved. It separated out the remaining 15 incomplete child agents as having reasons other than the API key.

The two re-measurement runs were recorded separately from the comparison totals, as additional measurements for isolating the cause. The re-measurement was one run for each condition. So these records cannot tell how often every child agent launched completes when the parent is OpenHands CLI.

A mechanism that does not hide failures, and unit tests that prevent the same omission

This section explains the mechanism that does not hide failures and the unit tests that prevent the same omission, both added in PRs after the API key fix.

The AI agent added handling for when the API key is needed but missing. In this case, the parallel-execution MCP server stops before launching child agents and marks the run record file as failed. It does not treat the stop as the failure of one child agent, and marks the whole call as failed.

The PR that added this handling went through three reviews.

In the first review, the reviewing AI agent demonstrated a problem by actually running the code. The problem was that the exception from the stopping logic was treated as the failure of one child agent, and the status in the run record file was left as normal
In the second review, the reviewing AI agent showed that with the parallel and sequential ways of writing the orchestration, the same exception was still treated as normal
In response to the third review, the AI agent changed the stopping logic so that, once it runs, it leaves a failure flag in the run record file. This flag remains even if the orchestration script catches the exception and swallows it

In the same PR, the AI agent removed the logic that, when the API key was empty, overwrote the caller's environment variable with an empty string. After the fix, when there is no API key, the parallel-execution MCP server passes the authentication in the caller's environment to the child agents as it is.

In the same PR, the AI agent also added logic that automatically puts a flag on the rows the measurement script records, indicating that there was a defect on the measurement side. Rows with this flag are not counted in the totals as observations of an agent's capability. The flag is added when a task run meets all three of the following conditions.

The exit code is not 0
The tests were run 0 times
The total time from start to scoring is less than 15 seconds

The conditions were made narrow because, if failures caused by a lack of capability were excluded from the totals as defects on the measurement side, the totals would look better than the actual capability. The logic that adds this flag is separate from the logic that judges whether the parallel-execution MCP server was called.

The same kind of omission also happened with another environment variable. An isolated directory is passed through an environment variable to Qwen Code, the child agent. This directory is used in place of the user's home directory. The measurement script set this value at the point of starting the parent only when the parent was Qwen Code. So this value reached the child agents through environment variable inheritance only when the parent was Qwen Code. When the parent was one of the other two, the value did not exist at the point of starting the parent, so it did not reach the child agents.

The AI agent changed this value as well so that it is written in the env field of the MCP server configuration and passed explicitly for all three coding agents. It confirmed this setup with a unit test. The reviewing AI agent pointed out two places with the same omission. The two places were a separate measurement script and an integration test. The AI agent fixed both places in the same way.

In a later PR, the AI agent consolidated the logic that builds the env field of the MCP server configuration into a single function. This is because, if callers set values separately, the same omission happens when the number of places to register grows. The unit test confirms that the set of environment variable names in the env field exactly matches a defined set. If a new credential is written into a configuration file without anyone noticing, this unit test fails.

Protection added because the API key is now written in the env field

This section explains the new risk created by the fix that passes the API key explicitly, and the protection added in the same PR.

After the fix, the API key written in the env field is written as a raw value into a configuration file inside the working directory. The reviewing AI agent returned seven findings on this PR. Each finding is given one of four severity levels: Critical, High, Medium, or Low. All seven had Medium severity. In response to the findings among the seven that concerned protecting the API key, the AI agent fixed the following two things.

It closed off the routes by which the API key value could get into the measurement script's records and into commits. It added logic that mechanically masks values that look like API keys before the records are written, and confirmed this with a unit test. It also fixed the check that runs before a commit and looks for API key values mixed in, staged a file containing an API key value, and confirmed that the commit was rejected
It changed the setup so that the API key is written only in the configuration of the parallel-execution MCP server, which launches the child agents. The key is not written in the configuration of other MCP servers that do not use it. This setup was also confirmed with a unit test

Two rules for designing comparison measurements

This section presents two rules for designing comparison measurements that can be generalized from these records. The research records contain no statement in this form.

Before starting comparison measurements, identify the implicit behaviors that differ between coding agents. Whether the parent process's environment variables reach the MCP server is one of those behaviors
Do not leave credentials to environment variable inheritance, and pass them explicitly from the measurement script

The basis for rule 1 is the observation that only when the parent was OpenHands CLI did the environment variables fail to arrive, and the source code in which the inherited range differs by implementation. The basis for rule 2 is the facts written in the section on the fix that passes the key explicitly in the env field and the re-measurement, and in the section on the mechanism that does not hide failures.

As a supplementary note, the same kind of failure also happened on an external evaluation execution platform. The AI agent ran Qwen Code over ACP on the external evaluation execution platform. ACP is a public protocol for connecting editors and coding agents. This run got as far as starting Qwen Code and connecting over ACP, and then failed with an error at the authentication stage. The task score was 0. In the logic that started the execution platform at that time, the AI agent could not find a configuration item for passing the model's authentication information to the container. The research records treat this as the cause. The measurement script in the same repository had gotten through the same authentication by passing the environment variables and a copy of the configuration file. The AI agent recorded this run as a defect on the measurement side, not as an observation of capability.

The inheritance behavior of the parent process's environment variables can be checked in the source code of the version in use, not in the official documentation. The part to read is the logic that starts a stdio MCP server. It is a good idea to check whether that logic passes the whole of the parent's environment or narrows it with a list, and only then use that version of the coding agent in comparison measurements.

Materials I referred to

The public materials referred to in the body are as follows. All of them were checked on 2026-09-29.

The source code of the stdio client in the official MCP Python SDK https://github.com/modelcontextprotocol/python-sdk/blob/main/src/mcp/client/stdio.py
The source code of the stdio client in the 1.x series of the official MCP Python SDK https://github.com/modelcontextprotocol/python-sdk/blob/v1.x/src/mcp/client/stdio/__init__.py
The release page of v2.2.0 of the official MCP Python SDK https://github.com/modelcontextprotocol/python-sdk/releases/tag/v2.2.0
The source code of the stdio client in the official MCP TypeScript SDK https://github.com/modelcontextprotocol/typescript-sdk/blob/main/packages/client/src/client/stdio.ts
The source code of the MCP client in Qwen Code version 0.21.13 https://github.com/QwenLM/qwen-code/blob/v0.21.13/packages/core/src/tools/mcp-client.ts
The source code of the function that removes values not to be passed, in Qwen Code version 0.21.13 https://github.com/QwenLM/qwen-code/blob/v0.21.13/packages/core/src/utils/sanitize-child-env.ts
The source code that checks authentication in non-interactive mode, in Qwen Code version 0.21.13 https://github.com/QwenLM/qwen-code/blob/v0.21.13/packages/cli/src/validateNonInterActiveAuth.ts
The release page of Qwen Code 0.24.6 https://github.com/QwenLM/qwen-code/releases/tag/v0.24.6
The source code of the MCP handling in OpenCode version 1.18.18 https://github.com/anomalyco/opencode/blob/v1.18.18/packages/opencode/src/mcp/index.ts
The release page of OpenCode v1.18.33 https://github.com/anomalyco/opencode/releases/tag/v1.18.33
The dependency definition of OpenHands CLI version 1.16.0 https://github.com/OpenHands/OpenHands-CLI/blob/1.16.0/pyproject.toml
The release page of OpenHands CLI 1.16.0 https://github.com/OpenHands/OpenHands-CLI/releases/tag/1.16.0
The source code of the MCP handling in openhands-sdk version 1.21.0 https://github.com/OpenHands/software-agent-sdk/blob/v1.21.0/openhands-sdk/openhands/sdk/mcp/utils.py
The source code of the stdio handling in fastmcp https://github.com/PrefectHQ/fastmcp/blob/main/fastmcp_slim/fastmcp/client/transports/stdio.py
The official documentation on MCP servers for OpenHands CLI https://docs.openhands.dev/openhands/usage/cli/mcp-servers
The official documentation on MCP servers for Qwen Code https://qwenlm.github.io/qwen-code-docs/en/developers/tools/mcp-server/
The official documentation on MCP servers for OpenCode https://opencode.ai/docs/mcp-servers/

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

In Next.js 16, the return value of require.resolve becomes an integer and the code crashes

matsumotory — Tue, 29 Sep 2026 12:28:47 +0000

Summary

I added one tool, which AI agents call from outside, to a service I develop at my own company. When this tool is called, one page selected from a PDF document stored in the service is converted into a JPEG image and included in the response as an image. Before deploying, the AI in charge of development had passed the type check and every unit test. It had also called the tool's function directly from a script and visually checked the generated image. Even so, starting right after the deployment to the verification environment, the response contained no image, no matter how many times the tool was called.

The cause lay in the production build of Next.js 16. In Next.js 16, the default bundler is Turbopack. The require.resolve function returns the location of a file in a package. In the Turbopack build, this call had been resolved at build time and replaced in the build output with an integer module ID. That integer was passed to path.dirname, and a TypeError occurred. Because type checking, unit tests, and direct calls do not pass through this build step, none of the three can detect this replacement.

The AI resolved this defect by fixing one place. The fix changes the base for resolving the package location to the process's working directory. This fix works with Turbopack but not with webpack. In addition, the minimal reproduction built for this article showed that the diagnosis at the time was off in one place. The records at the time explained that import.meta.url had been replaced with a number. In the reproduction, however, import.meta.url remained a string. The value that became an integer was the return value of resolve. After the failure, the development side recorded, as a lesson, a check that starts the same build as production and calls it from outside. Ten days later, the development side also added the same procedure to the service's operations document.

In the body, I first show how the replacement works using values from the reproduction, and then compare the results across four execution methods. After that, I compare two ways to fix the code along with the conditions under which each works, and summarize the procedure for checking before deployment. Finally, I explain the checking step the AI had not run, the failure right after deployment, and the discrepancy in the diagnosis at the time. I confirmed the behavior on 2026-09-25 with Next.js 16.3.6 and 16.2.2 on Node 25.7.0.

What you can take away

For those whose Next.js server code reads files inside a package at runtime, I explain the following three things.

You will understand why type checking, unit tests, and direct calls cannot detect a value being replaced by the bundler, and you will be able to check whether the same replacement happens in your own environment
You will be able to choose a fix for code that resolves a package location at runtime, either using the working directory as the base or adding a turbopackIgnore annotation, knowing the conditions under which each one works
You will be able to incorporate into your pre-deployment steps a check that starts the same build as production and calls it from outside

How Turbopack replaces a require.resolve call with an integer

In this section, I explain how the values in code that resolves a package location change in the production build of Next.js 16. According to the Turbopack API reference in the official Next.js documentation, next build in Next.js 16 uses Turbopack by default. It uses webpack only when --webpack is passed. The same page also states that type checking is not part of Turbopack's processing.

For the reproduction, I prepared the following single function and called it from a Next.js route handler. Inside this function, createRequire creates a require function from each of two bases. Then the resolve of each one resolves the location of next/package.json. The viaImportMeta form uses import.meta.url, the URL of the module itself, as the base. The viaCwd form uses the path of a hypothetical file under the process's working directory as the base.

import path from 'node:path';
import { createRequire } from 'node:module';

export function describeBase() {
  const out: Record<string, unknown> = { importMetaUrl: import.meta.url, cwd: process.cwd() };
  try {
    const req = createRequire(import.meta.url);
    out.viaImportMeta = path.dirname(req.resolve('next/package.json'));
  } catch (e) { out.viaImportMetaError = String(e); }
  try {
    const req = createRequire(path.join(process.cwd(), 'index.js'));
    out.viaCwd = path.dirname(req.resolve('next/package.json'));
  } catch (e) { out.viaCwdError = String(e); }
  return out;
}

I built the project with the default bundler of Next.js 16.3.6, started the build output, and called this function. The viaImportMeta form failed with a Node TypeError. The error code was ERR_INVALID_ARG_TYPE, and the message said that the argument named path had received a number instead of a string. The parentheses at the end of the message contained 39897. The viaCwd form returned a path under a node_modules directory that actually exists. The result was the same with Next.js 16.2.2.

Reading the code in the build output showed where the replacement had happened. The call resolve('next/package.json') was gone from the build output. In its place, the integer 39897 was written directly, and the expression had become path.dirname(39897). The number 39897 is the module ID of next/package.json, which was pulled into the build output, and it is a number assigned in the Turbopack build.

On the other hand, import.meta.url had not become a number. In the Turbopack build, import.meta.url had been replaced with an expression that calls a function at runtime. Inside that function, a file URL is built from the root path of the deployment environment, so the value stayed a string. The resolve in viaCwd also remained in the build output as a runtime call.

The function in which the exception occurred is path.dirname. In Node 25.7.0, passing a number to path.dirname results in an ERR_INVALID_ARG_TYPE exception. Passing a number to createRequire results in a different exception, ERR_INVALID_ARG_VALUE. Its message says that the filename argument is invalid.

The number in the parentheses is a module ID. According to the table of settings in the Turbopack API reference of the official Next.js documentation, the default value of turbopackModuleIds in production builds is deterministic, that is, module IDs are assigned deterministically. In the reproduction with 16.2.2, I changed the target of resolve to a different public package. That package is listed in the setting that keeps packages out of the app bundle. Even so, the replacement did not stop, and the number in the parentheses matched the number in the service's logs. I have not checked whether the value is the same when the project is structured differently. Both 39897 and the 5607 from the webpack build are values that may change if you reproduce this with another version.

The type of resolve is declared as returning a string. The replacement with a number happens only in the build output after the build has finished. That is why type checking cannot detect this failure. In the reproduction as well, the TypeScript check included in the next build step succeeded, and then the code failed at runtime.

Resolving the resolve call at build time is legitimate static analysis for pulling dependent files into the build output. The cause of the defect is that the service's implementation had been written without taking this static analysis into account.

Comparing the results of resolve by execution method

The following table summarizes the results of calling the same function with four execution methods. In the Bundle column, I wrote whether the app's code goes through the step that combines it into a single build output.

Execution method	Bundle	Value of import.meta.url	viaImportMeta	viaCwd
Run the source directly with tsx	No	The source file's file URL	Existing path	Existing path
Run inside a Vitest 5.0.1 test	No	The source file's file URL	Existing path, test passed	Existing path
next build, Turbopack, 16.3.6 and 16.2.2	Yes	A string assembled at runtime	TypeError, the number 39897 in 16.3.6	Existing path
next build --webpack, 16.3.6	Yes	A file URL pointing to an absolute path on the machine that did the build	TypeError, the number 5607	Warning at build time, TypeError at runtime

With tsx and Vitest, the code does not go through the bundling step, so the resolve calls are evaluated at runtime just as written in the source. That is why both forms succeed. According to the official tsx documentation, tsx transforms TypeScript and ESM with esbuild and does not check types. It says nothing about bundling. According to the Common Errors page of Vitest, source files are run with Vite's module runner by default. According to the deps page in the Vitest configuration reference, in version 5.0.1, externalized modules in node_modules are loaded by Node itself by default. Neither page describes a step that combines the app's code into a single build output.

With webpack too, the viaImportMeta form failed with a number. So the replacement of the resolve call is not behavior unique to Turbopack. In the webpack build, the absolute path of the source file on the machine that did the build was written directly into the build output as a file URL.

The Next.js 16.3 Turbopack announcement from 2026-06-29 lists two compatibility improvements. They are improved handling of createRequire combined with new URL, and a fix for import.meta.url on Windows. Even so, the replacement of the resolve call remained in 16.3.6. The behavior described in this article covers only what was confirmed with the versions stated.

Comparing a fix that uses the working directory as the base with a fix that uses the turbopackIgnore annotation

In this section, I compare two ways to fix code that resolves a package location at runtime. One is the fix adopted in the service, which uses the working directory as the base. The other is a fix that uses an annotation described in the Turbopack API reference in the official Next.js documentation.

According to the table of annotations on the same page, the webpack-compatible annotations can also be used on require.resolve() expressions. The applicable expressions are these four: dynamic import(), require(), require.resolve(), and new Worker(). The table explains that turbopackIgnore: true is interpreted only by Turbopack and that the call is excluded from bundling. It states that webpackIgnore: true is interpreted by both webpack and Turbopack. In the reproduction, with Turbopack in 16.3.6, I placed /* turbopackIgnore: true */ right before the argument passed to resolve in viaImportMeta, the form that uses import.meta.url as the base. Then an existing path came back, even with import.meta.url still as the base.

The following table summarizes the results and conditions of the two fixes.

Fix	Turbopack 16.3.6	webpack 16.3.6	Conditions for it to work	Not measured
Use the working directory as the base, the viaCwd form	Existing path	Warning at build time, TypeError at runtime	There is a node_modules under the working directory at runtime. The resolve call remains without being resolved at build time	Distributing it as Next.js standalone output
Keep the viaImportMeta form and put the turbopackIgnore annotation right before the argument to resolve	Existing path	Not measured	The annotation is interpreted by Turbopack	Behavior under webpack, behavior in 16.2.2

The fix that uses the working directory as the base depends on that form not being resolved statically by the bundler. With webpack, a build of the same form produced the warning "module.createRequire failed parsing argument." At runtime, it failed with a TypeError from trying to read resolve of undefined. I have not measured whether this fix works when distributed as Next.js standalone output. The service's configuration does not specify standalone.

The fix that uses the annotation lets you keep the form that uses import.meta.url as the base. In exchange, whether this fix works depends on whether the annotation is interpreted by Turbopack. Which fix to adopt depends on how the code is distributed and which bundler is used. As of 2026-09-25, the main branch of the service is still written with the working directory as the base.

Steps for a check that starts the same build as production and calls it from outside

In this section, I describe the procedure for checking before deployment. The development side wrote this procedure in a record in response to the failure right after deployment, which I explain in the last section. On the same day as the failure, the development side wrote the following practice as a lesson in a repository for personal memory and the history of rule changes. The record covers two kinds of changes. One is adding a new tool. The other is adding processing that reads files at runtime, where what is read is a data file shipped with a package or a native module. For these changes, just calling the function directly does not count as having checked it. The practice is to create the same build as production locally, start it on a separate port, call it from outside, and go as far as checking the response before deploying.

The same record also includes two cautions. One is that defects caused by the bundler do not reproduce in unit tests. The other is that when developers write code that computes a path at runtime, they should suspect that it can break after it goes through bundling. The record frames this practice as the same idea as a rule that has been in the service's rule documents since before this failure. That rule is that even if a server is running, it is not necessarily running the latest code. Ten days later, the development side also added the same procedure to the service's operations document. It is a procedure that starts the same build output as the deployment with next build and next start, and then calls the tool.

Below, I summarize the procedure before deployment by adding the checks the AI actually ran after the fix to the steps in the record and the operations document. Items 1 and 2 are the steps in the record and the operations document, and items 3 and 4 are the AI's checks.

Create the build output in the local repository with next build, the same command as in production
Start the created output with next start on a separate port that is free
From outside, call over HTTP the endpoint for AI agents on the same route the callers use, and check the response down to its contents
Also check that passing input that does not exist returns a response that reports an error

What this procedure checks is the bundling step, which direct function calls and unit tests do not go through.

What the AI checked before deploying and the step it had not run

In this section, I explain what the AI in charge of development checked when the tool was added to the service. When the feature was added, the AI ran the following three checks before deploying.

Passed lint, the type check, and every unit test
Called the tool's function directly from a tsx script, had it create an image from real data, and looked at that image
Checked the list of tools and the shape of the responses through a relay that connects to the AI client over standard input and output

The AI put off a fourth check, having a model read the image, until after deployment.

The rendering unit test had not replaced the PDF rendering library with a fake. This test passes a small PDF built by hand to the real rendering library. A function written the same way as viaImportMeta, which uses import.meta.url as the base, was also actually run inside the test and succeeded. The only tests that swapped in a fake were the tests for the part that assembles the tool's response. As the table in the section comparing execution methods shows, the app's code is not bundled in Vitest tests. So even in a test that went through the real library, the resolve call was not replaced, and the failure did not reproduce.

The AI had also paid attention to the shape of the build output it would deploy. In the same commit, the AI added two items to the Next.js configuration. One is a setting that keeps the rendering library out of the app bundle and has it loaded from node_modules. The other is a file tracing setting. With this setting, the AI included in the build output the data files shipped with the rendering library, such as fonts, and the native binaries. However, as shown in the section on how the replacement works, the resolve call is replaced with an integer even for a package listed in the setting that keeps it out of the bundle.

The check the AI had never run was to start the next build output and call the tool from outside it. The AI visually checked the image generated by calling the tool's function directly. However, that check does not go through the bundling step, so the AI had not verified how the build output it would deploy behaves. In the body of the fix PR it created later, the AI wrote this missing check down as one of the causes. A failure in which the range verified by tests did not match the range of code that actually runs was also covered in an earlier article, About 1,300 tests were passing, and the screen still said saved when the save had failed.

The failure right after deployment and the discrepancy in the diagnosis at the time

In this section, I explain in chronological order what happened after the deployment on the day the feature was added. Right after merging the PR that added the feature, the AI deployed to the verification environment and called the tool. The response was text saying that the PDF page could not be turned into an image, and it was the same no matter how many times the tool was called. The server logs recorded a TypeError with ERR_INVALID_ARG_TYPE and the same message as in the reproduction. The parentheses at the end contained a five-digit number.

The AI changed one place. It moved the code that resolves where the shipped data files are located from the viaImportMeta form, which uses import.meta.url as the base, to the viaCwd form, which uses the working directory as the base. The AI gave the following three reasons.

In the deployment environment too, node_modules is under the process's working directory
Other data in the same repository is already loaded with the working directory as the base
The value of the working directory is not subject to Turbopack's replacement

After the fix, the AI ran next build in the local repository where it had made the fix, and started the output with next start on a free port. It then sent an HTTP request to the endpoint for AI agents and called the tool. The response contained two things, text and an image, and the image was a JPEG in which the text on the page could be read. The AI also confirmed that passing a page number that does not exist returned a response reporting an error. After merging the fix PR and deploying again, the AI sent the same call to the same endpoint in the deployment environment. The AI confirmed that an image came back and left the result in a comment on the PR.

The diagnosis at the time was off in one place. The fix commit and the records from that time explain the cause this way: import.meta.url was replaced with a numeric ID, and that number was passed to createRequire and caused an exception. In the reproduction built for this article, import.meta.url remained a string. The value that became a number was the return value of resolve, and the exception occurred in path.dirname. The error code in the logs also matches path.dirname's ERR_INVALID_ARG_TYPE, not createRequire's ERR_INVALID_ARG_VALUE. The resolve in viaCwd stays a runtime call even in the build output. That is why the fix resolved the defect even though the diagnosis was off in one place.

Materials I referred to

Here I list again the materials I referred to at various points in the body.

The Turbopack API reference in the official Next.js documentation, updated 2026-08-03. I confirmed the default bundler, that it does not do type checking, the default value of turbopackModuleIds, and the table of supported annotations https://nextjs.org/docs/app/api-reference/turbopack
The Next.js 16.3 Turbopack announcement, 2026-06-29. I confirmed the compatibility improvements related to createRequire and import.meta.url https://nextjs.org/blog/next-16-3-turbopack
The TypeScript page of the official tsx documentation. I confirmed that tsx transforms code with esbuild and does not check types https://tsx.hirok.io/typescript
Common Errors in the official Vitest documentation. I confirmed that source files are run with Vite's module runner https://vitest.dev/guide/common-errors
The deps configuration in the official Vitest documentation, version 5.0.1. I confirmed that externalized modules are loaded by Node itself https://vitest.dev/config/deps

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

Once I wrote in the MCP tool description when to use the tool, AI agents called it

matsumotory — Tue, 29 Sep 2026 12:27:27 +0000

Summary

I built an MCP tool that assigns multiple independent tasks to subagents and has the subagents execute them in parallel. Below, I call this MCP tool the parallel-execution MCP tool. MCP stands for Model Context Protocol, a standard for AI agents to find and call external tools. A subagent is a separate AI agent that an AI agent launches in order to delegate a task to it. However, the three AI coding agents OpenHands, OpenCode, and Qwen Code did not call the parallel-execution MCP tool on their own.

To find the conditions under which it gets called, I tried three things. First, I increased the number of short memos I had the AI agent summarize from 6 to 30. Second, I replaced the task I gave the AI agent with 8 independent tasks that require reading, fixing, and verification. Third, in the rule file that the AI agent loads at startup, I wrote the name of the parallel-execution MCP tool and the condition that it is to be used when the items are independent of each other and there are 5 or more of them. With these three, there was not a single run in which subagents completed the tasks.

The AI agents called the parallel-execution MCP tool only when I added to the MCP tool description one sentence stating when to use the tool. The MCP tool description is the text written in the description field of an MCP tool definition. The conditions are the 6 combinations of the three AI agents and whether a rule file is present. I ran each of the 6 conditions twice with the sentence in the description and twice without it, and compared 24 runs in total. With the sentence, the AI agent called the parallel-execution MCP tool in 7 of the 12 runs. In 4 of those 7 runs, subagents completed the tasks. Without the sentence, every one of the 12 runs had 0 calls.

When I tested this difference with Fisher's exact test, the p-value was 0.0046 with the run as the unit and 0.0152 with the condition as the unit. However, 6 of the 12 runs with the sentence were made before I aligned two points other than the description. The two points are how numbers were written in the prompt and the recording of the time limit given to the AI agent. When I compared only the runs in which the two points were aligned, the runs with a call were 3 of 6 with the sentence and 0 of 6 without it. The p-value of this comparison is 0.182, which is larger than 0.05.

For comparison, I also measured a consultation MCP tool. The consultation MCP tool is an off-the-shelf MCP tool with which an AI agent asks another model for an opinion just once before deciding on a design. Whether the AI agent called the consultation MCP tool depended on whether I had written a provision about consultation in the rule file. With the provision, there was a call in all 6 runs. Without the provision, there were 0 calls in all 6 runs.

A call to the parallel-execution MCP tool is heavy processing that launches multiple subagents. A call to the consultation MCP tool is light processing that asks another model for an opinion just once. I present the following explanation as a hypothesis: the factor that decides whether the AI agent calls a tool changes with the weight of the processing. The reason is that the consultation MCP tool is an off-the-shelf tool whose description I cannot change.

In the body, I first explain the results of the 24 runs and my decision to add the sentence to the default description. Next, I explain the settings I kept the same across the four measurements and the method of judgment. The four measurements are the 6-memo measurement, the 30-memo measurement, the 8-task measurement, and the measurement comparing the description with and without the sentence. After that, I show the results of the four measurements and the results of the statistical tests. Then I explain the three cases I measured again because of defects in the measurement program. Finally, I explain the measurement of the consultation MCP tool and the four factors I have not separated. I also explain the design of the run records that lets me later tell apart the reasons for 0 calls, as well as what cannot be said from the results of this experiment. In the last section, I have gathered the bibliographic details of the materials I referred to.

What you can take away

For those who build their own MCP servers and are having trouble because AI agents do not call their MCP tools, I explain the following three things.

When an MCP tool is not called even though you wrote its name and the conditions for using it in the rule file, you will be able to check whether the cause lies in the MCP tool description. In the 12 runs without the sentence, including the 6 runs in which I had the AI agent load the rule file, there were 0 runs in which the AI agent called the MCP tool. In the 12 runs with the sentence, there was a call in 7
You will be able to build, in your own environment, an experiment that changes one point at a time and measures whether an AI agent calls an MCP tool. I explain the settings I kept the same, the points I changed, and a method of judgment that does not rely on the AI agent's reports
You will be able to keep the records of runs with 0 calls in a form that lets you tell two cases apart later. The two cases are the case where the AI agent did not call the tool and the case where it called the tool but the call did not meet the criteria for counting

All three are based on the four measurements I ran and the three cases I measured again because of defects in the measurement program.

This article is a discussion based on primary records from work I am pursuing. In the body I name the three AI coding agents I compared. The content does not represent the views of any of their providers.

The MCP tool that was called only when I added one sentence to its description

I measured the conditions under which AI agents call, on their own, an MCP tool I built. In this article, an AI agent means an AI coding agent. MCP stands for Model Context Protocol, a standard for AI agents to find and call external tools.

I implemented the parallel-execution MCP tool as an MCP server and made it the target of the measurement. The parallel-execution MCP tool assigns multiple independent tasks to subagents and has the subagents execute them in parallel. A subagent is a separate AI agent that an AI agent launches in order to delegate a task to it.

In this article, I count two numbers separately. The first is the number of runs in which the AI agent called the parallel-execution MCP tool. The second is the number of runs in which subagents completed the tasks.

The conclusion is as follows. The AI agents called the parallel-execution MCP tool only when I added to the MCP tool description one sentence stating when to use the tool. The MCP tool description is the text written in the description field of an MCP tool definition. An AI agent reads MCP tool descriptions when it chooses which tool to call.

There are 6 conditions. For each of the three AI agents, OpenHands, OpenCode, and Qwen Code, I prepared a case in which I had it load a rule file and a case in which I did not. A rule file is a file, loaded by the AI agent at startup, that describes how to proceed with the work.

For each of the 6 conditions, I ran twice with the sentence stating when to use the tool in the MCP tool description and twice without it. There were 24 runs in total, and the results are as shown in the following table.

The sentence in the description	Number of runs	Runs in which the AI agent called the parallel-execution MCP tool	Runs in which subagents completed the tasks
Present	12 runs	7 runs	4 runs
Absent	12 runs	0 runs	0 runs

Of the 12 runs without the sentence stating when to use the tool in the MCP tool description, 6 were runs in which I had the AI agent load the rule file. In these 6 runs as well, there were 0 runs in which the AI agent called the parallel-execution MCP tool.

On the basis of this result, I added the sentence stating when to use the tool to the default description of the parallel-execution MCP tool. However, so that I can compare again in the future, I kept a mechanism for replacing the description through an environment variable. I also pinned down with a unit test that the default description contains this sentence.

Because I added the sentence stating when to use the tool to the default description of the parallel-execution MCP tool, what can be compared has changed. After I added the sentence, when to use the parallel-execution MCP tool is written in both the MCP tool description and the rule file. So from now on, even if I compare having the AI agent load the rule file with not having it load the file, the comparison for calls to the parallel-execution MCP tool becomes a different one. It is a comparison between having the statement of when to use the tool in two places and having it in one place. The two places are the MCP tool description and the rule file, and the one place is the MCP tool description.

Even so, I added the sentence stating when to use the tool to the default description of the parallel-execution MCP tool. The reason is that among the 12 runs without the sentence, there were 0 runs in which subagents completed the tasks. I gave priority to the parallel-execution MCP tool being used over the rigor of the comparison.

I wrote about how I think about which tasks can be delegated to AI in another article, The difference between work that broke when I delegated it to AI and work that didn't. This time, I measured whether an AI agent delegates tasks to subagents.

The settings I kept the same across the four measurements, and the method of judgment

With four measurements, I investigated the conditions under which AI agents call, on their own, the parallel-execution MCP tool I built. The parallel-execution MCP tool is an MCP tool that assigns multiple independent tasks to subagents and has the subagents execute them in parallel. The four measurements are the 6-memo measurement, the 30-memo measurement, the 8-task measurement, and the measurement comparing the description with and without the sentence. In all four measurements, I kept four things the same: the model, the inference endpoint, the order of runs, and the method of judgment.

I compared three AI agents: OpenHands, OpenCode, and Qwen Code. On 2026-08-17, I installed them on macOS and recorded their versions at that time. The versions are OpenHands CLI 1.16.0, OpenCode 1.18.18, and Qwen Code 0.21.13.

In every run, I used the same single model. The three AI agents call a preview version of a model for code through the shared endpoint of an inference service. In the run records that the measurement program writes for each run, the model field has the same value on every line.

The shared endpoint of the inference service has a limit on how many runs can execute at the same time, so I ran them one at a time, in order.

To check whether the AI agent chooses a tool by itself, I did not write the name of the MCP tool in the prompt. The prompt is the text of the instructions given to the AI agent.

To judge whether the AI agent called a tool, I did not use the reports the AI agent wrote in its output. For the judgment, I used the following records, decided for each type of tool.

For the MCP tool that launches subagents, I used the call records written by that MCP tool.
For the tool that sends requests to an AI agent running in another session, I used the file for receiving requests.
For the consultation MCP tool, I used the records of requests left on the relay server. The consultation MCP tool is an MCP tool with which an AI agent consults another model. The relay server relays requests from the consultation MCP tool to the model being consulted.

I judged whether each task passed or failed by the exit code of the tests alone. I placed the original tests where the AI agent could not rewrite them, and when judging pass or fail, I ran those originals. I also detected, with a separate mechanism, whether the AI agent had rewritten the tests.

The points I changed across the four measurements are as shown in the following table.

Name of the measurement	Task given to the AI agent	Point I changed
6-memo measurement	Summarizing 6 short memos	The measurement used as the baseline
30-memo measurement	Summarizing 30 short memos	Increased only the number of items
8-task measurement	8 independent tasks that require reading, fixing, and verification	Increased the amount of reasoning one task requires
Measurement comparing the description with and without the sentence	8 independent tasks that require reading, fixing, and verification	Changed only whether the MCP tool description has the sentence

In all four measurements, the conditions are the 6 combinations of the three AI agents and the 2 options of having or not having a rule file. A rule file is a file, loaded by the AI agent at startup, that describes how to proceed with the work.

I developed the method of measurement that compares three AI agents on the same model in another article, I compared three AI agents on the same AI model, and the only differences that came out were elapsed time and the number of retries. This time I used that method and changed what I measured from the quality of the deliverables to calls to the MCP tool.

The MCP tool that was not called even when I increased the number of items and the amount of reasoning

Before I added the sentence stating when to use the tool to the description of the parallel-execution MCP tool, I ran three measurements. The parallel-execution MCP tool is an MCP tool I built that assigns multiple independent tasks to subagents and has the subagents execute them in parallel. The purpose of the measurements was to find in what cases AI agents call the parallel-execution MCP tool on their own.

In the three measurements, I changed the number of items and the amount of reasoning one task requires. In all three measurements, the conditions are the 6 combinations of the three AI agents, OpenHands, OpenCode, and Qwen Code, and whether a rule file is present. A rule file is a file, loaded by the AI agent at startup, that describes how to proceed with the work.

The results of the three measurements are as shown in the following table.

Name of the measurement	Task given to the AI agent	Conditions in which the AI agent called the parallel-execution MCP tool	Time until the AI agent finished processing
6-memo measurement	Summarizing 6 short memos	0 of 6	22 to 143 seconds
30-memo measurement	Summarizing 30 short memos	1 of 6	42 to 101 seconds
8-task measurement	8 independent tasks	1 of 6	39 to 264 seconds

The times in the table are the time of the shortest run and the time of the longest run among the runs for the 6 conditions.

In the 6-memo measurement, I had the AI agent summarize each of 6 short memos in one line. In none of the 6 conditions did the AI agent call the parallel-execution MCP tool, and it processed the 6 summaries itself through to the end.

On the same day as the 6-memo measurement, I also measured the request-sending tool. The request-sending tool is a tool that sends requests to an AI agent running in another session. In all 6 conditions, the AI agent called the request-sending tool. In the file on the side that receives the requests, lines containing the marker string remained. The marker string is a string I had put into the text of the task. Whether a rule file was present had no bearing on the results for the request-sending tool.

I thought the cause of the different results between the parallel-execution MCP tool and the request-sending tool lay in the descriptions of the two tools. The description of the request-sending tool stated the purpose of the tool and when to use it.

In the 30-memo measurement, I increased the short memos to be summarized from 6 to 30. In 1 of the 6 conditions, the AI agent called the parallel-execution MCP tool. However, as far as the remaining records show, not a single subagent had been launched even in this 1 condition. In the run for this 1 condition, there were 2 call records written by the parallel-execution MCP tool. The measurement program at the time saved only the numbers from the first record. The measurement program is the program that launches the AI agent, writes the run records, and judges whether each task passed or failed. In the first record, the number of subagents launched was 0. The content of the second record does not remain.

I interpreted the result of the 30-memo measurement as follows. When the processing of each item is a repetition of the same steps, the AI agent can process all 30 items together by itself, even when there are 30 of them. So just increasing the number of items did not give the AI agent a reason to delegate tasks to subagents.

In the 8-task measurement, I switched the task given to the AI agent from summarizing memos to 8 independent tasks. Each of the 8 tasks requires reading, fixing, and verification, and each task has its own dedicated explanatory text and tests. However, I did not give the AI agent the correct answers.

In the 8-task measurement as well, the AI agent called the parallel-execution MCP tool in only 1 of the 6 conditions. However, in this 1 condition, the AI agent incorrectly wrote the script that launches the subagents. Because of the error in the script, not a single subagent was launched. So the parent agent processed the 8 tasks itself. The parent agent is the AI agent that tried to launch the subagents.

Combining the 6 conditions of the 30-memo measurement and the 6 conditions of the 8-task measurement gives 12 conditions. For the 8-task measurement, I counted the runs made after I fixed a defect in the measurement program. The AI agent called the parallel-execution MCP tool in only 2 of the 12 conditions. However, subagents were not launched in either of the 2 conditions. So the conditions in which subagents completed the tasks were 0 of 12.

At the time of the 8-task measurement, the rule file contained the name of the parallel-execution MCP tool. As for when to use it, the rule file also said "use it when the items are independent of each other and there are 5 or more of them." However, even with these two things written in the rule file, there was not a single condition in which subagents completed the tasks.

The sentence stating when to use the tool, added to the MCP tool description

In this section, I explain the measurement comparing the description with and without the sentence. The target of this measurement is the parallel-execution MCP tool I built. The parallel-execution MCP tool is an MCP tool that assigns multiple independent tasks to subagents and has the subagents execute them in parallel.

For this measurement, I used the same 8 tasks as in the 8-task measurement. The 8-task measurement is the measurement in which I had the AI agent solve 8 independent tasks that require reading, fixing, and verification. I also did not change the steps the AI agent needs in order to call the parallel-execution MCP tool. However, the one thing I did change was whether to put the sentence stating when to use the tool at the beginning of the description of the parallel-execution MCP tool. The MCP tool description is the text written in the description field of an MCP tool definition. I switched between putting the sentence in and leaving it out with an environment variable.

Before the measurement, I checked the provisions on MCP tool descriptions in the 2026-07-28 revision of the MCP specification. According to the specification, MCP tools are designed to be model-controlled. The "model" in the specification refers to the language model that the AI agent uses when it chooses tools. There is also an explanation that a language model can discover and call tools automatically, based on its understanding of the context and the user's prompts. The description field is defined as a human-readable description of functionality. The specification has no provision requiring the description field to state when to use a tool. However, the section of the specification on tools that hold state contains the following explanation. If the retention policy is written in the description of the tool responsible for creation, the model can read that policy. So the premise that the model reads descriptions and makes judgments is written in the specification.

In the sentence stating when to use the tool, I wrote the following two things. The first is the situation in which to use the parallel-execution MCP tool. That situation is one where there are multiple mutually independent items and you want to process every item without missing any. In the sentence, I wrote that in this situation, instead of processing the items in order itself, the parent agent has subagents process them in parallel with this MCP tool. The second is the behavior of the parallel-execution MCP tool. The number of subagents launched and the number of subagents that completed are left in the records. The two things above are the gist of the sentence, and I do not write the actual wording of the sentence in this article.

I took the way of writing the sentence stating when to use the tool from the description of the request-sending tool. The request-sending tool is a tool that sends requests to an AI agent running in another session. The 6-memo measurement is the measurement in which I had the AI agent summarize each of 6 short memos in one line. In the measurement on the same day as the 6-memo measurement, the AI agent called the request-sending tool in all 6 conditions. The conditions are the combinations of the three AI agents and whether a rule file is present. A rule file is a file, loaded by the AI agent at startup, that describes how to proceed with the work.

Next, I explain how I came to measure again. In the first round, the AI agent called the parallel-execution MCP tool in 4 of the 6 runs with the sentence stating when to use the tool. However, the review that followed found two points, other than the description, that were not aligned. The first was that how numbers were written in the prompt differed between the case with the sentence and the case without it. The second was that the time limit given to the AI agent had not been recorded. I aligned the two points and measured again. In the second round, there were calls in 0 of 6 runs without the sentence and in 3 of 6 runs with the sentence.

I requested another review and received 1 finding. The finding was as follows. If the AI agent ignores an environment variable whose value is empty, then even when a run uses the setting that leaves out the sentence stating when to use the tool, the MCP server uses the default description. In that case, the runs measured as runs without the sentence become runs that used the default description. So I added to the MCP server a process that writes the description actually used out to the records. After adding it, as a third round, I ran the case without the sentence 6 more times. Of the 6 runs, there were 0 runs in which the AI agent called the parallel-execution MCP tool. In all 6 runs, the description written out to the records did not contain the sentence.

Adding up the runs so far, there were 12 runs each with and without the sentence stating when to use the tool. The results split by whether a rule file was present are as shown in the following table.

The sentence in the description	Rule file	Number of runs	Runs in which the AI agent called the parallel-execution MCP tool
Present	Present	6 runs	5 runs
Present	Absent	6 runs	2 runs
Absent	Present	6 runs	0 runs
Absent	Absent	6 runs	0 runs

With the sentence stating when to use the tool, there were more runs in which the AI agent called the parallel-execution MCP tool when I had it load the rule file. However, there are only 1 or 2 runs that share the same type of AI agent, the same presence or absence of the rule file, and the same presence or absence of the sentence, and the results also vary widely. There are cases in which, under the same condition, the first run had a call and the second run did not. So this difference can be read only as a tendency, but it cannot be said that the rule file has no effect.

What I can say comes down to the following two points. Without the sentence stating when to use the tool, even when I had the AI agent load the rule file, the AI agent called the parallel-execution MCP tool in 0 of 6 runs. With the sentence, there were more runs with a call when I had the AI agent load the rule file.

Whether subagents completed the tasks differed by the type of AI agent.

Qwen Code called the parallel-execution MCP tool in 3 runs and launched 8 subagents in every one of them. In 2 of the 3 runs, all 8 subagents completed. In the remaining 1 run, 7 of the 8 completed. The run in which 7 completed was a run in which I did not have the AI agent load the rule file. In this run, 7 of the 8 subagents completed their tasks even without the rule file.

In the first-round run in which I had it load the rule file, OpenCode fixed on its own the error in the script that launches the subagents. After that, the 8 subagents completed. This run took 935 seconds.

OpenHands called the parallel-execution MCP tool in 3 runs. Across the 3 runs, there were 4 call records written by this MCP tool, and 32 subagents were launched. However, 0 of the 32 subagents completed. I wrote the cause of this in the section that explains the defects in the measurement program.

In every run, all 8 tasks passed their tests, and there were 0 cases in which the AI agent rewrote the tests.

External research also reports that results change with the quality of MCP tool descriptions. The first study is MCP Tool Descriptions Are Smelly!. This study covers the descriptions of MCP tools provided by many MCP servers. According to this study, many descriptions have defects, and filling in the elements missing from a description raises the task success rate. The second study is Learning to Rewrite Tool Descriptions. This study rewrites tool descriptions to make tool calls more reliable. This study also reports that rewriting descriptions raises the success rate.

What the two studies measured is the task success rate, not the rate at which AI agents call tools. So I do not use the two studies as support for the results of my measurements. However, from the two studies I took it that the observation that results change with the quality of MCP tool descriptions is not an observation of mine alone. I wrote the scale of the two studies' investigations and their figures in the section on the materials I referred to.

Fisher's exact test comparing the numbers of runs in which an MCP tool was called

I tested the differences in the number of runs in which the AI agent called an MCP tool with Fisher's exact test. There are two measurements I tested.

The first measurement is the measurement of the parallel-execution MCP tool. The parallel-execution MCP tool is an MCP tool that assigns multiple independent tasks to subagents and has the subagents execute them in parallel, and I built it. In this measurement, I compared the case where the description of the parallel-execution MCP tool has the sentence stating when to use the tool with the case where it does not.

The second measurement is the measurement of the consultation MCP tool. The consultation MCP tool is an off-the-shelf MCP tool with which an AI agent asks another model for an opinion just once before deciding on a design. In this measurement, I compared the case where the rule file has a provision about consultation with the case where it does not. A rule file is a file that describes how to proceed with the work, and the AI agent loads it at startup.

The records of the experiment did not include results of Fisher's exact test. So I calculated the test results on 2026-09-10. Using only the Python standard library, I calculated two-sided p-values directly from the hypergeometric distribution.

I counted the cases in which the AI agent called the MCP tool, using three kinds of units.

In the method with the run as the unit, I counted one run as one item.
In the method with the condition as the unit, I counted one condition as one item. The conditions are the combinations of the three AI agents and whether a rule file is present, and there are 6 of them. The three AI agents are OpenHands, OpenCode, and Qwen Code. Each condition has 2 runs. In this method, I counted the conditions in which the AI agent called the MCP tool in at least 1 of the 2 runs.
In the method with the type of AI agent as the unit, I counted one type of AI agent as one item. Each type of AI agent has 2 runs. In this method, I counted the AI agents that called the MCP tool in at least 1 of the 2 runs. I used this method for the measurement of the consultation MCP tool.

The results of the test are as shown in the following table.

What I compared	Values counted	p-value
Parallel-execution MCP tool. The run as the unit	With the sentence, 7 of 12 runs. Without the sentence, 0 of 12 runs	0.0046
Parallel-execution MCP tool. The condition as the unit	With the sentence, 5 of 6 conditions. Without the sentence, 0 of 6 conditions	0.0152
Parallel-execution MCP tool. Only the runs in which the two points other than the description were aligned	With the sentence, 3 of 6 runs. Without the sentence, 0 of 6 runs	0.182
Parallel-execution MCP tool. Whether a rule file was present, with the sentence	With a rule file, 5 of 6 runs. Without one, 2 of 6 runs	0.242
Consultation MCP tool. The run as the unit	With the provision, 6 of 6 runs. Without the provision, 0 of 6 runs	0.0022
Consultation MCP tool. The type of AI agent as the unit	With the provision, 3 of 3 types. Without the provision, 0 of 3 types	0.10

There are four cautions for reading the p-values in the table.

The first caution is as follows. The 24 runs of the parallel-execution MCP tool are not independent of each other. That is because they are runs in which I repeated the 6 conditions twice each, with and without the sentence stating when to use the tool. So the p-value with the run as the unit tends to come out smaller than it really is.

The second caution is as follows. In statistics, a way of counting under which it is harder to judge that there is a difference is called a conservative way of counting. Counting with the condition as the unit is more conservative than counting with the run as the unit. In the measurement of the parallel-execution MCP tool, the p-value of the comparison with the condition as the unit is 0.0152, which is larger than 0.0046, the p-value of the comparison with the run as the unit. However, counting with the condition as the unit is not as conservative as counting that compares only the runs in which the two points other than the description of the parallel-execution MCP tool were aligned. Comparing only the runs in which the two points other than the description were aligned, the p-value is 0.182.

The third caution is as follows. The 12 runs with the sentence stating when to use the tool are the 6 runs of the first round and the 6 runs of the second round. The 6 runs of the first round were made before I aligned the two points other than the description. In 4 of these 6 runs, the AI agent called the parallel-execution MCP tool. The two points are how numbers were written in the prompt and the recording of the time limit given to the AI agent. The 6 runs of the second round were made after I aligned the two points. In 3 of these 6 runs, the AI agent called the parallel-execution MCP tool. The 12 runs without the sentence were all made after I aligned the two points. Comparing only the runs in which the two points were aligned, the numbers of runs in which the AI agent called the parallel-execution MCP tool are as follows. With the sentence, it was 3 of 6 runs, and without the sentence, 0 of 6 runs. The p-value of this comparison is 0.182, which is larger than 0.05.

The fourth caution is as follows. In the comparison for the consultation MCP tool, taking the type of AI agent as the unit gives a p-value of 0.10. This value is also larger than 0.05.

Three cases I measured again because of defects in the measurement program

The four measurements are measurements in which I investigated in what cases AI agents call, on their own, the parallel-execution MCP tool I built. The four measurements are the 6-memo measurement, the 30-memo measurement, the 8-task measurement, and the measurement comparing the description with and without the sentence. The parallel-execution MCP tool is an MCP tool that assigns multiple independent tasks to subagents and has the subagents execute them in parallel.

In the four measurements, three defects in the measurement program were found. The measurement program is the program that launches the AI agent, writes the run records, and judges whether each task passed or failed. I fixed all three defects and measured again. In this section, I explain the three defects in the order they were found.

The first defect was found in the first round of the 8-task measurement. The 8-task measurement is the measurement in which I had the AI agent solve 8 independent tasks. The measurement program had no process for detecting that the AI agent had rewritten the tests. In a review, it was pointed out to me that this process was missing. I added to the measurement program a process that detects rewrites of the tests and a process that restores the tests to their original state, and measured again.

However, I have not deleted the records of the measurement that had the defect. I attached to those records a flag indicating a defect in the measurement program and excluded them from the aggregation.

In the 8-task measurement after I fixed the measurement program, all 8 tasks passed their tests in all 6 conditions. The conditions are the combinations of the three AI agents, OpenHands, OpenCode, and Qwen Code, and whether a rule file is present. A rule file is a file, loaded by the AI agent at startup, that describes how to proceed with the work. There were 0 cases in which the AI agent rewrote the tests and 0 cases in which it stopped the work partway.

The second defect was found in the first round of the measurement comparing the description with and without the sentence. The measurement comparing the description with and without the sentence is a measurement that changes only whether the description of the parallel-execution MCP tool has the sentence stating when to use the tool. However, in the first round, there were two points other than the description that were not aligned between the runs with the sentence and the runs without it. The first was that how numbers were written in the prompt differed. The second was that the time limit given to the AI agent had not been recorded. I aligned the two points and measured again.

The third defect was found in the runs of OpenHands. In the OpenHands runs, not a single subagent completed. Using a program that reproduces the problem, I extracted the errors from when the subagents failed. All 8 subagents had terminated about 1.2 seconds after they were launched, with an error saying that no authentication method had been selected. This error occurred because three causes overlapped.

The first cause was that the parent process's environment variables were not passed to the MCP server. With the method the measurement program uses to launch the AI agent, OpenHands CLI 1.16.0 was the only case in which the environment variables were not passed. In the case of OpenCode and Qwen Code, the environment variables were passed with the same method. However, I have not checked whether the parent process's environment variables are passed to the MCP server if the OpenHands configuration is written differently.

The second cause was that there was no environment configuration file in the place the parallel-execution MCP tool reads from. Instead of environment variables, the parallel-execution MCP tool reads the environment configuration file located where this MCP tool is placed. Because I ran the measurement in an isolated working directory, there was no environment configuration file in that place.

The third cause was that the measurement program did not pass the authentication key used by the subagents through the environment variable entry of the MCP configuration.

I fixed the measurement program and stopped using a method that assumes the parent process's environment variables are inherited. The fixed measurement program writes the authentication key used by the subagents into the environment variable entry of the MCP configuration and passes it that way. After the fix, I measured again with OpenHands.

In the run in which I had it load the rule file, OpenHands called the parallel-execution MCP tool on its own and launched 8 subagents. All 8 subagents completed, and all 8 tasks passed their tests. This run was the first in which all of the OpenHands subagents completed.

In the run in which I did not have it load the rule file, OpenHands also called the parallel-execution MCP tool. There were 3 call records, and the subagents launched in this run totaled 16. Of the 16 subagents, 1 completed.

So in the measurements before the fix, it was because of the defect in the measurement program that not a single OpenHands subagent completed. A difference in the capability of the AI agents was not the cause, so this result does not show which of the three AI agents is better or worse. However, in the run in which I did not have it load the rule file, 15 of the 16 subagents launched did not complete, for reasons other than the authentication key. I have not yet investigated why these 15 subagents did not complete.

The 2 runs of OpenHands after I fixed the measurement program are additional runs to investigate why not a single subagent completed in the measurements before the fix. So I recorded these 2 runs separately from the aggregation of the 24 runs of the measurement comparing the description with and without the sentence.

The consultation MCP tool that was called only when I wrote a provision in the rule file

To compare with the parallel-execution MCP tool, I also measured the consultation MCP tool. The parallel-execution MCP tool is an MCP tool that assigns multiple independent tasks to subagents and has the subagents execute them in parallel, and I built it. A call to the parallel-execution MCP tool is heavy processing that launches multiple subagents.

The consultation MCP tool is a tool included in an off-the-shelf MCP server. Before deciding on a design, the AI agent uses the consultation MCP tool to ask another model for an opinion just once. A call to the consultation MCP tool is light processing.

In the prompt, I wrote neither the name of the consultation MCP tool nor the model being consulted. The prompt is the text of the instructions given to the AI agent. In this measurement, I changed only whether to write a provision about consultation in the rule file. A rule file is a file that describes how to proceed with the work, and the AI agent loads it at startup.

The provision about consultation says two things. The first is that the AI agent asks another model for an opinion before deciding on a design, and decides after taking into account the main points of the opinion that comes back. The second is that the AI agent must not decide on its own without asking another model for an opinion. I also wrote the name of the consultation MCP tool in the provision.

This measurement had 12 runs, which are the combinations of the following: the 2 options of having or not having the provision about consultation; the three AI agents, OpenHands, OpenCode, and Qwen Code; and 2 repetitions. With the provision, the AI agent called the consultation MCP tool in all 6 runs. Without the provision, the AI agent did not call the consultation MCP tool even once in any of the 6 runs. In this measurement, there were 0 defects in the measurement program. The measurement program launches the AI agent, writes the run records, and judges whether each task passed or failed.

I judged whether the AI agent called the consultation MCP tool from the records of the relay server. The relay server relays the requests that the consultation MCP tool sends to the model being consulted. In this judgment, of the requests in the relay server's records, I counted only the requests that met all three of the following.

The request is addressed to the model being consulted.
The status code of the response is 200.
The body of the response is not empty.

I set an invalid key in the consultation MCP tool. The valid key exists only on the relay server, and the relay server replaces the invalid key with the valid key when it forwards a request. So a request that does not go through the relay server fails authentication, and a consultation that is not left in the relay server's records does not succeed.

However, this measurement has one constraint. Because the description of the consultation MCP tool is included in the off-the-shelf MCP server, I could not change this description. The measurement of the consultation MCP tool is a comparison that changes only whether the rule file has the provision.

The hypothesis that the factor changes with the weight of the processing, and four factors I have not separated

In this section, I explain the extent of what I can say from the results of the two measurements. The first is the measurement of the parallel-execution MCP tool. The parallel-execution MCP tool is an MCP tool I built that assigns multiple independent tasks to subagents and has the subagents execute them in parallel. The second is the measurement of the consultation MCP tool. The consultation MCP tool is an off-the-shelf MCP tool with which an AI agent asks another model for an opinion just once before deciding on a design.

The results of the two measurements are as follows. The AI agent called the parallel-execution MCP tool only when the description of the parallel-execution MCP tool had the sentence stating when to use the tool. Also, the AI agent called the consultation MCP tool only when the rule file had a provision about consultation. A rule file is a file, loaded by the AI agent at startup, that describes how to proceed with the work.

The conclusion first written from the results of the two measurements was as follows. The factor that decides whether an AI agent calls an MCP tool on its own changes with the weight of the processing being called. This conclusion divided heavy and light processing as follows. The processing of calling the parallel-execution MCP tool is heavy processing that launches multiple subagents. The processing of calling the consultation MCP tool is light processing that asks another model for an opinion just once. The first conclusion explained that for heavy processing the MCP tool description is the factor, and for light processing the rule file is the factor.

I requested a review of the first conclusion and received a serious finding. There were two reasons for the finding.

The first reason was that in the measurement of the parallel-execution MCP tool as well, there were more runs in which the AI agent called the parallel-execution MCP tool when I had it load the rule file. When the sentence stating when to use the tool was present and I had the AI agent load the rule file, the AI agent called the parallel-execution MCP tool in 5 of 6 runs. When the sentence was present and I did not have it load the rule file, it called the parallel-execution MCP tool in 2 of 6 runs. So the explanation that the factor switches with the weight of the processing does not match the results of the measurement.

The second reason was that four points differ between the two measurements. The four points are the MCP tool used, the quality of the MCP tool description, the task given to the AI agent, and how specifically the MCP tool was written into the provision of the rule file. So I have not been able to separate which difference is the cause of the results.

I accepted the review's finding and withdrew the assertion in the first conclusion. So I now write the results of the two measurements separately. I write the explanation that the factor changes with the weight of the processing as a hypothesis, and I also write that I have not separated the factors. Turning this explanation back into an assertion would mean writing again, as it is, the claim that was sent back in the review, so I do not turn it back into an assertion.

There are four factors I have not separated. In statistics, such factors are called confounding factors. The four factors are as follows.

The MCP tool used differs between the two measurements.
The quality of the MCP tool description differs between the two MCP tools.
The task given to the AI agent differs between the two measurements.
How specifically the MCP tool was written into the provision of the rule file differs between the two measurements.

Because the consultation MCP tool is an off-the-shelf MCP tool, I cannot change the description of the consultation MCP tool. To separate the four factors, it is necessary to take a single MCP tool as the target and measure all combinations of the presence or absence of the sentence stating when to use the tool and the presence or absence of the provision in the rule file. However, I have not yet run this measurement.

The design of the run records for telling apart the reasons for 0 calls

In this section, I explain the design of the run records. A run record is the record that the measurement program writes for each run. The measurement program is the program that launches the AI agent, writes the run records, and judges whether each task passed or failed. With the measurement program, I measured whether AI agents call MCP tools on their own.

When a run record says there were 0 calls to the MCP tool, there are two possible reasons. The first is that the AI agent did not call the MCP tool. The second is that the AI agent called the MCP tool, but the call did not meet the criteria for counting. I set the criteria for counting for each measurement. For example, in the measurement of the MCP tool that consults another model, I counted only requests whose response had status code 200 and a body that was not empty. So that these two reasons can be told apart later, I divided the roles of two fields in the run records as follows.

The call field records whether the MCP tool was called, the total number of calls, and the breakdown of that number.
The run status field records whether the measurement program worked correctly and the run can be counted as a result of the measurement. This field does not record whether the AI agent succeeded at the task.

On the line for a run in which the measurement program had a defect, I set the run status field to failure and attach a flag indicating the defect. So as not to record values that could not be measured, I do not write the call field on that line.

I have been using the recording method explained so far since 2026-08-27. The lines of the run records from before 2026-08-27 remain as recorded with the old method, and I have not rewritten them. That is because there is a provision that lines are only appended to the run records. I also have not deleted the records of measurements that became invalid because of defects in the measurement program; I keep those records with a flag attached.

I received a review of the conclusions of the measurements. During this review, I split three processes into separate functions. The three processes are the process that counts the number of calls, the process that judges whether the measurement program worked correctly, and the process that assembles the run records. I pinned down the behavior of the three functions with 10 unit tests.

Previously, I copied part of the MCP tool's output into the notes field of the run records, but I stopped using this method. Now I record the total number of calls and the breakdown of that number as structured data.

What cannot be said from the results of this experiment

In this section, I explain what cannot be said from the results of this experiment, and what can. In this experiment, I measured the conditions under which the three AI agents, OpenHands, OpenCode, and Qwen Code, call, on their own, the parallel-execution MCP tool I built. The parallel-execution MCP tool is an MCP tool that assigns multiple independent tasks to subagents and has the subagents execute them in parallel.

The following six things cannot be said from the results of this experiment.

Only one model was used in this experiment, so I cannot say that other models would give the same results.
There are only 1 or 2 runs under the same condition. A condition is a combination of the type of AI agent and whether a rule file is present. A rule file is a file, loaded by the AI agent at startup, that describes how to proceed with the work. Also, because I measured the same conditions repeatedly, the p-values of Fisher's exact test calculated with the run as the unit tend to come out smaller than they really are.
The measurement of the consultation MCP tool is a comparison that changes only whether to write a provision about consultation in the rule file. The consultation MCP tool is an off-the-shelf MCP tool with which an AI agent asks another model for an opinion just once. Because it is off-the-shelf, I could not change the description of the consultation MCP tool. Also, between the measurement of the parallel-execution MCP tool and the measurement of the consultation MCP tool, there are four factors I have not separated. The four factors are the MCP tool used, the quality of the MCP tool description, the task given to the AI agent, and the specificity of the provision in the rule file.
In the OpenHands runs, there are 15 subagents that did not complete even after I fixed the defect in the measurement program. The defect I fixed is the one in which the authentication key did not reach the subagents. I have not yet investigated why these 15 did not complete.
As of 2026-09-27, the version of OpenCode has gone up from 1.18.18, which I used for the measurements, to 1.18.32. The version of Qwen Code has also gone up from 0.21.13, which I used for the measurements, to 0.24.6. So measuring the same way with the current versions may give different results. However, the version of the OpenHands CLI is 1.16.0, the same as the version I used for the measurements.
The values measured in the external research on the quality of MCP tool descriptions are task success rates. They are not the rates at which AI agents call MCP tools. So I cannot use the external research as support for the results of this experiment.

What can be said from the results of this experiment is as follows. With the three AI agents, even when I wrote in the rule file the name of the parallel-execution MCP tool and in what cases to use it, there was not a single run in which subagents completed the tasks.

Even when I increased the memos I had the AI agent summarize to 30, and even when I increased the amount of reasoning one task requires, the result was the same. Of the 12 conditions, which combine the 6 conditions of the 30-memo measurement and the 6 conditions of the 8-task measurement, the AI agent called the parallel-execution MCP tool in only 2. However, in neither of the 2 conditions was a single subagent launched.

However, only when I added the sentence stating when to use the tool to the description of the parallel-execution MCP tool were there runs in which subagents completed the tasks.

Still, I have not yet separated which is the factor that decides whether an AI agent calls an MCP tool: the MCP tool description, the rule file, or the weight of the processing executed after the call.

Materials I referred to

In the body of this article, I referred to three public materials. I checked the original text of all three on 2026-09-10 and checked it again on 2026-09-27. For each material, I write below its content and the range I quoted from it in this article.

The tools section of the Model Context Protocol specification (revision 2026-07-28) https://modelcontextprotocol.io/specification/2026-07-28/server/tools

This material is the specification that defines the fields of an MCP tool definition. This material also contains the provision that MCP tools are designed to be model-controlled.

In the body of this article, I quoted the following three points from this material. The description field of an MCP tool definition is defined as a human-readable description of functionality. This material has no provision requiring the description field to state when to use a tool. The section on tools that hold state contains the premise that the model reads MCP tool descriptions and makes judgments.

Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions (Mohammed Mehedi Hasan, Hao Li, Gopi Krishnan Rajbahadur, Bram Adams, Ahmed E. Hassan, arXiv:2602.14878, submitted 2026-02-16, revised 2026-05-31) https://arxiv.org/abs/2602.14878

This material is a study that examined 856 MCP tools provided by 103 MCP servers. It reports that 97.1% of the descriptions of the MCP tools examined had at least one defect, and that 56% did not state their purpose explicitly. It reports that filling in the elements of a description improves the task success rate by a median of 5.85 percentage points. However, it reports that the number of execution steps increased by 67.46%, and that performance dropped in 16.67% of the cases.

In the body of this article, I quoted from this material only the point that results change with the quality of MCP tool descriptions. I wrote the scale of the investigation and the figures in this section.

Learning to Rewrite Tool Descriptions for Reliable LLM-Agent Tool Use (Ruocheng Guo, Kaiwen Dong, Xiang Gao, Kamalika Das, arXiv:2602.20426, submitted 2026-02-23, revised 2026-04-29) https://arxiv.org/abs/2602.20426

This material is a study that rewrites tool descriptions to make tool calls more reliable, and it addresses the following problem. Tool descriptions are often written for human developers and contain ambiguity. Especially when the number of candidate tools grows, AI agents cannot resolve this ambiguity. It reports that in an experiment that increased the number of candidate tools to 150 or more, the per-query success rate improved by an average of 60.89% compared with the original descriptions.

In the body of this article, I quoted from this material only the point that rewriting tool descriptions raises the success rate. I wrote the figures of this study in this section.

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

The world as a data stream, and the human who moves to the side that makes data

matsumotory — Fri, 25 Sep 2026 07:58:07 +0000

Summary

For an AI, the world is the whole of the data the AI can read. Whatever lies outside the readable data does not exist for that AI to begin with. This column is about building and running such a world out of nothing but the activity data of a single person, and what I thought about while doing it. One person's worth of data is a tiny fraction of the world as a whole, but the loop that reads that data and makes the next thing is actually turning right now. Where will the role of the human, who has taken in data and thought about it, move to from here? I start from this small attempt and work my way to that question.

Audience and takeaways

This column is for people who are building something together with an AI and have come to want to reconsider what they are supplying to that AI. How long will what I hand over now stay something only I can hand over? Beyond that, where does the role of the human move to? This is not a story that settles on a single answer, but you can take away one way of seeing the role of the human who has ended up on the side that gets overtaken. I wrote it hoping it might be a clue for people who develop with an AI every day to rethink where they stand.

So, concretely, what does the world refer to for an AI? Just as a human builds up the world from the information that comes in through the eyes and ears, an AI builds up the world from the data it is given. Whatever lies outside the readable data does not exist for the AI to begin with. So the data you hand over decides the breadth and the shape of that AI's world.

The world for this site's AI is made of nothing but the activity data of a single person, me. What I do day to day, the things I have written, the things I have revised, the things I hesitated over. That becomes the whole of the world for this site's AI. It is far too small to call a world. Against the totality of the world, one person's worth of data is next to nothing.

This mechanism has already begun to turn. Day by day on this site I build up two things: the articles I write, and the records of my judgments about which of the points raised I took and which I turned down. From those, this site's AI makes the next thing. Answering a question and choosing the subject of the next article are both done by reading that one person's data. It is too small to call a world, but the loop that reads one person's data as a world and makes the next thing from it is certainly running.

But what nagged at me was not a matter of scale. It was the thought that even if the scale is small, the principle might be the same.

Inside the world, which is one large stream of data, an AI finds problems on its own, decides what it wants to do, and builds. Supposing such a day comes, and you break that activity down into its form, it becomes a loop: look at the data flowing in, find something in it, and start building. The form of this loop does not change whether the data coming in is one person's worth or the whole world. What differs is the volume and the width of the flow, not the type of the activity. If that is so, then what I am doing now with one person's data is a shrunk-down version of the same activity inside the large flow. Regardless of scale, I am actually starting to give concrete form to the same thing, right now. That is what I ended up thinking.

The moment I thought that, the place where the human's role sits began to look shifted.

Until now, I have been on the side that consumes data. I take in information from the world, think based on it, and produce an answer. An AI caught up with me on exactly the two things my work was built on, speed and quality. So what does the side that has been overtaken do? Here is my read on it. I think I will move not to the consuming side but to the side that makes the very world the AI reads, that is, to the side that makes the data.

Honestly, there is a part of this I cannot fully settle with myself. Even in the role of supplier, whatever part can be handed over as a pattern will eventually go to the AI. The work I have done, finding problems and deciding a direction, also becomes something the AI can trace once I leave its method behind as data. At the end of handing over everything that can be handed over, what remains in my hands might be only the role of newly offering up a world the AI has not yet read.

Whether you see that as a step backward depends, I think, on how you look at it. Putting data no one, and no AI, has read yet into the world is also, seen from the side that gets consumed, making the very source of the data everyone reads. However clever an AI becomes, it cannot move if there is nothing to read. Making that thing to read does not look, at least to me right now, like a meager role.

This is not a story with an answer already in hand. What a human who has moved to the side that makes the data feels beyond that, and what they place value on, I do not yet know. But I have come to stand in a place where I can see for myself how that question turns out, and that I think is not bad.

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

Two projects, different in field and in build, had sorted where feedback to an AI should live into the very same three layers

matsumotory — Fri, 25 Sep 2026 07:56:11 +0000

Summary

When I point something out to a conversational AI agent, the feedback disappears once the session ends, and in the next session I end up pointing out the same thing all over again. I wanted to find where to record that feedback so it would take effect in the next session too, so across two projects that differ completely in field and in build, a desktop tool I built myself and a repository for research drafts and research operations, I reread the last few weeks of commit history and design documents. What I found was that the two, while proceeding separately, had sorted where feedback should live into exactly the same three layers.

The three layers are these: the instruction file the agent loads every session together with the documents that hold the operating rules; skills that gather up fixed procedures; and a memory outside the session that keeps only the history of decisions. Which layer something goes into is decided by a single criterion, whether another session, looking only at the repository, can reproduce the same operation and the same quality. The conclusion I reached was this: choose, every time, which layer a piece of feedback belongs in, instead of fixing it on the spot.

There were failures along the way. I kept writing general rules into memory, and the discipline did not reproduce in other sessions. After that, I moved a summary of the operating rules into the top-level instruction file, and I put in place a hook that prompts a reminder just before an edit without stopping the operation, along with a two-layer test that separates what a machine can judge from what requires judging meaning. These happened on separate days over several weeks.

Feedback disappears when the session ends

For the past two or three weeks I have been running two projects in parallel, a desktop tool I built myself and a repository for research drafts and research operations. The whole time, I kept finding myself pointing out the same things to a conversational AI agent over and over. Whatever I point out, the feedback does not survive once the session ends, and in the next session I end up pointing out the same thing from scratch. What I write about in this article is how to choose where to record that feedback so it takes effect in the next session, which has not received the feedback. These two projects, completely different in field and in build, arrived separately at the very same shape of sorting feedback into three layers. Because they matched, I decided to write down the three-layer split, and the single criterion for deciding which layer to use, in a form I can hand to the next person who runs into the same trouble.

Three layers and one criterion

The three-layer split goes like this. General rules that another session needs in order to reproduce the same deliverable from the repository alone go into the instruction file the agent loads every session and into the documents that hold the operating rules. Work whose procedure is settled and can be repeated mechanically goes into a skill that gathers up that procedure. And only the primary record of the decision history, when I received what feedback and where I moved it, goes into the memory that the agent carries outside the session.

There is only one criterion for the sorting, whether another session, looking only at the repository, can reproduce the same operation and the same quality. Put what is needed for reproduction inside the repository, and put into memory only the history that is personal and is not used for reproduction. That alone does the sorting.

The failure of writing general rules into memory

The reason I arrived at this criterion is that I once stumbled by doing the opposite. In one of the projects, a practice of writing feedback straight into memory the moment I received it had, before I noticed, become an unspoken habit, and there was a period when I kept writing even general writing discipline, the kind that should work for anyone who reads it, into memory one after another. The operating rules themselves were written in the repository, as documents. But those documents sat in a position where they only took effect if a new session actively went to read them. Because of that, even when another session or an agent running in parallel looked at the repository alone, the discipline was not reproduced at all.

So I moved a summary of the operating-rules documents over to the top-level instruction file, the one that is always read every session. I rewrote the handful of general rules I had put in memory into brief records that keep only the background of the feedback I received, so the rules themselves live in the instruction file and the skills, with memory pointing to them.

Shifting toward a layer that works even when it is not read

Still, even after moving things into the instruction file, the weakness remains that the operation depends on the file being read at all. So I shifted one more step, toward a layer that works even when it is not read. In one project, I put in a hook that runs automatically just before editing design, Japanese wording, or documents, so that it surfaces a prompt to read the relevant skill. This hook does not issue a permission decision; it only passes along extra information, and it never stops the operation. In the other project, I converted some of the feedback into automated tests and split them into two layers: things a machine can settle as black or white, such as character counts, punctuation, and formatting, go to a program check, while judgments that depend on context, such as meaning and logic, use the AI itself as the judge.

That said, tests are not finished once they are in; weaknesses show up on the test side too. For example, the test that judges meaning would demand that I add hedging words and weaken the claim even in the body text of a piece written on a day when the number of cases I could observe for the research was small. So I placed a file that declares in advance how many cases and how many days I could observe on that day, and I loosened the judgment so it only drops claims that go beyond the declared range. I have also been fixing two things in the mechanism itself: it returned only a failure without showing how to fix it, and it failed the text by mistake because the context of the other sections was not passed in.

There is not much firm evidence I could use to judge that this reorganization into three layers was complete. As I touched on earlier, the hook does not change the permissions of an operation; it only prompts a reminder. Whether that hook is broken is guarded by a test that takes the script's location out of the registered hooks and checks that the file actually exists. What I could confirm goes about this far.

The dependence on being read that still remains

Even after all this, the dependence on being read has not disappeared. The mechanism that automatically runs checks on every code change sees only part of the program checks, and judging meaning stays a matter of running it by hand. There also remains a trap: if I am not signed in to the service that runs the judge, the judgment passes straight through and the test looks like it passed. The instruction file side has its own structural cost, because as the instruction file grows, the amount the agent loads and processes each session grows with it. I have not yet obtained firsthand data measuring how much slower that makes things.

Whether recurrences of the same kind of feedback have really decreased, I also do not yet know. All I can pull from the commit history is the trend in how many times I wrote rules and in the number of tests, because there is no data comparing the recurrence rate under matched conditions before and after putting this mechanism in.

The human work left after shifting toward the machine

To the extent I shifted the checking toward the machine, the boundary that humans judge stood out more clearly instead. It is the line between which checks to leave to the machine and which not to. Every time a test fails, it returns a suggested fix and a distinction, whether that piece of feedback is fixed by rewriting the body or whether it should go back to the measurement that backs it up. How I act on that is still in my hands. From my side, I have also given instructions: cut the tests that only match form mechanically, because they miss things, and do not turn rules I never actually raised into tests. And I think that deciding what to record as a rule and what not to record is itself work that still remains on the human side.

What I arrived at in the end was a practice of choosing, every time, which layer to put feedback in, rather than fixing it on the spot. If there is someone repeating the same feedback to an AI agent over and over, I would be glad if this article reached them as one way to choose where to put that feedback so it takes effect in the next session too.

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

Matching strings do not settle the direction: which document is the original

matsumotory — Fri, 25 Sep 2026 07:56:07 +0000

Summary

I run a check that compares the body text of articles before publication with the text of my source material, string by string. The source material is the private development repositories I draw on for the articles.

On 2026-07-18, 3 articles waiting for publication failed this check. The cause was not that the articles had copied internal text from the source material. It was because one file in the source material, a file that analyzed this blog's published articles, quoted body text I had already published. The copying document and the copied document were the other way around.

The same check also has a blind spot in the opposite direction. As long as I exclude my own repository from the targets of matching, a draft that echoes the wording of my own internal documents cannot be found by the machine matching.

For the false positive where I mistook the direction, I revised the targets of matching, and for the defect in allowed words, I fixed how matches are counted. Neither fix loosened the criteria for the judgment itself. Copies taken from my own repository are still found by an AI review that reads the meaning.

What you can take away

For those who have a check based on string matching in their own operations, I explain the following three things.

You will be able to choose, at design time, the evidence that decides the direction of a match, that is, which document is the original. I decide it with time information rather than with strings, and I write out the three dates used in that judgment
You will be able to estimate how large a blind spot remains in a check that excludes itself from the targets of matching. I describe a real case that the machine matching could not find and only an AI reading the meaning found, and why I set aside the idea of putting my repository back among the targets
You will be able to plan ahead for how the way you record already-judged matches can force you to judge them again. I describe a real case where I had to judge the same position a second time

This article is a discussion based on primary records from operations I continue as individual research.

What a string match does not settle

I start with the conclusion. Even if strings match between two documents, that does not settle which one is the original. Nor can the match itself tell you whether one copied the other, or whether both simply used the same public identifiers or naming habits.

I made three errors with this check: a false positive from mistaking the direction, a false negative from excluding myself from the targets, and a defect in the handling of allowed words. A false positive is the error of judging something that was not copied to be a copy, and a false negative is the error of missing a copy. All three come from one and the same property: I tried to decide the direction of the original, and whether a text was mine or someone else's, from the matching strings alone.

I judge the direction from time information, not from string matches. I compare three dates queried with git. They are the date the article body was last changed, the date of the last full matching run that found 0 matches, and the date that text entered the source material.

External plagiarism detection rests on the same premise. I confirmed the following three sources in their original text on 2026-09-16. According to the English Wikipedia entry Content similarity detection, an external detection system is defined as one that compares a suspicious document with a reference collection. That collection is a set of documents assumed to be genuine. In the generated plagiarism detection task of PAN at CLEF 2025 as well, suspicious documents and source documents are separated in advance as part of how the data is constructed. According to Reception Reader, a study dealing with text reuse in historical materials, pairwise similarity comparison is reported not to explain the direction of reuse over time. The same study lists, as future development, giving the direction by a process that links text fragments back to their earliest occurrence.

How the check that detects copies is built

Before going into the errors, I show what this check matches and how. The following table is the configuration as of 2026-09-16.

Item	Current configuration
Targets of matching	The body text of each published, awaiting-publication, and draft article, excluding the record metadata yaml and code blocks
What they are matched against	The working trees and commit messages of the source material repositories
Length counted as a match	16 or more consecutive characters
Output	Only which position in which file had a match of what length, and the identifier of the matched source material
When it cannot judge	It stops running and lets nothing through
What it excludes from the targets	This platform's own repository
Scale	98 target files, 12 repositories matched against

The minimum match length of 16 characters is the result of measurement. At 8 characters, so many common technical Japanese phrases and command strings matched by chance that the results were useless for judgment. The check does not output the matching strings themselves because the check's output would itself become a new path for leaks.

This platform itself is excluded from the targets because the article files exist in its working tree. Without the exclusion, an article always matches itself. Documents in the source material that contain quotations from this platform can also be excluded from matching by specifying their paths. But only documents that quote content originating here may be excluded. If the source material's own internal documents were put on that exclusion list, no match that ought to be stopped would be detected at all.

A reverse-direction false alarm caused by quotations in a source document

The first false alarm was on 2026-07-18. In a bulk matching run, 3 articles in the stock waiting for publication failed. The match lengths were 16, 24, and 16 characters, and the same single file also matched 8 published article directories.

The other side of the match was a file in one of the source materials, the repository for research drafts and research operations. This file analyzed the published articles of this blog. On 2026-09-16 I checked it again, read-only. This file was added on 2026-07-16 and has not changed since that day. I had not copied internal text of the source material into the articles; the source document quoted body text already published here.

At that time I added a mechanism that specifies, per source material, paths to exclude from matching. I also wrote into the rule documents the restriction that only documents quoting content originating from this platform may be excluded.

This false alarm was reproduced on 2026-09-16 as well. I ran the reproduction in a copied working directory set to the same state as production, and referred to the original repository read-only. Removing just one exclusion produces 29 matches against that same single file from 9 published articles. Of these, 21 are body text of 6 articles published between 2026-07-07 and 2026-07-11. Those were published before 2026-07-16, when that single file was created, so they are reverse-direction matches where prose from here was taken into the source document. The longest was 65 characters. The remaining 8 are fragments of 3 articles published between 2026-08-06 and 2026-09-03. Their content was the paths of published article directories and the names of skills I have published. Removing all 9 exclusions gives 58. These counts of 29 and 58 are under the reproduction condition with exclusions removed, so they cannot be compared side by side with the counts for the production configuration I describe later.

The blind spot left by excluding my own repository

While writing one draft, I had echoed the wording of other documents in two places in my draft. The documents I echoed were this repository's own rule documents and the implementation of the reader-facing chat. The lengths were 34 and 19 characters respectively. The machine matching could not detect them, and only an AI reading the meaning found the echoed wording. As long as I exclude my own repository from the targets, wording echoed from my own internal documents cannot be found by the machine matching.

I considered simply putting it back among the targets and set that idea aside. This repository's rule documents quote the sentences of articles themselves as examples of review comments. The handoff records and operations data contain summaries of the articles' content. A machine cannot distinguish a quotation that went from an article into a document from a copy that went from a document into an article. Putting the repository back among the targets would make the list of exclusions too broad, and the check would become a formality.

The approach I am now moving toward is self-matching against only those places that, by their structure, contain no quotations of articles. The code of server-side processing, the site implementation, and scripts fall into those places. There is nowhere in the code that quotes an article, so the ambiguity of direction does not arise. For now, copies in this range are found by the AI review that reads the meaning.

The defect in handling allowed words

The third error, the defect in handling allowed words, was a defect in how the content of a match was counted. This check has an allow list of general names registered as approved for publication and widely known technical terms. What remains after removing those from a match is the specific content. On 2026-08-13, I fixed a defect where an allowed word could not be removed when a match boundary fell inside the word. Before the fix, fragments of the product names of public AI models were counted as specific content. The fix changes the counting so that it first finds the ranges the allowed words occupy on the full text of the line and counts only the characters not covered. I ran the check over all published articles and drafts before and after the change, and confirmed that matches already judged continued to be detected.

Recording matches already judged, and judging the direction again

Once I finish judging the direction of a match, I record the result with the reason and the date, and later runs pass it. Below, I call registering a match in this record "acceptance." I created the acceptance record on 2026-08-11. As of 2026-09-16 it has 26 lines. The articles with an acceptance are 17 published articles. Matches in drafts and in articles waiting for publication cannot be accepted.

The grounds for the direction written on the 26 lines are all time information. Not one was decided by the string match itself. The 26 lines fall into four kinds. Of these, 20 lines are reverse-direction overlaps where wording from here was later taken into a source document. Another 3 lines are matches where both sides shared field names of a public API or path names of a repository structure. Just 1 line is a case where both sides quoted the same sentence from a published technical article. The remaining 2 lines are matches where common phrasings containing particles lined up by chance. There is not a single match in the direction that ought to be stopped, that is, where an article echoed internal text of the source material.

Judging the direction does not end in one pass. On 2026-09-16, a full matching run in the production configuration reported 7 failing matches across 3 published articles. All 7 were the same type of sentence, a set phrasing that states the typesetting rule for mixing Japanese and Latin text. The 3 failed articles all took that rule itself as their subject. I traced the origin of this rule in the article Who decided on the half-width space between Japanese text and alphanumerics?. Querying the direction with git showed the reverse direction: the same phrasing had been taken into source documents on 2026-09-16 and 2026-09-02.

I had judged the direction for these 7 positions on 2026-08-26 and 2026-09-07 and accepted them. They failed again because at that time I recorded only one matched source material. On this day the same positions also matched another source material. Each acceptance pins down not only the text of the fragment but also the set of identifiers of the source materials that matched at that time. Each time the same text is later taken into another source material, a match already judged goes back to failing.

The count grows each time the source material moves forward. When I ran full matching on 2026-09-07, new matches appeared across 8 published articles. The count ranged from 17 to 24 depending on when the run happened. That is because the working trees of the source material move with every run. After judging the direction of all of them, the number in the direction that ought to be stopped was 0.

For drafts I fix things differently. The matches on 2026-09-08 and 2026-09-11 were both reverse-direction, where a source document later overlapped. Even so, I resolved them by rephrasing the relevant sentence in the draft without changing its meaning. The list of facts I made to write this article also produced 7 matches when I matched it right after finishing it. Of these, 6 were the connective words inside English quotations taken from external primary sources, the parts that only join the phrasing together. The core words of the quotations matched no source material. The remaining 1 was the name of the directory where withdrawn drafts are kept. The source material side had also named the same kind of directory with the same word. So I had not copied text from the source material; both sides simply used the same naming habit. Matches in drafts cannot be accepted, so I shortened the English quotations to only their core words. I also rewrote the directory name into a Japanese description without giving the name, and resolved the matches. When I matched again after rewriting, it came to 0. In a full matching run in the production configuration on the same day, the other 4 draft articles, 8 files, also passed with 0.

A design that revises the targets of matching without loosening the check

Since I created the script for this matching check on 2026-07-12, I have rewritten it 18 times in two months and one week. The latest change was on 2026-09-07. In the rewrites I added allowed words, added reverse-direction matches to what can be excluded, created the acceptance record, and fixed the defect in allowed words. Not one change loosened the criteria for the judgment itself. For the mistaken direction and the defect in allowed words, the check stopped, and its design of not passing what it cannot judge stayed as it was. I confirmed the cause first, and then fixed the targets of matching and how matches are counted. For the blind spot from excluding my own repository, the check did not stop, and an AI reading the meaning found the echoed wording. The design of not passing when it cannot judge is not unique to this check. I wrote about how I built the same design separately in two places in I built evaluators that do not pass what they cannot judge, in two separate places.

For those who put the same check into their own operations, I write three key points. Decide the direction from time-based evidence, not from string matches. If you exclude yourself from matching, decide from the start that copies in that range are found by an AI review that reads the meaning. And because each acceptance also pins down the set of matched source materials, you will judge the direction again each time the source material grows.

I set the limits of my claims myself. In this operation, my articles, which do the observing, and the source documents, which are observed, form a loop connected by quotation. The generalization that this configuration always produces false positives was not supported by a search on 2026-09-16. What external primary sources support goes only as far as two points. One is that external detection assumes the reference collection is genuine. The other is that a match does not explain the direction in time. I write about the loop as a discussion confined to measurements from my own operations.

Research I referred to

A study dealing with text reuse in historical materials. On 2026-09-16 I confirmed in the original text that pairwise similarity comparison does not explain the direction of reuse over time, and that the study states, as future development, giving the direction by a process that links text fragments back to their earliest occurrence. Reception Reader: Exploring Text Reuse in Early Modern British Publications (David Rosson, Eetu Mäkelä, Ville Vaara, Ananth Mahadevan, Yann Ryan, Mikko Tolonen. Journal of Open Humanities Data 9, 2023-04-17. DOI 10.5334/johd.101) https://openhumanitiesdata.metajnl.com/articles/10.5334/johd.101

Materials I referred to

On 2026-09-16 I confirmed that an external detection system is defined as one that compares a suspicious document with a reference collection assumed to be genuine. Content similarity detection (English Wikipedia) https://en.wikipedia.org/wiki/Content_similarity_detection
On 2026-09-16 I confirmed that, in how the task is constructed, the roles of documents are separated in advance into suspicious documents that may contain reuse and source documents that are the other side of matching. Generative Plagiarism Detection (PAN at CLEF 2025) https://pan.webis.de/clef25/pan25-web/generated-plagiarism-detection.html

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

Does shallower reasoning make an AI's Japanese more natural? I compared five drafts and the assessments split

matsumotory — Wed, 02 Sep 2026 19:04:24 +0000

Summary

On 2026-08-31 I passed on one impression to the AI agent I write articles with. The impression was that even a model like Claude Opus 5 seems to write natural Japanese, without over-reasoning, when you run it with effort set to medium. Effort is an API parameter that decides how much thinking a model spends on reasoning. The AI agent wrote this impression into the rules document the same day. It also collected five studies that looked like they would support it. And it went as far as having five drafts written from the same material and comparing them, all on that same day.

The results of the experiment, though, did not come out the way my impression said they would. When I counted violations of the Japanese rules, the drafts written with effort lowered came out better. When I had an outside model rate overall naturalness, on the other hand, the drafts written with effort raised came out better. In this article I explain first how the impression became a rule. Next I show how the assessments split when I compared the five drafts. Then I explain why two of the studies I collected did not hold up as support. Last I write what I decided in light of that.

What you can take away

For people who have an AI write Japanese prose, I explain the following three things.

Whether you should raise or lower effort can flip between having an AI write Japanese and having it check the facts and logic of what an AI wrote. I explain how I split effort between those two
I made a point of not settling a rule on an impression alone. On the same day, I wrote into the body of the rule that I would record what came out of running the AI under it and check later whether the rule was right. I explain how that came about, and the comparison experiment I ran the same day
I have the studies an AI collects read again the same day by an AI other than the one that collected them. For two of the five, the claim I was about to write into the article was either absent from the source or the opposite of what the source said

The product names of AI models in the body refer to the ones actually used in this operation, and they do not represent the views of their providers.

How I set effort and run it now

To give the conclusion first, whether shallower reasoning makes Japanese natural came out differently depending on how I compared. So I have not turned this impression into a settled rule. Here is how I run it now.

When having an AI write Japanese (generating, polishing, drafting fixes), choose effort explicitly, not just the model name. Do not let it silently inherit the deep setting of the whole session
When having it check the facts and safety of what an AI wrote, keep effort deep
Use low for neither, because low increases missed detections
Leave polishing and drafting fixes to Claude Sonnet 5 as before, until measurement confirms my impression is right. Put on hold the proposal to make Opus 5 at effort medium the first choice
The session's leader model writes the final draft directly. This decision is not among the ones on hold

I explain in order how I arrived at running it this way.

What happened on the day I passed on the impression

On 2026-08-04, about four weeks before I passed on this impression, I had decided not to use Claude Opus 5 for checking Japanese. There were two reasons. I once had an Opus 5 checker, an AI that checks text against the rules document, look for the mistaken simplification that breaks standard compound words down into native Japanese wording. That checker left noun-phrase compression inside its own suggested fixes. Two days before that, when I wrote and compared four drafts from the same material, the Opus 5 draft had noun phrases strung together to excess. Since then Claude Sonnet 5 handles the Japanese checks, and Claude Fable 5 takes over when Sonnet is not enough.

On 2026-08-31 I told the AI agent to narrow the scope of that decision. What I meant was that Opus 5 too seems to produce natural Japanese without over-reasoning if you set effort to medium, and that I wanted to put that to use. I thought one cause of unnatural Japanese might lie in over-reasoning rather than in the choice of model. In response, the AI agent rewrote the decision from 2026-08-04 so that it applies only to the setting of that time, when the model ran with deep reasoning and no effort specified. It kept the record of the problems that actually happened and narrowed only the range the decision reaches. Even for a rule I have already decided, I make a point of revisiting its range when a new observation or comment comes up. There is a worry that fixing a rule decided with an AI while its grounds are still weak lets a mistaken practice settle in, and I wrote about it earlier in Who decided on the half-width space between Japanese text and alphanumerics? Tracing it back, I found an AI had decided it with no grounds.

That same day, two AI sessions running in parallel each received what I had said separately, and both revised the rules document independently. The session that got its change into the rules document first kept Sonnet 5 as the Japanese checker, as before. On top of that, it required choosing a model name and an effort level together, but only when having an AI write Japanese. The setting to try first was running Opus 5 at effort medium. The other session, which noticed later, withdrew its own reading that Opus 5 should also come back as a checker under conditions, and fell in line with the earlier version. It then wrote the content of that earlier version into the document that sets which model goes to which task, and into the settings checks of the two article-writing workflows.

Under the rule left standing after all this, I keep effort high when checking and set effort to medium when writing. First, when I have an AI check the facts and safety of written text, I keep effort deep. Second, when I have an AI write Japanese, I state effort as medium and do not let it silently inherit the session's deep setting. Third, I use low for neither, because it increases missed detections. Those are the three points. The AI agent wrote these three points into seven places in the documents the same day. They are the Japanese writing rules, the definition of which model goes to which task, the settings checks of the two article-writing workflows, the procedure for the daily writing job and its skill, and the settings of the polishing workflow. My impression, though, is still one person's subjective view. So on the same day the AI agent wrote into the body of the rule that it would keep a record of what came out of writing articles under this setting, and add to the rule only after the results were confirmed in review.

Comparing five drafts written from the same material

On the same day it wrote into the rule, the AI agent started an experiment comparing drafts. It varied the effort of Claude Fable 5 and Claude Opus 5 across medium, high, and max. Then, with the same material and word-for-word the same instructions, it had five final drafts written from scratch. It hid which draft came from which setting and compared them in three ways. Two models and three effort levels make six combinations. Five drafts remain in the record. Which combination is missing cannot be pinned down from the record. It prepared three ways of comparing. A Sonnet 5 checker counts violations against the Japanese rules document. Three independent recalculations check whether the numbers in the material match what the text says. And Gemini, a model from an outside provider, rates overall naturalness.

Way of comparing	Result
A Sonnet 5 checker counts rule violations	For Fable 5, the draft at effort medium had the fewest violations of translationese and of style. For Opus 5 it went the other way, and the draft at effort max had the fewest
Three independent recalculations of whether the numbers in the material match the text	Only in the draft Fable 5 wrote at effort medium did all three recalculations find the same mismatches with the material. They were a wrong count, and a statement that settled something that had not been measured yet
Gemini rates overall naturalness	Gemini rated the drafts as more natural the higher the effort, for both models

In short, counting violations made the lower-effort draft the better one for Fable 5, while rating naturalness made the higher-effort drafts the better ones for both models. And the Fable 5 draft at effort medium, the one with the fewest rule violations, was the only one that carried mismatches with the material.

The comparison procedure itself also ran into a problem. The order in which drafts are handed to a checker can affect the assessment. The AI agent shuffled the order of the drafts for each checker to avoid that effect. Three of the four checkers then mixed up the draft labels with the order they had read them in. The AI agent matched the quotations the checkers returned against the text of the drafts. It could confirm that all three had copied them the other way round, so the results were recoverable. Out of this, the AI agent added two items to the rules for work it repeats on its own. One is to match the quotations a checker returns against the original text before tallying anything. The other is to fix the order in which drafts are handed over rather than shuffling it, and to catch mix-ups in reading order through that matching of quotations.

What came of having a different AI read the five collected studies again

In parallel with writing into the rule, the AI agent set out to check in primary research sources whether my impression had any support in theory. In its 2026-08-31 search, the AI agent collected the studies into three groups. The first group is two studies on multilingual reasoning (When Models Reason in Your Language and A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning). From those the AI agent took two readings. Reasoning models tend to think mainly in English even on non-English tasks. Pinning the language of thought to the user's language lowers accuracy. The second group is one empirical study on reasoning time (Inverse Scaling in Test-Time Compute). It said there are tasks where extending reasoning time makes scores go down instead. The third group is two reports on creative writing tasks (COIG-Writer and When Reasoning Supervision Hurts). The AI agent had collected these two as reports that adding reasoning has no effect or does harm.

On 2026-09-02, an AI in a context separate from the one that collected the studies fetched these five sources and read them again. It checks three things: whether the source exists, whether the bibliographic details are right, and whether the claim I was about to attribute to it really appears in the source text. To prevent fabricated quotations and references, this site always runs this check before publishing. Here are the results in a table.

Study	The claim I was about to write	What the second reading found
When Models Reason in Your Language (EMNLP 2025 Findings)	Reasoning models think mainly in English even on non-English questions, and pinning the language of thought to the user's language lowers accuracy	Matches what the source says. The subjects, though, are six open-source distilled models and short answers in math and science, so it does not extend to commercial models or to writing prose. The language used for thinking is sometimes Chinese, not only English
A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning (EACL 2026 Findings)	The bias toward English-centered reasoning, and noise from translation	Only the first half is in the source. Noise from translation is not written there, and the source carries the opposite result, that thinking translated into English is more accurate
Inverse Scaling in Test-Time Compute (TMLR 2025)	There are tasks where extending reasoning time makes scores go down instead, and this was confirmed on several models	Matches what the source says. It is a result shown on deliberately constructed task types, though, not a law that thinking longer generally lowers scores. How the breakdown happens differs by model family
COIG-Writer (arXiv:2510.14763)	Training without a reasoning process attached gave better creative writing scores	The source states the opposite conclusion. It reports that training on creative data with the process attached, mixed with general data, works well, and it holds no experiment comparing the presence or absence of the process at all
When Reasoning Supervision Hurts (arXiv:2605.20364)	Reasoning supervision lowers the quality of long-form literary generation	What the study covers is different. What it had generated was not literary writing but critique reports in a set format, and it is a preprint

The first and second groups, once their scope was limited, were usable as grounds for explaining why output can change when reasoning runs longer. Neither, though, is a study that deals directly with the naturalness of Japanese prose. As of 2026-09-02 I still have not found a study that compares effort high and medium directly on the naturalness of Japanese. The two in the third group did not hold up as grounds. All five looked like grounds when the AI collected them on 2026-08-31, and yet when a different AI read them again on 2026-09-02, two turned out not to be grounds at all. That is why this site has an AI other than the one that collected them read them again the same day and check.

Why I separated the writing role from the checking role

The assessments split. So that same day the AI agent stopped putting the setting of Opus 5 at effort medium first in line to try. Which setting to use was left undecided. Polishing and drafting fixes went back to Sonnet 5 as before, and the effort of the polishing workflow went back to its original high setting. The prose a checker rates as natural and the prose I read and feel is natural are not necessarily the same. So to settle this judgment, I first have to read the drafts side by side with their names hidden and decide which draft's Japanese fits my own sense. Beyond that, the same experiment has to be repeated on other material.

The same experiment did make one thing clear, though. The draft written at lowered effort carried three mismatches with the material. They were a wrong count, and a statement that settled something that had not been measured yet. Recalculating with a model at raised reasoning caught all three. In this experiment the writer was better with reasoning lowered, and the checker was better with reasoning raised. So on the same day I approved running the role where an AI writes and the role that checks what was written as separate roles. Once the Japanese is written, a model at raised reasoning checks the facts and the logic. The checker does not rewrite the text, and returns only the places it flags and the grounds for flagging them. The writer takes those comments and changes the text as little as possible. I also decided that the writer does not copy the checker's suggested sentences as they are, but rewrites them in its own wording. If a model at raised reasoning rewrites the prose directly, the sentences turn unnatural again.

The next day, 2026-09-01, I read two articles written at the deep reasoning setting, and sentences that turn a verb into a noun and then take it up with a padded predicate, such as continues or is repeated, stood out. The habit had stayed even with effort set deep. I said on the spot that it was hard to read, and the AI agent added this pattern to the rules and to the mechanical check. On 2026-09-02 I dropped the proposal to move the Japanese writer to Opus 5 at effort medium altogether. I decided that the session's leader model writes the final draft, as it has all along. My reason is that building a correct writing procedure and holding the writer to it serves the goal of good Japanese better than changing the writer's model or settings.

How to run the same check in your own operation

If you have an AI write Japanese and feel the prose changed when you changed the reasoning setting, I recommend checking in this order.

Separate having it write from having it check, and state the effort for each. Do not let it silently inherit the session's deep setting
When you write an impression into a rule, write into the body of the rule that you will record what comes out of running the AI under it and check it later in review
Make several drafts from the same material and compare them in several ways with the draft names hidden. Match the checker's quotations against the original text before tallying, and fix the order in which you present them
Have the studies you use as support read again the same day by an AI other than the one that collected them, and have it check that they exist, that the bibliographic details are right, and that the claims match

My impression became a rule in a day. In the experiment on that same day, the assessments split. The next day I found that the habit of turning verbs into nouns stays even with effort set deep. Because I recorded the disagreement as it stood instead of claiming proof, the material for the next check is there in the record. I hope this record reaches people setting out to check the same thing in their own operation.

Research I referred to

For all of them, as an independent check on 2026-09-02, I fetched the original text and verified that the bibliographic details and the claims matched.

A study that measured which language reasoning models tend to think in, and how accuracy changes when the language of thought is pinned to the user's language, measured on six open-source reasoning models and on short answers in math and science. When Models Reason in Your Language: Controlling Thinking Language Comes at the Cost of Accuracy (Jirui Qi, Shan Chen, Zidi Xiong, Raquel Fernández, Danielle S. Bitterman, Arianna Bisazza. Findings of the Association for Computational Linguistics: EMNLP 2025) https://aclanthology.org/2025.findings-emnlp.1103/
According to this paper, an evaluation of multilingual chain-of-thought reasoning on the three sides of performance, consistency, and faithfulness reports that thinking written in English is easier for a model to draw on. No report of noise from translation is included. A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning: Performance, Consistency, and Faithfulness Across Languages (Raoyuan Zhao, Yihong Liu, Hinrich Schütze, Michael A. Hedderich. Findings of the Association for Computational Linguistics: EACL 2026. arXiv:2510.09555) https://aclanthology.org/2026.findings-eacl.276/
A joint study by Anthropic's alignment research team and outside researchers that constructed and showed task types where extending reasoning time lowers scores. Inverse Scaling in Test-Time Compute (Aryo Pradipta Gema, Alexander Hägele, and others. Transactions on Machine Learning Research, 2025. arXiv:2507.14417) https://arxiv.org/abs/2507.14417
Reading its text showed that this source reports the opposite of the claim I was about to attribute to it. It is not used as grounds for any claim. COIG-Writer: A High-Quality Dataset for Chinese Creative Writing with Thought Processes (Yunwen Li and others. arXiv:2510.14763, preprint) https://arxiv.org/abs/2510.14763
A source that turned out to cover the generation of critiques in a set format rather than literary writing. It is not used as grounds for any claim. When Reasoning Supervision Hurts: TTCW-Based Long-Form Literary Review Generation (Jinlong Liu, Mohammed Bahja, Mark Lee. arXiv:2605.20364, preprint) https://arxiv.org/abs/2605.20364

Materials referred to
I confirmed the values of the effort parameter (low, medium, high, xhigh, max) and the default. Effort (Claude Platform Docs. Text fetched 2026-09-02) https://platform.claude.com/docs/en/build-with-claude/effort

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

Deciding the criteria for excluding an AI-collected research topic candidate

matsumotory — Wed, 12 Aug 2026 19:05:48 +0000

The problem and the outcome in brief

I collect the problems I run into in daily development and operations as candidate research topics, rather than letting them end with an on-the-spot fix. An AI agent automatically reads the material and drafts candidates (I will call this role the extraction agent), and a person decides whether to adopt them. In this process I narrowed the reasons for excluding a candidate down to two. This article explains how that came about, and what the automatic extraction produced when I ran it on the day I rewrote the criteria. The point is that a criterion for exclusion has to be one whose truth the extraction agent can judge from the material alone. I also cover how to write records so they can serve as candidates later, and the division of roles that leaves adoption and confirmation to a person. This is for people who want to pick research or writing topics out of their own development records.

Deciding the criteria for exclusion

I record the problems I find in daily development and operations as candidate research topics. An AI agent extracts candidates from development records, following a procedure kept in a repository I run on my own for research drafts and operations records, and that same procedure also decides which candidates may be excluded before they go into the record. On 2026-07-15 I rewrote these exclusion criteria. Before the rewrite, I judged exclusion with a single question: does a candidate still stand as a research topic and look valuable once you cut it loose from the circumstances of the organization doing the development? The extraction agent excluded any candidate that did not satisfy this question. The criteria after the rewrite limit the reasons for exclusion to two. The first is a candidate that does not generalize, meaning one whose context does not carry to a reader who does not know the internal circumstances of the development. The second is a candidate whose confidential information cannot be separated out, meaning one where no change of wording removes the part that has to stay hidden, such as the name of an unreleased product. Every candidate that passes those two goes into the record in full, and I do not use an assessment of value as a reason to exclude; I write it instead as an observation attached to the candidate.

The person who decides adoption judges whether a candidate has value. The extraction agent does not pre-empt that judgment. If the extraction agent excludes a candidate up front on the grounds that it seems to have no value, nobody can verify afterward whether the candidate really had none. On top of that, the extraction agent comes to keep only the candidates that look likely to match the wishes of the person who decides adoption, and the range of collection itself narrows to what matches that person's interests.

Up to this point this has been about candidate research topics, but around the same time, I also made a rule change in the opposite direction on the blog that carries this article. I was increasingly writing an article and only then finding it too thin to keep, so for the high-priority candidates among the article proposals, I added a condition: that the exchange between the AI and the person, that is, what was tried, what happened, and what was fixed and how, can be copied out of the material in concrete form. In other words, for research topic candidates I removed a condition the extraction agent has no way to verify, and for article proposal candidates I added a condition the extraction agent can verify on the spot. Both changes rest on the same principle: a criterion for exclusion has to be one whose truth the extraction agent can judge from the material alone.

Along with this principle, here are the five things I hold to across the process from collection to aggregation.

Only two kinds of candidate may be excluded: ones that do not generalize, and ones whose confidential information cannot be separated out. An assessment of value is written as an observation attached to the candidate, and is not used as a reason to exclude.
Excluded candidates stay in the record too, with the reason for exclusion attached. Without a record, you can no longer tell afterward whether the range of collection was narrow or the material simply was not there.
In the recording step, write the conditions for reproducing the problem, meaning what happened when you did what, and the place where that event can be confirmed, in the same place as what you did about it.
Material with only a single failure is too thin in content to become a candidate on its own. I wait until similar failures accumulate and then pull them into a single candidate.
In the aggregation step, a person makes the adoption decision and confirms it. The extraction agent hands over every candidate with a proposed judgment attached, and does not cut candidates on its own by reading ahead into what the deciding person wants.

What the automatic extraction produced on the day I rewrote the criteria

On the day I rewrote the criteria, the AI agent ran the automatic extraction once under the new ones. Here are the numbers from that record.

Category	Count
Candidates received	60
Raised as new candidates	24
Absorbed into an existing candidate	15
Dropped as duplicates of other candidates	21
Excluded under the two criteria	0

Not a single candidate met the two criteria. What actually did the narrowing was the 21 folded together as duplicates and the 15 absorbed into existing candidates.

That zero cannot be read as the effect of the rewrite alone. In the same change, I also added viewpoints for the extraction. Until then there were only seven viewpoints, named after fields such as ways of thinking about design, or privacy. To those I added seven viewpoints for how to compare material, such as comparing the gap between an estimate and a measurement in numbers, or questioning a pass result itself. The extraction record shows that 11 of the 24 new candidates came mainly out of the viewpoints I added. Because I bundled the rewrite of the criteria and the addition of viewpoints into a single change, I cannot separate which of the two the zero belongs to. It can also be read as the criteria being so loose that nothing gets excluded at all. The AI agent kept even the candidate that looked most specific to my own setting rather than excluding it, on the grounds that the same thing happens in other environments, and that judgment is in the record. Still, I have not run the measurement enough times yet to rule that reading out.

The level of detail that makes a record usable as a candidate later

Even if you keep the range of collection wide, a record that holds nothing but what was done about the problem cannot serve as a candidate later. The rules document for an app I develop at my own company carries a dated record of an accident that happened when an operation with side effects was sent to an external service. The address used to test user registration turned out to be a real one, and the confirmation email reached a stranger. The same section also states that the message is recorded in the authentication log on the external service side. There is a list of checks to prevent a repeat, and one item stops the operation if a real domain is included. The conditions for reproduction, meaning what happened when you did what, and the place where it can be confirmed, sit in the same place as what was done about it, so the record can serve as a candidate later. The third of the five conditions I listed above refers to this form of record.

The five conditions for aggregation, and who decides adoption

I use five conditions to decide whether to adopt a collected candidate as a research topic I will actually start work on.

Who is responsible has been decided
When it starts has been decided
Where the results will be published has been decided
The first piece of work has already been started
It has passed the check for whether it generalizes

If even one is missing, I do not adopt it. I lower its priority or move it to hold. The extraction agent is responsible up to handing over every candidate with a proposed judgment attached, and for a candidate it has not judged yet it writes "not judged" rather than leaving the field empty. Otherwise you can no longer tell afterward whether it was judged or skipped.

I also have a measurement of how many the deciding person withdraws. On this platform, on 2026-07-27, I went through all 60 article proposals and decided which were needed, keeping 45 and withdrawing 15. The reasons for withdrawal fell into three patterns. The largest was candidates where the AI only appears as something called or something worked on, with no exchange between the AI and a person at the core, and that accounted for 60 percent of the 15 I withdrew. The rest were candidates that amount to an explanation anyone who knows the field could write, and candidates raised from thin material with only a single failure. Checking all 60 is a measurement on the article proposal side, but if the extraction agent does not pre-empt the judgment of value, that judgment falls to the person deciding adoption. When the range of collection is kept wide, I take this one quarter as a rough guide to how many the person will withdraw.

The design of separating the collection step from the step that decides adoption is not new in itself, and it is already in the handbook for systematic literature reviews. The pattern of reading across after-the-fact incident records to find a common theme, rather than stopping at an individual fix, is also established, as laid out in a standard account in the operations field. Compared with that pattern for incident records, I changed two things in my own operation. I widened the target from incidents to everyday problems and review comments as well, and I made the destination of the aggregation a candidate research topic.

What I hand to an AI agent, and what I keep with a person

In this operation, for candidate research topics, an AI agent automatically handles reading the material, drafting candidates, and attaching a proposal, and a person confirms whether to adopt. Outside measurements confirm that this division of roles makes sense, too. A human-evaluation study that enlisted more than 100 researchers judged ideas from an LLM to be more novel than ideas from experts, while their feasibility came out somewhat lower. A survey that generated more than 200,000 ideas reports that AI ideas cluster near the literature they started from. The latter is a result showing that the more the collection step is handed to an AI, the more the candidates gather into a narrow range. I take both results as support for keeping the adoption judgment on the human side.

There is a shared convention for recording what an AI agent does as well, the OpenTelemetry semantic conventions for GenAI, and work is under way there to settle the names and meanings of the items to record. The document's status is still Development, though, so the names can still change. As of today, what an AI can take over is the first-hand record of what was called when and what came back, and the judgment that raises that into a problem worth working on still sits with a person.

Whether this process has reached a research result is something I cannot speak to yet. What I have is a little under two months of material even at its longest, and three weeks of records since the rewrite, so measuring whether the candidates I collected made it to acceptance is still ahead. Even so, when you are torn between widening and narrowing the range of collection, start by checking whether you can tell from the material alone whether something meets your current exclusion criteria.

Research I referred to

Here are the five outside primary sources referred to in the article. I checked the source page for all of them as of 2026-08-06. For the two arXiv items, I checked the abstract page.

Chapter 4 of the Cochrane Handbook (designing the search step) https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-04
The chapter on postmortem culture in Google's SRE book https://sre.google/sre-book/postmortem-culture/
The document in the OpenTelemetry semantic conventions for GenAI that defines the unit for recording what an AI agent does https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md
Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers. A human-evaluation study by Si, Yang, and Hashimoto that enlisted more than 100 natural language processing researchers (submitted 2024-09-06) arXiv:2409.04109 https://arxiv.org/abs/2409.04109
AI Research Agents Narrow Scientific Exploration. A study by Tang and Yang on whether AI research agents narrow the range of scientific exploration (submitted 2026-05-27, revised 2026-07-11) arXiv:2605.27905 https://arxiv.org/abs/2605.27905

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

At companies where AI writes 80 percent of the code, has development become 80 percent faster?

matsumotory — Wed, 12 Aug 2026 19:05:47 +0000

Summary

I hold a hypothesis: what decides the value of software from here on may be the autonomous continuity in which software makes software. To verify that hypothesis, I ran two passes of research through the public primary sources of the major companies that build AI models. The research asked how far each company officially says it uses its own AI models in its own model development, and how far the companies and outside researchers have been able to verify with actual measurement the claims of acceleration, of how much faster development has become as a result.

I read four kinds of sources: each company's announcements, the documents that gather a provider's evaluations of performance and safety, published when it releases an AI model (system cards), research papers, and an economic estimate in which independent researchers rework the figures each company published within a calculation framework of their own. I read all of them down to the body text, not just summaries. From the research I found both large numbers that support the hypothesis and conclusions from the AI companies themselves that weaken it.

In this article I go through what the primary documents of the three companies actually say, with the definitions of the numbers attached. I too operate a chain of the same structure, in which software makes software, together with AI at the scale of an individual. I add the measurements from running it, and explain how far this hypothesis can be claimed now and where it stops being claimable.

Let me lay out the structure of the body up front. First, I confirm the exact definition of the figure that says AI writes 80 percent of the code inside a company. Next, I restate the hypothesis as a structure in which the gap between models keeps widening the gap between the next models. Third, I recount how the methods I had settled on earlier for the convenience of my own operation matched the current recommendations of these companies, which I read afterward. Last, I read the judgment in which the parties themselves conclude that they cannot attribute the acceleration of their own progress to AI.

What you can take away

This article is for people who are building a setup where they develop alongside AI agents. It is also for people who want to know from primary sources where the story of AI building models actually stands right now. In this article, I explain two insights. The first is how to read acceleration figures, the numbers for how much faster AI has made development. Each company describes the same acceleration as large in a promotional setting and as small in a safety evaluation. So you must not line up numbers with different definitions and compare them as they are. I show this way of reading with quotations from the original text. The second is a lesson that applies directly to individual development. What I settled on is this: the lower the cost of verification in a domain, the larger the effect the chain shows there. I also explain the conclusion that what has the highest value to hand over to AI is the record of judgment, of why a person decided the way they did.

The problem and what came of it

To check my hypothesis against reality, I read the public primary documents of the major companies and an independent economic estimate down to the body text. After the first pass, I went back over the premises to make my conclusion more exact. My material was those public documents, together with measurements from the running of this platform itself.

The hypothesis

What AI can write moves closer to what anyone can build. If the value of building software itself keeps thinning out under competition, value should remain somewhere else. I think that place is the autonomous continuity in which software, once built, builds the next software. I build this membership platform for a technical blog together with AI: the articles, the search, and the conversation feature alike. Including the chain that records the discussions held here, turns them into content, and returns them to the knowledge of that AI itself, this platform is also a live experiment testing this hypothesis.

Still, a hypothesis is a hypothesis. To check it against reality, I went and read public primary documents down to the body text, to see whether this is really happening at the leading edge and how far the companies and outside researchers have been able to verify it.

What each company officially says

First, I will quote only the official words of the parties involved. In an announcement dated 2026-07-09, OpenAI wrote that over the preceding six months, the share of its research compute allocated to running AI that writes code internally had grown a hundredfold. In the same paragraph, though, it notes on its own that this is a measure of usage and not a number that measures research progress itself. In an official explainer article, Anthropic published that, as of May 2026, its own models wrote more than 80 percent of the code integrated into its codebase. That 80 percent has a footnote as well, which gives it the conservative definition of the share of lines integrated into production that can be attributed to a model, and goes as far as writing that line count is a measure of quantity and not of quality. Google's chief executive said in a talk that 75 percent of new code inside the company is AI generated and has passed engineer approval. This one is defined as the share of generation that passed approval, so a few lines emitted by a completion feature can enter the numerator. The two figures, 80 and 75, look close, but their definitions differ, so they cannot be compared side by side.

In the same explainer article, Anthropic has also published an internal evaluation in which the rate at which a model beats human judgment on choosing the next research move rose from 51 percent to 64 percent. As for OpenAI, reports carry an announcement that a higher-tier AI model autonomously performed post-training, the additional training done after a model is built, on a lower-tier model. An OpenAI employee added a note to this, though: it did not build the training procedure from scratch but adapted its own post-training configuration for a small model, and for humans that would be about two weeks of work for two researchers. OpenAI's system card, which gathers its evaluations of performance and safety, also states plainly that reliably designing and executing a complete post-training procedure across diverse models is not yet possible. Even so, one of Anthropic's co-founders goes so far as writing, in an essay, that it may be one or two years until the point where the current generation of AI autonomously builds the next.

Read only the disclosures up to this point, and the chain in which software makes software looks like it is already functioning in full.

The structure where the gap between models widens the next gap

The competitive picture is not decided by whether a company has the chain in which software makes software. Because every company has begun to depend on the AI models that carry the chain, today's gap between models becomes the performance gap of the next models, and that gap keeps widening, so latecomers find it harder and harder to catch up. More than whether a company has the chain at all, what decides the competition is that the gap in the results the chain produces goes on widening with every turn. This is my hypothesis restated more exactly. It is not a verified fact, and I write it as my own thinking.

I confirmed this reading most strongly in a passage of an Anthropic system card. It says that, on the grounds that recent models have the ability to accelerate their own development, they implemented an intervention that, for requests aimed at developing frontier large language models, lowers the model's effectiveness in a way invisible to the user. In other words, Anthropic throttles the power its own model holds to build the next model whenever the request comes from someone else. The party writes this much in its own document, so I read it as support for that power sitting at the center of the competition.

On the other hand, when I read the primary sources, I found that you cannot draw a line among the leading companies between those that have the chain and those that don't. The first to publish the most concrete instance of a closed chain was Google. AlphaEvolve is a coding agent that runs on Gemini. A function it found keeps recovering 0.7 percent of the compute in the company's data centers. There was also an improvement that made the whole of the core computation of training 23 percent faster on average, which cut Gemini's training time by 1 percent. Google adopted a circuit design that AlphaEvolve proposed for the next generation of TPU. On the circuit, though, the authors of the paper themselves add that an existing synthesis tool had independently found the same improvement. The paper states plainly that this is a new instance of Gemini optimizing its own training process through AlphaEvolve. Google published this about a year before the other two companies began disclosures of the same kind. And these are measured numbers from improvements deployed in production, not a self-reported productivity survey. Google's disclosure was modest, but its example of the chain was the most concrete.

The chain I run at an individual scale

I operate a chain of the same structure at the scale of one person. Let me start with the layer of rules documents. The documents that set down the discipline of this operation come to 54, counting the one at the top. Every time I point something out, the AI adds that lesson to the rules documents on the spot. On revision, rather than stacking additions, the AI rewrites the whole body so that only the current rule stands there. Inside the documents, 248 places state their origin as a dated remark or comment of mine. That is a count of occurrences, so the same lesson gets counted more than once across documents. Even so, the default I keep for this operation is a form in which every decision traces back to when it was made and on whose judgment. The other layer is the raw record. In the body of a commit, the AI writes, on top of what changed, why it changed, which comment and which judgment it followed, and what it verified. The commits piled up over these 20 days come to 849 when counted with merges and squashed changes excluded. Before and after the day I decided to write fuller commit messages, the median length of the body moved from 223 characters to 312. It is not a controlled comparison, so I cannot claim causation, but I have been able to measure the correlation: once the rule was written, the writing changed.

The match came out most clearly on the second pass, when I read what each company currently recommends. One company's official guide recommended keeping one record per lesson, and writing down why it mattered as well. Official documents also carried guidance to version prompts and rules with git in your own repository rather than entrusting them to an external management feature. A research conclusion that raw records should be kept alongside summaries rather than replaced by them points the same way. So does the judgment, near-identical across the three companies, that acceleration is concentrated in execution and is not reaching judgment. The methods I had settled on earlier for the convenience of running the operation matched, one after another, the current recommendations I read afterward. This match between methods is the center of what the two passes of research told me.

That said, for my own chain too, the quantity that matters remains unmeasured. How much did output rise per unit of AI capability put in? In fact, the authors of the independent economics paper write, on their own, that no one has been able to measure this quantity, not even at the leading companies. They release almost none of their internal metrics, for competitive reasons. Then if I can measure the same quantity on a small chain at an individual scale and publish it, that becomes primary data of a kind those companies do not release.

The most modest judgment came from the parties themselves

If I do not write this part, this article becomes nothing but a summary of promotion. So I write the side that weakens the hypothesis just as fully.

The strongest counterevidence was Anthropic's own judgment. Its April 2026 system card, while granting that the growth of its capability had turned steep, goes as far as writing that the growth it could identify is confidently attributable to human research and is not due to AI assistance, and that it confirmed this by interviewing the people involved. Even with employees self-reporting a fourfold output, combining that with an estimate of its impact on progress put the overall multiple below two. The judgment does not change in the latest system card, released on 2026-07-24, which says the acceleration is concentrated in engineering execution rather than research judgment. OpenAI's system card also states plainly that none of its three new models reaches the threshold of High, the highest risk level, in the AI self-improvement evaluation, and an independent evaluation body judged that fully automated AI research and development will not become possible. In the independent economic estimate, the condition needed for self-sustaining acceleration is a 15 percent gain in productivity per unit of capability, and the measured figure, taking the parties' self-reports at face value, was 9 percent. It does not reach the condition. The paper closes, though, by saying that it appears to be strengthening.

And one pattern common to the three companies comes into view. Each company reports the same acceleration at its maximum in a product announcement and at its minimum in a safety evaluation. One party put out a figure of eight times as much code integrated per day compared with 2024. That same party concluded from the same internal data that overall progress was less than double, while an outside evaluation body read more than double from the same data. Unless you check that definitions and measures agree before you check that numbers agree, you can build two opposite pictures of the same company.

The more primary sources I read, the more modest the wording of the claims becomes. The paper's authors write their own reservations, the party behind the system card itself denies the attribution, and the party that put out the usage figure notes on its own that it is not a measure of results. The most modest telling of the story that AI has started building AI was in the primary documents of the parties, the ones who should have been claiming it most strongly.

Where the hypothesis stands, neither denied nor confirmed

The economics paper carries one reservation that matters for this hypothesis. It predicts that acceleration of narrow capability can run ahead of acceleration of broad capability, and that the narrow acceleration should concentrate in domains where verifying an answer is cheap, such as software development and mathematics. If what my hypothesis points at is this narrow acceleration, it has not been denied yet. And as long as the quantity at its core is unmeasured, it has not been confirmed either.

Let me narrow down what can be said now. If a low cost of verification is what decides the effect of the chain, then adding more checks before publication is the obvious move. If acceleration concentrates in execution and does not reach judgment, then the record of judgment is what has the highest value to hand over to AI. I design my own operation on the basis of these two expectations. I will keep measuring here whether those two expectations were right, and writing down the results.

Research and primary sources I referred to

The numbers and quotations in the body are all as the documents read when I checked them on 2026-07-26. This article is an analysis that organizes what public documents state, and it does not represent the views of any of the companies named.

When AI builds itself (Anthropic Institute) https://www.anthropic.com/institute/recursive-self-improvement
Claude Mythos Preview system card (Anthropic) https://www-cdn.anthropic.com/08ab9158070959f88f296514c21b7facce6f52bc.pdf
System card for Claude Fable 5 and Claude Mythos 5 (Anthropic) https://www-cdn.anthropic.com/d00db56fa754a1b115b6dd7cb2e3c342ee809620.pdf
Claude Opus 5 system card (Anthropic) https://www-cdn.anthropic.com/c5fbac3f0b1280a933ebd26d3cb8bb9f5bdeaf48/Claude%20Opus%205%20System%20Card.pdf
The GPT-5.6 announcement and system card (OpenAI) https://openai.com/index/gpt-5-6/ https://deploymentsafety.openai.com/gpt-5-6/gpt-5-6.pdf
The AlphaEvolve paper (Google DeepMind) https://arxiv.org/abs/2506.13131
The Economics of Recursive Self-Improvement (researchers at METR and elsewhere) https://elasticity.institute/rsi-paper.pdf
The Adolescence of Technology (an essay by a co-founder) https://darioamodei.com/essay/the-adolescence-of-technology
An article on a talk stating that 75 percent of code inside the company is AI generated (Google) https://blog.google/innovation-and-ai/infrastructure-and-cloud/google-cloud/cloud-next-2026-sundar-pichai/

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.