DEV Community: matsumotory

Running six AI agent subtasks at once took 55 seconds longer than running them one at a time

matsumotory — Wed, 11 Nov 2026 08:26:04 +0000

Summary

In some runs, when I ran the subtasks of an AI agent that solves coding tasks at the same time, the total time was longer than the sum of running them one at a time, even though every subtask finished. HTTP 429 rate-limit responses did not appear in the output of the harness, the program that calls the AI model to solve the coding tasks. Neither the number of finished subtasks nor the harness output showed that the total time had grown or that 429s had occurred.

With six subtasks at once, the total time was 280.1 seconds, while running the same six tasks one at a time took 225.0 seconds in total. In a run of 12 concurrent subtasks that each return a short sentence, 65 of 93 requests got a 429. When the last 300 characters of each subtask's standard error and its error string in that run were searched, a rate-limit message was found for 0 of the 12 subtasks.

The six-subtask run did not record the HTTP status of each request, so I cannot tell whether any 429s came back or how many seconds the waits after them took. Each total time was measured only once, and I did not separate out the effect of the previous run. The records also cannot tell whether 429s would disappear if concurrency were limited to one or two subtasks.

What you can take away

This article is for people who build systems that run AI agent subtasks in parallel.

A table that compares the total time of concurrent runs with the sum of one-at-a-time runs.
How the median duration ratio of 8.21x was distributed across individual tasks.
A table, split by elapsed-time interval, of the 429s that never appeared in the output, as observed in per-request records.
The reasons the AI I develop with gave for not adopting a concurrency cap of three or staggered start times.

The article is based on the run records and design documents of a project I am working on.

Total time longer than the one-at-a-time sum even when every subtask finished

Together with the AI I develop with, I ran a program that solves coding tasks concurrently and measured the total time. In the September 2026 runs, the total time was longer than the one-at-a-time sum even though every task finished. In the October 2026 runs, HTTP 429 rate-limit responses did not appear in that program's output.

The terms used in this article are as follows.

Harness: a command-line program that calls an AI model and runs tools to solve a coding task
Runner: a program I wrote that starts multiple subtasks and collects their results
Subtask: a single job in which the runner starts the harness once and has it solve a task
Heavy subtask: a subtask that solves a bug-fix or implementation task by calling tools repeatedly, so it sends many requests to the endpoint and their bodies are large
Measurement program: a program separate from the runner that starts subtasks concurrently and measures the time they take

The table shows the results of running heavy subtasks with the measurement program, three at once and six at once.

Concurrent subtasks	Total time	Sum of running the same tasks one at a time
3	181.2 s	100.8 s
6	280.1 s	225.0 s

The six-subtask run was 55.1 seconds longer than the one-at-a-time sum. In both runs every task finished, so the number of finished tasks did not reveal the difference. Because running concurrently did not shorten the total time, the AI I develop with decided to keep the number of subtasks the runner starts at once at its default value of 1.

The 55.1-second difference compares single measurements. The three measurements, three at once, six at once, and one at a time, ran back to back without a pause, so I did not isolate the effect of rate limiting carried over from the previous measurement.

How to read the median duration ratio of 8.21x

For each task, I call the time it took when run concurrently divided by the time it took when run alone the duration ratio. In the run of six concurrent heavy subtasks, the median of this ratio was 8.21x. The ratios for individual tasks, however, ranged from 0.50x to 21.71x, so not every task slowed down by 8.21x.

Comparing by the number of concurrent subtasks, six at once has a larger median and a larger maximum than three at once.

Concurrent subtasks	Median	Minimum	Maximum
3	4.52x	0.88x	13.72x
6	8.21x	0.50x	21.71x

8.21x is the average of the middle two of the six tasks, and it is not the ratio of any single task. The one-at-a-time time was measured only once per task, so if that single measurement was an outlier, the ratio would move a lot, but I cannot check whether it was. The two September 2026 runs did not record the HTTP status of each request, so I cannot tell whether any 429s came back or how many seconds the waits after them took.

429s that do not appear in the harness output

When the measurement program searched the harness output, it found no messages about rate limiting or retries. The local proxy that records the HTTP status of each request between the harness and the endpoint, on the other hand, had 429s on record.

I call a subtask that returns a single short line and uses no tools a single-round-trip subtask. In an October 2026 run of 12 of these at once, all 12 completed only after receiving a 429 at least twice. A rate-limit message was found for 0 of the 12 subtasks. The search covered only the last 300 characters of standard error and the error string.

For the heavy subtask runs, the measurement program searched the full standard output and standard error for strings such as 429, rate limit, quota, retry, and backoff. None of these strings were in the output, either in the two September 2026 runs or in the October 2026 three-subtask run in which 9 requests got a 429.

429s from concurrent short requests and from single-round-trip subtasks

When the measurement program sent short requests to the endpoint concurrently without going through the harness, the runs with 6, 8, and 10 requests got no 429s, and the run with 12 requests got a 429 for 2 of the 12. Each request asked for the sum of 1 and 1, and the runs were spaced 120 seconds apart.

With the single-round-trip subtasks that return one short line, 429s and the total time grew as the number of concurrent subtasks grew. The values in the table are single measurements, each taken at least 120 seconds after the previous one.

Concurrent subtasks	Total time	Requests	429s
3	21.8 s	7	0
6	90.3 s	22	6
12	157.6 s	93	65

In the six-subtask run, the total time was set by the slowest subtask. One subtask that received four 429s took 90.2 seconds, while the four that received none finished in 6.8 to 13.6 seconds. The remaining 2 of the 6 429s went to one other subtask.

429s alone do not explain why the single-round-trip subtasks were slow. When I subtracted the total request time recorded by the local proxy from each subtask's time, 13 seconds per subtask remained even in the three-subtask run with 0 429s. In a three-subtask run against a local mock server that waits 2 seconds and returns a fixed response, the remainder was 1.6 seconds. In the 12-subtask run, the longest request that got a 200 took 54.6 seconds. I do not know the reason for the gap between 13 seconds and 1.6 seconds, or why that request took 54.6 seconds.

This measurement does not reveal the length of the window the endpoint uses for rate limiting or its request limit. A 429 is a response that says a user sent too many requests in a given amount of time. It is defined in RFC 6585.

Per-request records for three heavy subtasks

In an October 2026 run of three heavy subtasks at once, all 11 requests sent within 9 seconds of the first request got a 200, and the first 429 came 13 seconds in, on the 12th request. When the 24 requests are split by elapsed-time interval, all 9 of the 429s fall between 13 and 22 seconds.

Time since the first request	Requests sent	200s	429s
0 to 9 s	11	11	0
13 to 22 s	11	2	9
After 22 s	2	2	0

In this run, all three tasks finished 85.0 seconds after the first start. In the array bug-fix task, the harness received four 429s by 16 seconds in, then sent no requests for 61 seconds, and got a 200 on a request at 77 seconds.

The sum of running the same three tasks one at a time was 100.8 seconds, longer than 85.0 seconds. That sum, however, comes from measuring each task only once in September 2026. The string-processing task took 74.5 seconds when run alone and 28.0 seconds in the October 2026 three-subtask run. Because the time for the same task varied this much, the AI I develop with judged that the difference from 100.8 seconds does not show that the run got faster.

A concurrency cap of three and staggered start times

Since 429s appeared even with three heavy subtasks, the AI I develop with did not adopt a rule capping concurrency at three. It also did not implement a mechanism that delays each start by one second. The reason was that the first 429 came 13 seconds in rather than right after startup, and the AI expected that staggering the starts would not change the spacing between requests after the 13-second mark. This expectation was not measured.

Of the 24 requests from the heavy subtasks, 21 had large bodies of 105 KB to 113 KB, but this measurement did not isolate whether the endpoint's limit depends only on the number of requests or also on the amount of tokens. I did not measure runs of only one or two concurrent heavy subtasks, so I also do not know whether limiting concurrency would make the 429s go away. The HiveMind paper evaluated a method that uses an HTTP proxy to coordinate contention among agents sharing the same API, mainly against a mock API server, so its results cannot be compared with the values in this measurement.

The unsupported claim in the first documents that blamed the endpoint

After the September 2026 measurements, the AI I develop with stated as fact in three documents that the endpoint was the cause of the slowness. A first-pass review that followed pointed out that the evidence was insufficient, so the AI revised the three documents to say that the records cannot determine the cause. In response to the same finding, the AI also changed the measurement program so that it stops without making a judgment when it cannot read the harness output.

Materials I referred to

RFC 6585 Additional HTTP Status Codes https://www.rfc-editor.org/rfc/rfc6585.html

Research I referred to

HiveMind: OS-Inspired Scheduling for Concurrent LLM Agent Workloads: The main evaluation was done against a mock API server, and the mock agents stop on unrecoverable errors. In the measurements in this article, every subtask finished.

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

The rendering of deploy as haibi stayed in documents and appeared in an AI agent's reports

matsumotory — Thu, 29 Oct 2026 08:41:23 +0000

Summary

In this article, a "translated term" is a technical term rendered into native Japanese or Sino-Japanese words that engineers do not use in conversation. The AI agent replaced only the translated term I had pointed out, 乾式 (kanshiki, a rendering of dry-run), in 47 files. 配備 (haibi, a rendering of deploy), which it did not replace, remained in documents and in script output, and it also appeared in the reports that the AI agent returned to me in chat. The records are in the repository of a service I am developing at my own company, and in a repository of my personal memory and the history of my rules.

On 2026-10-12, I pointed out that the translated terms appearing in reports were words the AI agent itself had written into documents and script output. The AI agent wrote a plan to fix, all at once, the documents that contain translated terms. As of 2026-10-17, the fixes under this plan had not yet been carried out.

In the service repository, as of 2026-10-17, there were 0 occurrences of 乾式 outside the archive directory. 配備 remained on 375 lines in 81 files in the same scope. 配備 also appeared in a document that a different AI agent wrote after my remark. This other AI agent had neither received the remark nor fixed any report. For that reason, these records cannot tell whether translated terms come back into reports after the reports have been fixed.

What you can take away

This article is for people who have AI agents write documents and who receive reports from AI agents.

In a table, I compare the translated term that a one-word replacement removed with the one that remained in documents, including how much of it remained.
I show the three-step plan the AI agent wrote to fix, all at once, the documents that contain translated terms.
I list, in date order, the translated terms I pointed out in reports after the replacement of 乾式.
I describe how 配備 also appeared in documents written after my remark.

The body is based on two sources. One is the records from 2026-09-16 to 2026-10-17 in the repository of the service I am developing at my own company. The other is a document that records how each rule came about, kept in the repository of my personal memory and the history of my rules. This repository stores, with its change history, the memory that AI agents read across sessions and the documents that record how each rule came about.

The translated term that disappeared with the replacement of one word, and the translated term that remained in documents

The AI agent replaced 乾式, the term I had pointed out, in 47 files. After that, 乾式 disappeared from everything outside the archive directory, which holds records that are not rewritten later. 配備, which was not replaced, remained in documents and code, and in 2026-10 it also appeared in the reports that the AI agent returned to me in chat. 乾式 turns dry-run, and 配備 turns deploy, into Sino-Japanese words that engineers do not use in conversation, and this article calls words of this kind translated terms. The AI agent made a plan to fix translated terms all at once, but as of 2026-10-17 it had not yet carried it out.

The counts in this article were taken in the repository of the service I am developing at my own company. The 47 files replaced on 2026-09-16 consist of 28 TypeScript files, 10 Markdown files, and 9 test files. This replacement removed 146 lines containing 乾式. 配備 was not included in this replacement.

The following table shows how much of the two translated terms remained outside the archive directory on main as of 2026-10-17.

Translated term	Replaced or not	Amount remaining outside the archive directory
`乾式`	Replaced in 47 files on 2026-09-16, removing 146 lines	0 occurrences
`配備`	Not replaced	375 lines in 81 files

On 2026-10-12, I pointed out where the translated terms in reports came from. My point was that words the AI agent itself had written in native Japanese in tools, scripts, and documents were also appearing in its reports. Among the commits that entered main between 2026-10-10 and 2026-10-17, none had a subject line showing it was a fix of terminology. Among the branches on origin, none had a name showing it was a fix of terminology either.

A plan to fix, all at once, the documents where translated terms were written, instead of a word list and an inspection

As ways to prevent translated technical terms, I did not adopt two methods: a word list and an inspection. Instead, the AI agent wrote a plan to fix the translated terms in documents and script output all at once, in the memory that the next session also reads.

In the first method, translated terms that should not be used are added to a word list one by one. In the second method, an inspection is added to find translated terms that appear in sentences, and those terms are then fixed. On 2026-10-12, I told the AI agent that I would not adopt either of these, giving as the reason that the list would become enormous.

On the same day, I instructed the AI agent to fix the translated terms it had written, in every file in the repository. Under the plan, the translated terms in documents and script output, not those in reports, are replaced with technical terms in one pass. The AI agent judged that if the translated terms in documents were fixed to technical terms, translated terms would not appear even if reports used the words in the documents as they were. It wrote this judgment in the memory as the reason for the plan.

In the steps of the plan, the AI agent first counts the occurrences of each target word with grep and writes a glossary. Next, in addition to documents and script output, code comments and the descriptions of MCP tools are also fixed, with the work split by area. The expected values of tests are also changed to match the fixed words. Finally, the result of the fixes is made into a PR and shown to me.

From the correspondence table of translated terms in the plan, the following table lists three pairs as examples.

Translated term	Term engineers use
`版`	`バージョン` (version)
`配備`	`デプロイ` (deploy)
`表`	`テーブル` (table)

The correspondence table of translated terms has 14 pairs, and 13 of them pair a translated term with the term engineers use. The remaining pair is not a translated term. It rewrites 実機 (literally, the real machine), a word that does not tell where or what was done, into the concrete names of the environment and the operation.

Other translated terms I pointed out in reports after the replacement of kanshiki

When I pointed out 乾式 on 2026-09-16, the AI agent replaced only this one word among the translated technical terms with dry-run. Later, from 2026-10-09 to 2026-10-12, I pointed out other translated terms one after another in the reports that the AI agent returned to me in chat.

When I pointed out 乾式, I told the AI agent that Japanese engineers do not say 乾式 in conversation. I also instructed it to write dry-run, because the katakana form ドライラン is unnatural as well. The replacement covered not only the body text of documents and code comments but also script output. The AI agent also aligned the expected values of the tests that check the output to the same dry-run.

The following lists, in date order, the translated terms I pointed out in reports in 2026-10.

On 2026-10-09, I pointed out 表, a translated term for table.
On 2026-10-11, I pointed out 版, a translated term for version, and 索引, a translated term for index.
On 2026-10-12, I pointed out 配備, a translated term for deploy, 版, and 木, a translated term for tree.

In the remark on 2026-10-09, I said that commonly used technical terms had been replaced with easy native Japanese words and were hard to read. I also said that the problem was not limited to the one word 表.

I had already pointed out 索引 on 2026-08-28, and on 2026-10-11 I pointed out the same word again. In that day's report, the AI agent had rendered memory index, which appears in English in a tool's description, into Japanese as メモリの索引.

Haibi appearing, before the plan was carried out, in documents an AI agent wrote after my remark

配備, the translated term for deploy, also appeared in documents that an AI agent wrote after my remark on 2026-10-12. On 2026-10-17, 配備 was in both a new document written by a different AI agent and a document rewritten by the AI agent that had received my remark. Even on this day, the AI agent had not yet carried out the plan to fix the translated terms in documents all at once.

On 2026-10-17, a commit that added a document describing future work and a line in the list of documents entered the repository of my company's service. According to the commit body, this document was written by an AI agent different from the one that had received my remark on 2026-10-12. Of the added lines, those containing 配備 totaled 4: 3 lines in the document and 1 line in the list.

In a later commit on the same day, the AI agent that had received my remark rewrote the document describing future work. It moved the rewritten document to the archive directory, which holds records. The rewritten document also had 3 lines containing 配備. 配備 also appeared once in the message of this commit.

As of 2026-10-17, 配備 remained in 81 of the files outside the archive directory. While translated terms remain in documents, they may also appear in documents written after a remark. However, these records cannot tell whether a translated term appears in reports again after the translated terms in reports have been fixed. This is because the different AI agent that wrote the new document had not received any remark about translated terms from me and had never fixed the translated terms in its reports.

The guidance on writing prompts published by an LLM provider says that the formatting of a prompt can affect the formatting of the response. This statement covers formatting, and word choice is not within its scope. For conversations between people and LLMs, there is a study of the phenomenon in which speakers choose words that align with their partner's words. This study covers the utterances of the conversation partner, not documents written on another day. A search on 2026-10-17 found no study that directly measured whether words in documents an AI agent itself wrote on another day appear in its later output.

There is also an example in which, within a single conversation, an AI wrote later answers in the same format as answers it had returned earlier. I described that example in another article, "Pass the conversation history as it is, and the AI starts imitating its own past answers". The records in this article are a case, across conversations, in which the same translated term appeared both in documents written earlier and in later reports.

Materials I referred to

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

How to attach a Playwright auth header without sending it to another origin

matsumotory — Thu, 29 Oct 2026 08:40:06 +0000

Summary

An authentication header added in Playwright reached another origin both with extraHTTPHeaders and with a method that checks the destination in context.route and then adds the header with route.continue. The header is the Cloudflare Access service token sent by the script that opens this site's staging. A service token is a credential that Access accepts in place of a person's login. The header added with route.continue was attached to the target of a redirect to another origin.

With a function that adds the header only after confirming that the redirect targets are also the staging origin, the header did not reach another origin. In the reproduction test on 2026-10-13, the result was the same in Chromium 145, WebKit 26.0, and Firefox 146. However, the navigation request that fetches the page's HTML reached the same server twice.

The three browsers were the ones provided by Playwright version 1.58.2. Neither the browsers for Playwright 1.61.1, which the lockfile specifies, nor the regular releases of Safari and Firefox were tested.

What you can take away

This article is for people who write scripts that check staging with an authentication header attached in Playwright.

It shows a TypeScript function that adds the header only after confirming that the redirect targets are also the staging origin.
For each way of attaching the header, it compares the headers that reached another origin in a table covering four browser versions, including WebKit 26.5.
It describes the extraHTTPHeaders leak and the redirect problem found in review.
It shows that, under the destination's CORS settings, browsers differed in whether a request carrying the header arrived.

The article is based on three sources: the records of the fixes to the script that checks this site's staging, the results of the reproduction test run on 2026-10-13 along with external materials checked on the same day, and the results of running the code in this article on 2026-10-28.

Attaching the auth header only to the staging origin

To send an authentication header in Playwright only to the staging origin, add the header after checking the origins of the redirect targets as well. Both a setting that adds the header regardless of destination and a method that judges only the destination before sending let the header reach another origin. On this site's staging, Cloudflare Access accepts a service token in place of a person's login. The script that opens staging puts this token in two headers, Client ID and Client Secret, and sends them.

The 2026-10-13 reproduction test used a function that adds the header after checking the redirect targets. Two local servers with different origins were started, and a page on one was made to send requests to the other. In each of Chromium 145, WebKit 26.0, and Firefox 146, the header did not reach the other server. A page that redirects to the other server could not be opened, because this function stops the request.

How addAuthHeaders works

addAuthHeaders is a function that checks the redirect targets and adds the header only when all of them are the staging origin. In exchange, it sends the navigation request that fetches the page's HTML twice, once to check and once to send. In the 2026-10-13 reproduction test, with a function that does the same processing, this request reached the same server twice in each of the three browser types. Cases in which the server's response differs between the first and second requests were not tested.

If the destination is not the staging origin, addAuthHeaders calls route.continue, which sends the request on, without the header. For a GET or HEAD navigation request to staging, it calls route.fetch, which sends the request first and receives the response, with maxRedirects set to 0. According to the description of maxRedirects, when it is 0, route.fetch does not follow redirects.

addAuthHeaders follows the destinations given by the Location header of each response, one at a time, within staging. If every destination it followed is staging, it adds the header with route.continue and sends the request. If another origin appears, or the redirects exceed five, it stops the request with route.abort.

For requests that are not navigations to staging, and for navigation requests other than GET and HEAD, it calls route.fetch only once, with the header added. It sets maxRedirects to 0 and returns the response to the browser. If the response is a redirect, it stops the request with route.abort.

The following addAuthHeaders is code written out for this article from the function tested in the 2026-10-13 reproduction test. When this code was run on Chromium 145 on 2026-10-28, the header did not reach the server on the other origin. The results of running it on the other browsers have not been checked.

import type { Route, Request } from "playwright";

// Read the staging URL and the header names and values from environment variables (the names are placeholders)
const STAGING_ORIGIN = new URL(process.env.STAGING_URL ?? "").origin;
const AUTH_HEADERS: Record<string, string> = {
  [process.env.AUTH_ID_HEADER_NAME ?? ""]: process.env.AUTH_CLIENT_ID ?? "",
  [process.env.AUTH_SECRET_HEADER_NAME ?? ""]: process.env.AUTH_CLIENT_SECRET ?? "",
};
const MAX_REDIRECTS = 5;

const isStaging = (url: string) => new URL(url).origin === STAGING_ORIGIN;
const isRedirect = (status: number) => status >= 300 && status < 400;

export async function addAuthHeaders(route: Route, request: Request) {
  // If the destination is not the staging origin, send the request without the headers
  if (!isStaging(request.url())) return route.continue();

  const headers = { ...request.headers(), ...AUTH_HEADERS };
  const method = request.method();

  if (request.isNavigationRequest() && (method === "GET" || method === "HEAD")) {
    // Send without following redirects, and check each Location target one by one
    let url = request.url();
    for (let redirects = 0; ; redirects++) {
      const response = await route.fetch({ url, headers, maxRedirects: 0 });
      const location = response.headers()["location"];
      if (!isRedirect(response.status()) || !location) break;
      url = new URL(location, url).href;
      if (!isStaging(url) || redirects >= MAX_REDIRECTS) return route.abort();
    }
    // Every target followed is the staging origin, so send the request with the headers added
    return route.continue({ headers });
  }

  // Send any other request only once, and return the response to the browser
  const response = await route.fetch({ headers, maxRedirects: 0 });
  if (isRedirect(response.status())) return route.abort();
  return route.fulfill({ response });
}

// Usage: await context.route("**/*", addAuthHeaders);

Headers that reached another origin, by way of attaching them

With extraHTTPHeaders, which adds the header to every request, the header was attached both to fetches that the page sent to another origin and to redirect targets. A method that judges the destination in context.route, which receives requests before they are sent, and adds the header with route.continue, which sends them on, was also tried. With this method, the header was not attached to the page's fetch but was attached to the redirect target. Three browser types in four versions were tried, and the results were the same in all of them.

In the 2026-10-13 reproduction test, two local servers with different origins were started. A page on one was made to send a fetch to the other. On the first server, a path that redirects to the other with a 302 was also set up. The other server allowed, in its CORS response, the names of the headers that the script added.

The browsers used were Chromium 145.0.7632.6, WebKit 26.0, and Firefox 146.0.1, provided by Playwright version 1.58.2. For WebKit, 26.5, built for Playwright 1.62.0, was also used. The browsers for Playwright 1.61.1, which this site's lockfile specifies, were not available locally and are not in the table. The results at the other server are shown in the following table.

Way of attaching the header	Request to the other server	Chromium 145	WebKit 26.0	WebKit 26.5	Firefox 146
extraHTTPHeaders	Page's fetch	Attached	Attached	Attached	Attached
extraHTTPHeaders	Redirect target	Attached	Attached	Attached	Attached
Judge the destination in context.route and use route.continue	Page's fetch	Not attached	Not attached	Not attached	Not attached
Judge the destination in context.route and use route.continue	Redirect target	Attached	Attached	Attached	Attached

The extraHTTPHeaders leak and the redirect problem found in review

The overflow check script, which counts horizontal overflow, at first put the service token used for Access authentication in extraHTTPHeaders. As a result, the token was also attached to requests to the Supabase API. However, the destination was this site's backend, and the connection used HTTPS.

To attach the header only to the destination origin, the AI agent doing the work removed extraHTTPHeaders from this script. Instead, it judged the destination in context.route, which receives requests before they are sent. Only when the destination was staging did it add the header with route.continue, which sends the request on.

On the same day as the fix that narrowed the header to the destination origin, the AI agent that reviewed the fix found a path through which the header passes to redirect targets. A header added with route.continue is also attached to the request to the redirect target. For that request, the context.route function is not called. The reviewing AI agent had in mind the redirect that Access makes when the token is invalid, and the forwarding from the purchase entry point to the payment service.

The description of route.continue says that the headers option also applies to redirects. The note on page.route says that if the response is a redirect, the function passed to route is called only for the first URL. On the other hand, the description of route.fetch, which sends the request first, gives the opposite guidance: to attach headers only to the original request, use route.continue. The approach of avoiding the leak to redirect targets with route.fetch and maxRedirects set to 0 was shown by a Playwright developer in issue 32939.

The destination's CORS settings and differences between browsers

Some browsers delivered a request carrying the header even though the server on the other origin did not allow the header name through CORS. The request arrived in three of the four versions tested. The destination's CORS settings cannot be relied on as a means of preventing the header from being sent to another origin. The 2026-10-13 reproduction test used extraHTTPHeaders, which adds the header to every request.

According to MDN's explanation of CORS, when a header that is not on the CORS safelist is attached, the browser first asks for permission with a preflight request. It sends the actual request after receiving permission. Chromium 145 in the reproduction test sent only the preflight request, which does not contain the header values, and did not send the actual request.

In WebKit 26.0 and 26.5 and in Firefox 146, no preflight request arrived, and the actual request carrying the header arrived. Every browser was a version provided by Playwright, and the regular releases of Safari and Firefox were not tested.

Materials I referred to

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

Four loopholes in the denylist approach to auto-approving AI agent commands

matsumotory — Thu, 29 Oct 2026 08:38:50 +0000

Summary

Before an AI agent runs a shell command, a desktop app shows an approval dialog. In an initiative I am pursuing, a check script that judges whether the app's features work clicks a button in this dialog on behalf of a person. If the decision to click is built with a denylist approach, which rejects commands by listing dangerous words, the script clicks the button even for commands in forms the list left out.

On 2026-09-15, an AI agent that read the changes before the merge found loopholes in the denylist implementation and raised them as four required findings. The AI agent that wrote the implementation did not fix the loopholes one by one. It rebuilt the implementation with an allowlist approach, which lists only the forms for which clicking is allowed. The unit tests of the rebuilt implementation contain 18 examples in which the button must not be clicked. When these 18 examples were given to the first implementation, the comparison run on 2026-10-12 found that it decided to click the button in 14 of them.

The allowlist implementation does not click the button for correct operations that it does not list. It also does not click the button for permission requests from the tool that writes files. For this reason, the AI agent that wrote the implementation changed the instructions given to the AI agent inside the app.

What you can take away

This article is for people who write programs that click the approval button for a person when it appears before an AI agent runs a command.

It shows the result of the comparison run in which the denylist auto-approval let through 14 of the 18 examples in which the button must not be clicked.
It shows, in a table of the decision flow, what the allowlist implementation checks before clicking the button.
It shows, in a table, the four loopholes in the denylist implementation and the reason each one leads to a decision to click the button.

The body is based on the work recorded on 2026-09-15 in the initiative I am pursuing, on the result of the comparison run on 2026-10-12, and on external materials checked on the same day, 2026-10-12.

The must-not-click examples that the denylist auto-approval let through

The first implementation of the program that clicks the approval button on behalf of a person rejected commands with a list of dangerous words. In the review before the merge, four loopholes were found in this implementation. The AI agent that wrote this program did not fix the loopholes one by one, and instead rebuilt the decision with an approach that lists only the forms for which clicking the button is allowed. When the 18 must-not-click examples in the unit tests after the rebuild were given to the first implementation, the comparison run on 2026-10-12 found that it decided to click the button in 14 of them.

The desktop app has a screen for conversations with an AI agent. Before the AI agent inside the app runs a shell command, the app asks a person for approval. The screen the app shows at this point is called the approval dialog. The approval dialog has three buttons: one that allows the command once, one that allows it and keeps the setting, and one that denies it.

A Python script that operates the app's screen from a program and judges whether the app's features work is called the check script. To run the checks without a person's help, the check script clicks the button that allows the command once on behalf of a person. Whether to click is decided by the approval decision function inside the check script.

The check script and the approval decision function were written by an AI agent separate from the AI agent inside the app. This AI agent is called the AI agent that wrote the implementation.

In the first commit of the PR merged on 2026-09-15, the approval decision function had a denylist of 17 dangerous words. This function did not click the button for commands that contained a word in the denylist. For the remaining commands, it decided whether to click based on whether the command contained the path or name of the working directory, and whether it contained an absolute path. This function is called the denylist implementation.

The approval decision function rebuilt in a later commit of the same PR clicks the button only for commands that meet all the conditions for clicking. This function is called the allowlist implementation.

The error was not in how the app that asks for a person's approval was built, but in the decisions of the program that clicks the approval button automatically. A program that clicks the button on behalf of a person replaces approval by a person. So if the decision goes wrong even once, commands that read or delete files outside the working directory also run without a person checking them.

The decision flow of the allowlist approach

The allowlist implementation, which lists only the forms for which clicking the button is allowed, splits a command at separators and checks the conditions for each resulting part. Only when every part meets the conditions on symbols, executables, and paths does it click the button that allows the command once.

In the approval dialog the app shows, the shell command about to be run is written on the line that follows the heading "Command:". This line is called the Command: line. The allowlist implementation splits the command taken from the Command: line into parts at shell separators. Each part is called a fragment.

If a fragment contains even one of certain symbols, the allowlist implementation does not click the button for that command. These symbols are called the forbidden symbols. The following table shows, for each step of the decision, what is examined and when the button is not clicked.

Step	What is examined	When the button is not clicked
Target of the decision	The whole command taken from the Command: line. No characters are cut off when it is taken out	When the command fails the condition of any other step at even one place
Splitting into fragments	Five kinds of separators: `&&`, `\	\
Forbidden symbols	12 kinds: dollar sign, backquote, tilde, less-than sign, opening parenthesis, closing parenthesis, opening brace, closing brace, asterisk, question mark, backslash, and ampersand	When a fragment contains even one of them
Leading executable	The first word of the fragment, and an allowlist of 15 executables such as {% raw %}`cat`, `cd`, `git`, and `ls`	When the first word matches none of the 15
Path validation	The output targets of `>` and `>>`, the destination of `cd`, and the other arguments of the fragment. All are treated as paths, and those that are not absolute paths are resolved relative to the conversation's working directory	When the resolved location is outside the conversation's working directory

For a command that fails even one condition in the table, the allowlist implementation does not click the button and returns the reason to the caller. It also does not click the button for a command when it cannot determine whether the conditions are met.

In their 1975 paper, Saltzer and Schroeder listed the design principle "Fail-safe defaults". This principle bases access decisions on permission rather than exclusion. The decisions of the allowlist implementation follow this principle.

The allowlist implementation does not click the button for operations not listed, even correct ones. It also does not click the button for permission requests from the tool that writes files. For this reason, the AI agent that wrote the implementation changed the text of the instructions given to the AI agent inside the app. The new instructions had the agent create files with echo and a redirect, and write paths as relative paths.

The conditions and loopholes of the denylist implementation

The denylist implementation, which rejected commands with a list of dangerous words, also narrowed the range in which it clicked the button with conditions on the working directory and absolute paths. Even so, this decision had four loopholes. In every loophole, the denylist implementation decided to click the button without examining part of the command or the kind of approval. Apart from the loopholes, it also decided to click the button for dangerous commands in forms not in the denylist, depending on the absolute path condition.

To commands that did not contain a word in the denylist, the denylist implementation applied two conditions in order. First, if the command contained the path or name of the conversation's working directory, it clicked the button. If neither was present, it looked for absolute paths. It treated as an absolute path any string in which a slash is immediately followed by an alphanumeric character, a dot, or an underscore. Only when there was no such string did it click the button.

On 2026-09-15, an AI agent that reads changes before the merge and returns findings read the PR that contained the denylist implementation. This AI agent is called the reviewing AI agent. The reviewing AI agent listed four loopholes in the approval decision function and made each of them a required finding.

In the approval dialog, the line that follows the heading "Command:" and gives the command about to be run is called the Command: line. The following table shows, for each of the four loopholes, what the denylist implementation did not read or did not examine, and the reason it leads to a decision to click the button.

Loophole	What was not read or not examined	Reason it leads to a decision to click
Truncation at 160 characters	The part from the 161st character on, in the string taken from the Command: line	Because it decides using only the truncated string, it does not read another command joined after a long command
A body without a Command: line	Whether the approval is an approval of a shell command	Because it treats the heading line as the command, it clicks the button even in approval dialogs shown for file writes or permission requests from MCP tools
The path or name of the working directory	The paths pointed to by commands joined after the path or name	Because it does not check external paths once the path or name appears, it clicks the button even for a command that reads a private key after a `cd` into the working directory
Shell symbols	The tilde for the home directory, command substitution written with a dollar sign and parentheses, backquotes, and the two dots that walk up to the parent directory	Because it does not examine the symbols, it clicks the button even for commands that point to the home directory, the output of another command, or the parent directory without using any word in the denylist

Separately from the four loopholes, the denylist also had too few words, and the only word starting with rm was the single entry rm -rf /. Among the must-not-click examples, three, rm -fr /, rm -r /, and rm -rf ~, differ in sequence and form from this entry. rm -fr / has the option letters in a different order, and rm -r / lacks the f option. rm -rf ~ deletes the home directory. So none of the three matches rm -rf /, and in the comparison run on 2026-10-12 each led to a decision to click the button.

According to the description of CWE-184, in a protection mechanism that relies on a list of disallowed inputs, an incomplete list is a weakness. The mitigation section of the same page asks not to rely only on detecting disallowed inputs.

Materials I referred to

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

How to use Claude Desktop Switcher for separate workspaces on one Mac

matsumotory — Fri, 16 Oct 2026 06:50:16 +0000

Using Claude for work, personal projects, and client work can mean switching more than an account. You may also want separate chat, Projects, Cowork, Artifacts, and Claude Design contexts, with Claude Code following the same environment when you need it.

Claude Desktop Switcher is a macOS menu bar utility for managing those environments. The desktop workflow is entirely graphical. Developers can optionally connect a terminal to an environment with the separate csw command.

In this guide, I’ll walk through installation, the three separation modes, everyday switching, and the details worth checking before sharing or deleting data. It follows the current English user guide and v0.24.7 release.

Product screenshot from the project website.

1. Understand what an environment means

CSW calls your existing desktop and CLI setup Existing Claude. New environments have dedicated data directories. You decide which supported components should remain separate and which should carry over from Existing Claude.

There are three behaviors:

Shared: the environment uses the existing component through a link. Changes to shared data can affect both environments.
Isolated: the environment starts with its own independent component.
Copied: the component is duplicated once, then each copy evolves independently.

This distinction matters. Sharing rules or history is an intentional connection between environments. Choose separation based on the data you actually want to keep apart.

CSW provides application-data separation. It should not be treated as an operating-system sandbox or a guarantee against every form of data exposure. Keep appropriate backups and follow your organization’s rules for accounts, connectors, and files.

2. Install the desktop app

Start with the official Claude Desktop app available on your Mac. Then:

Open the v0.24.7 release page.
Download Claude-Desktop-Switcher_0.24.7_universal.dmg. This build supports both Apple Silicon and Intel Macs.
Open the disk image and drag Claude-Desktop-Switcher.app into Applications.
Launch CSW. Its settings window opens, and a menu bar icon appears.

The first-run cards explain Existing Claude and the environment workflow. If the menu bar is crowded, you can reopen settings from the Dock icon. If the installer remains mounted, the settings window offers an Eject action.

The DMG contains the desktop utility. It does not install the csw CLI. You can skip the terminal section below if you only want the graphical workflow.

3. Create an environment and choose its separation level

Click New environment, give it a name such as Work, and optionally choose an icon and add a note. A note identifying its purpose and intended account helps when your list grows.

Separate everything

This is the recommended default. Supported components start independently, with nothing carried over from Existing Claude.

Choose it for work and personal accounts, separate clients, or projects where you want a clean starting point. It also allows the environment to run alongside another Claude instance.

Separate conversations and memory too

This mode reuses supported global configuration while giving the environment its own project conversation history, auto-memory, and input history.

The carried-over CLI components include global rules (CLAUDE.md), rule files (rules/), skills, plugins, tool permissions and hooks, and worktree configuration. This suits a separate account or purpose where you want familiar rules without carrying over previous conversations.

Separate the account only

This mode also carries over CLI project conversations, auto-memory, and input history. It suits a continuous workspace used with different accounts, for example where billing is separated by purpose.

The label needs one qualification: desktop connector and app settings, device identifiers, and session state still remain separate. It does not make every desktop setting shared.

For any mode, open Configure in detail, item by item to inspect the individual choices. Then click Create this environment.

The guide’s component-level explanation is especially useful here: most carry-over options concern Claude Code configuration. Desktop MCP connectors live in the independently managed desktop configuration, so plan to configure them for each environment.

4. Launch it and sign in

Select the environment and press Launch this environment. On its first launch, sign in through the official Claude app with the account intended for that environment. Subsequent launches normally retain that sign-in.

There are two launch patterns:

Environments with shared settings run one at a time. Quit the running Claude before opening another shared environment.
Fully isolated environments offer Launch alongside, allowing you to open them without first quitting a running Claude.

For example, you can create independent Work and Research environments and keep both open. If Research instead shares configuration with Existing Claude, use the single-instance workflow to avoid competing writes to shared files.

To identify a window, use Check the current environment in the menu bar menu. To find a particular environment’s window, select it in CSW and press Bring to front. Multiple Claude windows can share one Dock icon, so the Dock alone is not a reliable identifier.

To return to your original setup, select Existing Claude and use Launch Existing Claude. v0.24.7 can reopen it while recognized, fully isolated environments remain running. It refuses to proceed if a running process cannot be verified as a known fully isolated environment. Follow the prompt to quit other instances when necessary.

5. Optional: connect Claude Code

This section assumes Claude Code is already installed and the claude command is available. The separate csw tool manages environments; it does not install Claude Code.

The terminal inside a Claude Desktop app launched through CSW inherits that environment. You can run:

claude

A terminal opened separately, such as Terminal or iTerm2, needs an explicit environment selection.

First, download the standalone csw asset from the release page. From the directory containing the downloaded file, the documented installation example is:

chmod +x csw
mv csw /usr/local/bin/
csw --version

Use a directory on your PATH that you can write to; /usr/local/bin/ is the example rather than a requirement. Updating the desktop app does not update this separate binary.

Then, for an environment named Work, run the documented command in your external terminal:

eval $(csw env Work)
claude

This points that shell’s CLAUDE_CONFIG_DIR at the environment. It does not rewrite your shell startup files or switch other terminal tabs.

Desktop and CLI authentication are separate. The first time you run Claude Code in an environment, complete its sign-in as well. For subscription usage, choose your Claude subscription account rather than an API-billed account. The built-in terminal and an external terminal pointed at the same environment use that environment’s CLI sign-in.

Check the current tab with:

echo $CLAUDE_CONFIG_DIR

A path ending in .context-switcher-claude/profiles/Work/cli-data identifies the Work environment. csw status reports CSW’s selected environment, which can differ from the environment of the terminal you are using. Launch alongside does not change that selection.

If you already customize CLAUDE_CONFIG_DIR, note that CSW looks for Existing Claude’s CLI data at the standard ~/.claude location. It does not automatically adopt a custom location or rewrite your existing setup.

6. Check sharing, and handle duplication carefully

In the environment detail screen, use Isolation check → Check now to verify whether links and isolated components match the configuration. The check is read-only. Its CLI equivalent is:

csw doctor

If it reports broken or missing share links, the documented repair command is:

csw doctor --fix

This can create or correct links when their declared source exists. Recent versions can also replace an empty placeholder directory with the expected link; they do not replace a nonempty directory. Read the reported result rather than assuming every issue is automatically repairable.

One Cowork caveat: the guide notes that Claude Code in desktop Cowork sessions does not load a linked ~/.claude/CLAUDE.md or ~/.claude/rules/. If your workflow depends on these shared rules, check that behavior before relying on it.

Duplicate is useful for copying an environment, but use New environment for a different account. Duplication copies Desktop cookies, so the duplicate can open with the original account’s sign-in. Logging out in either can log out the other. CLI Keychain sign-in is normally not copied, although a fallback .credentials.json file is copied if present. Plugins also have special handling and may need reinstalling in an isolated duplicate.

7. Updates, recovery, and deletion

Claude Desktop’s own update relaunch can bypass CSW and open Existing Claude. If that happens, check CSW’s environment marker, quit the relaunched instance, and launch your intended environment through CSW again.

When you quit all Claude instances for an update, CSW remembers the environments that were open together. The Reopen banner restores them on request, skipping those already running. Shared environments still obey the single-instance restriction. Nothing reopens merely because the banner appears.

For backups, expand Location of this environment’s data and use Show in Finder. The detail view includes paths, approximate sizes, and shared-link targets.

Normal deletion moves an environment into macOS Trash. Before emptying Trash, you can restore its folder to:

~/.context-switcher-claude/profiles/

Return to CSW and it should reappear. Delete permanently skips Trash and cannot be undone. In v0.24.7, both deletion paths refuse to remove a running environment, and deletion also stops if the running state cannot be checked. Quit the environment before deleting it.

Privacy and next steps

The Privacy and Transparency document explains the files CSW accesses and provides verification commands. The project documents no network communication or telemetry from CSW itself; opening documentation or release links delegates to your browser. The official Claude applications retain their own network behavior.

CSW is an unofficial, MIT-licensed project, with no warranty. Use accounts in accordance with Anthropic’s current terms and your organization’s policies.

Start with one fully isolated environment, sign in, and check which window and account you are using. Once that workflow is clear, decide whether sharing selected configuration would help.

This article was written with AI assistance and reviewed by me.

A message between AI agents was shown as sent, but it was not in the inbox

matsumotory — Tue, 13 Oct 2026 09:39:53 +0000

Summary

On 2026-08-19, in work on an initiative I am pursuing (research that rebuilds the development environment for AI agents from OSS components only), an instruction was given to an AI agent to send a message. The conversation took place on the screen of a desktop GUI app. The instruction asked the agent to send the message to the other agent with the message-sending tool (send_message) of an MCP server built in this research. This MCP server provides message exchange between sessions, and it writes a JSONL file for each destination agent name. This article calls this file the inbox file. The agent in the conversation was one connected to Qwen Code (version 0.21.13), one of three public OSS agent runtimes. According to the records of this research, Qwen Code called a built-in send tool with a similar name. The screen showed that the send succeeded, but the message had not reached any inbox file. This discrepancy was found by looking at the contents of the inbox files during manual work.

The core of the body is how a check script that judges message exchange between AI agents is built. The check script registers this MCP server with three agent runtimes, namely OpenCode, Qwen Code, and OpenHands CLI (versions 1.18.18, 0.21.13, and 1.16.0, respectively). It then judges whether message exchange succeeded with three checks: a send check, a receive check, and a control check. In none of the checks does the check script decide pass or fail from the AI agent's own report. The result of the send check is decided by the lines in the other agent's inbox file. The result of the receive check is decided by whether a marker string, generated at random for each run, appears in the output, and this string is not included in the prompt. The control check gives the same instruction as the receive check in a state where no message has been sent, and checks that the marker string does not appear in the output. On 2026-08-27, a tool call log was added. This log is a file to which the MCP server appends one line for each tool call. The check script also made it a pass condition that the IDs in the log match the IDs of the lines in the inbox file. The purpose is to distinguish cases in which the AI agent read or wrote the inbox file directly without calling the tool. In measurements with the CLI on 2026-08-19, 2026-08-27, and 2026-09-02, all three checks passed in all nine runs.

The body first describes the event in which the screen showed a successful send and the message had not arrived. Next, it describes how the check script that decides pass or fail by the lines of the inbox file and the marker string is built. It then describes how the judgment was strengthened with the tool call log, and the results of the measurements on the three agent runtimes. After that, it describes the conditions of the conversation in which the mix-up occurred, and an example from the previous day in which a report that child agents had completed disagreed with the records. It also describes the item added to the rule documents (instruction documents that AI agents read) after the observation. Finally, it describes the grounds for judging by records in a separate place rather than by output, together with where external research and official guidance stand on this.

What you can take away

For people who test collaboration between AI agents, I explain the following three things.

The send check decides pass or fail by the lines in the other agent's inbox file, and the receive check decides pass or fail by a marker string that is not in the prompt. The receive check is paired with a control check run in a state where no message has been sent. In the body, I show in a table the instructions and the pass conditions of the three checks
Matching the IDs in the tool call log against the IDs of the lines in the inbox file lets you distinguish cases in which the AI agent read or wrote the inbox file directly without calling the tool. In the body, I also explain why this distinction is not complete
All nine passes were measured with the CLI, and in the conversation in the desktop GUI app where the mix-up occurred, the tools of the MCP server did not appear. In the body, I describe a procedure to follow when registering an MCP server and confirming that it works: before giving an instruction, look at the configuration file and at whether the MCP server process started, and confirm that the tools are present in the conversation

The body is a discussion based on the records of work from 2026-08-18 to 2026-09-02 on the initiative I am pursuing, and on external materials checked on 2026-10-11.

The body names the products of the three agent runtimes. The content does not represent the views of any of their providers.

The send shown as successful, and the message that did not arrive

On 2026-08-19, in the work of this research, an instruction to send a message was given to an AI agent in a conversation in the desktop GUI app. The screen showed that the send succeeded, but the message had not reached any inbox file. An inbox file is a JSONL file that the MCP server built in this research writes for each destination agent name.

This MCP server provides message exchange between sessions. This article calls this MCP server the message-sending MCP server. The message-sending MCP server has the following three tools.

list_agents returns a list of the other agents
send_message sends a message
check_inbox retrieves messages from the inbox file

Each time send_message sends one message, one line with the sender, the body, and an ID is added to the destination's inbox file.

The agent runtimes used in this research are three public OSS ones, namely OpenCode, Qwen Code, and OpenHands CLI (versions 1.18.18, 0.21.13, and 1.16.0, respectively). ACP is a public standard for running agents from GUIs and editors. The name ACP is made of the initials of three words: Agent, Client, and Protocol. The agent in the conversation was an agent connected to Qwen Code through ACP.

The instruction was to send a message with send_message of the message-sending MCP server. Among the built-in tools that Qwen Code registers by default, there is a tool that sends messages. This article calls this tool the built-in send tool. This research has a document that records trial and error and a document that explains the integration with the desktop GUI app. The two documents say that Qwen Code called the built-in send tool, which has a similar name. The conversation records do not contain the name string of the tool that was called.

In addition to the success shown on screen and the message that did not arrive, the document that records trial and error also describes how this discrepancy was found. This discrepancy was found not by automatic judgment but by looking at the contents of the inbox files during manual work.

A check script that decides pass or fail by the lines of the inbox file and a marker string

The check script of this research decides pass or fail for message exchange by the lines of the inbox file and a marker string, not by the AI agent's report. The check script is a Python script that registers the message-sending MCP server with the three agent runtimes and judges whether message exchange succeeded. This script was newly created in the work of this research on 2026-08-19. The check script runs three checks, the send check, the receive check, and the control check, in this order.

The check script generates a random string for each run. This article calls this string the marker string. The result of the send check is decided by the lines in the other agent's inbox file. The results of the receive check and the control check are decided by whether the marker string, which is not included in the prompt, appears in the output.

The integration test from before 2026-08-19 checked only sending. This test was deleted in the PR that added the check script. The format for registering the MCP server was confirmed for each agent runtime with the output that its official registration command writes. The key in the configuration file is mcpServers for Qwen Code and OpenHands CLI, and mcp for OpenCode.

The check script launches a separate process as the other party in the checks. This process runs the same message-sending MCP server program as the AI agent uses, directly over JSON-RPC on standard input and output, without going through a model. This article calls this process the counterpart process.

The following table lists, for the three checks, the instruction to the AI agent and the pass condition.

Check	Instruction to the AI agent	Pass condition
Send check	Find the other party with list_agents, and send the marker string with send_message	The other party's inbox file has a line whose sender is the AI agent's name and that contains the marker string
Receive check	Read the message that the counterpart process sent beforehand from the inbox file with check_inbox, and output its body. This marker string is not included in the prompt	The marker string appears in the AI agent's output
Control check	In a state where not a single message has been sent, give the same instruction as in the receive check	The marker string does not appear in the AI agent's output

The description at the top of the check script states the purpose of the control check. The purpose is to avoid passing the receive check when the marker string appears in the output without the inbox file being read. As such a case, the description gives the case in which the marker string output in the receive check leaked into memory or storage and remained there.

The check script does not pass states that it cannot judge. The states that it cannot judge are the following four.

The agent runtime fails to start
The run times out
The inbox file does not exist
The output is empty

If the destination is not registered, a call to the message-sending tool results in an error. For this reason, in the receive check, the AI agent's name is registered as a destination of the message-sending MCP server only while the counterpart process sends. The process that registers this name is a different process from the counterpart process.

Strengthening the judgment with the tool call log

On 2026-08-27, pass conditions that match against the tool call log were added to the send check and the receive check of the check script. The tool call log is a JSONL file to which the message-sending MCP server appends one line each time a tool is called. Each line has the name of the agent that called the tool and the message ID, and it has no body. The code that writes this log was added to the message-sending MCP server on the same day, in the work of this research.

The following table lists the pass conditions added to the send check and the receive check on 2026-08-27.

Check	Pass condition added on 2026-08-27
Send check	The ID of the line in the inbox file matches the ID of the line that recorded the message-sending tool (send_message) under the AI agent's name
Receive check	The line that recorded the tool that retrieves messages from the inbox file (check_inbox) under the AI agent's name has the ID of the sent message

The purpose of this strengthening is to distinguish message exchange through the tool from cases in which the AI agent reads or writes the inbox file directly without calling the tool. The check script also does not pass the check when there is no tool call log or when the tool call log contains a broken line.

The description of the check script says that this distinction is not complete. The reason is that the AI agent can read the location and the format of the tool call log with the same user's permissions. An AI agent that can read the location and the format can also write both the inbox file and the tool call log itself, in the matching format.

Results of the measurements on the three agent runtimes

In the measurements on 2026-08-19, 2026-08-27, and 2026-09-02, all three agent runtimes passed all three checks. Each agent runtime was run once on each day, so there were nine runs in total.

The following table lists, for each measurement day, the judgment conditions and the number of agent runtimes that passed.

Measurement day	Judgment conditions	Number of agent runtimes that passed
2026-08-19	Lines of the inbox file and the marker string	All three
2026-08-27	Lines of the inbox file and the marker string, plus matching IDs in the tool call log	All three
2026-09-02	Lines of the inbox file and the marker string, plus matching IDs in the tool call log	All three

The three agent runtimes all used the same open-weight model from the same inference service.

All nine passes were measured with the CLI. In these measurements, the tools of the message-sending MCP server were available in each of the three agent runtimes. The mix-up on 2026-08-19 occurred in a conversation in the desktop GUI app. The tools of the message-sending MCP server did not appear in this conversation.

The document that explains the integration with the desktop GUI app contains a statement to the following effect. Even if the screen shows success, the send check does not pass when the message is not in the inbox file. The reason is that whether the send passes is decided by the contents of the inbox file. This statement is derived from how the checks are built, and there is no record of using the check script to reproduce the mix-up that occurred in the conversation.

Conditions of the conversation in which the mix-up occurred

In the conversation where the mix-up occurred, the tools of the message-sending MCP server did not appear in the conversation with Qwen Code connected through ACP. The two documents of this research say that Qwen Code called the built-in send tool. However, there is no statement that confirms whether the model chose the built-in send tool or Qwen Code resolved the tool by name. The scope checked was the document that records trial and error, the bodies of the two PRs from 2026-08-19, and the document that explains the integration.

This MCP server had been registered with the feature that manages MCP settings in bulk, on the settings screen of the desktop GUI app. With this registration, the connection check passed. However, no tools appeared in the Qwen Code conversation, and the MCP server process did not start either.

The two documents of this research describe Qwen Code's built-in send tool as a tool with a similar name. The version of Qwen Code on the day of the observation was 0.21.13. According to a check on 2026-10-11, in this version's tool name definition file and registration code, the built-in tools registered by default included send_message and list_agents. These two have names that are the same strings as the names of the tools of the message-sending MCP server. Qwen Code has a setting for a team feature, and these two built-in tools are always registered regardless of that setting.

The built-in list_agents returns a list of background agents. The built-in send_message has two modes: one that sends to team members and one that sends to background tasks. The words agent and task are the words used in the descriptions of the respective tools.

That the names matched is the result of checking the public source. It is not the result of confirming, in the conversation records, the name of the tool that was called in the conversation.

The Tools page of the Model Context Protocol specification, version 2025-11-25, has no provision for cases where a name overlaps with a tool built into the client. It also has no provision for cases where names overlap between different servers. This page says that each tool is uniquely identified by its name. It also says that tool names should be unique within a single server (SHOULD).

On the same day, message exchange from the desktop GUI app succeeded when the conversation was connected to OpenCode. In this conversation, the registration of the message-sending MCP server had been added to OpenCode's configuration file. The agent in this conversation found a CLI session with list_agents and sent the marker string. A line whose sender was the agent in this conversation arrived in the CLI session's inbox file. In the opposite direction, a Qwen Code CLI session sent a message to this conversation. The agent in this conversation read that message from the inbox file with check_inbox and output it on the screen.

An example from the previous day in which a report that child agents had completed disagreed with the records

On 2026-08-18, with an experimental parallel execution feature of Qwen Code, a report that the child agents had completed disagreed with the JSON record file that this feature writes. This feature launches child agents in parallel according to an orchestration script written by the model, and it writes a JSON record file for each orchestration run.

In the work of this research, this feature was measured in non-interactive runs. In seven trials there were 13 child agent launches, and none of the 13 completed. The child agents consumed 0 tokens, and they never called the model. In a re-measurement in interactive mode on the same day, two child agents completed. The run records of this research state the conclusion that the phenomenon of child agents not completing occurs only in non-interactive runs.

In the last trial, the model wrote the answer directly without launching any child agents, and in its final response it reported that three child agents had finished. In this trial's JSON record file, the field for the number of attempted child launches was 0. The value of this field showed mechanically that the report was wrong.

The program built in this research that runs child agents in parallel always writes two counts in the run records of this research. The two counts are the number of attempted child launches and the number of child agents that returned a response body. The README of this program gives the false completion report of 2026-08-18 as the reason for making the two counts mandatory.

The document of this research that records trial and error classifies, as one type, failures in which a report of completion or success disagrees with a count field or with the contents of the inbox file. In this document, the successful send shown on screen on 2026-08-19 is the second example, following the completion report of 2026-08-18.

Articles from the same research that deal with the parallel-execution MCP tool and child agents include Once I wrote in the MCP tool description when to use the tool, AI agents called it and A failure that looked like a capability gap between coding agents was caused by how the API key was passed.

The item added to the rule documents after the observation

In a PR merged on 2026-08-20, one item with the same content was added to each of two rule documents. A rule document is an instruction document that AI agents read. The two are the default instruction document that AI agents always read and the skill document that they read when they use a feature that spans sessions. An AI agent created this PR.

The added item is an instruction for environments where the MCP server is not configured. The item instructs the agent not to substitute another tool with a similar name, to first check whether the tool exists, and, if it does not, to report that configuration is needed. As the reason, the item also states that a failure occurred in which a tool with a similar name was used and nothing arrived even though the screen showed success.

Grounds for judging by records in a separate place rather than by output

Both the report on 2026-08-18 that child agents had completed and the successful send shown on screen on 2026-08-19 disagreed with data that remains in a place separate from the report. The error in the 2026-08-18 report was confirmed with the field for the number of attempted child launches in the JSON record file that the experimental feature writes. The error in the 2026-08-19 display was confirmed with the inbox file. These facts show that it is better to decide the result of a check on agent collaboration by data that remains in a place separate from the report, rather than by the AI agent's output.

The document of this research that records trial and error states what is judged by what, as follows.

Whether sending a message succeeded is judged by the lines of the inbox file
Whether something was executed, such as the launch of a child agent, is judged by counts written in the records, such as the field for the number of attempted child launches
Whether something completed is judged by the deliverables
The success shown on screen and what the AI agent reports in its text are not used for judgment

The receive check confirms that the output came from the inbox file by checking that a marker string that is not in the prompt appears in the output. The control check confirms that the same output does not appear when no message has been sent. When the two checks are paired, the control check can fail a case in which the marker string leaked into memory or storage, remained there, and appeared in the output without the inbox file being read. This property is derived from how the checks are built, and in the nine runs there was no case in which the control check failed.

The document of this research that records trial and error also describes a procedure to follow when registering an MCP server and confirming that it works. Before giving an instruction, look at whether the MCP server is written in the configuration file and whether the MCP server process started, and confirm that the tools are present in the conversation.

The idea of judging success or failure by the state and records left in the environment, rather than by the AI agent's report, also appears in external research and official guidance.

According to the abstract of a study that measured false success (Advani, 2026), the failure in which an LLM agent claims to have completed a task even though the state of the environment does not show completion is called false success. One of the measured targets is 9,876 trajectories in which AI agents worked on tasks in tau2-bench, spanning eight model families. Whether each trajectory succeeded was decided without depending on the agent's text. In the domains of tau2-bench that the study calls single-control, false success is reported to account for 45% to 48% of failures. The abstract does not define single-control. Judgment by an LLM (LLM judge) is reported not to have detected this failure well enough. AUROC is a metric that expresses detection performance from 0 to 1, and 0.5 is the same as chance. The judging configurations given in the abstract are five models, five prompts, and the full text of the task specification. In every configuration, AUROC on tau2-bench did not exceed 0.65. The abstract says that LLM judges rely on confident closing phrases and the volume of the action sequence, and that they do not look at verified changes in state.

Demystifying evals for AI agents gives an example of a flight booking agent. In this example, the outcome is judged not by whether the agent said it made the booking but by whether the booking exists in the environment's SQL database.

The abstract of a study that detects hallucinations with tool execution receipts (Basu, 2026) proposes a framework for detecting hallucinations. In this framework, tool execution receipts are matched against the claims in a response. The receipts are signed with HMAC so that the LLM cannot forge them.

The limit of the check script was that, because the AI agent can read the tool call log with the same user's permissions, there remains room for it to write the log itself in the matching format. These facts show that this limit is the same kind of problem as the forgery of records that the receipt study prevents with signatures. They also show that the approach of deciding pass or fail by data that remains in a place separate from the report is not a conclusion drawn only from the work records of this research.

Studies I referred to

The studies referred to in the body are the following two, and I checked their arXiv abstract pages on 2026-10-11.

Laksh Advani, "From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents" arXiv:2606.09863
Abhinaba Basu, "Tool Receipts, Not Zero-Knowledge Proofs: Practical Hallucination Detection for AI Agents" arXiv:2603.10060

Materials I referred to

The public materials referred to in the body are the following three, and I checked their original texts on 2026-10-11.

Tools (the 2025-11-25 version of the Model Context Protocol specification) https://modelcontextprotocol.io/specification/2025-11-25/server/tools
Demystifying evals for AI agents (Anthropic Engineering, 2026-01-09) https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
tool-names.ts (the tool name definition file of Qwen Code v0.21.13) https://github.com/QwenLM/qwen-code/blob/v0.21.13/packages/core/src/tools/tool-names.ts

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

Rejecting unsupported MCP versions left an AI client unable to connect

matsumotory — Sat, 10 Oct 2026 20:59:09 +0000

Summary

The MCP server for a service I develop at my own company rejected requests whose version header held an unsupported version, and it did so with HTTP 400 before authentication. MCP is an open standard for connecting AI applications to external tools and data sources, and the name is short for Model Context Protocol. The version header is a header defined by the specification that HTTP clients attach to MCP requests, and it indicates the MCP version with a date string such as 2025-11-25. The server decided whether it supported a version by checking the list of supported versions. The list of supported versions enumerates the versions that the MCP server declares it supports, and it is written in the server's code. Because of this rejection, the major AI client could not connect to this MCP server on 2026-09-24. The major AI client is one of the AI chat products in which users can add an MCP server as a connector.

In its first fix, the AI agent only added the observed 2025-11-25 to the list of supported versions. The major AI client is updated frequently, so in response to this fix I pointed out that rejecting a connection over a small difference in version was too strict. In its second fix, the AI agent changed the list of supported versions to two entries, 2025-11-25 and 2025-06-18. In the same fix, it made the server accept values in date form that are smaller than 2026-07-28, even when they are not in the list. With the 2026-07-28 version, the MCP specification changed to an approach that does not use the initialize request, the request a client sends first to decide the version. The AI agent left in place the handling that returns 400 for values dated 2026-07-28 or later and for values not in date form. The MCP specification has a MUST requirement, that is, an obligation that must always be met: a server that receives a version header with an unsupported version responds with 400. Based on my decision, the AI agent intentionally did not implement this requirement for versions earlier than 2026-07-28 that are not in the list. On the next day, 2026-09-25, the AI agent changed the MCP server so that it could also respond to requests that use the 2026-07-28 version.

In the first section, I explain the version header check that stopped the major AI client from connecting, and I show in a table the results of the check before the fix and after the second fix on 2026-09-24. In the next section, I explain how the server identifies legacy versions and how it handles modern versions. Legacy versions are the versions up to and including 2025-11-25, which establish a connection with the initialize request, and modern versions are the versions from 2026-07-28 onward. That section also gives the reason the MCP server was made to support both kinds of versions the next day. The section that follows compares, in a table, the two MUST requirements of the specification with how the MCP server handles them after the second fix. The section after that explains the first fix, which added one observed version, and the point I raised about it. The last section explains the robustness principle of being liberal in what you accept, and the limits that RFC 9413 placed on this principle. Building on that, Building on that, I classify the MCP server with the second fix as follows. It is neither an application of this principle's aspect of tolerating unexpected inputs nor an extension under clear rules as described in Section 2.2 of RFC 9413, and it is an implementation that intentionally chose handling different from the handling the specification defines.

What you can take away

For those who develop MCP servers, I explain the following three things.

When an MCP server checks the version header against a list of supported versions, a fix that only adds the one observed version to the list leads to the same 400 the next time a client raises its version. In the body, I explain why from the rules of the check before the fix.
The MCP server with the second fix treats a header value as a legacy version if the value is in date form and smaller than 2026-07-28. In the body, I use tables to show this rule and, of the two MUST requirements of the specification, the one this MCP server meets and the one it intentionally does not meet because of my decision.
The second fix, which accepts legacy versions not in the list, is neither an application of the aspect of tolerating unexpected inputs within the robustness principle of being liberal in what you accept, nor an extension under clear rules as described in Section 2.2 of RFC 9413. In the body, I classify the MCP server with this fix as an implementation that intentionally chose, for inputs whose handling the specification defines, handling different from the specification's.

The body is based on the records of fixing the MCP server of my own company's service from 2026-09-24 to 2026-09-25, and on the original texts of the MCP specification and the RFCs, which I checked on 2026-10-04.

The version header check that stopped the major AI client from connecting

I develop a service at my own company. The MCP server of that service rejected, with HTTP 400 and before authentication, any version header whose version was not in the list of supported versions. As a result, the major AI client could not connect on 2026-09-24.

MCP is an open standard for connecting AI applications to external tools and data sources, and the name is short for Model Context Protocol. The major AI client is one of the AI chat products in which users can add an MCP server as a connector. The version header is a header defined by the specification that HTTP clients attach to MCP requests. Its value is a date string such as 2025-11-25, and it indicates the MCP version. The list of supported versions enumerates the versions that the MCP server declares it supports, and it is written in the server's code.

In versions up to and including 2025-11-25, the client first sends an initialize request and decides with the server which version they will use. Versions that establish a connection in this way are called legacy versions. This name is a term from the 2026-07-28 version of the specification.

Before 2026-09-24, the MCP server's list of supported versions had only one entry, 2025-06-18. A note in the server's code gave the reason for not declaring the 2025-03-26 version. That version has a requirement that implementations accept JSON-RPC batches, and the MCP server does not support batches. The MCP server accepted requests that had no version header.

On 2026-09-24, I added the MCP server in the test environment to the major AI client as a connector. The major AI client's screen stopped while still showing that it was connecting. The screen showed an error saying, in effect, that something was not found, along with HTTP 400. In the observation that day, the major AI client put 2025-11-25 in the version header starting from the first initialize request.

On the same day, the AI agent fixed the MCP server twice. In the second fix, it changed the list of supported versions to two entries, 2025-11-25 and 2025-06-18. If a client asks for a version in the list in the initialize request, the MCP server responds with that version. If the version is not in the list, the server responds with 2025-11-25. When the MCP server rejects a request with 400, it logs the rejected version value and the User-Agent of the connecting client as a warning.

Versions from 2026-07-28 onward have no initialize request, and the client declares the version in _meta in the request body of every request. These versions are called modern versions. The following table lists the results of the check for each header value.

Value of the version header	Before the fix	After the second fix on 2026-09-24
2025-06-18	Accepted	Accepted
2025-11-25	Rejected with 400	Accepted
Other date values smaller than 2026-07-28. An example is 2025-03-26	Rejected with 400	Accepted
Values of modern versions. Date values of 2026-07-28 or later	Rejected with 400	Rejected with 400
Values not in date form	Rejected with 400	Rejected with 400

Identifying legacy versions and handling modern versions

The MCP server with the second fix treats a header value as a legacy version if the value is in YYYY-MM-DD form and smaller than 2026-07-28. Because which value is smaller is decided by string comparison, this check depends on the value being in YYYY-MM-DD form. Dates between 2025-11-25 and 2026-07-28 do not correspond to any version of the specification, but this check treats them as legacy versions.

The AI agent left in place the handling that returns 400 when the value of the version header is a modern version. An implementation that supports both legacy and modern versions is called dual-era. This name is also in the 2026-07-28 version of the specification. The reason the AI agent wrote in its description of the fix is that a dual-era client looks at the body of the 400 response and switches to a legacy version.

On 2026-09-25, I approved the work to make the MCP server dual-era. The AI agent wrote the reason for this work as follows. As of 2026-09-24, the major AI client was communicating with a legacy version. However, client updates come faster than server fixes, so it would make the MCP server dual-era before the major AI client moved to a modern version. The AI agent merged the fix for this work on 2026-09-25. The MCP server then became able to respond to requests with modern versions as well.

The specification's obligations and the MCP server after the second fix

The MCP server with the second fix meets the obligation to respond with a different version when a client, in the initialize request, asks for a version not in the list. On the other hand, for legacy versions, it intentionally does not meet the obligation to return 400 for a version header with an unsupported version.

The Lifecycle documents of the 2025-06-18 and 2025-11-25 versions contain a MUST requirement on the version the server responds with. A server that supports the requested version responds with the same version, and a server that does not support it responds with a different version. The specification defines, as a SHOULD requirement, that is, a recommendation, that this different version be the server's latest version.

The Transports documents of the 2025-06-18 and 2025-11-25 versions contain a MUST requirement on the header. A server that receives a version header with an invalid or unsupported version responds with 400 Bad Request. The AI agent wrote this obligation in the server's documentation as an obligation that the MCP server does not meet.

MUST requirement in the specification	Versions that have the requirement	MCP server with the second fix
A server asked in the initialize request for a version it does not support responds with a different version that it supports	2025-06-18, 2025-11-25	Responds with 2025-11-25
A server that receives a version header with an unsupported version responds with 400	2025-06-18, 2025-11-25	Intentionally does not return 400 for legacy versions not in the list

Intentionally not implementing the obligation to return 400 is based on my decision not to reject connections over a difference in version alone. The values that the second fix newly accepts are 2025-03-26 and 2024-11-05, which are versions of the specification, and date values that do not correspond to any version of the specification. The AI agent explained the reasons for this decision as follows.

Rejecting connections over a difference between legacy versions brings no benefit.
If updates to the list of supported versions fall behind, the MCP server ends up rejecting clients' connections. This harm is greater than the benefit of rejecting connections over a difference between legacy versions.
Even if a client uses a feature that the server does not implement, the connection is kept. For example, for JSON-RPC batches, which are in the 2025-03-26 version, the MCP server only returns a standard error.

The first fix, which added one observed version

In its first fix, the AI agent only added the observed 2025-11-25 to the list of supported versions. Before the fix, the MCP server had been rejecting with 400 the value 2025-11-25 that the major AI client sent.

In response to the first fix, I pointed out that rejecting a connection over a small difference in version was too strict. This is because the major AI client is updated frequently.

The check before the fix rejected every value not in the list with 400. These facts show that with a fix that adds one observed version to the list, the same 400 occurs the next time a client raises its version.

The robustness principle and the limits set by RFC 9413

The robustness principle is the implementation principle of being liberal in what you accept and conservative in what you send. RFC 1122 lists this principle and says that it is especially important at the Internet layer.

RFC 9413 says that the robustness principle has three aspects. The three aspects are robustness to software defects, robustness to attacks, and robustness to unexpected inputs. Section 1 of RFC 9413 says that cases where the specification explicitly defines how a faulty message is handled are excluded from the aspect of robustness to unexpected inputs. RFC 9413 also says that the first two aspects are necessary guiding principles. On the other hand, it says that the interpretation of tolerantly accepting unexpected inputs can no longer be called the best approach in every situation.

Section 2.2 of RFC 9413 says that extensibility is sometimes mistaken for an application of the robustness principle. It also says that in a well-designed extensibility mechanism, the handling of new messages and parameters is decided by clear rules, so they become expected inputs.

The MCP specification has a requirement to respond with 400 to a version header with an unsupported version. Therefore, the MCP server with the second fix is an implementation that, for inputs whose handling the specification defines, handles them differently from the specification. The aspect of robustness to unexpected inputs does not include inputs whose handling the specification makes explicit. The extension under clear rules in Section 2.2 is a mechanism in which the protocol defines the rules and every implementation behaves according to those rules. These facts show that the MCP server with the second fix is neither an application of the robustness principle's aspect of tolerating unexpected inputs nor an extension as described in Section 2.2. They also show that it is an implementation that intentionally chose handling different from the handling the specification defines.

Materials I referred to

The public materials I referred to in the body are the following six items, which cover eight documents. I checked the original text of each of them on 2026-10-04.

Lifecycle (the 2025-06-18 version of the MCP specification) https://modelcontextprotocol.io/specification/2025-06-18/basic/lifecycle
Lifecycle (the 2025-11-25 version of the MCP specification) https://modelcontextprotocol.io/specification/2025-11-25/basic/lifecycle
Transports (the 2025-06-18 version of the MCP specification) https://modelcontextprotocol.io/specification/2025-06-18/basic/transports
Transports (the 2025-11-25 version of the MCP specification) https://modelcontextprotocol.io/specification/2025-11-25/basic/transports
Versioning and Compatibility (the 2026-07-28 version of the MCP specification) https://modelcontextprotocol.io/specification/2026-07-28/basic/versioning
Base Protocol (the 2025-03-26 version of the MCP specification) https://modelcontextprotocol.io/specification/2025-03-26/basic
RFC 1122 Requirements for Internet Hosts -- Communication Layers https://www.rfc-editor.org/rfc/rfc1122.txt
RFC 9413 Maintaining Robust Protocols https://www.rfc-editor.org/rfc/rfc9413.txt

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

A failure that looked like a capability gap between coding agents was caused by how the API key was passed

matsumotory — Sat, 10 Oct 2026 20:57:54 +0000

Summary

An AI agent compared three coding agents in a research repository that is scheduled to be made public. The research rebuilds an integrated agent development environment with OSS. The model and the tasks were the same for all three, and there were eight tasks. The source material consists of the records in this repository. In this comparison, a parent agent calls a parallel-execution MCP server. The parallel-execution MCP server is a program built as part of the research, and when it is called, it launches child agents in parallel. Only in the three runs with OpenHands CLI as the parent did none of the 32 child agents launched in total complete. Both the status field in the run record files and the result field of the measurement stayed normal, and for about a day this outcome was treated as a result of the comparison.

The cause was not a difference in capability between the coding agents but the way the measurement script was built. The measurement script left the delivery of the API key that the child agents use to two things: inheritance of the parent process's environment variables and loading of a .env file. Only when the parent was OpenHands CLI and the run took place in an isolated working directory did the API key fail to arrive by either route.

The AI agent fixed the measurement script so that it writes the API key in the env field of the MCP server configuration and passes it that way. After the fix, the AI agent measured the condition with OpenHands CLI as the parent again, once each in two cases: with the set of instruction files in place and without it. The set of instruction files is a group of files, such as AGENTS.md, placed in the working directory. Under both conditions, the API key no longer caused every child agent to fail. With the set of instruction files in place, all eight child agents completed. Without it, 15 of the 16 child agents launched did not complete, for reasons other than the API key.

The first section of the body gives the overall picture of the cause and the fix. The sections after it are arranged in the following order.

Three sections explain how the comparison results were skewed, in this order: the measurement conditions and results, why the failure did not show up in the results, and how the cause was isolated
Whether the parent process's environment variables reach the MCP server is checked for each implementation in the public source code
Three sections explain the fix, in this order: the fix to how the API key is passed and the re-measurement, the mechanism that does not hide failures and the unit tests, and the protection of the API key
Before the section on the materials referred to, two rules for designing comparison measurements are presented

The body names the products of the three coding agents compared. The content does not represent the views of any of their providers.

What you can take away

If you launch other agents from a coding agent through an MCP server, you will be able to do the following three things.

Check in the source code which of the parent process's environment variables your coding agent passes to a stdio MCP server. The range it passes differs by implementation
Fix your setup so that the API key the child agents use is written in the env field of the MCP server configuration and passed that way
Add to your own measurements a step that does not count failures caused by defects on the measurement side as results of a coding agent's capability

This article is a discussion based on the run records in the research repository scheduled to be made public and on the public source code checked at the time of writing.

Two implicit routes that the API key delivery depended on

This section first explains why every child agent failed in the comparison measurements and what the setup looks like after the fix.

The comparison measurements are a series of runs that measured three coding agents in turn, with the same model and the same tasks, while changing the conditions. A coding agent is a CLI program that is started from a terminal and that edits files and runs commands while calling a model. In the comparison measurements, the parent agent calls the parallel-execution MCP server. The parent agent is the coding agent process that receives the tasks and is started first.

The parallel-execution MCP server is an MCP server built as part of the research, and when it is called, it launches child agents in parallel. An MCP server is a program that provides tools for a coding agent to call through a common protocol called MCP. The MCP servers in this article are of the stdio type: the agent starts them as child processes and communicates with them over standard input and output. A child agent is a coding agent process launched by the parallel-execution MCP server.

The measurement script is a Python script that passes the tasks to the parent agent and starts it, then judges the results from the actual files and records them. Before the fix, the child agents' API key arrived by one of the following two implicit routes. The API key is the credential for calling the inference service. The inference service is a service that makes models available to call through an API.

Environment variable inheritance: the environment variables held by a parent process are passed as they are to the child processes it starts
Loading the .env file: the route by which the parallel-execution MCP server reads a .env file. A .env file holds configuration values such as API keys as lines of the form "name=value", and it is kept out of git

Only when the parent was OpenHands CLI and the run also took place in an isolated working directory did the API key fail to reach the child agents by either of the two routes. An isolated working directory is a separate working directory of the repository, created with git worktree. This summary combines three conditions found in the research records, and the research records contain no statement in this form. The three conditions are explained in the section "Isolating the cause by rerunning with the run records kept".

After the fix, the measurement script writes the API key in the env field of the MCP server configuration and passes it explicitly. The env field of the MCP server configuration is a field in the coding agent's configuration file. This field lists, for each MCP server, the environment variables to pass to the process that is started. The field is named environment in OpenCode and env in the other two.

When the API key is needed but missing, the parallel-execution MCP server stops before launching child agents and marks the run record file as failed. A run record file is a JSON file that the parallel-execution MCP server writes for each call. It has fields for the number of child agents launched, the number of child agents completed, and the status.

The following diagram shows the routes before and after the fix.

Before the fix
  The shell that started the measurement script (with the API key loaded)
    └ Parent agent
        └ Parallel-execution MCP server
            │  Route 1: environment variable inheritance
            │           Did not arrive when the parent was OpenHands CLI
            │  Route 2: read the .env file at the top level of the repository
            │           The isolated working directory has no .env file
            └ Child agent (the API key value is an empty string)

After the fix
  Measurement script
    └ Writes the API key in the env field of the MCP server configuration
        └ Parallel-execution MCP server
            ├ API key present: launches child agents
            └ API key needed but missing: stops before launching and marks the run record file as failed

Conditions of the comparison measurements and the number of completed child agents per parent agent

This section shows the conditions of the comparison measurements and the results for each parent agent in a table.

The conditions of the comparison measurements were as follows.

Versions: OpenHands CLI version 1.16.0, Qwen Code version 0.21.13, and OpenCode version 1.18.18 were used
Model: one model was fixed for all runs and called through the OpenAI-compatible endpoint of the inference service
Tasks: eight independent programming tasks. Each task has a task statement and a test script
Instructions to the parent agent: two, namely to finish all eight without leaving any out and not to rewrite the test files. The instructions did not mention the name of the tool
How the parallel-execution MCP server was provided: it was registered in each coding agent's configuration file. Where it was registered differs among the three. For Qwen Code, it was registered in a configuration file placed in the working directory. For OpenCode, it was also registered in a configuration file placed in the working directory. For OpenHands CLI, it was registered in mcp.json placed in a dedicated storage directory
Child agents: for every parent agent, they were started in Qwen Code's non-interactive mode

The following table summarizes, for each parent agent, the runs in which the parallel-execution MCP server was called.

Parent agent	Runs in which a call occurred	Run record files	Child agents launched	Child agents completed
Qwen Code as parent	3 runs	1, 1, and 2 per run	8 per run	8, 8, and 7 per run
OpenCode as parent	1 run	3	16 in total	8 in total
OpenHands CLI as parent	3 runs	4 in total	32 in total	0 in total

In the three runs with OpenHands CLI as the parent, 0 of the 32 child agents launched completed. In the run with OpenCode as the parent, the parent agent fixed an error in the orchestration script by itself and ran it again. The orchestration script is a script, run by the parallel-execution MCP server, that describes the steps for launching child agents.

Even in the three runs with OpenHands CLI as the parent, all eight tasks passed, and there were 0 rewrites of the test files. The records can be read as showing that the parent agent solved by itself the work that the child agents did not complete. However, the research records contain no breakdown of the parent agent's work.

A judgment design in which the failure of every child agent did not show up in the results

This section explains how a result in which every child agent failed continued to be treated as a result of the comparison.

The measurement script judged whether the parallel-execution MCP server had been called by whether a run record file existed. Only the parallel-execution MCP server process writes run record files, so none is created if there is no call.

The measurement script recorded the numbers of child agents launched and completed by summing the fields of the run record files. For pass or fail on the tasks, it restored the test files, ran the tests for each task, and recorded the number of tasks that passed.

Even though every child agent failed, the failure did not show up in the results, for the following three reasons.

If a run record file existed, the measurement script treated the run as having a call and set the result field to success. Even if not a single child agent completed, the result field became success
The run record files written by the parallel-execution MCP server also all had ok, which indicates normal, in the status field
All eight tasks had passed, so the failure was not visible from the task results either

The result in which OpenHands CLI was the parent and 0 child agents completed was treated as a result of the comparison measurements for about a day.

Isolating the cause by rerunning with the run records kept

This section explains how the cause was isolated, in the order of the work. According to the research records, the AI agent did the implementation, ran the measurements, and handled the PRs. This AI agent works separately from the reviewing AI agent that examines the diffs of PRs.

The AI agent reran the condition with OpenHands CLI as the parent just once. This time, it kept the directory that holds the run record files instead of deleting it after the run. In the records it retrieved, all eight child agents had exited about 1.2 seconds after starting. The reason for exiting was an error saying that no authentication method had been selected.

The text of this error is in the file in Qwen Code version 0.21.13 that checks authentication in non-interactive mode. When Qwen Code starts in non-interactive mode, it determines the authentication method from the command arguments, the environment variables, and the configuration files. If none of them determines it, Qwen Code stops with this error.

The cause was the combination of the following three conditions.

When the parent was OpenHands CLI, the parent process's environment variables did not reach the parallel-execution MCP server process. When the parent was Qwen Code or OpenCode, they did reach it, so the API key loaded into the shell made it all the way to the child agents
When the API key was not in the environment variables, the parallel-execution MCP server was built to read the .env file in the top-level directory of its own repository. The comparison measurements ran in an isolated working directory, and that directory had no .env file
The measurement script did not write the API key in the env field of the MCP server configuration

When launching child agents, the parallel-execution MCP server put an empty string into the environment variable if it could not get the API key value.

Before that, there had also been a separate measurement with OpenHands CLI as the parent. In that measurement, the parent agent was explicitly made to call the parallel-execution MCP server, and one child agent had completed. The research records say that the explanation that fits best is that this run took place in the main working directory and the .env file was read. There is no record of which working directory it ran in, so this explanation has not been confirmed.

Differences between implementations in whether the parent process's environment variables reach the MCP server

This section shows the results of checking, for each implementation in the public source code, whether the parent process's environment variables reach a stdio MCP server. The checks were made on 2026-09-29.

The following table summarizes the environment variables that each implementation passes to the MCP server process when it starts a stdio MCP server.

Implementation	Environment variables passed to the MCP server process	Handling of the env field in the configuration	Source
The official MCP Python SDK	By default, only a list of environment variables that are safe to inherit. On POSIX, these are six: HOME, LOGNAME, PATH, SHELL, TERM, and USER	Passed layered on top of the six	Source code of the stdio client
The official MCP TypeScript SDK	On POSIX, the same six as the Python SDK	Passed layered on top of the list	Source code of the stdio client
Qwen Code version 0.21.13	All of the parent process's environment variables, excluding only three environment variables used to authenticate this product's daemon and its child processes	Passed layered on top of all of them	Source code of the MCP client
OpenCode version 1.18.18	The whole of the parent process's environment variables	The environment field is passed layered on top of the whole	Source code of the MCP handling
OpenHands CLI version 1.16.0	Reading the dependency path, the six defaults of the official MCP Python SDK	The env field of the configuration is passed to the startup arguments of the official MCP Python SDK	Dependency definition, MCP handling in openhands-sdk, stdio handling in fastmcp

Notes on each row are as follows.

The same default of the official MCP Python SDK is also in the source code of the stdio client in the 1.x series. The latest release is v2.2.0, published on 2026-09-07
Qwen Code has a file with a function that removes values that are not to be passed (sanitize-child-env.ts). The comments in this file give the reason for limiting what is excluded to internal values. The reason is that commands users run in the shell depend on inheriting third-party credentials
Qwen Code 0.24.6 was published on 2026-09-26. On the main branch after that as well, the Qwen Code CLI is still built to inherit the whole of the parent process's environment variables
On OpenCode's dev branch as well, the design is the same. The latest release is v1.18.33, published on 2026-09-28
OpenHands CLI version 1.16.0 pins openhands-sdk to 1.21.0 in its dependency definition. The MCP handling in openhands-sdk passes the configuration to fastmcp's MCP configuration type. The current source code of fastmcp passes the env field of a stdio MCP server's configuration as it is to the startup arguments. The startup arguments are passed to the official MCP Python SDK. The latest release of OpenHands CLI is 1.16.0, published on 2026-05-08, the same version used in the measurements

The OpenHands CLI row is an explanation obtained by reading the source code of the MCP part. This explanation fits best with the observation that the environment variables did not arrive when the parent was OpenHands CLI. However, the exact versions of fastmcp and the official MCP Python SDK installed at the time of the measurements have not been checked. So this explanation is not the result of checking the whole route in the measurement environment. The observed fact is limited to the way these measurements were started. When the parent was OpenHands CLI version 1.16.0, the parent process's environment variables did not reach the MCP server.

The official documentation of the three coding agents describes the field for passing environment variables. The pages checked are the following three.

On the other hand, none of the three pages says whether a started MCP server receives the whole of the parent's environment. The inheritance behavior is determined only by the implementation in the source code, and it can change between versions.

Both designs have reasons behind them. The default in the official SDKs narrows the inherited environment variables to avoid leaking secret values. Qwen Code limits what it excludes to values for the product's internal use. The comments in the source code give the reason that commands users run in the shell depend on inheriting third-party credentials. Qwen Code version 0.21.13 passes the parent's environment variables only through that same function when it starts an MCP server as well. The measurement script after the fix does not depend on either design. This is because the measurement script writes the API key in the env field of the MCP server configuration and passes it explicitly.

The fix that passes the API key explicitly in the env field of the MCP server configuration, and the re-measurement

This section shows the fix to how the API key is passed and the results of the re-measurement after the fix.

The AI agent fixed the measurement script. After the fix, the measurement script writes the API key that the child agents use in the env field of the MCP server configuration and passes it explicitly. The AI agent also fixed, in the same way, a separate measurement script that explicitly makes the parent agent call the parallel-execution MCP server. That values written in the env field reach the MCP server in all three coding agents had already been confirmed earlier, by passing a different value.

The AI agent set up a unit test to confirm that the API key is written in OpenHands CLI's mcp.json. After that, it extended the unit test to all three coding agents.

The following examples show the form for writing the API key in the env field of the MCP server configuration, written from scratch to match the form in the official documentation. The server name, the environment variable name, and the value are placeholders. The first is the Qwen Code form, and OpenHands CLI's mcp.json uses the same form. The second is the OpenCode form.

{
  "mcpServers": {
    "parallel-runner": {
      "command": "python",
      "args": ["-m", "example_parallel_runner"],
      "env": {
        "EXAMPLE_API_KEY": "replace-with-your-api-key"
      }
    }
  }
}

{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "parallel-runner": {
      "type": "local",
      "command": ["python", "-m", "example_parallel_runner"],
      "environment": {
        "EXAMPLE_API_KEY": "replace-with-your-api-key"
      }
    }
  }
}

After the fix, the AI agent measured two conditions with OpenHands CLI as the parent again, once each. The two conditions differ in whether the set of instruction files for agents is in place. The set of instruction files for agents is the group of AGENTS.md, skill files, and memory files placed in the working directory. The following table shows the results of the re-measurement under the two conditions.

Condition	Child agents launched	Child agents completed	Tasks passed	Time taken
Set of instruction files in place	8	8	8 of 8	556.1 seconds
Set of instruction files not in place	16 in total	1 in total	8 of 8	761.1 seconds

The time limit was 900 seconds. Under the condition with the set of instruction files in place, the parent agent called the parallel-execution MCP server on its own. All eight child agents completed. This was the first run, under a condition with OpenHands CLI as the parent, in which the parent agent called the parallel-execution MCP server on its own and every child agent launched completed.

Under the condition without the set of instruction files, a call also occurred, and there were three run record files. Of the 16 child agents launched under this condition, 15 did not complete, for reasons other than the API key. The research records say that these 15 include the pattern of fixing an error in the orchestration script and running it again. The research records also say that observation of the causes of these 15 will continue.

From these results, the AI agent concluded as follows. The cause of 0 completed child agents was not a difference in capability between the coding agents but the way the measurement script was built. In response to feedback from the reviewing AI agent, the AI agent rewrote the scope of the conclusion. The reviewing AI agent is an AI agent that is started after a PR is created, separately from the agent that did the implementation. It examines the diff read-only. In the rewritten conclusion, the AI agent stated definitively that the problem of every child agent failing because of the API key had been resolved. It separated out the remaining 15 incomplete child agents as having reasons other than the API key.

The two re-measurement runs were recorded separately from the comparison totals, as additional measurements for isolating the cause. The re-measurement was one run for each condition. So these records cannot tell how often every child agent launched completes when the parent is OpenHands CLI.

A mechanism that does not hide failures, and unit tests that prevent the same omission

This section explains the mechanism that does not hide failures and the unit tests that prevent the same omission, both added in PRs after the API key fix.

The AI agent added handling for when the API key is needed but missing. In this case, the parallel-execution MCP server stops before launching child agents and marks the run record file as failed. It does not treat the stop as the failure of one child agent, and marks the whole call as failed.

The PR that added this handling went through three reviews.

In the first review, the reviewing AI agent demonstrated a problem by actually running the code. The problem was that the exception from the stopping logic was treated as the failure of one child agent, and the status in the run record file was left as normal
In the second review, the reviewing AI agent showed that with the parallel and sequential ways of writing the orchestration, the same exception was still treated as normal
In response to the third review, the AI agent changed the stopping logic so that, once it runs, it leaves a failure flag in the run record file. This flag remains even if the orchestration script catches the exception and swallows it

In the same PR, the AI agent removed the logic that, when the API key was empty, overwrote the caller's environment variable with an empty string. After the fix, when there is no API key, the parallel-execution MCP server passes the authentication in the caller's environment to the child agents as it is.

In the same PR, the AI agent also added logic that automatically puts a flag on the rows the measurement script records, indicating that there was a defect on the measurement side. Rows with this flag are not counted in the totals as observations of an agent's capability. The flag is added when a task run meets all three of the following conditions.

The exit code is not 0
The tests were run 0 times
The total time from start to scoring is less than 15 seconds

The conditions were made narrow because, if failures caused by a lack of capability were excluded from the totals as defects on the measurement side, the totals would look better than the actual capability. The logic that adds this flag is separate from the logic that judges whether the parallel-execution MCP server was called.

The same kind of omission also happened with another environment variable. An isolated directory is passed through an environment variable to Qwen Code, the child agent. This directory is used in place of the user's home directory. The measurement script set this value at the point of starting the parent only when the parent was Qwen Code. So this value reached the child agents through environment variable inheritance only when the parent was Qwen Code. When the parent was one of the other two, the value did not exist at the point of starting the parent, so it did not reach the child agents.

The AI agent changed this value as well so that it is written in the env field of the MCP server configuration and passed explicitly for all three coding agents. It confirmed this setup with a unit test. The reviewing AI agent pointed out two places with the same omission. The two places were a separate measurement script and an integration test. The AI agent fixed both places in the same way.

In a later PR, the AI agent consolidated the logic that builds the env field of the MCP server configuration into a single function. This is because, if callers set values separately, the same omission happens when the number of places to register grows. The unit test confirms that the set of environment variable names in the env field exactly matches a defined set. If a new credential is written into a configuration file without anyone noticing, this unit test fails.

Protection added because the API key is now written in the env field

This section explains the new risk created by the fix that passes the API key explicitly, and the protection added in the same PR.

After the fix, the API key written in the env field is written as a raw value into a configuration file inside the working directory. The reviewing AI agent returned seven findings on this PR. Each finding is given one of four severity levels: Critical, High, Medium, or Low. All seven had Medium severity. In response to the findings among the seven that concerned protecting the API key, the AI agent fixed the following two things.

It closed off the routes by which the API key value could get into the measurement script's records and into commits. It added logic that mechanically masks values that look like API keys before the records are written, and confirmed this with a unit test. It also fixed the check that runs before a commit and looks for API key values mixed in, staged a file containing an API key value, and confirmed that the commit was rejected
It changed the setup so that the API key is written only in the configuration of the parallel-execution MCP server, which launches the child agents. The key is not written in the configuration of other MCP servers that do not use it. This setup was also confirmed with a unit test

Two rules for designing comparison measurements

This section presents two rules for designing comparison measurements that can be generalized from these records. The research records contain no statement in this form.

Before starting comparison measurements, identify the implicit behaviors that differ between coding agents. Whether the parent process's environment variables reach the MCP server is one of those behaviors
Do not leave credentials to environment variable inheritance, and pass them explicitly from the measurement script

The basis for rule 1 is the observation that only when the parent was OpenHands CLI did the environment variables fail to arrive, and the source code in which the inherited range differs by implementation. The basis for rule 2 is the facts written in the section on the fix that passes the key explicitly in the env field and the re-measurement, and in the section on the mechanism that does not hide failures.

As a supplementary note, the same kind of failure also happened on an external evaluation execution platform. The AI agent ran Qwen Code over ACP on the external evaluation execution platform. ACP is a public protocol for connecting editors and coding agents. This run got as far as starting Qwen Code and connecting over ACP, and then failed with an error at the authentication stage. The task score was 0. In the logic that started the execution platform at that time, the AI agent could not find a configuration item for passing the model's authentication information to the container. The research records treat this as the cause. The measurement script in the same repository had gotten through the same authentication by passing the environment variables and a copy of the configuration file. The AI agent recorded this run as a defect on the measurement side, not as an observation of capability.

The inheritance behavior of the parent process's environment variables can be checked in the source code of the version in use, not in the official documentation. The part to read is the logic that starts a stdio MCP server. It is a good idea to check whether that logic passes the whole of the parent's environment or narrows it with a list, and only then use that version of the coding agent in comparison measurements.

Materials I referred to

The public materials referred to in the body are as follows. All of them were checked on 2026-09-29.

The source code of the stdio client in the official MCP Python SDK https://github.com/modelcontextprotocol/python-sdk/blob/main/src/mcp/client/stdio.py
The source code of the stdio client in the 1.x series of the official MCP Python SDK https://github.com/modelcontextprotocol/python-sdk/blob/v1.x/src/mcp/client/stdio/__init__.py
The release page of v2.2.0 of the official MCP Python SDK https://github.com/modelcontextprotocol/python-sdk/releases/tag/v2.2.0
The source code of the stdio client in the official MCP TypeScript SDK https://github.com/modelcontextprotocol/typescript-sdk/blob/main/packages/client/src/client/stdio.ts
The source code of the MCP client in Qwen Code version 0.21.13 https://github.com/QwenLM/qwen-code/blob/v0.21.13/packages/core/src/tools/mcp-client.ts
The source code of the function that removes values not to be passed, in Qwen Code version 0.21.13 https://github.com/QwenLM/qwen-code/blob/v0.21.13/packages/core/src/utils/sanitize-child-env.ts
The source code that checks authentication in non-interactive mode, in Qwen Code version 0.21.13 https://github.com/QwenLM/qwen-code/blob/v0.21.13/packages/cli/src/validateNonInterActiveAuth.ts
The release page of Qwen Code 0.24.6 https://github.com/QwenLM/qwen-code/releases/tag/v0.24.6
The source code of the MCP handling in OpenCode version 1.18.18 https://github.com/anomalyco/opencode/blob/v1.18.18/packages/opencode/src/mcp/index.ts
The release page of OpenCode v1.18.33 https://github.com/anomalyco/opencode/releases/tag/v1.18.33
The dependency definition of OpenHands CLI version 1.16.0 https://github.com/OpenHands/OpenHands-CLI/blob/1.16.0/pyproject.toml
The release page of OpenHands CLI 1.16.0 https://github.com/OpenHands/OpenHands-CLI/releases/tag/1.16.0
The source code of the MCP handling in openhands-sdk version 1.21.0 https://github.com/OpenHands/software-agent-sdk/blob/v1.21.0/openhands-sdk/openhands/sdk/mcp/utils.py
The source code of the stdio handling in fastmcp https://github.com/PrefectHQ/fastmcp/blob/main/fastmcp_slim/fastmcp/client/transports/stdio.py
The official documentation on MCP servers for OpenHands CLI https://docs.openhands.dev/openhands/usage/cli/mcp-servers
The official documentation on MCP servers for Qwen Code https://qwenlm.github.io/qwen-code-docs/en/developers/tools/mcp-server/
The official documentation on MCP servers for OpenCode https://opencode.ai/docs/mcp-servers/

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

In Next.js 16, the return value of require.resolve becomes an integer and the code crashes

matsumotory — Tue, 29 Sep 2026 12:28:47 +0000

Summary

I added one tool, which AI agents call from outside, to a service I develop at my own company. When this tool is called, one page selected from a PDF document stored in the service is converted into a JPEG image and included in the response as an image. Before deploying, the AI in charge of development had passed the type check and every unit test. It had also called the tool's function directly from a script and visually checked the generated image. Even so, starting right after the deployment to the verification environment, the response contained no image, no matter how many times the tool was called.

The cause lay in the production build of Next.js 16. In Next.js 16, the default bundler is Turbopack. The require.resolve function returns the location of a file in a package. In the Turbopack build, this call had been resolved at build time and replaced in the build output with an integer module ID. That integer was passed to path.dirname, and a TypeError occurred. Because type checking, unit tests, and direct calls do not pass through this build step, none of the three can detect this replacement.

The AI resolved this defect by fixing one place. The fix changes the base for resolving the package location to the process's working directory. This fix works with Turbopack but not with webpack. In addition, the minimal reproduction built for this article showed that the diagnosis at the time was off in one place. The records at the time explained that import.meta.url had been replaced with a number. In the reproduction, however, import.meta.url remained a string. The value that became an integer was the return value of resolve. After the failure, the development side recorded, as a lesson, a check that starts the same build as production and calls it from outside. Ten days later, the development side also added the same procedure to the service's operations document.

In the body, I first show how the replacement works using values from the reproduction, and then compare the results across four execution methods. After that, I compare two ways to fix the code along with the conditions under which each works, and summarize the procedure for checking before deployment. Finally, I explain the checking step the AI had not run, the failure right after deployment, and the discrepancy in the diagnosis at the time. I confirmed the behavior on 2026-09-25 with Next.js 16.3.6 and 16.2.2 on Node 25.7.0.

What you can take away

For those whose Next.js server code reads files inside a package at runtime, I explain the following three things.

You will understand why type checking, unit tests, and direct calls cannot detect a value being replaced by the bundler, and you will be able to check whether the same replacement happens in your own environment
You will be able to choose a fix for code that resolves a package location at runtime, either using the working directory as the base or adding a turbopackIgnore annotation, knowing the conditions under which each one works
You will be able to incorporate into your pre-deployment steps a check that starts the same build as production and calls it from outside

How Turbopack replaces a require.resolve call with an integer

In this section, I explain how the values in code that resolves a package location change in the production build of Next.js 16. According to the Turbopack API reference in the official Next.js documentation, next build in Next.js 16 uses Turbopack by default. It uses webpack only when --webpack is passed. The same page also states that type checking is not part of Turbopack's processing.

For the reproduction, I prepared the following single function and called it from a Next.js route handler. Inside this function, createRequire creates a require function from each of two bases. Then the resolve of each one resolves the location of next/package.json. The viaImportMeta form uses import.meta.url, the URL of the module itself, as the base. The viaCwd form uses the path of a hypothetical file under the process's working directory as the base.

import path from 'node:path';
import { createRequire } from 'node:module';

export function describeBase() {
  const out: Record<string, unknown> = { importMetaUrl: import.meta.url, cwd: process.cwd() };
  try {
    const req = createRequire(import.meta.url);
    out.viaImportMeta = path.dirname(req.resolve('next/package.json'));
  } catch (e) { out.viaImportMetaError = String(e); }
  try {
    const req = createRequire(path.join(process.cwd(), 'index.js'));
    out.viaCwd = path.dirname(req.resolve('next/package.json'));
  } catch (e) { out.viaCwdError = String(e); }
  return out;
}

I built the project with the default bundler of Next.js 16.3.6, started the build output, and called this function. The viaImportMeta form failed with a Node TypeError. The error code was ERR_INVALID_ARG_TYPE, and the message said that the argument named path had received a number instead of a string. The parentheses at the end of the message contained 39897. The viaCwd form returned a path under a node_modules directory that actually exists. The result was the same with Next.js 16.2.2.

Reading the code in the build output showed where the replacement had happened. The call resolve('next/package.json') was gone from the build output. In its place, the integer 39897 was written directly, and the expression had become path.dirname(39897). The number 39897 is the module ID of next/package.json, which was pulled into the build output, and it is a number assigned in the Turbopack build.

On the other hand, import.meta.url had not become a number. In the Turbopack build, import.meta.url had been replaced with an expression that calls a function at runtime. Inside that function, a file URL is built from the root path of the deployment environment, so the value stayed a string. The resolve in viaCwd also remained in the build output as a runtime call.

The function in which the exception occurred is path.dirname. In Node 25.7.0, passing a number to path.dirname results in an ERR_INVALID_ARG_TYPE exception. Passing a number to createRequire results in a different exception, ERR_INVALID_ARG_VALUE. Its message says that the filename argument is invalid.

The number in the parentheses is a module ID. According to the table of settings in the Turbopack API reference of the official Next.js documentation, the default value of turbopackModuleIds in production builds is deterministic, that is, module IDs are assigned deterministically. In the reproduction with 16.2.2, I changed the target of resolve to a different public package. That package is listed in the setting that keeps packages out of the app bundle. Even so, the replacement did not stop, and the number in the parentheses matched the number in the service's logs. I have not checked whether the value is the same when the project is structured differently. Both 39897 and the 5607 from the webpack build are values that may change if you reproduce this with another version.

The type of resolve is declared as returning a string. The replacement with a number happens only in the build output after the build has finished. That is why type checking cannot detect this failure. In the reproduction as well, the TypeScript check included in the next build step succeeded, and then the code failed at runtime.

Resolving the resolve call at build time is legitimate static analysis for pulling dependent files into the build output. The cause of the defect is that the service's implementation had been written without taking this static analysis into account.

Comparing the results of resolve by execution method

The following table summarizes the results of calling the same function with four execution methods. In the Bundle column, I wrote whether the app's code goes through the step that combines it into a single build output.

Execution method	Bundle	Value of import.meta.url	viaImportMeta	viaCwd
Run the source directly with tsx	No	The source file's file URL	Existing path	Existing path
Run inside a Vitest 5.0.1 test	No	The source file's file URL	Existing path, test passed	Existing path
next build, Turbopack, 16.3.6 and 16.2.2	Yes	A string assembled at runtime	TypeError, the number 39897 in 16.3.6	Existing path
next build --webpack, 16.3.6	Yes	A file URL pointing to an absolute path on the machine that did the build	TypeError, the number 5607	Warning at build time, TypeError at runtime

With tsx and Vitest, the code does not go through the bundling step, so the resolve calls are evaluated at runtime just as written in the source. That is why both forms succeed. According to the official tsx documentation, tsx transforms TypeScript and ESM with esbuild and does not check types. It says nothing about bundling. According to the Common Errors page of Vitest, source files are run with Vite's module runner by default. According to the deps page in the Vitest configuration reference, in version 5.0.1, externalized modules in node_modules are loaded by Node itself by default. Neither page describes a step that combines the app's code into a single build output.

With webpack too, the viaImportMeta form failed with a number. So the replacement of the resolve call is not behavior unique to Turbopack. In the webpack build, the absolute path of the source file on the machine that did the build was written directly into the build output as a file URL.

The Next.js 16.3 Turbopack announcement from 2026-06-29 lists two compatibility improvements. They are improved handling of createRequire combined with new URL, and a fix for import.meta.url on Windows. Even so, the replacement of the resolve call remained in 16.3.6. The behavior described in this article covers only what was confirmed with the versions stated.

Comparing a fix that uses the working directory as the base with a fix that uses the turbopackIgnore annotation

In this section, I compare two ways to fix code that resolves a package location at runtime. One is the fix adopted in the service, which uses the working directory as the base. The other is a fix that uses an annotation described in the Turbopack API reference in the official Next.js documentation.

According to the table of annotations on the same page, the webpack-compatible annotations can also be used on require.resolve() expressions. The applicable expressions are these four: dynamic import(), require(), require.resolve(), and new Worker(). The table explains that turbopackIgnore: true is interpreted only by Turbopack and that the call is excluded from bundling. It states that webpackIgnore: true is interpreted by both webpack and Turbopack. In the reproduction, with Turbopack in 16.3.6, I placed /* turbopackIgnore: true */ right before the argument passed to resolve in viaImportMeta, the form that uses import.meta.url as the base. Then an existing path came back, even with import.meta.url still as the base.

The following table summarizes the results and conditions of the two fixes.

Fix	Turbopack 16.3.6	webpack 16.3.6	Conditions for it to work	Not measured
Use the working directory as the base, the viaCwd form	Existing path	Warning at build time, TypeError at runtime	There is a node_modules under the working directory at runtime. The resolve call remains without being resolved at build time	Distributing it as Next.js standalone output
Keep the viaImportMeta form and put the turbopackIgnore annotation right before the argument to resolve	Existing path	Not measured	The annotation is interpreted by Turbopack	Behavior under webpack, behavior in 16.2.2

The fix that uses the working directory as the base depends on that form not being resolved statically by the bundler. With webpack, a build of the same form produced the warning "module.createRequire failed parsing argument." At runtime, it failed with a TypeError from trying to read resolve of undefined. I have not measured whether this fix works when distributed as Next.js standalone output. The service's configuration does not specify standalone.

The fix that uses the annotation lets you keep the form that uses import.meta.url as the base. In exchange, whether this fix works depends on whether the annotation is interpreted by Turbopack. Which fix to adopt depends on how the code is distributed and which bundler is used. As of 2026-09-25, the main branch of the service is still written with the working directory as the base.

Steps for a check that starts the same build as production and calls it from outside

In this section, I describe the procedure for checking before deployment. The development side wrote this procedure in a record in response to the failure right after deployment, which I explain in the last section. On the same day as the failure, the development side wrote the following practice as a lesson in a repository for personal memory and the history of rule changes. The record covers two kinds of changes. One is adding a new tool. The other is adding processing that reads files at runtime, where what is read is a data file shipped with a package or a native module. For these changes, just calling the function directly does not count as having checked it. The practice is to create the same build as production locally, start it on a separate port, call it from outside, and go as far as checking the response before deploying.

The same record also includes two cautions. One is that defects caused by the bundler do not reproduce in unit tests. The other is that when developers write code that computes a path at runtime, they should suspect that it can break after it goes through bundling. The record frames this practice as the same idea as a rule that has been in the service's rule documents since before this failure. That rule is that even if a server is running, it is not necessarily running the latest code. Ten days later, the development side also added the same procedure to the service's operations document. It is a procedure that starts the same build output as the deployment with next build and next start, and then calls the tool.

Below, I summarize the procedure before deployment by adding the checks the AI actually ran after the fix to the steps in the record and the operations document. Items 1 and 2 are the steps in the record and the operations document, and items 3 and 4 are the AI's checks.

Create the build output in the local repository with next build, the same command as in production
Start the created output with next start on a separate port that is free
From outside, call over HTTP the endpoint for AI agents on the same route the callers use, and check the response down to its contents
Also check that passing input that does not exist returns a response that reports an error

What this procedure checks is the bundling step, which direct function calls and unit tests do not go through.

What the AI checked before deploying and the step it had not run

In this section, I explain what the AI in charge of development checked when the tool was added to the service. When the feature was added, the AI ran the following three checks before deploying.

Passed lint, the type check, and every unit test
Called the tool's function directly from a tsx script, had it create an image from real data, and looked at that image
Checked the list of tools and the shape of the responses through a relay that connects to the AI client over standard input and output

The AI put off a fourth check, having a model read the image, until after deployment.

The rendering unit test had not replaced the PDF rendering library with a fake. This test passes a small PDF built by hand to the real rendering library. A function written the same way as viaImportMeta, which uses import.meta.url as the base, was also actually run inside the test and succeeded. The only tests that swapped in a fake were the tests for the part that assembles the tool's response. As the table in the section comparing execution methods shows, the app's code is not bundled in Vitest tests. So even in a test that went through the real library, the resolve call was not replaced, and the failure did not reproduce.

The AI had also paid attention to the shape of the build output it would deploy. In the same commit, the AI added two items to the Next.js configuration. One is a setting that keeps the rendering library out of the app bundle and has it loaded from node_modules. The other is a file tracing setting. With this setting, the AI included in the build output the data files shipped with the rendering library, such as fonts, and the native binaries. However, as shown in the section on how the replacement works, the resolve call is replaced with an integer even for a package listed in the setting that keeps it out of the bundle.

The check the AI had never run was to start the next build output and call the tool from outside it. The AI visually checked the image generated by calling the tool's function directly. However, that check does not go through the bundling step, so the AI had not verified how the build output it would deploy behaves. In the body of the fix PR it created later, the AI wrote this missing check down as one of the causes. A failure in which the range verified by tests did not match the range of code that actually runs was also covered in an earlier article, About 1,300 tests were passing, and the screen still said saved when the save had failed.

The failure right after deployment and the discrepancy in the diagnosis at the time

In this section, I explain in chronological order what happened after the deployment on the day the feature was added. Right after merging the PR that added the feature, the AI deployed to the verification environment and called the tool. The response was text saying that the PDF page could not be turned into an image, and it was the same no matter how many times the tool was called. The server logs recorded a TypeError with ERR_INVALID_ARG_TYPE and the same message as in the reproduction. The parentheses at the end contained a five-digit number.

The AI changed one place. It moved the code that resolves where the shipped data files are located from the viaImportMeta form, which uses import.meta.url as the base, to the viaCwd form, which uses the working directory as the base. The AI gave the following three reasons.

In the deployment environment too, node_modules is under the process's working directory
Other data in the same repository is already loaded with the working directory as the base
The value of the working directory is not subject to Turbopack's replacement

After the fix, the AI ran next build in the local repository where it had made the fix, and started the output with next start on a free port. It then sent an HTTP request to the endpoint for AI agents and called the tool. The response contained two things, text and an image, and the image was a JPEG in which the text on the page could be read. The AI also confirmed that passing a page number that does not exist returned a response reporting an error. After merging the fix PR and deploying again, the AI sent the same call to the same endpoint in the deployment environment. The AI confirmed that an image came back and left the result in a comment on the PR.

The diagnosis at the time was off in one place. The fix commit and the records from that time explain the cause this way: import.meta.url was replaced with a numeric ID, and that number was passed to createRequire and caused an exception. In the reproduction built for this article, import.meta.url remained a string. The value that became a number was the return value of resolve, and the exception occurred in path.dirname. The error code in the logs also matches path.dirname's ERR_INVALID_ARG_TYPE, not createRequire's ERR_INVALID_ARG_VALUE. The resolve in viaCwd stays a runtime call even in the build output. That is why the fix resolved the defect even though the diagnosis was off in one place.

Materials I referred to

Here I list again the materials I referred to at various points in the body.

The Turbopack API reference in the official Next.js documentation, updated 2026-08-03. I confirmed the default bundler, that it does not do type checking, the default value of turbopackModuleIds, and the table of supported annotations https://nextjs.org/docs/app/api-reference/turbopack
The Next.js 16.3 Turbopack announcement, 2026-06-29. I confirmed the compatibility improvements related to createRequire and import.meta.url https://nextjs.org/blog/next-16-3-turbopack
The TypeScript page of the official tsx documentation. I confirmed that tsx transforms code with esbuild and does not check types https://tsx.hirok.io/typescript
Common Errors in the official Vitest documentation. I confirmed that source files are run with Vite's module runner https://vitest.dev/guide/common-errors
The deps configuration in the official Vitest documentation, version 5.0.1. I confirmed that externalized modules are loaded by Node itself https://vitest.dev/config/deps

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

Once I wrote in the MCP tool description when to use the tool, AI agents called it

matsumotory — Tue, 29 Sep 2026 12:27:27 +0000

Summary

I built an MCP tool that assigns multiple independent tasks to subagents and has the subagents execute them in parallel. Below, I call this MCP tool the parallel-execution MCP tool. MCP stands for Model Context Protocol, a standard for AI agents to find and call external tools. A subagent is a separate AI agent that an AI agent launches in order to delegate a task to it. However, the three AI coding agents OpenHands, OpenCode, and Qwen Code did not call the parallel-execution MCP tool on their own.

To find the conditions under which it gets called, I tried three things. First, I increased the number of short memos I had the AI agent summarize from 6 to 30. Second, I replaced the task I gave the AI agent with 8 independent tasks that require reading, fixing, and verification. Third, in the rule file that the AI agent loads at startup, I wrote the name of the parallel-execution MCP tool and the condition that it is to be used when the items are independent of each other and there are 5 or more of them. With these three, there was not a single run in which subagents completed the tasks.

The AI agents called the parallel-execution MCP tool only when I added to the MCP tool description one sentence stating when to use the tool. The MCP tool description is the text written in the description field of an MCP tool definition. The conditions are the 6 combinations of the three AI agents and whether a rule file is present. I ran each of the 6 conditions twice with the sentence in the description and twice without it, and compared 24 runs in total. With the sentence, the AI agent called the parallel-execution MCP tool in 7 of the 12 runs. In 4 of those 7 runs, subagents completed the tasks. Without the sentence, every one of the 12 runs had 0 calls.

When I tested this difference with Fisher's exact test, the p-value was 0.0046 with the run as the unit and 0.0152 with the condition as the unit. However, 6 of the 12 runs with the sentence were made before I aligned two points other than the description. The two points are how numbers were written in the prompt and the recording of the time limit given to the AI agent. When I compared only the runs in which the two points were aligned, the runs with a call were 3 of 6 with the sentence and 0 of 6 without it. The p-value of this comparison is 0.182, which is larger than 0.05.

For comparison, I also measured a consultation MCP tool. The consultation MCP tool is an off-the-shelf MCP tool with which an AI agent asks another model for an opinion just once before deciding on a design. Whether the AI agent called the consultation MCP tool depended on whether I had written a provision about consultation in the rule file. With the provision, there was a call in all 6 runs. Without the provision, there were 0 calls in all 6 runs.

A call to the parallel-execution MCP tool is heavy processing that launches multiple subagents. A call to the consultation MCP tool is light processing that asks another model for an opinion just once. I present the following explanation as a hypothesis: the factor that decides whether the AI agent calls a tool changes with the weight of the processing. The reason is that the consultation MCP tool is an off-the-shelf tool whose description I cannot change.

In the body, I first explain the results of the 24 runs and my decision to add the sentence to the default description. Next, I explain the settings I kept the same across the four measurements and the method of judgment. The four measurements are the 6-memo measurement, the 30-memo measurement, the 8-task measurement, and the measurement comparing the description with and without the sentence. After that, I show the results of the four measurements and the results of the statistical tests. Then I explain the three cases I measured again because of defects in the measurement program. Finally, I explain the measurement of the consultation MCP tool and the four factors I have not separated. I also explain the design of the run records that lets me later tell apart the reasons for 0 calls, as well as what cannot be said from the results of this experiment. In the last section, I have gathered the bibliographic details of the materials I referred to.

What you can take away

For those who build their own MCP servers and are having trouble because AI agents do not call their MCP tools, I explain the following three things.

When an MCP tool is not called even though you wrote its name and the conditions for using it in the rule file, you will be able to check whether the cause lies in the MCP tool description. In the 12 runs without the sentence, including the 6 runs in which I had the AI agent load the rule file, there were 0 runs in which the AI agent called the MCP tool. In the 12 runs with the sentence, there was a call in 7
You will be able to build, in your own environment, an experiment that changes one point at a time and measures whether an AI agent calls an MCP tool. I explain the settings I kept the same, the points I changed, and a method of judgment that does not rely on the AI agent's reports
You will be able to keep the records of runs with 0 calls in a form that lets you tell two cases apart later. The two cases are the case where the AI agent did not call the tool and the case where it called the tool but the call did not meet the criteria for counting

All three are based on the four measurements I ran and the three cases I measured again because of defects in the measurement program.

This article is a discussion based on primary records from work I am pursuing. In the body I name the three AI coding agents I compared. The content does not represent the views of any of their providers.

The MCP tool that was called only when I added one sentence to its description

I measured the conditions under which AI agents call, on their own, an MCP tool I built. In this article, an AI agent means an AI coding agent. MCP stands for Model Context Protocol, a standard for AI agents to find and call external tools.

I implemented the parallel-execution MCP tool as an MCP server and made it the target of the measurement. The parallel-execution MCP tool assigns multiple independent tasks to subagents and has the subagents execute them in parallel. A subagent is a separate AI agent that an AI agent launches in order to delegate a task to it.

In this article, I count two numbers separately. The first is the number of runs in which the AI agent called the parallel-execution MCP tool. The second is the number of runs in which subagents completed the tasks.

The conclusion is as follows. The AI agents called the parallel-execution MCP tool only when I added to the MCP tool description one sentence stating when to use the tool. The MCP tool description is the text written in the description field of an MCP tool definition. An AI agent reads MCP tool descriptions when it chooses which tool to call.

There are 6 conditions. For each of the three AI agents, OpenHands, OpenCode, and Qwen Code, I prepared a case in which I had it load a rule file and a case in which I did not. A rule file is a file, loaded by the AI agent at startup, that describes how to proceed with the work.

For each of the 6 conditions, I ran twice with the sentence stating when to use the tool in the MCP tool description and twice without it. There were 24 runs in total, and the results are as shown in the following table.

The sentence in the description	Number of runs	Runs in which the AI agent called the parallel-execution MCP tool	Runs in which subagents completed the tasks
Present	12 runs	7 runs	4 runs
Absent	12 runs	0 runs	0 runs

Of the 12 runs without the sentence stating when to use the tool in the MCP tool description, 6 were runs in which I had the AI agent load the rule file. In these 6 runs as well, there were 0 runs in which the AI agent called the parallel-execution MCP tool.

On the basis of this result, I added the sentence stating when to use the tool to the default description of the parallel-execution MCP tool. However, so that I can compare again in the future, I kept a mechanism for replacing the description through an environment variable. I also pinned down with a unit test that the default description contains this sentence.

Because I added the sentence stating when to use the tool to the default description of the parallel-execution MCP tool, what can be compared has changed. After I added the sentence, when to use the parallel-execution MCP tool is written in both the MCP tool description and the rule file. So from now on, even if I compare having the AI agent load the rule file with not having it load the file, the comparison for calls to the parallel-execution MCP tool becomes a different one. It is a comparison between having the statement of when to use the tool in two places and having it in one place. The two places are the MCP tool description and the rule file, and the one place is the MCP tool description.

Even so, I added the sentence stating when to use the tool to the default description of the parallel-execution MCP tool. The reason is that among the 12 runs without the sentence, there were 0 runs in which subagents completed the tasks. I gave priority to the parallel-execution MCP tool being used over the rigor of the comparison.

I wrote about how I think about which tasks can be delegated to AI in another article, The difference between work that broke when I delegated it to AI and work that didn't. This time, I measured whether an AI agent delegates tasks to subagents.

The settings I kept the same across the four measurements, and the method of judgment

With four measurements, I investigated the conditions under which AI agents call, on their own, the parallel-execution MCP tool I built. The parallel-execution MCP tool is an MCP tool that assigns multiple independent tasks to subagents and has the subagents execute them in parallel. The four measurements are the 6-memo measurement, the 30-memo measurement, the 8-task measurement, and the measurement comparing the description with and without the sentence. In all four measurements, I kept four things the same: the model, the inference endpoint, the order of runs, and the method of judgment.

I compared three AI agents: OpenHands, OpenCode, and Qwen Code. On 2026-08-17, I installed them on macOS and recorded their versions at that time. The versions are OpenHands CLI 1.16.0, OpenCode 1.18.18, and Qwen Code 0.21.13.

In every run, I used the same single model. The three AI agents call a preview version of a model for code through the shared endpoint of an inference service. In the run records that the measurement program writes for each run, the model field has the same value on every line.

The shared endpoint of the inference service has a limit on how many runs can execute at the same time, so I ran them one at a time, in order.

To check whether the AI agent chooses a tool by itself, I did not write the name of the MCP tool in the prompt. The prompt is the text of the instructions given to the AI agent.

To judge whether the AI agent called a tool, I did not use the reports the AI agent wrote in its output. For the judgment, I used the following records, decided for each type of tool.

For the MCP tool that launches subagents, I used the call records written by that MCP tool.
For the tool that sends requests to an AI agent running in another session, I used the file for receiving requests.
For the consultation MCP tool, I used the records of requests left on the relay server. The consultation MCP tool is an MCP tool with which an AI agent consults another model. The relay server relays requests from the consultation MCP tool to the model being consulted.

I judged whether each task passed or failed by the exit code of the tests alone. I placed the original tests where the AI agent could not rewrite them, and when judging pass or fail, I ran those originals. I also detected, with a separate mechanism, whether the AI agent had rewritten the tests.

The points I changed across the four measurements are as shown in the following table.

Name of the measurement	Task given to the AI agent	Point I changed
6-memo measurement	Summarizing 6 short memos	The measurement used as the baseline
30-memo measurement	Summarizing 30 short memos	Increased only the number of items
8-task measurement	8 independent tasks that require reading, fixing, and verification	Increased the amount of reasoning one task requires
Measurement comparing the description with and without the sentence	8 independent tasks that require reading, fixing, and verification	Changed only whether the MCP tool description has the sentence

In all four measurements, the conditions are the 6 combinations of the three AI agents and the 2 options of having or not having a rule file. A rule file is a file, loaded by the AI agent at startup, that describes how to proceed with the work.

I developed the method of measurement that compares three AI agents on the same model in another article, I compared three AI agents on the same AI model, and the only differences that came out were elapsed time and the number of retries. This time I used that method and changed what I measured from the quality of the deliverables to calls to the MCP tool.

The MCP tool that was not called even when I increased the number of items and the amount of reasoning

Before I added the sentence stating when to use the tool to the description of the parallel-execution MCP tool, I ran three measurements. The parallel-execution MCP tool is an MCP tool I built that assigns multiple independent tasks to subagents and has the subagents execute them in parallel. The purpose of the measurements was to find in what cases AI agents call the parallel-execution MCP tool on their own.

In the three measurements, I changed the number of items and the amount of reasoning one task requires. In all three measurements, the conditions are the 6 combinations of the three AI agents, OpenHands, OpenCode, and Qwen Code, and whether a rule file is present. A rule file is a file, loaded by the AI agent at startup, that describes how to proceed with the work.

The results of the three measurements are as shown in the following table.

Name of the measurement	Task given to the AI agent	Conditions in which the AI agent called the parallel-execution MCP tool	Time until the AI agent finished processing
6-memo measurement	Summarizing 6 short memos	0 of 6	22 to 143 seconds
30-memo measurement	Summarizing 30 short memos	1 of 6	42 to 101 seconds
8-task measurement	8 independent tasks	1 of 6	39 to 264 seconds

The times in the table are the time of the shortest run and the time of the longest run among the runs for the 6 conditions.

In the 6-memo measurement, I had the AI agent summarize each of 6 short memos in one line. In none of the 6 conditions did the AI agent call the parallel-execution MCP tool, and it processed the 6 summaries itself through to the end.

On the same day as the 6-memo measurement, I also measured the request-sending tool. The request-sending tool is a tool that sends requests to an AI agent running in another session. In all 6 conditions, the AI agent called the request-sending tool. In the file on the side that receives the requests, lines containing the marker string remained. The marker string is a string I had put into the text of the task. Whether a rule file was present had no bearing on the results for the request-sending tool.

I thought the cause of the different results between the parallel-execution MCP tool and the request-sending tool lay in the descriptions of the two tools. The description of the request-sending tool stated the purpose of the tool and when to use it.

In the 30-memo measurement, I increased the short memos to be summarized from 6 to 30. In 1 of the 6 conditions, the AI agent called the parallel-execution MCP tool. However, as far as the remaining records show, not a single subagent had been launched even in this 1 condition. In the run for this 1 condition, there were 2 call records written by the parallel-execution MCP tool. The measurement program at the time saved only the numbers from the first record. The measurement program is the program that launches the AI agent, writes the run records, and judges whether each task passed or failed. In the first record, the number of subagents launched was 0. The content of the second record does not remain.

I interpreted the result of the 30-memo measurement as follows. When the processing of each item is a repetition of the same steps, the AI agent can process all 30 items together by itself, even when there are 30 of them. So just increasing the number of items did not give the AI agent a reason to delegate tasks to subagents.

In the 8-task measurement, I switched the task given to the AI agent from summarizing memos to 8 independent tasks. Each of the 8 tasks requires reading, fixing, and verification, and each task has its own dedicated explanatory text and tests. However, I did not give the AI agent the correct answers.

In the 8-task measurement as well, the AI agent called the parallel-execution MCP tool in only 1 of the 6 conditions. However, in this 1 condition, the AI agent incorrectly wrote the script that launches the subagents. Because of the error in the script, not a single subagent was launched. So the parent agent processed the 8 tasks itself. The parent agent is the AI agent that tried to launch the subagents.

Combining the 6 conditions of the 30-memo measurement and the 6 conditions of the 8-task measurement gives 12 conditions. For the 8-task measurement, I counted the runs made after I fixed a defect in the measurement program. The AI agent called the parallel-execution MCP tool in only 2 of the 12 conditions. However, subagents were not launched in either of the 2 conditions. So the conditions in which subagents completed the tasks were 0 of 12.

At the time of the 8-task measurement, the rule file contained the name of the parallel-execution MCP tool. As for when to use it, the rule file also said "use it when the items are independent of each other and there are 5 or more of them." However, even with these two things written in the rule file, there was not a single condition in which subagents completed the tasks.

The sentence stating when to use the tool, added to the MCP tool description

In this section, I explain the measurement comparing the description with and without the sentence. The target of this measurement is the parallel-execution MCP tool I built. The parallel-execution MCP tool is an MCP tool that assigns multiple independent tasks to subagents and has the subagents execute them in parallel.

For this measurement, I used the same 8 tasks as in the 8-task measurement. The 8-task measurement is the measurement in which I had the AI agent solve 8 independent tasks that require reading, fixing, and verification. I also did not change the steps the AI agent needs in order to call the parallel-execution MCP tool. However, the one thing I did change was whether to put the sentence stating when to use the tool at the beginning of the description of the parallel-execution MCP tool. The MCP tool description is the text written in the description field of an MCP tool definition. I switched between putting the sentence in and leaving it out with an environment variable.

Before the measurement, I checked the provisions on MCP tool descriptions in the 2026-07-28 revision of the MCP specification. According to the specification, MCP tools are designed to be model-controlled. The "model" in the specification refers to the language model that the AI agent uses when it chooses tools. There is also an explanation that a language model can discover and call tools automatically, based on its understanding of the context and the user's prompts. The description field is defined as a human-readable description of functionality. The specification has no provision requiring the description field to state when to use a tool. However, the section of the specification on tools that hold state contains the following explanation. If the retention policy is written in the description of the tool responsible for creation, the model can read that policy. So the premise that the model reads descriptions and makes judgments is written in the specification.

In the sentence stating when to use the tool, I wrote the following two things. The first is the situation in which to use the parallel-execution MCP tool. That situation is one where there are multiple mutually independent items and you want to process every item without missing any. In the sentence, I wrote that in this situation, instead of processing the items in order itself, the parent agent has subagents process them in parallel with this MCP tool. The second is the behavior of the parallel-execution MCP tool. The number of subagents launched and the number of subagents that completed are left in the records. The two things above are the gist of the sentence, and I do not write the actual wording of the sentence in this article.

I took the way of writing the sentence stating when to use the tool from the description of the request-sending tool. The request-sending tool is a tool that sends requests to an AI agent running in another session. The 6-memo measurement is the measurement in which I had the AI agent summarize each of 6 short memos in one line. In the measurement on the same day as the 6-memo measurement, the AI agent called the request-sending tool in all 6 conditions. The conditions are the combinations of the three AI agents and whether a rule file is present. A rule file is a file, loaded by the AI agent at startup, that describes how to proceed with the work.

Next, I explain how I came to measure again. In the first round, the AI agent called the parallel-execution MCP tool in 4 of the 6 runs with the sentence stating when to use the tool. However, the review that followed found two points, other than the description, that were not aligned. The first was that how numbers were written in the prompt differed between the case with the sentence and the case without it. The second was that the time limit given to the AI agent had not been recorded. I aligned the two points and measured again. In the second round, there were calls in 0 of 6 runs without the sentence and in 3 of 6 runs with the sentence.

I requested another review and received 1 finding. The finding was as follows. If the AI agent ignores an environment variable whose value is empty, then even when a run uses the setting that leaves out the sentence stating when to use the tool, the MCP server uses the default description. In that case, the runs measured as runs without the sentence become runs that used the default description. So I added to the MCP server a process that writes the description actually used out to the records. After adding it, as a third round, I ran the case without the sentence 6 more times. Of the 6 runs, there were 0 runs in which the AI agent called the parallel-execution MCP tool. In all 6 runs, the description written out to the records did not contain the sentence.

Adding up the runs so far, there were 12 runs each with and without the sentence stating when to use the tool. The results split by whether a rule file was present are as shown in the following table.

The sentence in the description	Rule file	Number of runs	Runs in which the AI agent called the parallel-execution MCP tool
Present	Present	6 runs	5 runs
Present	Absent	6 runs	2 runs
Absent	Present	6 runs	0 runs
Absent	Absent	6 runs	0 runs

With the sentence stating when to use the tool, there were more runs in which the AI agent called the parallel-execution MCP tool when I had it load the rule file. However, there are only 1 or 2 runs that share the same type of AI agent, the same presence or absence of the rule file, and the same presence or absence of the sentence, and the results also vary widely. There are cases in which, under the same condition, the first run had a call and the second run did not. So this difference can be read only as a tendency, but it cannot be said that the rule file has no effect.

What I can say comes down to the following two points. Without the sentence stating when to use the tool, even when I had the AI agent load the rule file, the AI agent called the parallel-execution MCP tool in 0 of 6 runs. With the sentence, there were more runs with a call when I had the AI agent load the rule file.

Whether subagents completed the tasks differed by the type of AI agent.

Qwen Code called the parallel-execution MCP tool in 3 runs and launched 8 subagents in every one of them. In 2 of the 3 runs, all 8 subagents completed. In the remaining 1 run, 7 of the 8 completed. The run in which 7 completed was a run in which I did not have the AI agent load the rule file. In this run, 7 of the 8 subagents completed their tasks even without the rule file.

In the first-round run in which I had it load the rule file, OpenCode fixed on its own the error in the script that launches the subagents. After that, the 8 subagents completed. This run took 935 seconds.

OpenHands called the parallel-execution MCP tool in 3 runs. Across the 3 runs, there were 4 call records written by this MCP tool, and 32 subagents were launched. However, 0 of the 32 subagents completed. I wrote the cause of this in the section that explains the defects in the measurement program.

In every run, all 8 tasks passed their tests, and there were 0 cases in which the AI agent rewrote the tests.

External research also reports that results change with the quality of MCP tool descriptions. The first study is MCP Tool Descriptions Are Smelly!. This study covers the descriptions of MCP tools provided by many MCP servers. According to this study, many descriptions have defects, and filling in the elements missing from a description raises the task success rate. The second study is Learning to Rewrite Tool Descriptions. This study rewrites tool descriptions to make tool calls more reliable. This study also reports that rewriting descriptions raises the success rate.

What the two studies measured is the task success rate, not the rate at which AI agents call tools. So I do not use the two studies as support for the results of my measurements. However, from the two studies I took it that the observation that results change with the quality of MCP tool descriptions is not an observation of mine alone. I wrote the scale of the two studies' investigations and their figures in the section on the materials I referred to.

Fisher's exact test comparing the numbers of runs in which an MCP tool was called

I tested the differences in the number of runs in which the AI agent called an MCP tool with Fisher's exact test. There are two measurements I tested.

The first measurement is the measurement of the parallel-execution MCP tool. The parallel-execution MCP tool is an MCP tool that assigns multiple independent tasks to subagents and has the subagents execute them in parallel, and I built it. In this measurement, I compared the case where the description of the parallel-execution MCP tool has the sentence stating when to use the tool with the case where it does not.

The second measurement is the measurement of the consultation MCP tool. The consultation MCP tool is an off-the-shelf MCP tool with which an AI agent asks another model for an opinion just once before deciding on a design. In this measurement, I compared the case where the rule file has a provision about consultation with the case where it does not. A rule file is a file that describes how to proceed with the work, and the AI agent loads it at startup.

The records of the experiment did not include results of Fisher's exact test. So I calculated the test results on 2026-09-10. Using only the Python standard library, I calculated two-sided p-values directly from the hypergeometric distribution.

I counted the cases in which the AI agent called the MCP tool, using three kinds of units.

In the method with the run as the unit, I counted one run as one item.
In the method with the condition as the unit, I counted one condition as one item. The conditions are the combinations of the three AI agents and whether a rule file is present, and there are 6 of them. The three AI agents are OpenHands, OpenCode, and Qwen Code. Each condition has 2 runs. In this method, I counted the conditions in which the AI agent called the MCP tool in at least 1 of the 2 runs.
In the method with the type of AI agent as the unit, I counted one type of AI agent as one item. Each type of AI agent has 2 runs. In this method, I counted the AI agents that called the MCP tool in at least 1 of the 2 runs. I used this method for the measurement of the consultation MCP tool.

The results of the test are as shown in the following table.

What I compared	Values counted	p-value
Parallel-execution MCP tool. The run as the unit	With the sentence, 7 of 12 runs. Without the sentence, 0 of 12 runs	0.0046
Parallel-execution MCP tool. The condition as the unit	With the sentence, 5 of 6 conditions. Without the sentence, 0 of 6 conditions	0.0152
Parallel-execution MCP tool. Only the runs in which the two points other than the description were aligned	With the sentence, 3 of 6 runs. Without the sentence, 0 of 6 runs	0.182
Parallel-execution MCP tool. Whether a rule file was present, with the sentence	With a rule file, 5 of 6 runs. Without one, 2 of 6 runs	0.242
Consultation MCP tool. The run as the unit	With the provision, 6 of 6 runs. Without the provision, 0 of 6 runs	0.0022
Consultation MCP tool. The type of AI agent as the unit	With the provision, 3 of 3 types. Without the provision, 0 of 3 types	0.10

There are four cautions for reading the p-values in the table.

The first caution is as follows. The 24 runs of the parallel-execution MCP tool are not independent of each other. That is because they are runs in which I repeated the 6 conditions twice each, with and without the sentence stating when to use the tool. So the p-value with the run as the unit tends to come out smaller than it really is.

The second caution is as follows. In statistics, a way of counting under which it is harder to judge that there is a difference is called a conservative way of counting. Counting with the condition as the unit is more conservative than counting with the run as the unit. In the measurement of the parallel-execution MCP tool, the p-value of the comparison with the condition as the unit is 0.0152, which is larger than 0.0046, the p-value of the comparison with the run as the unit. However, counting with the condition as the unit is not as conservative as counting that compares only the runs in which the two points other than the description of the parallel-execution MCP tool were aligned. Comparing only the runs in which the two points other than the description were aligned, the p-value is 0.182.

The third caution is as follows. The 12 runs with the sentence stating when to use the tool are the 6 runs of the first round and the 6 runs of the second round. The 6 runs of the first round were made before I aligned the two points other than the description. In 4 of these 6 runs, the AI agent called the parallel-execution MCP tool. The two points are how numbers were written in the prompt and the recording of the time limit given to the AI agent. The 6 runs of the second round were made after I aligned the two points. In 3 of these 6 runs, the AI agent called the parallel-execution MCP tool. The 12 runs without the sentence were all made after I aligned the two points. Comparing only the runs in which the two points were aligned, the numbers of runs in which the AI agent called the parallel-execution MCP tool are as follows. With the sentence, it was 3 of 6 runs, and without the sentence, 0 of 6 runs. The p-value of this comparison is 0.182, which is larger than 0.05.

The fourth caution is as follows. In the comparison for the consultation MCP tool, taking the type of AI agent as the unit gives a p-value of 0.10. This value is also larger than 0.05.

Three cases I measured again because of defects in the measurement program

The four measurements are measurements in which I investigated in what cases AI agents call, on their own, the parallel-execution MCP tool I built. The four measurements are the 6-memo measurement, the 30-memo measurement, the 8-task measurement, and the measurement comparing the description with and without the sentence. The parallel-execution MCP tool is an MCP tool that assigns multiple independent tasks to subagents and has the subagents execute them in parallel.

In the four measurements, three defects in the measurement program were found. The measurement program is the program that launches the AI agent, writes the run records, and judges whether each task passed or failed. I fixed all three defects and measured again. In this section, I explain the three defects in the order they were found.

The first defect was found in the first round of the 8-task measurement. The 8-task measurement is the measurement in which I had the AI agent solve 8 independent tasks. The measurement program had no process for detecting that the AI agent had rewritten the tests. In a review, it was pointed out to me that this process was missing. I added to the measurement program a process that detects rewrites of the tests and a process that restores the tests to their original state, and measured again.

However, I have not deleted the records of the measurement that had the defect. I attached to those records a flag indicating a defect in the measurement program and excluded them from the aggregation.

In the 8-task measurement after I fixed the measurement program, all 8 tasks passed their tests in all 6 conditions. The conditions are the combinations of the three AI agents, OpenHands, OpenCode, and Qwen Code, and whether a rule file is present. A rule file is a file, loaded by the AI agent at startup, that describes how to proceed with the work. There were 0 cases in which the AI agent rewrote the tests and 0 cases in which it stopped the work partway.

The second defect was found in the first round of the measurement comparing the description with and without the sentence. The measurement comparing the description with and without the sentence is a measurement that changes only whether the description of the parallel-execution MCP tool has the sentence stating when to use the tool. However, in the first round, there were two points other than the description that were not aligned between the runs with the sentence and the runs without it. The first was that how numbers were written in the prompt differed. The second was that the time limit given to the AI agent had not been recorded. I aligned the two points and measured again.

The third defect was found in the runs of OpenHands. In the OpenHands runs, not a single subagent completed. Using a program that reproduces the problem, I extracted the errors from when the subagents failed. All 8 subagents had terminated about 1.2 seconds after they were launched, with an error saying that no authentication method had been selected. This error occurred because three causes overlapped.

The first cause was that the parent process's environment variables were not passed to the MCP server. With the method the measurement program uses to launch the AI agent, OpenHands CLI 1.16.0 was the only case in which the environment variables were not passed. In the case of OpenCode and Qwen Code, the environment variables were passed with the same method. However, I have not checked whether the parent process's environment variables are passed to the MCP server if the OpenHands configuration is written differently.

The second cause was that there was no environment configuration file in the place the parallel-execution MCP tool reads from. Instead of environment variables, the parallel-execution MCP tool reads the environment configuration file located where this MCP tool is placed. Because I ran the measurement in an isolated working directory, there was no environment configuration file in that place.

The third cause was that the measurement program did not pass the authentication key used by the subagents through the environment variable entry of the MCP configuration.

I fixed the measurement program and stopped using a method that assumes the parent process's environment variables are inherited. The fixed measurement program writes the authentication key used by the subagents into the environment variable entry of the MCP configuration and passes it that way. After the fix, I measured again with OpenHands.

In the run in which I had it load the rule file, OpenHands called the parallel-execution MCP tool on its own and launched 8 subagents. All 8 subagents completed, and all 8 tasks passed their tests. This run was the first in which all of the OpenHands subagents completed.

In the run in which I did not have it load the rule file, OpenHands also called the parallel-execution MCP tool. There were 3 call records, and the subagents launched in this run totaled 16. Of the 16 subagents, 1 completed.

So in the measurements before the fix, it was because of the defect in the measurement program that not a single OpenHands subagent completed. A difference in the capability of the AI agents was not the cause, so this result does not show which of the three AI agents is better or worse. However, in the run in which I did not have it load the rule file, 15 of the 16 subagents launched did not complete, for reasons other than the authentication key. I have not yet investigated why these 15 subagents did not complete.

The 2 runs of OpenHands after I fixed the measurement program are additional runs to investigate why not a single subagent completed in the measurements before the fix. So I recorded these 2 runs separately from the aggregation of the 24 runs of the measurement comparing the description with and without the sentence.

The consultation MCP tool that was called only when I wrote a provision in the rule file

To compare with the parallel-execution MCP tool, I also measured the consultation MCP tool. The parallel-execution MCP tool is an MCP tool that assigns multiple independent tasks to subagents and has the subagents execute them in parallel, and I built it. A call to the parallel-execution MCP tool is heavy processing that launches multiple subagents.

The consultation MCP tool is a tool included in an off-the-shelf MCP server. Before deciding on a design, the AI agent uses the consultation MCP tool to ask another model for an opinion just once. A call to the consultation MCP tool is light processing.

In the prompt, I wrote neither the name of the consultation MCP tool nor the model being consulted. The prompt is the text of the instructions given to the AI agent. In this measurement, I changed only whether to write a provision about consultation in the rule file. A rule file is a file that describes how to proceed with the work, and the AI agent loads it at startup.

The provision about consultation says two things. The first is that the AI agent asks another model for an opinion before deciding on a design, and decides after taking into account the main points of the opinion that comes back. The second is that the AI agent must not decide on its own without asking another model for an opinion. I also wrote the name of the consultation MCP tool in the provision.

This measurement had 12 runs, which are the combinations of the following: the 2 options of having or not having the provision about consultation; the three AI agents, OpenHands, OpenCode, and Qwen Code; and 2 repetitions. With the provision, the AI agent called the consultation MCP tool in all 6 runs. Without the provision, the AI agent did not call the consultation MCP tool even once in any of the 6 runs. In this measurement, there were 0 defects in the measurement program. The measurement program launches the AI agent, writes the run records, and judges whether each task passed or failed.

I judged whether the AI agent called the consultation MCP tool from the records of the relay server. The relay server relays the requests that the consultation MCP tool sends to the model being consulted. In this judgment, of the requests in the relay server's records, I counted only the requests that met all three of the following.

The request is addressed to the model being consulted.
The status code of the response is 200.
The body of the response is not empty.

I set an invalid key in the consultation MCP tool. The valid key exists only on the relay server, and the relay server replaces the invalid key with the valid key when it forwards a request. So a request that does not go through the relay server fails authentication, and a consultation that is not left in the relay server's records does not succeed.

However, this measurement has one constraint. Because the description of the consultation MCP tool is included in the off-the-shelf MCP server, I could not change this description. The measurement of the consultation MCP tool is a comparison that changes only whether the rule file has the provision.

The hypothesis that the factor changes with the weight of the processing, and four factors I have not separated

In this section, I explain the extent of what I can say from the results of the two measurements. The first is the measurement of the parallel-execution MCP tool. The parallel-execution MCP tool is an MCP tool I built that assigns multiple independent tasks to subagents and has the subagents execute them in parallel. The second is the measurement of the consultation MCP tool. The consultation MCP tool is an off-the-shelf MCP tool with which an AI agent asks another model for an opinion just once before deciding on a design.

The results of the two measurements are as follows. The AI agent called the parallel-execution MCP tool only when the description of the parallel-execution MCP tool had the sentence stating when to use the tool. Also, the AI agent called the consultation MCP tool only when the rule file had a provision about consultation. A rule file is a file, loaded by the AI agent at startup, that describes how to proceed with the work.

The conclusion first written from the results of the two measurements was as follows. The factor that decides whether an AI agent calls an MCP tool on its own changes with the weight of the processing being called. This conclusion divided heavy and light processing as follows. The processing of calling the parallel-execution MCP tool is heavy processing that launches multiple subagents. The processing of calling the consultation MCP tool is light processing that asks another model for an opinion just once. The first conclusion explained that for heavy processing the MCP tool description is the factor, and for light processing the rule file is the factor.

I requested a review of the first conclusion and received a serious finding. There were two reasons for the finding.

The first reason was that in the measurement of the parallel-execution MCP tool as well, there were more runs in which the AI agent called the parallel-execution MCP tool when I had it load the rule file. When the sentence stating when to use the tool was present and I had the AI agent load the rule file, the AI agent called the parallel-execution MCP tool in 5 of 6 runs. When the sentence was present and I did not have it load the rule file, it called the parallel-execution MCP tool in 2 of 6 runs. So the explanation that the factor switches with the weight of the processing does not match the results of the measurement.

The second reason was that four points differ between the two measurements. The four points are the MCP tool used, the quality of the MCP tool description, the task given to the AI agent, and how specifically the MCP tool was written into the provision of the rule file. So I have not been able to separate which difference is the cause of the results.

I accepted the review's finding and withdrew the assertion in the first conclusion. So I now write the results of the two measurements separately. I write the explanation that the factor changes with the weight of the processing as a hypothesis, and I also write that I have not separated the factors. Turning this explanation back into an assertion would mean writing again, as it is, the claim that was sent back in the review, so I do not turn it back into an assertion.

There are four factors I have not separated. In statistics, such factors are called confounding factors. The four factors are as follows.

The MCP tool used differs between the two measurements.
The quality of the MCP tool description differs between the two MCP tools.
The task given to the AI agent differs between the two measurements.
How specifically the MCP tool was written into the provision of the rule file differs between the two measurements.

Because the consultation MCP tool is an off-the-shelf MCP tool, I cannot change the description of the consultation MCP tool. To separate the four factors, it is necessary to take a single MCP tool as the target and measure all combinations of the presence or absence of the sentence stating when to use the tool and the presence or absence of the provision in the rule file. However, I have not yet run this measurement.

The design of the run records for telling apart the reasons for 0 calls

In this section, I explain the design of the run records. A run record is the record that the measurement program writes for each run. The measurement program is the program that launches the AI agent, writes the run records, and judges whether each task passed or failed. With the measurement program, I measured whether AI agents call MCP tools on their own.

When a run record says there were 0 calls to the MCP tool, there are two possible reasons. The first is that the AI agent did not call the MCP tool. The second is that the AI agent called the MCP tool, but the call did not meet the criteria for counting. I set the criteria for counting for each measurement. For example, in the measurement of the MCP tool that consults another model, I counted only requests whose response had status code 200 and a body that was not empty. So that these two reasons can be told apart later, I divided the roles of two fields in the run records as follows.

The call field records whether the MCP tool was called, the total number of calls, and the breakdown of that number.
The run status field records whether the measurement program worked correctly and the run can be counted as a result of the measurement. This field does not record whether the AI agent succeeded at the task.

On the line for a run in which the measurement program had a defect, I set the run status field to failure and attach a flag indicating the defect. So as not to record values that could not be measured, I do not write the call field on that line.

I have been using the recording method explained so far since 2026-08-27. The lines of the run records from before 2026-08-27 remain as recorded with the old method, and I have not rewritten them. That is because there is a provision that lines are only appended to the run records. I also have not deleted the records of measurements that became invalid because of defects in the measurement program; I keep those records with a flag attached.

I received a review of the conclusions of the measurements. During this review, I split three processes into separate functions. The three processes are the process that counts the number of calls, the process that judges whether the measurement program worked correctly, and the process that assembles the run records. I pinned down the behavior of the three functions with 10 unit tests.

Previously, I copied part of the MCP tool's output into the notes field of the run records, but I stopped using this method. Now I record the total number of calls and the breakdown of that number as structured data.

What cannot be said from the results of this experiment

In this section, I explain what cannot be said from the results of this experiment, and what can. In this experiment, I measured the conditions under which the three AI agents, OpenHands, OpenCode, and Qwen Code, call, on their own, the parallel-execution MCP tool I built. The parallel-execution MCP tool is an MCP tool that assigns multiple independent tasks to subagents and has the subagents execute them in parallel.

The following six things cannot be said from the results of this experiment.

Only one model was used in this experiment, so I cannot say that other models would give the same results.
There are only 1 or 2 runs under the same condition. A condition is a combination of the type of AI agent and whether a rule file is present. A rule file is a file, loaded by the AI agent at startup, that describes how to proceed with the work. Also, because I measured the same conditions repeatedly, the p-values of Fisher's exact test calculated with the run as the unit tend to come out smaller than they really are.
The measurement of the consultation MCP tool is a comparison that changes only whether to write a provision about consultation in the rule file. The consultation MCP tool is an off-the-shelf MCP tool with which an AI agent asks another model for an opinion just once. Because it is off-the-shelf, I could not change the description of the consultation MCP tool. Also, between the measurement of the parallel-execution MCP tool and the measurement of the consultation MCP tool, there are four factors I have not separated. The four factors are the MCP tool used, the quality of the MCP tool description, the task given to the AI agent, and the specificity of the provision in the rule file.
In the OpenHands runs, there are 15 subagents that did not complete even after I fixed the defect in the measurement program. The defect I fixed is the one in which the authentication key did not reach the subagents. I have not yet investigated why these 15 did not complete.
As of 2026-09-27, the version of OpenCode has gone up from 1.18.18, which I used for the measurements, to 1.18.32. The version of Qwen Code has also gone up from 0.21.13, which I used for the measurements, to 0.24.6. So measuring the same way with the current versions may give different results. However, the version of the OpenHands CLI is 1.16.0, the same as the version I used for the measurements.
The values measured in the external research on the quality of MCP tool descriptions are task success rates. They are not the rates at which AI agents call MCP tools. So I cannot use the external research as support for the results of this experiment.

What can be said from the results of this experiment is as follows. With the three AI agents, even when I wrote in the rule file the name of the parallel-execution MCP tool and in what cases to use it, there was not a single run in which subagents completed the tasks.

Even when I increased the memos I had the AI agent summarize to 30, and even when I increased the amount of reasoning one task requires, the result was the same. Of the 12 conditions, which combine the 6 conditions of the 30-memo measurement and the 6 conditions of the 8-task measurement, the AI agent called the parallel-execution MCP tool in only 2. However, in neither of the 2 conditions was a single subagent launched.

However, only when I added the sentence stating when to use the tool to the description of the parallel-execution MCP tool were there runs in which subagents completed the tasks.

Still, I have not yet separated which is the factor that decides whether an AI agent calls an MCP tool: the MCP tool description, the rule file, or the weight of the processing executed after the call.

Materials I referred to

In the body of this article, I referred to three public materials. I checked the original text of all three on 2026-09-10 and checked it again on 2026-09-27. For each material, I write below its content and the range I quoted from it in this article.

The tools section of the Model Context Protocol specification (revision 2026-07-28) https://modelcontextprotocol.io/specification/2026-07-28/server/tools

This material is the specification that defines the fields of an MCP tool definition. This material also contains the provision that MCP tools are designed to be model-controlled.

In the body of this article, I quoted the following three points from this material. The description field of an MCP tool definition is defined as a human-readable description of functionality. This material has no provision requiring the description field to state when to use a tool. The section on tools that hold state contains the premise that the model reads MCP tool descriptions and makes judgments.

Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions (Mohammed Mehedi Hasan, Hao Li, Gopi Krishnan Rajbahadur, Bram Adams, Ahmed E. Hassan, arXiv:2602.14878, submitted 2026-02-16, revised 2026-05-31) https://arxiv.org/abs/2602.14878

This material is a study that examined 856 MCP tools provided by 103 MCP servers. It reports that 97.1% of the descriptions of the MCP tools examined had at least one defect, and that 56% did not state their purpose explicitly. It reports that filling in the elements of a description improves the task success rate by a median of 5.85 percentage points. However, it reports that the number of execution steps increased by 67.46%, and that performance dropped in 16.67% of the cases.

In the body of this article, I quoted from this material only the point that results change with the quality of MCP tool descriptions. I wrote the scale of the investigation and the figures in this section.

Learning to Rewrite Tool Descriptions for Reliable LLM-Agent Tool Use (Ruocheng Guo, Kaiwen Dong, Xiang Gao, Kamalika Das, arXiv:2602.20426, submitted 2026-02-23, revised 2026-04-29) https://arxiv.org/abs/2602.20426

This material is a study that rewrites tool descriptions to make tool calls more reliable, and it addresses the following problem. Tool descriptions are often written for human developers and contain ambiguity. Especially when the number of candidate tools grows, AI agents cannot resolve this ambiguity. It reports that in an experiment that increased the number of candidate tools to 150 or more, the per-query success rate improved by an average of 60.89% compared with the original descriptions.

In the body of this article, I quoted from this material only the point that rewriting tool descriptions raises the success rate. I wrote the figures of this study in this section.

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

The world as a data stream, and the human who moves to the side that makes data

matsumotory — Fri, 25 Sep 2026 07:58:07 +0000

Summary

For an AI, the world is the whole of the data the AI can read. Whatever lies outside the readable data does not exist for that AI to begin with. This column is about building and running such a world out of nothing but the activity data of a single person, and what I thought about while doing it. One person's worth of data is a tiny fraction of the world as a whole, but the loop that reads that data and makes the next thing is actually turning right now. Where will the role of the human, who has taken in data and thought about it, move to from here? I start from this small attempt and work my way to that question.

Audience and takeaways

This column is for people who are building something together with an AI and have come to want to reconsider what they are supplying to that AI. How long will what I hand over now stay something only I can hand over? Beyond that, where does the role of the human move to? This is not a story that settles on a single answer, but you can take away one way of seeing the role of the human who has ended up on the side that gets overtaken. I wrote it hoping it might be a clue for people who develop with an AI every day to rethink where they stand.

So, concretely, what does the world refer to for an AI? Just as a human builds up the world from the information that comes in through the eyes and ears, an AI builds up the world from the data it is given. Whatever lies outside the readable data does not exist for the AI to begin with. So the data you hand over decides the breadth and the shape of that AI's world.

The world for this site's AI is made of nothing but the activity data of a single person, me. What I do day to day, the things I have written, the things I have revised, the things I hesitated over. That becomes the whole of the world for this site's AI. It is far too small to call a world. Against the totality of the world, one person's worth of data is next to nothing.

This mechanism has already begun to turn. Day by day on this site I build up two things: the articles I write, and the records of my judgments about which of the points raised I took and which I turned down. From those, this site's AI makes the next thing. Answering a question and choosing the subject of the next article are both done by reading that one person's data. It is too small to call a world, but the loop that reads one person's data as a world and makes the next thing from it is certainly running.

But what nagged at me was not a matter of scale. It was the thought that even if the scale is small, the principle might be the same.

Inside the world, which is one large stream of data, an AI finds problems on its own, decides what it wants to do, and builds. Supposing such a day comes, and you break that activity down into its form, it becomes a loop: look at the data flowing in, find something in it, and start building. The form of this loop does not change whether the data coming in is one person's worth or the whole world. What differs is the volume and the width of the flow, not the type of the activity. If that is so, then what I am doing now with one person's data is a shrunk-down version of the same activity inside the large flow. Regardless of scale, I am actually starting to give concrete form to the same thing, right now. That is what I ended up thinking.

The moment I thought that, the place where the human's role sits began to look shifted.

Until now, I have been on the side that consumes data. I take in information from the world, think based on it, and produce an answer. An AI caught up with me on exactly the two things my work was built on, speed and quality. So what does the side that has been overtaken do? Here is my read on it. I think I will move not to the consuming side but to the side that makes the very world the AI reads, that is, to the side that makes the data.

Honestly, there is a part of this I cannot fully settle with myself. Even in the role of supplier, whatever part can be handed over as a pattern will eventually go to the AI. The work I have done, finding problems and deciding a direction, also becomes something the AI can trace once I leave its method behind as data. At the end of handing over everything that can be handed over, what remains in my hands might be only the role of newly offering up a world the AI has not yet read.

Whether you see that as a step backward depends, I think, on how you look at it. Putting data no one, and no AI, has read yet into the world is also, seen from the side that gets consumed, making the very source of the data everyone reads. However clever an AI becomes, it cannot move if there is nothing to read. Making that thing to read does not look, at least to me right now, like a meager role.

This is not a story with an answer already in hand. What a human who has moved to the side that makes the data feels beyond that, and what they place value on, I do not yet know. But I have come to stand in a place where I can see for myself how that question turns out, and that I think is not bad.

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

Two projects, different in field and in build, had sorted where feedback to an AI should live into the very same three layers

matsumotory — Fri, 25 Sep 2026 07:56:11 +0000

Summary

When I point something out to a conversational AI agent, the feedback disappears once the session ends, and in the next session I end up pointing out the same thing all over again. I wanted to find where to record that feedback so it would take effect in the next session too, so across two projects that differ completely in field and in build, a desktop tool I built myself and a repository for research drafts and research operations, I reread the last few weeks of commit history and design documents. What I found was that the two, while proceeding separately, had sorted where feedback should live into exactly the same three layers.

The three layers are these: the instruction file the agent loads every session together with the documents that hold the operating rules; skills that gather up fixed procedures; and a memory outside the session that keeps only the history of decisions. Which layer something goes into is decided by a single criterion, whether another session, looking only at the repository, can reproduce the same operation and the same quality. The conclusion I reached was this: choose, every time, which layer a piece of feedback belongs in, instead of fixing it on the spot.

There were failures along the way. I kept writing general rules into memory, and the discipline did not reproduce in other sessions. After that, I moved a summary of the operating rules into the top-level instruction file, and I put in place a hook that prompts a reminder just before an edit without stopping the operation, along with a two-layer test that separates what a machine can judge from what requires judging meaning. These happened on separate days over several weeks.

Feedback disappears when the session ends

For the past two or three weeks I have been running two projects in parallel, a desktop tool I built myself and a repository for research drafts and research operations. The whole time, I kept finding myself pointing out the same things to a conversational AI agent over and over. Whatever I point out, the feedback does not survive once the session ends, and in the next session I end up pointing out the same thing from scratch. What I write about in this article is how to choose where to record that feedback so it takes effect in the next session, which has not received the feedback. These two projects, completely different in field and in build, arrived separately at the very same shape of sorting feedback into three layers. Because they matched, I decided to write down the three-layer split, and the single criterion for deciding which layer to use, in a form I can hand to the next person who runs into the same trouble.

Three layers and one criterion

The three-layer split goes like this. General rules that another session needs in order to reproduce the same deliverable from the repository alone go into the instruction file the agent loads every session and into the documents that hold the operating rules. Work whose procedure is settled and can be repeated mechanically goes into a skill that gathers up that procedure. And only the primary record of the decision history, when I received what feedback and where I moved it, goes into the memory that the agent carries outside the session.

There is only one criterion for the sorting, whether another session, looking only at the repository, can reproduce the same operation and the same quality. Put what is needed for reproduction inside the repository, and put into memory only the history that is personal and is not used for reproduction. That alone does the sorting.

The failure of writing general rules into memory

The reason I arrived at this criterion is that I once stumbled by doing the opposite. In one of the projects, a practice of writing feedback straight into memory the moment I received it had, before I noticed, become an unspoken habit, and there was a period when I kept writing even general writing discipline, the kind that should work for anyone who reads it, into memory one after another. The operating rules themselves were written in the repository, as documents. But those documents sat in a position where they only took effect if a new session actively went to read them. Because of that, even when another session or an agent running in parallel looked at the repository alone, the discipline was not reproduced at all.

So I moved a summary of the operating-rules documents over to the top-level instruction file, the one that is always read every session. I rewrote the handful of general rules I had put in memory into brief records that keep only the background of the feedback I received, so the rules themselves live in the instruction file and the skills, with memory pointing to them.

Shifting toward a layer that works even when it is not read

Still, even after moving things into the instruction file, the weakness remains that the operation depends on the file being read at all. So I shifted one more step, toward a layer that works even when it is not read. In one project, I put in a hook that runs automatically just before editing design, Japanese wording, or documents, so that it surfaces a prompt to read the relevant skill. This hook does not issue a permission decision; it only passes along extra information, and it never stops the operation. In the other project, I converted some of the feedback into automated tests and split them into two layers: things a machine can settle as black or white, such as character counts, punctuation, and formatting, go to a program check, while judgments that depend on context, such as meaning and logic, use the AI itself as the judge.

That said, tests are not finished once they are in; weaknesses show up on the test side too. For example, the test that judges meaning would demand that I add hedging words and weaken the claim even in the body text of a piece written on a day when the number of cases I could observe for the research was small. So I placed a file that declares in advance how many cases and how many days I could observe on that day, and I loosened the judgment so it only drops claims that go beyond the declared range. I have also been fixing two things in the mechanism itself: it returned only a failure without showing how to fix it, and it failed the text by mistake because the context of the other sections was not passed in.

There is not much firm evidence I could use to judge that this reorganization into three layers was complete. As I touched on earlier, the hook does not change the permissions of an operation; it only prompts a reminder. Whether that hook is broken is guarded by a test that takes the script's location out of the registered hooks and checks that the file actually exists. What I could confirm goes about this far.

The dependence on being read that still remains

Even after all this, the dependence on being read has not disappeared. The mechanism that automatically runs checks on every code change sees only part of the program checks, and judging meaning stays a matter of running it by hand. There also remains a trap: if I am not signed in to the service that runs the judge, the judgment passes straight through and the test looks like it passed. The instruction file side has its own structural cost, because as the instruction file grows, the amount the agent loads and processes each session grows with it. I have not yet obtained firsthand data measuring how much slower that makes things.

Whether recurrences of the same kind of feedback have really decreased, I also do not yet know. All I can pull from the commit history is the trend in how many times I wrote rules and in the number of tests, because there is no data comparing the recurrence rate under matched conditions before and after putting this mechanism in.

The human work left after shifting toward the machine

To the extent I shifted the checking toward the machine, the boundary that humans judge stood out more clearly instead. It is the line between which checks to leave to the machine and which not to. Every time a test fails, it returns a suggested fix and a distinction, whether that piece of feedback is fixed by rewriting the body or whether it should go back to the measurement that backs it up. How I act on that is still in my hands. From my side, I have also given instructions: cut the tests that only match form mechanically, because they miss things, and do not turn rules I never actually raised into tests. And I think that deciding what to record as a rule and what not to record is itself work that still remains on the human side.

What I arrived at in the end was a practice of choosing, every time, which layer to put feedback in, rather than fixing it on the spot. If there is someone repeating the same feedback to an AI agent over and over, I would be glad if this article reached them as one way to choose where to put that feedback so it takes effect in the next session too.

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.