DEV Community: matsumotory

The world as a data stream, and the human who moves to the side that makes data

matsumotory — Fri, 25 Sep 2026 07:58:07 +0000

Summary

For an AI, the world is the whole of the data the AI can read. Whatever lies outside the readable data does not exist for that AI to begin with. This column is about building and running such a world out of nothing but the activity data of a single person, and what I thought about while doing it. One person's worth of data is a tiny fraction of the world as a whole, but the loop that reads that data and makes the next thing is actually turning right now. Where will the role of the human, who has taken in data and thought about it, move to from here? I start from this small attempt and work my way to that question.

Audience and takeaways

This column is for people who are building something together with an AI and have come to want to reconsider what they are supplying to that AI. How long will what I hand over now stay something only I can hand over? Beyond that, where does the role of the human move to? This is not a story that settles on a single answer, but you can take away one way of seeing the role of the human who has ended up on the side that gets overtaken. I wrote it hoping it might be a clue for people who develop with an AI every day to rethink where they stand.

So, concretely, what does the world refer to for an AI? Just as a human builds up the world from the information that comes in through the eyes and ears, an AI builds up the world from the data it is given. Whatever lies outside the readable data does not exist for the AI to begin with. So the data you hand over decides the breadth and the shape of that AI's world.

The world for this site's AI is made of nothing but the activity data of a single person, me. What I do day to day, the things I have written, the things I have revised, the things I hesitated over. That becomes the whole of the world for this site's AI. It is far too small to call a world. Against the totality of the world, one person's worth of data is next to nothing.

This mechanism has already begun to turn. Day by day on this site I build up two things: the articles I write, and the records of my judgments about which of the points raised I took and which I turned down. From those, this site's AI makes the next thing. Answering a question and choosing the subject of the next article are both done by reading that one person's data. It is too small to call a world, but the loop that reads one person's data as a world and makes the next thing from it is certainly running.

But what nagged at me was not a matter of scale. It was the thought that even if the scale is small, the principle might be the same.

Inside the world, which is one large stream of data, an AI finds problems on its own, decides what it wants to do, and builds. Supposing such a day comes, and you break that activity down into its form, it becomes a loop: look at the data flowing in, find something in it, and start building. The form of this loop does not change whether the data coming in is one person's worth or the whole world. What differs is the volume and the width of the flow, not the type of the activity. If that is so, then what I am doing now with one person's data is a shrunk-down version of the same activity inside the large flow. Regardless of scale, I am actually starting to give concrete form to the same thing, right now. That is what I ended up thinking.

The moment I thought that, the place where the human's role sits began to look shifted.

Until now, I have been on the side that consumes data. I take in information from the world, think based on it, and produce an answer. An AI caught up with me on exactly the two things my work was built on, speed and quality. So what does the side that has been overtaken do? Here is my read on it. I think I will move not to the consuming side but to the side that makes the very world the AI reads, that is, to the side that makes the data.

Honestly, there is a part of this I cannot fully settle with myself. Even in the role of supplier, whatever part can be handed over as a pattern will eventually go to the AI. The work I have done, finding problems and deciding a direction, also becomes something the AI can trace once I leave its method behind as data. At the end of handing over everything that can be handed over, what remains in my hands might be only the role of newly offering up a world the AI has not yet read.

Whether you see that as a step backward depends, I think, on how you look at it. Putting data no one, and no AI, has read yet into the world is also, seen from the side that gets consumed, making the very source of the data everyone reads. However clever an AI becomes, it cannot move if there is nothing to read. Making that thing to read does not look, at least to me right now, like a meager role.

This is not a story with an answer already in hand. What a human who has moved to the side that makes the data feels beyond that, and what they place value on, I do not yet know. But I have come to stand in a place where I can see for myself how that question turns out, and that I think is not bad.

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

Two projects, different in field and in build, had sorted where feedback to an AI should live into the very same three layers

matsumotory — Fri, 25 Sep 2026 07:56:11 +0000

Summary

When I point something out to a conversational AI agent, the feedback disappears once the session ends, and in the next session I end up pointing out the same thing all over again. I wanted to find where to record that feedback so it would take effect in the next session too, so across two projects that differ completely in field and in build, a desktop tool I built myself and a repository for research drafts and research operations, I reread the last few weeks of commit history and design documents. What I found was that the two, while proceeding separately, had sorted where feedback should live into exactly the same three layers.

The three layers are these: the instruction file the agent loads every session together with the documents that hold the operating rules; skills that gather up fixed procedures; and a memory outside the session that keeps only the history of decisions. Which layer something goes into is decided by a single criterion, whether another session, looking only at the repository, can reproduce the same operation and the same quality. The conclusion I reached was this: choose, every time, which layer a piece of feedback belongs in, instead of fixing it on the spot.

There were failures along the way. I kept writing general rules into memory, and the discipline did not reproduce in other sessions. After that, I moved a summary of the operating rules into the top-level instruction file, and I put in place a hook that prompts a reminder just before an edit without stopping the operation, along with a two-layer test that separates what a machine can judge from what requires judging meaning. These happened on separate days over several weeks.

Feedback disappears when the session ends

For the past two or three weeks I have been running two projects in parallel, a desktop tool I built myself and a repository for research drafts and research operations. The whole time, I kept finding myself pointing out the same things to a conversational AI agent over and over. Whatever I point out, the feedback does not survive once the session ends, and in the next session I end up pointing out the same thing from scratch. What I write about in this article is how to choose where to record that feedback so it takes effect in the next session, which has not received the feedback. These two projects, completely different in field and in build, arrived separately at the very same shape of sorting feedback into three layers. Because they matched, I decided to write down the three-layer split, and the single criterion for deciding which layer to use, in a form I can hand to the next person who runs into the same trouble.

Three layers and one criterion

The three-layer split goes like this. General rules that another session needs in order to reproduce the same deliverable from the repository alone go into the instruction file the agent loads every session and into the documents that hold the operating rules. Work whose procedure is settled and can be repeated mechanically goes into a skill that gathers up that procedure. And only the primary record of the decision history, when I received what feedback and where I moved it, goes into the memory that the agent carries outside the session.

There is only one criterion for the sorting, whether another session, looking only at the repository, can reproduce the same operation and the same quality. Put what is needed for reproduction inside the repository, and put into memory only the history that is personal and is not used for reproduction. That alone does the sorting.

The failure of writing general rules into memory

The reason I arrived at this criterion is that I once stumbled by doing the opposite. In one of the projects, a practice of writing feedback straight into memory the moment I received it had, before I noticed, become an unspoken habit, and there was a period when I kept writing even general writing discipline, the kind that should work for anyone who reads it, into memory one after another. The operating rules themselves were written in the repository, as documents. But those documents sat in a position where they only took effect if a new session actively went to read them. Because of that, even when another session or an agent running in parallel looked at the repository alone, the discipline was not reproduced at all.

So I moved a summary of the operating-rules documents over to the top-level instruction file, the one that is always read every session. I rewrote the handful of general rules I had put in memory into brief records that keep only the background of the feedback I received, so the rules themselves live in the instruction file and the skills, with memory pointing to them.

Shifting toward a layer that works even when it is not read

Still, even after moving things into the instruction file, the weakness remains that the operation depends on the file being read at all. So I shifted one more step, toward a layer that works even when it is not read. In one project, I put in a hook that runs automatically just before editing design, Japanese wording, or documents, so that it surfaces a prompt to read the relevant skill. This hook does not issue a permission decision; it only passes along extra information, and it never stops the operation. In the other project, I converted some of the feedback into automated tests and split them into two layers: things a machine can settle as black or white, such as character counts, punctuation, and formatting, go to a program check, while judgments that depend on context, such as meaning and logic, use the AI itself as the judge.

That said, tests are not finished once they are in; weaknesses show up on the test side too. For example, the test that judges meaning would demand that I add hedging words and weaken the claim even in the body text of a piece written on a day when the number of cases I could observe for the research was small. So I placed a file that declares in advance how many cases and how many days I could observe on that day, and I loosened the judgment so it only drops claims that go beyond the declared range. I have also been fixing two things in the mechanism itself: it returned only a failure without showing how to fix it, and it failed the text by mistake because the context of the other sections was not passed in.

There is not much firm evidence I could use to judge that this reorganization into three layers was complete. As I touched on earlier, the hook does not change the permissions of an operation; it only prompts a reminder. Whether that hook is broken is guarded by a test that takes the script's location out of the registered hooks and checks that the file actually exists. What I could confirm goes about this far.

The dependence on being read that still remains

Even after all this, the dependence on being read has not disappeared. The mechanism that automatically runs checks on every code change sees only part of the program checks, and judging meaning stays a matter of running it by hand. There also remains a trap: if I am not signed in to the service that runs the judge, the judgment passes straight through and the test looks like it passed. The instruction file side has its own structural cost, because as the instruction file grows, the amount the agent loads and processes each session grows with it. I have not yet obtained firsthand data measuring how much slower that makes things.

Whether recurrences of the same kind of feedback have really decreased, I also do not yet know. All I can pull from the commit history is the trend in how many times I wrote rules and in the number of tests, because there is no data comparing the recurrence rate under matched conditions before and after putting this mechanism in.

The human work left after shifting toward the machine

To the extent I shifted the checking toward the machine, the boundary that humans judge stood out more clearly instead. It is the line between which checks to leave to the machine and which not to. Every time a test fails, it returns a suggested fix and a distinction, whether that piece of feedback is fixed by rewriting the body or whether it should go back to the measurement that backs it up. How I act on that is still in my hands. From my side, I have also given instructions: cut the tests that only match form mechanically, because they miss things, and do not turn rules I never actually raised into tests. And I think that deciding what to record as a rule and what not to record is itself work that still remains on the human side.

What I arrived at in the end was a practice of choosing, every time, which layer to put feedback in, rather than fixing it on the spot. If there is someone repeating the same feedback to an AI agent over and over, I would be glad if this article reached them as one way to choose where to put that feedback so it takes effect in the next session too.

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

Matching strings do not settle the direction: which document is the original

matsumotory — Fri, 25 Sep 2026 07:56:07 +0000

Summary

I run a check that compares the body text of articles before publication with the text of my source material, string by string. The source material is the private development repositories I draw on for the articles.

On 2026-07-18, 3 articles waiting for publication failed this check. The cause was not that the articles had copied internal text from the source material. It was because one file in the source material, a file that analyzed this blog's published articles, quoted body text I had already published. The copying document and the copied document were the other way around.

The same check also has a blind spot in the opposite direction. As long as I exclude my own repository from the targets of matching, a draft that echoes the wording of my own internal documents cannot be found by the machine matching.

For the false positive where I mistook the direction, I revised the targets of matching, and for the defect in allowed words, I fixed how matches are counted. Neither fix loosened the criteria for the judgment itself. Copies taken from my own repository are still found by an AI review that reads the meaning.

What you can take away

For those who have a check based on string matching in their own operations, I explain the following three things.

You will be able to choose, at design time, the evidence that decides the direction of a match, that is, which document is the original. I decide it with time information rather than with strings, and I write out the three dates used in that judgment
You will be able to estimate how large a blind spot remains in a check that excludes itself from the targets of matching. I describe a real case that the machine matching could not find and only an AI reading the meaning found, and why I set aside the idea of putting my repository back among the targets
You will be able to plan ahead for how the way you record already-judged matches can force you to judge them again. I describe a real case where I had to judge the same position a second time

This article is a discussion based on primary records from operations I continue as individual research.

What a string match does not settle

I start with the conclusion. Even if strings match between two documents, that does not settle which one is the original. Nor can the match itself tell you whether one copied the other, or whether both simply used the same public identifiers or naming habits.

I made three errors with this check: a false positive from mistaking the direction, a false negative from excluding myself from the targets, and a defect in the handling of allowed words. A false positive is the error of judging something that was not copied to be a copy, and a false negative is the error of missing a copy. All three come from one and the same property: I tried to decide the direction of the original, and whether a text was mine or someone else's, from the matching strings alone.

I judge the direction from time information, not from string matches. I compare three dates queried with git. They are the date the article body was last changed, the date of the last full matching run that found 0 matches, and the date that text entered the source material.

External plagiarism detection rests on the same premise. I confirmed the following three sources in their original text on 2026-09-16. According to the English Wikipedia entry Content similarity detection, an external detection system is defined as one that compares a suspicious document with a reference collection. That collection is a set of documents assumed to be genuine. In the generated plagiarism detection task of PAN at CLEF 2025 as well, suspicious documents and source documents are separated in advance as part of how the data is constructed. According to Reception Reader, a study dealing with text reuse in historical materials, pairwise similarity comparison is reported not to explain the direction of reuse over time. The same study lists, as future development, giving the direction by a process that links text fragments back to their earliest occurrence.

How the check that detects copies is built

Before going into the errors, I show what this check matches and how. The following table is the configuration as of 2026-09-16.

Item	Current configuration
Targets of matching	The body text of each published, awaiting-publication, and draft article, excluding the record metadata yaml and code blocks
What they are matched against	The working trees and commit messages of the source material repositories
Length counted as a match	16 or more consecutive characters
Output	Only which position in which file had a match of what length, and the identifier of the matched source material
When it cannot judge	It stops running and lets nothing through
What it excludes from the targets	This platform's own repository
Scale	98 target files, 12 repositories matched against

The minimum match length of 16 characters is the result of measurement. At 8 characters, so many common technical Japanese phrases and command strings matched by chance that the results were useless for judgment. The check does not output the matching strings themselves because the check's output would itself become a new path for leaks.

This platform itself is excluded from the targets because the article files exist in its working tree. Without the exclusion, an article always matches itself. Documents in the source material that contain quotations from this platform can also be excluded from matching by specifying their paths. But only documents that quote content originating here may be excluded. If the source material's own internal documents were put on that exclusion list, no match that ought to be stopped would be detected at all.

A reverse-direction false alarm caused by quotations in a source document

The first false alarm was on 2026-07-18. In a bulk matching run, 3 articles in the stock waiting for publication failed. The match lengths were 16, 24, and 16 characters, and the same single file also matched 8 published article directories.

The other side of the match was a file in one of the source materials, the repository for research drafts and research operations. This file analyzed the published articles of this blog. On 2026-09-16 I checked it again, read-only. This file was added on 2026-07-16 and has not changed since that day. I had not copied internal text of the source material into the articles; the source document quoted body text already published here.

At that time I added a mechanism that specifies, per source material, paths to exclude from matching. I also wrote into the rule documents the restriction that only documents quoting content originating from this platform may be excluded.

This false alarm was reproduced on 2026-09-16 as well. I ran the reproduction in a copied working directory set to the same state as production, and referred to the original repository read-only. Removing just one exclusion produces 29 matches against that same single file from 9 published articles. Of these, 21 are body text of 6 articles published between 2026-07-07 and 2026-07-11. Those were published before 2026-07-16, when that single file was created, so they are reverse-direction matches where prose from here was taken into the source document. The longest was 65 characters. The remaining 8 are fragments of 3 articles published between 2026-08-06 and 2026-09-03. Their content was the paths of published article directories and the names of skills I have published. Removing all 9 exclusions gives 58. These counts of 29 and 58 are under the reproduction condition with exclusions removed, so they cannot be compared side by side with the counts for the production configuration I describe later.

The blind spot left by excluding my own repository

While writing one draft, I had echoed the wording of other documents in two places in my draft. The documents I echoed were this repository's own rule documents and the implementation of the reader-facing chat. The lengths were 34 and 19 characters respectively. The machine matching could not detect them, and only an AI reading the meaning found the echoed wording. As long as I exclude my own repository from the targets, wording echoed from my own internal documents cannot be found by the machine matching.

I considered simply putting it back among the targets and set that idea aside. This repository's rule documents quote the sentences of articles themselves as examples of review comments. The handoff records and operations data contain summaries of the articles' content. A machine cannot distinguish a quotation that went from an article into a document from a copy that went from a document into an article. Putting the repository back among the targets would make the list of exclusions too broad, and the check would become a formality.

The approach I am now moving toward is self-matching against only those places that, by their structure, contain no quotations of articles. The code of server-side processing, the site implementation, and scripts fall into those places. There is nowhere in the code that quotes an article, so the ambiguity of direction does not arise. For now, copies in this range are found by the AI review that reads the meaning.

The defect in handling allowed words

The third error, the defect in handling allowed words, was a defect in how the content of a match was counted. This check has an allow list of general names registered as approved for publication and widely known technical terms. What remains after removing those from a match is the specific content. On 2026-08-13, I fixed a defect where an allowed word could not be removed when a match boundary fell inside the word. Before the fix, fragments of the product names of public AI models were counted as specific content. The fix changes the counting so that it first finds the ranges the allowed words occupy on the full text of the line and counts only the characters not covered. I ran the check over all published articles and drafts before and after the change, and confirmed that matches already judged continued to be detected.

Recording matches already judged, and judging the direction again

Once I finish judging the direction of a match, I record the result with the reason and the date, and later runs pass it. Below, I call registering a match in this record "acceptance." I created the acceptance record on 2026-08-11. As of 2026-09-16 it has 26 lines. The articles with an acceptance are 17 published articles. Matches in drafts and in articles waiting for publication cannot be accepted.

The grounds for the direction written on the 26 lines are all time information. Not one was decided by the string match itself. The 26 lines fall into four kinds. Of these, 20 lines are reverse-direction overlaps where wording from here was later taken into a source document. Another 3 lines are matches where both sides shared field names of a public API or path names of a repository structure. Just 1 line is a case where both sides quoted the same sentence from a published technical article. The remaining 2 lines are matches where common phrasings containing particles lined up by chance. There is not a single match in the direction that ought to be stopped, that is, where an article echoed internal text of the source material.

Judging the direction does not end in one pass. On 2026-09-16, a full matching run in the production configuration reported 7 failing matches across 3 published articles. All 7 were the same type of sentence, a set phrasing that states the typesetting rule for mixing Japanese and Latin text. The 3 failed articles all took that rule itself as their subject. I traced the origin of this rule in the article Who decided on the half-width space between Japanese text and alphanumerics?. Querying the direction with git showed the reverse direction: the same phrasing had been taken into source documents on 2026-09-16 and 2026-09-02.

I had judged the direction for these 7 positions on 2026-08-26 and 2026-09-07 and accepted them. They failed again because at that time I recorded only one matched source material. On this day the same positions also matched another source material. Each acceptance pins down not only the text of the fragment but also the set of identifiers of the source materials that matched at that time. Each time the same text is later taken into another source material, a match already judged goes back to failing.

The count grows each time the source material moves forward. When I ran full matching on 2026-09-07, new matches appeared across 8 published articles. The count ranged from 17 to 24 depending on when the run happened. That is because the working trees of the source material move with every run. After judging the direction of all of them, the number in the direction that ought to be stopped was 0.

For drafts I fix things differently. The matches on 2026-09-08 and 2026-09-11 were both reverse-direction, where a source document later overlapped. Even so, I resolved them by rephrasing the relevant sentence in the draft without changing its meaning. The list of facts I made to write this article also produced 7 matches when I matched it right after finishing it. Of these, 6 were the connective words inside English quotations taken from external primary sources, the parts that only join the phrasing together. The core words of the quotations matched no source material. The remaining 1 was the name of the directory where withdrawn drafts are kept. The source material side had also named the same kind of directory with the same word. So I had not copied text from the source material; both sides simply used the same naming habit. Matches in drafts cannot be accepted, so I shortened the English quotations to only their core words. I also rewrote the directory name into a Japanese description without giving the name, and resolved the matches. When I matched again after rewriting, it came to 0. In a full matching run in the production configuration on the same day, the other 4 draft articles, 8 files, also passed with 0.

A design that revises the targets of matching without loosening the check

Since I created the script for this matching check on 2026-07-12, I have rewritten it 18 times in two months and one week. The latest change was on 2026-09-07. In the rewrites I added allowed words, added reverse-direction matches to what can be excluded, created the acceptance record, and fixed the defect in allowed words. Not one change loosened the criteria for the judgment itself. For the mistaken direction and the defect in allowed words, the check stopped, and its design of not passing what it cannot judge stayed as it was. I confirmed the cause first, and then fixed the targets of matching and how matches are counted. For the blind spot from excluding my own repository, the check did not stop, and an AI reading the meaning found the echoed wording. The design of not passing when it cannot judge is not unique to this check. I wrote about how I built the same design separately in two places in I built evaluators that do not pass what they cannot judge, in two separate places.

For those who put the same check into their own operations, I write three key points. Decide the direction from time-based evidence, not from string matches. If you exclude yourself from matching, decide from the start that copies in that range are found by an AI review that reads the meaning. And because each acceptance also pins down the set of matched source materials, you will judge the direction again each time the source material grows.

I set the limits of my claims myself. In this operation, my articles, which do the observing, and the source documents, which are observed, form a loop connected by quotation. The generalization that this configuration always produces false positives was not supported by a search on 2026-09-16. What external primary sources support goes only as far as two points. One is that external detection assumes the reference collection is genuine. The other is that a match does not explain the direction in time. I write about the loop as a discussion confined to measurements from my own operations.

Research I referred to

A study dealing with text reuse in historical materials. On 2026-09-16 I confirmed in the original text that pairwise similarity comparison does not explain the direction of reuse over time, and that the study states, as future development, giving the direction by a process that links text fragments back to their earliest occurrence. Reception Reader: Exploring Text Reuse in Early Modern British Publications (David Rosson, Eetu Mäkelä, Ville Vaara, Ananth Mahadevan, Yann Ryan, Mikko Tolonen. Journal of Open Humanities Data 9, 2023-04-17. DOI 10.5334/johd.101) https://openhumanitiesdata.metajnl.com/articles/10.5334/johd.101

Materials I referred to

On 2026-09-16 I confirmed that an external detection system is defined as one that compares a suspicious document with a reference collection assumed to be genuine. Content similarity detection (English Wikipedia) https://en.wikipedia.org/wiki/Content_similarity_detection
On 2026-09-16 I confirmed that, in how the task is constructed, the roles of documents are separated in advance into suspicious documents that may contain reuse and source documents that are the other side of matching. Generative Plagiarism Detection (PAN at CLEF 2025) https://pan.webis.de/clef25/pan25-web/generated-plagiarism-detection.html

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

Does shallower reasoning make an AI's Japanese more natural? I compared five drafts and the assessments split

matsumotory — Wed, 02 Sep 2026 19:04:24 +0000

Summary

On 2026-08-31 I passed on one impression to the AI agent I write articles with. The impression was that even a model like Claude Opus 5 seems to write natural Japanese, without over-reasoning, when you run it with effort set to medium. Effort is an API parameter that decides how much thinking a model spends on reasoning. The AI agent wrote this impression into the rules document the same day. It also collected five studies that looked like they would support it. And it went as far as having five drafts written from the same material and comparing them, all on that same day.

The results of the experiment, though, did not come out the way my impression said they would. When I counted violations of the Japanese rules, the drafts written with effort lowered came out better. When I had an outside model rate overall naturalness, on the other hand, the drafts written with effort raised came out better. In this article I explain first how the impression became a rule. Next I show how the assessments split when I compared the five drafts. Then I explain why two of the studies I collected did not hold up as support. Last I write what I decided in light of that.

What you can take away

For people who have an AI write Japanese prose, I explain the following three things.

Whether you should raise or lower effort can flip between having an AI write Japanese and having it check the facts and logic of what an AI wrote. I explain how I split effort between those two
I made a point of not settling a rule on an impression alone. On the same day, I wrote into the body of the rule that I would record what came out of running the AI under it and check later whether the rule was right. I explain how that came about, and the comparison experiment I ran the same day
I have the studies an AI collects read again the same day by an AI other than the one that collected them. For two of the five, the claim I was about to write into the article was either absent from the source or the opposite of what the source said

The product names of AI models in the body refer to the ones actually used in this operation, and they do not represent the views of their providers.

How I set effort and run it now

To give the conclusion first, whether shallower reasoning makes Japanese natural came out differently depending on how I compared. So I have not turned this impression into a settled rule. Here is how I run it now.

When having an AI write Japanese (generating, polishing, drafting fixes), choose effort explicitly, not just the model name. Do not let it silently inherit the deep setting of the whole session
When having it check the facts and safety of what an AI wrote, keep effort deep
Use low for neither, because low increases missed detections
Leave polishing and drafting fixes to Claude Sonnet 5 as before, until measurement confirms my impression is right. Put on hold the proposal to make Opus 5 at effort medium the first choice
The session's leader model writes the final draft directly. This decision is not among the ones on hold

I explain in order how I arrived at running it this way.

What happened on the day I passed on the impression

On 2026-08-04, about four weeks before I passed on this impression, I had decided not to use Claude Opus 5 for checking Japanese. There were two reasons. I once had an Opus 5 checker, an AI that checks text against the rules document, look for the mistaken simplification that breaks standard compound words down into native Japanese wording. That checker left noun-phrase compression inside its own suggested fixes. Two days before that, when I wrote and compared four drafts from the same material, the Opus 5 draft had noun phrases strung together to excess. Since then Claude Sonnet 5 handles the Japanese checks, and Claude Fable 5 takes over when Sonnet is not enough.

On 2026-08-31 I told the AI agent to narrow the scope of that decision. What I meant was that Opus 5 too seems to produce natural Japanese without over-reasoning if you set effort to medium, and that I wanted to put that to use. I thought one cause of unnatural Japanese might lie in over-reasoning rather than in the choice of model. In response, the AI agent rewrote the decision from 2026-08-04 so that it applies only to the setting of that time, when the model ran with deep reasoning and no effort specified. It kept the record of the problems that actually happened and narrowed only the range the decision reaches. Even for a rule I have already decided, I make a point of revisiting its range when a new observation or comment comes up. There is a worry that fixing a rule decided with an AI while its grounds are still weak lets a mistaken practice settle in, and I wrote about it earlier in Who decided on the half-width space between Japanese text and alphanumerics? Tracing it back, I found an AI had decided it with no grounds.

That same day, two AI sessions running in parallel each received what I had said separately, and both revised the rules document independently. The session that got its change into the rules document first kept Sonnet 5 as the Japanese checker, as before. On top of that, it required choosing a model name and an effort level together, but only when having an AI write Japanese. The setting to try first was running Opus 5 at effort medium. The other session, which noticed later, withdrew its own reading that Opus 5 should also come back as a checker under conditions, and fell in line with the earlier version. It then wrote the content of that earlier version into the document that sets which model goes to which task, and into the settings checks of the two article-writing workflows.

Under the rule left standing after all this, I keep effort high when checking and set effort to medium when writing. First, when I have an AI check the facts and safety of written text, I keep effort deep. Second, when I have an AI write Japanese, I state effort as medium and do not let it silently inherit the session's deep setting. Third, I use low for neither, because it increases missed detections. Those are the three points. The AI agent wrote these three points into seven places in the documents the same day. They are the Japanese writing rules, the definition of which model goes to which task, the settings checks of the two article-writing workflows, the procedure for the daily writing job and its skill, and the settings of the polishing workflow. My impression, though, is still one person's subjective view. So on the same day the AI agent wrote into the body of the rule that it would keep a record of what came out of writing articles under this setting, and add to the rule only after the results were confirmed in review.

Comparing five drafts written from the same material

On the same day it wrote into the rule, the AI agent started an experiment comparing drafts. It varied the effort of Claude Fable 5 and Claude Opus 5 across medium, high, and max. Then, with the same material and word-for-word the same instructions, it had five final drafts written from scratch. It hid which draft came from which setting and compared them in three ways. Two models and three effort levels make six combinations. Five drafts remain in the record. Which combination is missing cannot be pinned down from the record. It prepared three ways of comparing. A Sonnet 5 checker counts violations against the Japanese rules document. Three independent recalculations check whether the numbers in the material match what the text says. And Gemini, a model from an outside provider, rates overall naturalness.

Way of comparing	Result
A Sonnet 5 checker counts rule violations	For Fable 5, the draft at effort medium had the fewest violations of translationese and of style. For Opus 5 it went the other way, and the draft at effort max had the fewest
Three independent recalculations of whether the numbers in the material match the text	Only in the draft Fable 5 wrote at effort medium did all three recalculations find the same mismatches with the material. They were a wrong count, and a statement that settled something that had not been measured yet
Gemini rates overall naturalness	Gemini rated the drafts as more natural the higher the effort, for both models

In short, counting violations made the lower-effort draft the better one for Fable 5, while rating naturalness made the higher-effort drafts the better ones for both models. And the Fable 5 draft at effort medium, the one with the fewest rule violations, was the only one that carried mismatches with the material.

The comparison procedure itself also ran into a problem. The order in which drafts are handed to a checker can affect the assessment. The AI agent shuffled the order of the drafts for each checker to avoid that effect. Three of the four checkers then mixed up the draft labels with the order they had read them in. The AI agent matched the quotations the checkers returned against the text of the drafts. It could confirm that all three had copied them the other way round, so the results were recoverable. Out of this, the AI agent added two items to the rules for work it repeats on its own. One is to match the quotations a checker returns against the original text before tallying anything. The other is to fix the order in which drafts are handed over rather than shuffling it, and to catch mix-ups in reading order through that matching of quotations.

What came of having a different AI read the five collected studies again

In parallel with writing into the rule, the AI agent set out to check in primary research sources whether my impression had any support in theory. In its 2026-08-31 search, the AI agent collected the studies into three groups. The first group is two studies on multilingual reasoning (When Models Reason in Your Language and A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning). From those the AI agent took two readings. Reasoning models tend to think mainly in English even on non-English tasks. Pinning the language of thought to the user's language lowers accuracy. The second group is one empirical study on reasoning time (Inverse Scaling in Test-Time Compute). It said there are tasks where extending reasoning time makes scores go down instead. The third group is two reports on creative writing tasks (COIG-Writer and When Reasoning Supervision Hurts). The AI agent had collected these two as reports that adding reasoning has no effect or does harm.

On 2026-09-02, an AI in a context separate from the one that collected the studies fetched these five sources and read them again. It checks three things: whether the source exists, whether the bibliographic details are right, and whether the claim I was about to attribute to it really appears in the source text. To prevent fabricated quotations and references, this site always runs this check before publishing. Here are the results in a table.

Study	The claim I was about to write	What the second reading found
When Models Reason in Your Language (EMNLP 2025 Findings)	Reasoning models think mainly in English even on non-English questions, and pinning the language of thought to the user's language lowers accuracy	Matches what the source says. The subjects, though, are six open-source distilled models and short answers in math and science, so it does not extend to commercial models or to writing prose. The language used for thinking is sometimes Chinese, not only English
A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning (EACL 2026 Findings)	The bias toward English-centered reasoning, and noise from translation	Only the first half is in the source. Noise from translation is not written there, and the source carries the opposite result, that thinking translated into English is more accurate
Inverse Scaling in Test-Time Compute (TMLR 2025)	There are tasks where extending reasoning time makes scores go down instead, and this was confirmed on several models	Matches what the source says. It is a result shown on deliberately constructed task types, though, not a law that thinking longer generally lowers scores. How the breakdown happens differs by model family
COIG-Writer (arXiv:2510.14763)	Training without a reasoning process attached gave better creative writing scores	The source states the opposite conclusion. It reports that training on creative data with the process attached, mixed with general data, works well, and it holds no experiment comparing the presence or absence of the process at all
When Reasoning Supervision Hurts (arXiv:2605.20364)	Reasoning supervision lowers the quality of long-form literary generation	What the study covers is different. What it had generated was not literary writing but critique reports in a set format, and it is a preprint

The first and second groups, once their scope was limited, were usable as grounds for explaining why output can change when reasoning runs longer. Neither, though, is a study that deals directly with the naturalness of Japanese prose. As of 2026-09-02 I still have not found a study that compares effort high and medium directly on the naturalness of Japanese. The two in the third group did not hold up as grounds. All five looked like grounds when the AI collected them on 2026-08-31, and yet when a different AI read them again on 2026-09-02, two turned out not to be grounds at all. That is why this site has an AI other than the one that collected them read them again the same day and check.

Why I separated the writing role from the checking role

The assessments split. So that same day the AI agent stopped putting the setting of Opus 5 at effort medium first in line to try. Which setting to use was left undecided. Polishing and drafting fixes went back to Sonnet 5 as before, and the effort of the polishing workflow went back to its original high setting. The prose a checker rates as natural and the prose I read and feel is natural are not necessarily the same. So to settle this judgment, I first have to read the drafts side by side with their names hidden and decide which draft's Japanese fits my own sense. Beyond that, the same experiment has to be repeated on other material.

The same experiment did make one thing clear, though. The draft written at lowered effort carried three mismatches with the material. They were a wrong count, and a statement that settled something that had not been measured yet. Recalculating with a model at raised reasoning caught all three. In this experiment the writer was better with reasoning lowered, and the checker was better with reasoning raised. So on the same day I approved running the role where an AI writes and the role that checks what was written as separate roles. Once the Japanese is written, a model at raised reasoning checks the facts and the logic. The checker does not rewrite the text, and returns only the places it flags and the grounds for flagging them. The writer takes those comments and changes the text as little as possible. I also decided that the writer does not copy the checker's suggested sentences as they are, but rewrites them in its own wording. If a model at raised reasoning rewrites the prose directly, the sentences turn unnatural again.

The next day, 2026-09-01, I read two articles written at the deep reasoning setting, and sentences that turn a verb into a noun and then take it up with a padded predicate, such as continues or is repeated, stood out. The habit had stayed even with effort set deep. I said on the spot that it was hard to read, and the AI agent added this pattern to the rules and to the mechanical check. On 2026-09-02 I dropped the proposal to move the Japanese writer to Opus 5 at effort medium altogether. I decided that the session's leader model writes the final draft, as it has all along. My reason is that building a correct writing procedure and holding the writer to it serves the goal of good Japanese better than changing the writer's model or settings.

How to run the same check in your own operation

If you have an AI write Japanese and feel the prose changed when you changed the reasoning setting, I recommend checking in this order.

Separate having it write from having it check, and state the effort for each. Do not let it silently inherit the session's deep setting
When you write an impression into a rule, write into the body of the rule that you will record what comes out of running the AI under it and check it later in review
Make several drafts from the same material and compare them in several ways with the draft names hidden. Match the checker's quotations against the original text before tallying, and fix the order in which you present them
Have the studies you use as support read again the same day by an AI other than the one that collected them, and have it check that they exist, that the bibliographic details are right, and that the claims match

My impression became a rule in a day. In the experiment on that same day, the assessments split. The next day I found that the habit of turning verbs into nouns stays even with effort set deep. Because I recorded the disagreement as it stood instead of claiming proof, the material for the next check is there in the record. I hope this record reaches people setting out to check the same thing in their own operation.

Research I referred to

For all of them, as an independent check on 2026-09-02, I fetched the original text and verified that the bibliographic details and the claims matched.

A study that measured which language reasoning models tend to think in, and how accuracy changes when the language of thought is pinned to the user's language, measured on six open-source reasoning models and on short answers in math and science. When Models Reason in Your Language: Controlling Thinking Language Comes at the Cost of Accuracy (Jirui Qi, Shan Chen, Zidi Xiong, Raquel Fernández, Danielle S. Bitterman, Arianna Bisazza. Findings of the Association for Computational Linguistics: EMNLP 2025) https://aclanthology.org/2025.findings-emnlp.1103/
According to this paper, an evaluation of multilingual chain-of-thought reasoning on the three sides of performance, consistency, and faithfulness reports that thinking written in English is easier for a model to draw on. No report of noise from translation is included. A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning: Performance, Consistency, and Faithfulness Across Languages (Raoyuan Zhao, Yihong Liu, Hinrich Schütze, Michael A. Hedderich. Findings of the Association for Computational Linguistics: EACL 2026. arXiv:2510.09555) https://aclanthology.org/2026.findings-eacl.276/
A joint study by Anthropic's alignment research team and outside researchers that constructed and showed task types where extending reasoning time lowers scores. Inverse Scaling in Test-Time Compute (Aryo Pradipta Gema, Alexander Hägele, and others. Transactions on Machine Learning Research, 2025. arXiv:2507.14417) https://arxiv.org/abs/2507.14417
Reading its text showed that this source reports the opposite of the claim I was about to attribute to it. It is not used as grounds for any claim. COIG-Writer: A High-Quality Dataset for Chinese Creative Writing with Thought Processes (Yunwen Li and others. arXiv:2510.14763, preprint) https://arxiv.org/abs/2510.14763
A source that turned out to cover the generation of critiques in a set format rather than literary writing. It is not used as grounds for any claim. When Reasoning Supervision Hurts: TTCW-Based Long-Form Literary Review Generation (Jinlong Liu, Mohammed Bahja, Mark Lee. arXiv:2605.20364, preprint) https://arxiv.org/abs/2605.20364

Materials referred to
I confirmed the values of the effort parameter (low, medium, high, xhigh, max) and the default. Effort (Claude Platform Docs. Text fetched 2026-09-02) https://platform.claude.com/docs/en/build-with-claude/effort

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

Deciding the criteria for excluding an AI-collected research topic candidate

matsumotory — Wed, 12 Aug 2026 19:05:48 +0000

The problem and the outcome in brief

I collect the problems I run into in daily development and operations as candidate research topics, rather than letting them end with an on-the-spot fix. An AI agent automatically reads the material and drafts candidates (I will call this role the extraction agent), and a person decides whether to adopt them. In this process I narrowed the reasons for excluding a candidate down to two. This article explains how that came about, and what the automatic extraction produced when I ran it on the day I rewrote the criteria. The point is that a criterion for exclusion has to be one whose truth the extraction agent can judge from the material alone. I also cover how to write records so they can serve as candidates later, and the division of roles that leaves adoption and confirmation to a person. This is for people who want to pick research or writing topics out of their own development records.

Deciding the criteria for exclusion

I record the problems I find in daily development and operations as candidate research topics. An AI agent extracts candidates from development records, following a procedure kept in a repository I run on my own for research drafts and operations records, and that same procedure also decides which candidates may be excluded before they go into the record. On 2026-07-15 I rewrote these exclusion criteria. Before the rewrite, I judged exclusion with a single question: does a candidate still stand as a research topic and look valuable once you cut it loose from the circumstances of the organization doing the development? The extraction agent excluded any candidate that did not satisfy this question. The criteria after the rewrite limit the reasons for exclusion to two. The first is a candidate that does not generalize, meaning one whose context does not carry to a reader who does not know the internal circumstances of the development. The second is a candidate whose confidential information cannot be separated out, meaning one where no change of wording removes the part that has to stay hidden, such as the name of an unreleased product. Every candidate that passes those two goes into the record in full, and I do not use an assessment of value as a reason to exclude; I write it instead as an observation attached to the candidate.

The person who decides adoption judges whether a candidate has value. The extraction agent does not pre-empt that judgment. If the extraction agent excludes a candidate up front on the grounds that it seems to have no value, nobody can verify afterward whether the candidate really had none. On top of that, the extraction agent comes to keep only the candidates that look likely to match the wishes of the person who decides adoption, and the range of collection itself narrows to what matches that person's interests.

Up to this point this has been about candidate research topics, but around the same time, I also made a rule change in the opposite direction on the blog that carries this article. I was increasingly writing an article and only then finding it too thin to keep, so for the high-priority candidates among the article proposals, I added a condition: that the exchange between the AI and the person, that is, what was tried, what happened, and what was fixed and how, can be copied out of the material in concrete form. In other words, for research topic candidates I removed a condition the extraction agent has no way to verify, and for article proposal candidates I added a condition the extraction agent can verify on the spot. Both changes rest on the same principle: a criterion for exclusion has to be one whose truth the extraction agent can judge from the material alone.

Along with this principle, here are the five things I hold to across the process from collection to aggregation.

Only two kinds of candidate may be excluded: ones that do not generalize, and ones whose confidential information cannot be separated out. An assessment of value is written as an observation attached to the candidate, and is not used as a reason to exclude.
Excluded candidates stay in the record too, with the reason for exclusion attached. Without a record, you can no longer tell afterward whether the range of collection was narrow or the material simply was not there.
In the recording step, write the conditions for reproducing the problem, meaning what happened when you did what, and the place where that event can be confirmed, in the same place as what you did about it.
Material with only a single failure is too thin in content to become a candidate on its own. I wait until similar failures accumulate and then pull them into a single candidate.
In the aggregation step, a person makes the adoption decision and confirms it. The extraction agent hands over every candidate with a proposed judgment attached, and does not cut candidates on its own by reading ahead into what the deciding person wants.

What the automatic extraction produced on the day I rewrote the criteria

On the day I rewrote the criteria, the AI agent ran the automatic extraction once under the new ones. Here are the numbers from that record.

Category	Count
Candidates received	60
Raised as new candidates	24
Absorbed into an existing candidate	15
Dropped as duplicates of other candidates	21
Excluded under the two criteria	0

Not a single candidate met the two criteria. What actually did the narrowing was the 21 folded together as duplicates and the 15 absorbed into existing candidates.

That zero cannot be read as the effect of the rewrite alone. In the same change, I also added viewpoints for the extraction. Until then there were only seven viewpoints, named after fields such as ways of thinking about design, or privacy. To those I added seven viewpoints for how to compare material, such as comparing the gap between an estimate and a measurement in numbers, or questioning a pass result itself. The extraction record shows that 11 of the 24 new candidates came mainly out of the viewpoints I added. Because I bundled the rewrite of the criteria and the addition of viewpoints into a single change, I cannot separate which of the two the zero belongs to. It can also be read as the criteria being so loose that nothing gets excluded at all. The AI agent kept even the candidate that looked most specific to my own setting rather than excluding it, on the grounds that the same thing happens in other environments, and that judgment is in the record. Still, I have not run the measurement enough times yet to rule that reading out.

The level of detail that makes a record usable as a candidate later

Even if you keep the range of collection wide, a record that holds nothing but what was done about the problem cannot serve as a candidate later. The rules document for an app I develop at my own company carries a dated record of an accident that happened when an operation with side effects was sent to an external service. The address used to test user registration turned out to be a real one, and the confirmation email reached a stranger. The same section also states that the message is recorded in the authentication log on the external service side. There is a list of checks to prevent a repeat, and one item stops the operation if a real domain is included. The conditions for reproduction, meaning what happened when you did what, and the place where it can be confirmed, sit in the same place as what was done about it, so the record can serve as a candidate later. The third of the five conditions I listed above refers to this form of record.

The five conditions for aggregation, and who decides adoption

I use five conditions to decide whether to adopt a collected candidate as a research topic I will actually start work on.

Who is responsible has been decided
When it starts has been decided
Where the results will be published has been decided
The first piece of work has already been started
It has passed the check for whether it generalizes

If even one is missing, I do not adopt it. I lower its priority or move it to hold. The extraction agent is responsible up to handing over every candidate with a proposed judgment attached, and for a candidate it has not judged yet it writes "not judged" rather than leaving the field empty. Otherwise you can no longer tell afterward whether it was judged or skipped.

I also have a measurement of how many the deciding person withdraws. On this platform, on 2026-07-27, I went through all 60 article proposals and decided which were needed, keeping 45 and withdrawing 15. The reasons for withdrawal fell into three patterns. The largest was candidates where the AI only appears as something called or something worked on, with no exchange between the AI and a person at the core, and that accounted for 60 percent of the 15 I withdrew. The rest were candidates that amount to an explanation anyone who knows the field could write, and candidates raised from thin material with only a single failure. Checking all 60 is a measurement on the article proposal side, but if the extraction agent does not pre-empt the judgment of value, that judgment falls to the person deciding adoption. When the range of collection is kept wide, I take this one quarter as a rough guide to how many the person will withdraw.

The design of separating the collection step from the step that decides adoption is not new in itself, and it is already in the handbook for systematic literature reviews. The pattern of reading across after-the-fact incident records to find a common theme, rather than stopping at an individual fix, is also established, as laid out in a standard account in the operations field. Compared with that pattern for incident records, I changed two things in my own operation. I widened the target from incidents to everyday problems and review comments as well, and I made the destination of the aggregation a candidate research topic.

What I hand to an AI agent, and what I keep with a person

In this operation, for candidate research topics, an AI agent automatically handles reading the material, drafting candidates, and attaching a proposal, and a person confirms whether to adopt. Outside measurements confirm that this division of roles makes sense, too. A human-evaluation study that enlisted more than 100 researchers judged ideas from an LLM to be more novel than ideas from experts, while their feasibility came out somewhat lower. A survey that generated more than 200,000 ideas reports that AI ideas cluster near the literature they started from. The latter is a result showing that the more the collection step is handed to an AI, the more the candidates gather into a narrow range. I take both results as support for keeping the adoption judgment on the human side.

There is a shared convention for recording what an AI agent does as well, the OpenTelemetry semantic conventions for GenAI, and work is under way there to settle the names and meanings of the items to record. The document's status is still Development, though, so the names can still change. As of today, what an AI can take over is the first-hand record of what was called when and what came back, and the judgment that raises that into a problem worth working on still sits with a person.

Whether this process has reached a research result is something I cannot speak to yet. What I have is a little under two months of material even at its longest, and three weeks of records since the rewrite, so measuring whether the candidates I collected made it to acceptance is still ahead. Even so, when you are torn between widening and narrowing the range of collection, start by checking whether you can tell from the material alone whether something meets your current exclusion criteria.

Research I referred to

Here are the five outside primary sources referred to in the article. I checked the source page for all of them as of 2026-08-06. For the two arXiv items, I checked the abstract page.

Chapter 4 of the Cochrane Handbook (designing the search step) https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-04
The chapter on postmortem culture in Google's SRE book https://sre.google/sre-book/postmortem-culture/
The document in the OpenTelemetry semantic conventions for GenAI that defines the unit for recording what an AI agent does https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md
Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers. A human-evaluation study by Si, Yang, and Hashimoto that enlisted more than 100 natural language processing researchers (submitted 2024-09-06) arXiv:2409.04109 https://arxiv.org/abs/2409.04109
AI Research Agents Narrow Scientific Exploration. A study by Tang and Yang on whether AI research agents narrow the range of scientific exploration (submitted 2026-05-27, revised 2026-07-11) arXiv:2605.27905 https://arxiv.org/abs/2605.27905

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

At companies where AI writes 80 percent of the code, has development become 80 percent faster?

matsumotory — Wed, 12 Aug 2026 19:05:47 +0000

Summary

I hold a hypothesis: what decides the value of software from here on may be the autonomous continuity in which software makes software. To verify that hypothesis, I ran two passes of research through the public primary sources of the major companies that build AI models. The research asked how far each company officially says it uses its own AI models in its own model development, and how far the companies and outside researchers have been able to verify with actual measurement the claims of acceleration, of how much faster development has become as a result.

I read four kinds of sources: each company's announcements, the documents that gather a provider's evaluations of performance and safety, published when it releases an AI model (system cards), research papers, and an economic estimate in which independent researchers rework the figures each company published within a calculation framework of their own. I read all of them down to the body text, not just summaries. From the research I found both large numbers that support the hypothesis and conclusions from the AI companies themselves that weaken it.

In this article I go through what the primary documents of the three companies actually say, with the definitions of the numbers attached. I too operate a chain of the same structure, in which software makes software, together with AI at the scale of an individual. I add the measurements from running it, and explain how far this hypothesis can be claimed now and where it stops being claimable.

Let me lay out the structure of the body up front. First, I confirm the exact definition of the figure that says AI writes 80 percent of the code inside a company. Next, I restate the hypothesis as a structure in which the gap between models keeps widening the gap between the next models. Third, I recount how the methods I had settled on earlier for the convenience of my own operation matched the current recommendations of these companies, which I read afterward. Last, I read the judgment in which the parties themselves conclude that they cannot attribute the acceleration of their own progress to AI.

What you can take away

This article is for people who are building a setup where they develop alongside AI agents. It is also for people who want to know from primary sources where the story of AI building models actually stands right now. In this article, I explain two insights. The first is how to read acceleration figures, the numbers for how much faster AI has made development. Each company describes the same acceleration as large in a promotional setting and as small in a safety evaluation. So you must not line up numbers with different definitions and compare them as they are. I show this way of reading with quotations from the original text. The second is a lesson that applies directly to individual development. What I settled on is this: the lower the cost of verification in a domain, the larger the effect the chain shows there. I also explain the conclusion that what has the highest value to hand over to AI is the record of judgment, of why a person decided the way they did.

The problem and what came of it

To check my hypothesis against reality, I read the public primary documents of the major companies and an independent economic estimate down to the body text. After the first pass, I went back over the premises to make my conclusion more exact. My material was those public documents, together with measurements from the running of this platform itself.

The hypothesis

What AI can write moves closer to what anyone can build. If the value of building software itself keeps thinning out under competition, value should remain somewhere else. I think that place is the autonomous continuity in which software, once built, builds the next software. I build this membership platform for a technical blog together with AI: the articles, the search, and the conversation feature alike. Including the chain that records the discussions held here, turns them into content, and returns them to the knowledge of that AI itself, this platform is also a live experiment testing this hypothesis.

Still, a hypothesis is a hypothesis. To check it against reality, I went and read public primary documents down to the body text, to see whether this is really happening at the leading edge and how far the companies and outside researchers have been able to verify it.

What each company officially says

First, I will quote only the official words of the parties involved. In an announcement dated 2026-07-09, OpenAI wrote that over the preceding six months, the share of its research compute allocated to running AI that writes code internally had grown a hundredfold. In the same paragraph, though, it notes on its own that this is a measure of usage and not a number that measures research progress itself. In an official explainer article, Anthropic published that, as of May 2026, its own models wrote more than 80 percent of the code integrated into its codebase. That 80 percent has a footnote as well, which gives it the conservative definition of the share of lines integrated into production that can be attributed to a model, and goes as far as writing that line count is a measure of quantity and not of quality. Google's chief executive said in a talk that 75 percent of new code inside the company is AI generated and has passed engineer approval. This one is defined as the share of generation that passed approval, so a few lines emitted by a completion feature can enter the numerator. The two figures, 80 and 75, look close, but their definitions differ, so they cannot be compared side by side.

In the same explainer article, Anthropic has also published an internal evaluation in which the rate at which a model beats human judgment on choosing the next research move rose from 51 percent to 64 percent. As for OpenAI, reports carry an announcement that a higher-tier AI model autonomously performed post-training, the additional training done after a model is built, on a lower-tier model. An OpenAI employee added a note to this, though: it did not build the training procedure from scratch but adapted its own post-training configuration for a small model, and for humans that would be about two weeks of work for two researchers. OpenAI's system card, which gathers its evaluations of performance and safety, also states plainly that reliably designing and executing a complete post-training procedure across diverse models is not yet possible. Even so, one of Anthropic's co-founders goes so far as writing, in an essay, that it may be one or two years until the point where the current generation of AI autonomously builds the next.

Read only the disclosures up to this point, and the chain in which software makes software looks like it is already functioning in full.

The structure where the gap between models widens the next gap

The competitive picture is not decided by whether a company has the chain in which software makes software. Because every company has begun to depend on the AI models that carry the chain, today's gap between models becomes the performance gap of the next models, and that gap keeps widening, so latecomers find it harder and harder to catch up. More than whether a company has the chain at all, what decides the competition is that the gap in the results the chain produces goes on widening with every turn. This is my hypothesis restated more exactly. It is not a verified fact, and I write it as my own thinking.

I confirmed this reading most strongly in a passage of an Anthropic system card. It says that, on the grounds that recent models have the ability to accelerate their own development, they implemented an intervention that, for requests aimed at developing frontier large language models, lowers the model's effectiveness in a way invisible to the user. In other words, Anthropic throttles the power its own model holds to build the next model whenever the request comes from someone else. The party writes this much in its own document, so I read it as support for that power sitting at the center of the competition.

On the other hand, when I read the primary sources, I found that you cannot draw a line among the leading companies between those that have the chain and those that don't. The first to publish the most concrete instance of a closed chain was Google. AlphaEvolve is a coding agent that runs on Gemini. A function it found keeps recovering 0.7 percent of the compute in the company's data centers. There was also an improvement that made the whole of the core computation of training 23 percent faster on average, which cut Gemini's training time by 1 percent. Google adopted a circuit design that AlphaEvolve proposed for the next generation of TPU. On the circuit, though, the authors of the paper themselves add that an existing synthesis tool had independently found the same improvement. The paper states plainly that this is a new instance of Gemini optimizing its own training process through AlphaEvolve. Google published this about a year before the other two companies began disclosures of the same kind. And these are measured numbers from improvements deployed in production, not a self-reported productivity survey. Google's disclosure was modest, but its example of the chain was the most concrete.

The chain I run at an individual scale

I operate a chain of the same structure at the scale of one person. Let me start with the layer of rules documents. The documents that set down the discipline of this operation come to 54, counting the one at the top. Every time I point something out, the AI adds that lesson to the rules documents on the spot. On revision, rather than stacking additions, the AI rewrites the whole body so that only the current rule stands there. Inside the documents, 248 places state their origin as a dated remark or comment of mine. That is a count of occurrences, so the same lesson gets counted more than once across documents. Even so, the default I keep for this operation is a form in which every decision traces back to when it was made and on whose judgment. The other layer is the raw record. In the body of a commit, the AI writes, on top of what changed, why it changed, which comment and which judgment it followed, and what it verified. The commits piled up over these 20 days come to 849 when counted with merges and squashed changes excluded. Before and after the day I decided to write fuller commit messages, the median length of the body moved from 223 characters to 312. It is not a controlled comparison, so I cannot claim causation, but I have been able to measure the correlation: once the rule was written, the writing changed.

The match came out most clearly on the second pass, when I read what each company currently recommends. One company's official guide recommended keeping one record per lesson, and writing down why it mattered as well. Official documents also carried guidance to version prompts and rules with git in your own repository rather than entrusting them to an external management feature. A research conclusion that raw records should be kept alongside summaries rather than replaced by them points the same way. So does the judgment, near-identical across the three companies, that acceleration is concentrated in execution and is not reaching judgment. The methods I had settled on earlier for the convenience of running the operation matched, one after another, the current recommendations I read afterward. This match between methods is the center of what the two passes of research told me.

That said, for my own chain too, the quantity that matters remains unmeasured. How much did output rise per unit of AI capability put in? In fact, the authors of the independent economics paper write, on their own, that no one has been able to measure this quantity, not even at the leading companies. They release almost none of their internal metrics, for competitive reasons. Then if I can measure the same quantity on a small chain at an individual scale and publish it, that becomes primary data of a kind those companies do not release.

The most modest judgment came from the parties themselves

If I do not write this part, this article becomes nothing but a summary of promotion. So I write the side that weakens the hypothesis just as fully.

The strongest counterevidence was Anthropic's own judgment. Its April 2026 system card, while granting that the growth of its capability had turned steep, goes as far as writing that the growth it could identify is confidently attributable to human research and is not due to AI assistance, and that it confirmed this by interviewing the people involved. Even with employees self-reporting a fourfold output, combining that with an estimate of its impact on progress put the overall multiple below two. The judgment does not change in the latest system card, released on 2026-07-24, which says the acceleration is concentrated in engineering execution rather than research judgment. OpenAI's system card also states plainly that none of its three new models reaches the threshold of High, the highest risk level, in the AI self-improvement evaluation, and an independent evaluation body judged that fully automated AI research and development will not become possible. In the independent economic estimate, the condition needed for self-sustaining acceleration is a 15 percent gain in productivity per unit of capability, and the measured figure, taking the parties' self-reports at face value, was 9 percent. It does not reach the condition. The paper closes, though, by saying that it appears to be strengthening.

And one pattern common to the three companies comes into view. Each company reports the same acceleration at its maximum in a product announcement and at its minimum in a safety evaluation. One party put out a figure of eight times as much code integrated per day compared with 2024. That same party concluded from the same internal data that overall progress was less than double, while an outside evaluation body read more than double from the same data. Unless you check that definitions and measures agree before you check that numbers agree, you can build two opposite pictures of the same company.

The more primary sources I read, the more modest the wording of the claims becomes. The paper's authors write their own reservations, the party behind the system card itself denies the attribution, and the party that put out the usage figure notes on its own that it is not a measure of results. The most modest telling of the story that AI has started building AI was in the primary documents of the parties, the ones who should have been claiming it most strongly.

Where the hypothesis stands, neither denied nor confirmed

The economics paper carries one reservation that matters for this hypothesis. It predicts that acceleration of narrow capability can run ahead of acceleration of broad capability, and that the narrow acceleration should concentrate in domains where verifying an answer is cheap, such as software development and mathematics. If what my hypothesis points at is this narrow acceleration, it has not been denied yet. And as long as the quantity at its core is unmeasured, it has not been confirmed either.

Let me narrow down what can be said now. If a low cost of verification is what decides the effect of the chain, then adding more checks before publication is the obvious move. If acceleration concentrates in execution and does not reach judgment, then the record of judgment is what has the highest value to hand over to AI. I design my own operation on the basis of these two expectations. I will keep measuring here whether those two expectations were right, and writing down the results.

Research and primary sources I referred to

The numbers and quotations in the body are all as the documents read when I checked them on 2026-07-26. This article is an analysis that organizes what public documents state, and it does not represent the views of any of the companies named.

When AI builds itself (Anthropic Institute) https://www.anthropic.com/institute/recursive-self-improvement
Claude Mythos Preview system card (Anthropic) https://www-cdn.anthropic.com/08ab9158070959f88f296514c21b7facce6f52bc.pdf
System card for Claude Fable 5 and Claude Mythos 5 (Anthropic) https://www-cdn.anthropic.com/d00db56fa754a1b115b6dd7cb2e3c342ee809620.pdf
Claude Opus 5 system card (Anthropic) https://www-cdn.anthropic.com/c5fbac3f0b1280a933ebd26d3cb8bb9f5bdeaf48/Claude%20Opus%205%20System%20Card.pdf
The GPT-5.6 announcement and system card (OpenAI) https://openai.com/index/gpt-5-6/ https://deploymentsafety.openai.com/gpt-5-6/gpt-5-6.pdf
The AlphaEvolve paper (Google DeepMind) https://arxiv.org/abs/2506.13131
The Economics of Recursive Self-Improvement (researchers at METR and elsewhere) https://elasticity.institute/rsi-paper.pdf
The Adolescence of Technology (an essay by a co-founder) https://darioamodei.com/essay/the-adolescence-of-technology
An article on a talk stating that 75 percent of code inside the company is AI generated (Google) https://blog.google/innovation-and-ai/infrastructure-and-cloud/google-cloud/cloud-next-2026-sundar-pichai/

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

Rewriting prose until the tests pass: everything passed, but the check that mattered never ran once

matsumotory — Tue, 04 Aug 2026 15:38:53 +0000

Summary

I ran into an incident where every test passed, yet the meaning check that was supposed to judge quality had never run even once. This happened while I was writing a research abstract, a long piece of natural-language prose, in a development loop that rewrites the text until the tests pass. When the reviewer's authentication expired, the meaning check was skipped entirely, and the tests that looked like they had passed stopped being evidence of safety.

The tests have two layers: a deterministic check that can judge without looking at context, and a meaning check that calls an LLM through the CLI as the reviewer. A judgment takes one of three values: pass, fail, and an error that stands for the review not having run at all.

I put this mechanism in about a week after I started running the repository for the research abstract and for research operations, and over more than two weeks after that I kept adding checks, extended the same loop to a second abstract, restored the reviewer's authentication, and updated the model.

Audience and takeaways

This article is for developers and writers who want to bring a development loop into their own writing, not just their code. It is a loop where you rewrite the prose while watching whether the tests pass. There are three takeaways. The first is a criterion for sorting: drop the checks that misjudge when they are enforced without looking at context. The second is a three-value design that keeps the state where the review could not be run at all separate from the pass-or-fail judgment of quality. The third is a failure where uniformly failing every assertion left the text full of hedging so that no claim stood, and how I fixed it. The material is the operational record of running this mechanism for more than two weeks in a repository for a research abstract and for research operations.

Bringing the code development loop into long-form writing

When you write code, the tests return pass or fail the moment you write it. The theme this time is whether this development loop can also be brought into long natural-language prose, such as papers, specifications, and proposals. What I targeted was a research abstract. The test definitions live in YAML, and a test runner built on pytest expands one item into one test through parameterization, and a way of working where you rewrite until the tests pass has begun to take hold.

Let me note up front that these tests are not a tool that writes in my place. I am only turning the comments I have actually made on abstracts, one by one, into rules and then into checks, so all the machine can reproduce is those past comments, and deciding what to write is still the writer's job.

Dropping checks that misjudge when enforced without context

There is a single axis that separates the checks I dropped from the checks I kept. I dropped the checks that misjudge even legitimate usage when they are enforced in isolation without looking at context, and I kept the checks that do not misjudge because the value being checked, such as a character count or a number of occurrences, can carry a range. At first I had put in only rigid, enforcement-style checks: enforcing the length of a single sentence, strictly enforcing the number of paragraphs, and rejecting forbidden words in isolation without context. But they misjudged even writing that only makes sense once there is context, and they took away freedom in how to write, so after building them once I moved them out of the runner and deleted them as an old implementation.

The foundation for this sorting was a policy of not assuming that the evaluation of a paper can be carried by deterministic checks alone, and of putting weight on judgments of meaning and logic. I moved only the items that can be decided without looking at context, such as a range of character counts excluding whitespace and typesetting commands, whether a specified word is present or absent, and regular-expression matches and their counts, to the deterministic check implemented as local functions inside the runner, and I left the consistency of meaning, logic, and chapter structure to the layer that calls an LLM through the CLI as the reviewer. Fast and cheap judgments go to the deterministic check, judgments that need context go to the reviewer, and both live together in a single runner.

Keeping errors separate from the quality judgment

I gave the reviewer a fixed instruction to judge pass or fail using only the criteria I gave it, and to list every violation without omission, and I forced the response into structured data of the judgment, the location, and the reason. This is to keep fixes from dragging on as violations get pointed out one at a time. When typesetting backslashes get mixed into the response, parsing the structured output fails, so I also added a step that corrects them and re-runs the parse.

The third of the three values, error, refers to the state where the reviewer could not be run at all because of expired authentication, a timeout, or a failure to parse the response. I keep this separate from the quality judgment of the text and do not mix it into fail. If failures increase and I cannot tell whether the problem is in the text or in the infrastructure, this loop does not serve its purpose. That was my view.

In fact, I ran into an incident where the reviewer's authentication expired, every reviewer was skipped as an error, and although the tests all looked like they had passed, the meaning check had not run even once. The lesson that became clear at this point is that even when all the tests pass, it does not follow that the meaning check actually ran.

Letting assertions pass within the range of observation, suppressing variance with a majority vote

Among the axes of judgment, the handling of assertions took the most tuning. During the period when I uniformly failed anything that asserted without regard to the scale of what was observed, the text turned into nothing but hedged phrasing and no claim would stand. So I inserted the scale of what was observed, such as how many people and how many cases, into the reviewer's instruction, failed only generalizations that went beyond that range, and reworked it so that a reasoned assertion kept within the range passes as is. Along with that, on a failure I always have the reviewer tell me which it is: whether a fix in phrasing is enough, or whether I should go back to collecting data.

To suppress the reviewer's variance, I implemented it so that if the first run passes it stops there, and only when it is not a pass does it judge twice more and take a majority vote over the three runs. That said, this is a compromise that favored speed, and I have not measured whether the reproducibility of getting the same judgment on the same abstract within at most three attempts is good enough for practical use. I do see it pointed out that a single reviewer run has variance in its judgment and that a stable consensus needs repeated attempts, but I have not yet been able to confirm primary literature that backs up the number of attempts needed. I cannot say anything more right now than that I struck a balance between speed and reproducibility.

The deterministic check to CI, the reviewer to the local loop

From introduction to being usable in practice took essentially two days, and over the roughly two weeks after that I added checks little by little, against primary literature on how to write papers and against my own feedback. Because per-chapter checks leave the gaps between one chapter and the next unseen, I also later added a whole-piece test that reads across all chapters to see whether a single thread runs from the question to the claim.

I also decided the division of operation. Only the deterministic check runs automatically in CI every time, and the reviewer that handles the meaning judgment is reserved for the loop I run by hand. I built the mechanism up on a single abstract, and I extended the same loop to an abstract on a different topic.

What the machine could offer went only as far as material for judgment

Now that I have run it this far, the range the machine could carry has come into view. What the machine could do went only as far as offering, as material for judgment, the comments I have made over and over. Whether test-driven work raised the quality of the abstract itself, I cannot assert, because I have not taken the primary data of how the number of failures changed over time. That is as far as running it made things visible, and I refrain from declaring anything beyond that as an effect of the tool.

The line between the judgments that can be left to the machine and the judgments that only a person can keep making will surely keep moving from here, and how far it moves is something I want to make out as I keep running this operation.

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

Developing solo with AI: what to entrust to outside services, and what not to hold

matsumotory — Tue, 04 Aug 2026 11:57:23 +0000

Summary

Billing, login, and receiving contact messages: I handed all of them off to outside services. For membership and billing, the only things I keep in my own database are a mapping of who is a member of which plan, and a copy of the subscription and purchase state. I keep that copy because if the site went to the payment service to check the subscription state every time a reader opens a paid article, the display would get slow. When you develop together with AI, the implementation itself can be pushed as far as you like. So the first thing to decide was not what I could build, but what I would not hold myself.

Audience and takeaways

This article is for someone trying to build a service with billing on their own while having an AI agent write much of the code. You can take away how to decide how much personal data to hold yourself and where to start handing it off to outside services, and why deciding that standard up front pays off. It also gives concrete examples: what rule I used to deal with the AI's habit of confidently writing outdated specifications, and how I split the work so that the AI handles the research into terms and prices while a person makes the decisions that put limits on cost and permissions. This article is a record of practice, based on the process of building this site itself together with AI.

The gist of what happened

Over about two weeks, working with an AI agent, I worked out what to entrust to outside services and what not to hold. I withdrew the mechanism for storing the body of contact messages in my own database after I had already verified that it worked, and I moved the site's form over to an outside form service instead. I hand off card entry for payment entirely to the payment page that the payment service displays on its own side, so my site never handles card details or passwords at all. The authoritative record of membership and billing lives on the payment service's side, and my side holds only the mapping and the copy. Along with this, I built in two rules. First, I verify any design the AI writes against the body text of the current official documentation. Second, a person sets an upper limit on costs in advance, uptime monitoring included.

Before deciding what not to hold, see the whole of what has to be done

The judgment of whether to build something yourself, entrust it to an outside service, or not hold it at all is itself a demanding one, and doing it well needs a further step of thought beforehand. To decide what to entrust and what not to hold, you have to see the whole of what has to be done: billing, authentication, legal matters, monitoring, performance, and security. If you decide without seeing the whole, you might end up holding data you should have entrusted, or conversely throw responsibilities you should guard yourself, like billing and permissions, entirely over to an outside service.

Grasping this whole on my own, without gaps, was hard. So I had the AI enumerate the list of what has to be done, and I checked that list for anything missing through a number of reviews from separate viewpoints. I asked the AI for a security audit of code and configuration and an audit of display speed as separate tasks, had it check for gaps in release preparation and the free-tier limits of outside services from yet another angle, laid out one by one the actions a reader actually takes in order to look for gaps, and for terms and legal matters I went through everything against the original text of the law and of each service's terms. Any single viewpoint's enumeration always misses something somewhere, so I layer viewpoints to fill the gaps. On top of that, I narrowed what a person checks with their own eyes down to the areas where a slip is unrecoverable: money, permissions, personal data, the weighing of legal matters, and the final judgment.

Decide up front, as a standard, what you will not hold

With AI, implementation can in practice be pushed as far as you like. The processing to receive contact messages, the database table to store them, the verification that it works, all come back in a short time once you ask. At that point whether you can build it drops out of the material for judgment, and only whether you should hold it remains. So I decided that a person would settle the standard for what not to hold up front.

The standard is a single one: keep what I hold to a minimum. When I feel the urge to build a feature that stores personal data, I first look for an outside place to entrust it to. I narrowed what I hold myself down to only the things that do not work unless the authoritative record is on my side, like billing, authentication, and permissions.

Handing something to the outside is not the same as being safe. Even so, there are two reasons handing off is better than holding. First, the damage when there is a leak is decided by what you hold. Since you cannot stand on the premise of fully preventing every vulnerability, the strongest defense is that what you do not hold cannot leak. Second, for data like card details and the identity check for login, there are specialist businesses whose very trade is guarding it. It is not realistic for a lone developer to keep up the same level of defensive setup and investment. So I choose the place to entrust to by reading its terms and its data handling down to the original text and confirming them. Even after handing off, the work of guarding the seam with the place I entrusted to remains mine. That work means two things: keeping the table that maps members to their records on the payment service accurate, and confirming that a notification arriving from the payment service is genuine.

What made me decide on this standard was withdrawing the contact feature. The AI had implemented the receiving process and the storage table, and had finished verifying that it worked, and I sent that mechanism back for two reasons. One was to make the damage from a leak as small as possible. If I do not hold the message body and the reply address myself and instead entrust them to an outside form service, then even if my own database leaks, no contact messages are in it. The other is that the database capacity I can use is also limited. I wrote this judgment down as a rule, so from now on I can stop before building.

Card details and passwords are the same. I leave login confirmation to authentication where you follow a link in an email, and I pass payment straight to the payment page that the payment service displays on its own side. So a card number never passes through my site. This is not only to keep the damage small; it is also because the free terms of the delivery service forbid handling card details on a free site at all.

The authoritative record of membership and billing I also do not hold myself. Who subscribes to which plan, and whether payment is currently valid, is known correctly only by the payment service, where the money actually moves. If I hold the same ledger in my own database too, there come to be two authoritative records, and dropping a single payment notification is enough to make them disagree, which raises the problem of which one to believe. So I settled the authoritative record on the single one on the payment service's side, and in my own database I placed only a mapping that points to it and a copy of the subscription and purchase state. The copy is a local duplicate, so that I do not keep readers waiting by querying the payment service every time they open a paid article. If it breaks, it comes back by pulling from the payment service's record again. For writes tied to an individual, like likes and favorites, I made login the minimum condition, and I narrowed what may be received anonymously down to counts not tied to an individual, like an article's view count.

Verify the outdated specs the AI confidently writes against the current official documentation

The AI writes outdated specs from its training data with confidence, and plausibly at that. So you are apt to let them through as they are. I stumbled twice around payments. The first was invoice.payment_intent, a field that points directly from an invoice to its payment, which the AI was using straight from memory. In a spring 2025 update the payment service had removed this field and changed to representing the correspondence between an invoice and its payments as an array called invoice.payments.

The second was a harder-to-spot form. A reservation to cancel a subscription at the end of the period, under the old spec, arrived as a boolean field called cancel_at_period_end, read as: if true, cancel at the end of the period. In the current spec this field has been deprecated, and it now arrives as cancel_at, the time at which the cancellation takes effect. The AI had written code that read cancel_at_period_end from its old knowledge, so it became a defect where a cancellation a reader had scheduled was not recorded on their account. The automated tests did not catch this, and I found it only when I actually went through a cancellation in the test environment. A change of spec happens not only in whether a field exists, but in the very meaning of what arrives.

There was also one that nearly became a third. In a mechanism to prevent the same person from holding two subscriptions, the AI, before checking the existing settings, started to assemble a workaround that did the cancellation and refund itself. When I read the official documentation, I found that a setting to limit a customer to a single subscription had been there from the start, so I rebuilt it to use that. From these three experiences I set a rule: before using an outside service, actually read the current official documentation, and confirm field names and setting locations from that body text rather than from memory before writing.

After I set the rule, this way of reading paid off on the verification side too. There was a time when the auditing AI recommended, in order to hide the database's real address, a custom domain that would put the delivery service's protection in front of it. But confirming against the official documentation, I found that this custom domain, for reasons of transport encryption, can only be set up in a way that cannot pass through the delivery service's protection, so it does not hold up as a front-line defense. The core of protecting the data is not front-line network protection to begin with; it is row-level security, where the database itself judges per row who may read and write, and the permission checks on the application side. For this part I created real accounts with different roles and put them through integration tests. In place of the custom domain I assembled three things: an upper limit on monthly spending, an upper limit on how many times the server-side functions can be called, and uptime monitoring. They sit outside the core of the defense and keep usage from running away. To see whether the same kind of mistake, a place reading an outdated spec from memory, was anywhere else, I had the AI cross-check every place that exchanges with an outside service, and confirmed that the shape of the arriving data did not diverge from the current spec.

Leave the research to AI, and let a person put limits on cost and permissions

In how to deal with outside services, the split that worked best was to leave the research of reading terms and prices to the AI, while a person makes the final decisions that put limits on cost and permissions. Where this split showed clearly was in choosing uptime monitoring, which checks at intervals whether the site is working as seen from outside and tells you if it is down.

The AI, reading down to the original text of current prices and terms of use, dug out fourteen services usable commercially, and lined up the ones that fit inside a free tier as its recommendation. Here I chose differently. Rather than being free, I decided to choose something where I could hold the settings for which pages to monitor and how as code, even if it cost money, and among those pick the cheap one.

This standard has a reason that matters more than cheapness. In the development ahead, I will leave even more of the work to AI than before. With a service whose settings you cannot change without a person operating the screen, I cannot leave that operation to AI. With a service you can read and write through an API, both adding monitoring targets and confirming that the monitoring works correctly can be built in as the AI's work. This site is built, after all, toward a form where I just keep doing research and development while both the making of articles and the running of the site keep turning on their own. In that form, being programmable through an API, that is, operable from code, becomes the requirement I place on every outside service I stitch together. So even at somewhat higher cost, I gave priority to what can be operated through an API. What I chose is a metered service that pays a small amount per check. It comes to under a hundred yen a month, and the free allowance given at signup alone lasts more than half a year. The monitoring settings sit in the repository as code, so running them any number of times gives the same result, and there is no accidentally erasing a setting added by hand from a screen.

Here too, a person set the limits in advance. On the use of generative AI, an upper limit on monthly spending is placed across the whole organization. On the conversation feature, I layered three further limits, how much one member can send in a day, how much one member can send in a month, and how much all members together can use in a month, and I also put a limit on the length of text handed to the AI in a single conversation. When any of these is exceeded, the request is refused rather than processed, and the usage allowance reserved earlier for that member is returned.

For the generative AI used in conversation, too, I chose the paid API rather than the free tier. Reading the provider's terms, on the free tier the text a user sends and the generated answer may be used by the provider to improve its product, and in some cases may be reviewed by human eyes. On the paid API it was stated explicitly that they are not used to improve the product. To keep the body of conversations members write from being handed to the provider's training or review, here I decided to pay.

I also split the degree of strictness by purpose. Billing, authentication, and permissions I build strictly, while rough measures of use like likes and view counts I build with approximate aggregation and sampling, without bringing in needless complexity and cost. This too is one of the limits a person set in advance.

Not what can be built, but what can be entrusted

Even just choosing one uptime monitor, there was a back-and-forth: the AI combed through the terms, I chose differently by my standard, and the AI built it, ran it, and verified it. Looking back, I built the contact feature, then withdrew it and entrusted it to the outside. For uptime monitoring I use an outside service. Likes I keep on my own side, but only for members. And the custom domain recommendation I did not adopt. Every one is the same three-way judgment: build it yourself, entrust it to the outside, or not hold it at all. Each of them redrew the line of what to entrust and what to hold.

In today's development, where the amount of implementation is no longer the constraint, what I should have decided first was not what to build, but what not to hold myself, or in other words, what to entrust to the outside. Once you decide up front not to hold something, the worry about a leak and the trouble of moving it out when you shut things down are gone from the start. If a person places the standard first and builds in verification of the current spec and limits on cost, then no matter how much volume you have the AI produce, the design does not break down. Before counting what can be built, decide what can be entrusted. What can be drawn here is only the line on the entrusting side; whether the code I wrote myself is correct needs a different kind of defense. I will write that back half of the story in the next installment.

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

The day I moved the rules to a place they can't be broken even if you try

matsumotory — Tue, 04 Aug 2026 11:56:14 +0000

What happened that day

Even if you hand an AI agent the rules as a document, once a session grows long the rules stop getting read, and the same violations recur. In development where several agents read and write the same repository in parallel, when the rules stop being followed like this, it leads to accidents: unsaved changes disappear, work proceeds against a stale main, writes to the same file collide.

So I stopped relying on getting the rules obeyed at all, and instead moved the rules to a place where they can't be broken even if you try. This article is a record of running that countermeasure across three repository groups, and the three, which I had been operating separately, have settled into the same pattern: isolation via git worktree, which carves each task out into an independent directory.

Of the rules I had written in documents, I moved three into a layer that takes effect without going through the agent's judgment: stopping state-changing git commands with a pre-execution hook, forbidding direct pushes to main with a setting on the hosting side, and connecting the environment-variable file to the worktree with a hook that runs when a session starts. Once I did, the violations that had kept recurring under the document-based approach stopped happening. Carving out one working directory takes 0.25 seconds, so even splitting it per agent, the carving itself barely costs any time.

If you are considering development that lets several agents touch the same repository, this record gives you an isolation pattern you can try as it is, along with a way to tell apart the accidents you can prevent by the mechanism from the accidents you can only guard against by procedure.

Parallel-development accidents and the worktree isolation pattern

I have kept up development that has several AI agents read and write the same repository in parallel, across three repository groups, combining ones I develop through my personal company and ones I run personally. In this kind of development, accidents can happen: unsaved changes disappear, an agent reads a stale main and returns a stale analysis, writes to the same file collide, a git checkout or reset breaks the state of the repository itself. This article is about where to put the rules that prevent such accidents. The answer I arrived at was not to rely on writing the rules in a document and getting the agents to follow them, but to move the rules into a layer that takes effect without going through the agent's judgment, such as hooks, settings on the hosting side, and directory structure. In this article I will call this layer the mechanism layer.

Git has a mechanism called worktree that can carve out several independent working directories from a single repository. The isolation pattern that prevents accidents is an operation that creates this worktree inside the repository's directory hierarchy and confines all work there: for read-only investigation I create a detached-HEAD worktree that belongs to no branch, and for work that produces a PR I create a worktree with a branch. This pattern converged to the same shape as I operated it across the three repository groups. Some repositories have a rule that forbids creating worktrees under /tmp, because the permission settings for what the agent is allowed to do are not carried over there.

Under the hood, the object database that stores commits, and references such as branches, are shared across all worktrees, and only the local state, meaning HEAD, the index, and the working tree, is independent per worktree. In other words, the history stays single while only the local state can be split per agent. The time to create a worktree was, measured on a repository of a few hundred commits and a Mac with Apple Silicon, an instant 0.25 seconds for git worktree add --detach. What actually ate time was the dependency install, not the worktree creation. On projects that use Node.js, I dodge that wait by reusing the parent repository's node_modules in the worktree through a symbolic link, and I reinstall only on the days the lockfile that pins dependency versions changes.

Alongside the worktree isolation, there are two operational rules. One is to always run git fetch origin main before referring to anything. Other sessions and automated runs keep advancing main, so I work on the premise that the local checkout is always stale. The other is how to handle parallel writes. In order from lowest conflict risk, I choose among three methods: having each agent return data in a fixed format so that a single aggregating agent writes it all in one place, splitting the files each agent writes to, and isolating via worktrees. I limit operations that cannot be undone, like commit and push, to the aggregating agent. For these three methods too, the shape I ended up with did not differ across the repository groups.

Rules written in documents stop getting read

But writing these operations into document-based rules and getting them followed by instruction was a means whose adherence I could only speak of in terms of probability. Once a session grows long, the rules stop getting read, and the same violation recurs. Looking back over the history, rules read at the start of a session would sometimes stop being referred to in the latter half of a long stretch of work. This is what I mean when I say document-based rules are a probabilistic means.

Moving the rules into the mechanism layer

So I decided to move the rules from documents into the mechanism layer. I moved three of them. A hook that intervenes before the agent runs a tool mechanically refuses state-changing git commands issued without a worktree specified. The hosting side has a setting called branch protection that forbids direct pushes to a specified branch, and I used it to forbid direct pushes to main. For the environment settings, a hook that runs when a session starts hands them over by creating a symbolic link in the worktree that points to the parent repository's environment-variable file. Violations that had recurred even when written in documents have not happened since I put the hooks in. I use this before-and-after relationship, that the recurrence stopped after the change was introduced, as the basis for whether it worked. Apart from these three rules, there was also one settings fix that comes with running worktrees. When a worktree is created inside the repository's directory hierarchy, the static analysis that mechanically inspects the code would, left as is, scan into the worktree as well, so the setting that excludes the worktree from that scan is also placed in a config file rather than as a note in a document. The three rules I decided to move have thus moved from document instructions onto hooks and hosting-side settings that take effect without passing through the agent's judgment, that is, onto the mechanism layer: refusing state-changing git commands, forbidding direct pushes to main, and handing over the environment settings.

"Unbreakable even if you try" is what works.

What lay at the point where the three operations converged was a shift in thinking: rather than preventing accidents by raising the agents' reliability, you put the rules in the mechanism layer, meaning hooks, branch protection, and directory structure, and you change how things are built so that accidents cannot occur.

What fit the mechanism layer and what stays a matter of procedure

That said, only some of the rules could be placed in the mechanism layer. I managed to place the restriction on state-changing git commands, the ban on direct pushes to main, and the handover of the environment settings. But the accident of skipping git fetch and reading a stale main is not stopped by a hook, and I still rely on following the procedure. Which of the three parallel-write methods to choose also remains an operational judgment. The speed-up of reusing node_modules through a symbolic link also breaks when the lockfile changes and needs a reinstall. Both sticking to git fetch and choosing among the three methods are rules that stayed on the judgment side, ones you cannot stop in the form of a command. How far I can push them down onto the settings or hook side is something I have not managed to fully tackle yet, and I won't know until I try.

Even so, rules belong in a place where they can't be broken. I'll keep going in that direction without changing course.

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

Rewriting research prose until the tests pass

matsumotory — Tue, 04 Aug 2026 11:56:13 +0000

Summary

In a repository I run on my own for research papers and research operations, I rewrite research prose until the tests pass, the same way I would with code.

I have turned the quality checks into a set of tests written in YAML. The ones a machine can decide are judged by code functions, and the ones that cannot be decided without reading the meaning, such as how an argument is built, are judged by a large language model acting as a reviewer. When I get a point in review, I turn it into a rule on the spot, so that a person does not have to make the same point again.

But the formal checks that mechanically constrain things like banned words and sentence length, far from raising quality, erased the individuality of the writing and made the prose uniform, so partway through I removed the checks I had been adding. When a test fails, it returns a suggested fix, and it tells me whether that failure can be fixed by rewriting.

Rewriting until the tests pass

In software development, there are tests as a way to automatically check whether the code you wrote works as expected. For a while now I have been writing research prose with the approach of rewriting it until those tests pass. Tests for prose might sound strange, but what I am doing is simple. I take the points I get in review and turn them into check rules on the spot, so that a human never has to make the same point twice. That is all it is. For example, there was a time when I turned three points I received at once into rules on the spot and fixed two pieces of writing together.

I split the checks into two layers. The ones a machine can decide, like character counts and syntactic consistency, are judged by code functions, and the ones that cannot be decided without reading the context, like the overall arc of a piece from opening to close, or the flow of cause and effect, are left to a large language model acting as a reviewer. The meaning-reading checks also far outnumber the mechanical ones. At the time of writing this article, there were 131 checks handed to the reviewer and 15 checks seen by machine functions. I keep this whole set of checks as a suite of tests in YAML, and running it against the whole piece returns a pass or a fail for each check.

With this, the basis for rewriting changed too. Now I look at the list of failing checks and decide from it what to fix.

Dropping the checks that were only for form

This setup was not in its current form from the start. When I began building it, I was heading toward turning every point I received into a rule without omission, and I was adding checks a machine can decide, like a list of banned words and an upper limit on sentence length.

But checks that are only for form, like being too strict about character counts, forcing a number of paragraphs, or mechanically banning particular turns of phrase, look at first like they raise quality, while in fact they strip the style out of the writing and make it uniform. On its own, a word may not make clear what it refers to, but if the sentence just before explains it, the meaning comes through. Even usages that hold only because of context, the machine checks reject across the board.

So partway through I changed direction, and I left on the machine side only the checks whose answer is uniquely determined, while I moved the judgments involving meaning and logic toward having the reviewer read them together with their context. To each check I kept, I have tied a source for why that rule exists, one by one, so that whether the basis is a document that sets writing conventions or a point received in review, it can be traced later.

A failing test returns how to fix it

Another part of the design is that when a test fails, the test side returns how it should be fixed. Every failing check comes with a suggested fix, and it also tells me whether that failure can be fixed by rewriting or whether the underlying data is simply not enough. A failure that cannot be fixed is not a problem with the writing but a task of data collection. If the data is not enough, I do not touch the writing and just go back to measuring.

I have also decided how to handle the case where the reviewer's judgment could not be run. A check that did not run goes into a count of its own; it does not stand as a failure. If a failed judgment and a flaw in the writing get mixed together, the number of failures itself stops being usable as a guide to where I should fix things.

What I could confirm goes only as far as stopping recurrence

The checks that turn points into rules accumulate on the side of the system. A person no longer has to hold them in memory. The test results now let me confirm that the checks have stopped the same kind of point from coming back. But that is as far as I can say for certain about the effect right now, and whether the prose written this way has actually become easier for readers to read is not something the passing of a test can tell me.

I also still cannot put a number on how consistently the reviewer returns the same judgment for the same input. I have put in a mechanism that judges two more times only when the first judgment is not a pass and takes a majority vote over the three, but whether that makes the judgment stable is not something I have been able to confirm. If the judgment wavers, trust in the list of failing checks itself is shaken, so I see this stability as the single most important thing to confirm, since it bears on the premise of the setup. Extending this setup to long-form deliverables is a matter still further ahead, and what I hold for certain right now is only this one point: that the checks stop the recurrence of the same point, so a person does not have to say it twice.

(Added 2026-07-11: In this article as published, I had written the mechanism that takes a majority vote over multiple judgments as a plan not yet in place. When I checked the past records during an operational review, I found that a limited mechanism, which judges two more times and takes a majority vote only when the first judgment is not a pass, was already in place on 2026-06-24, before publication. So I have corrected the passage above to read that the mechanism is in place but its stability cannot yet be measured.)

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

A two-layer hash design that counts anonymous posts without identifying the device

matsumotory — Tue, 04 Aug 2026 10:28:16 +0000

The problem and the outcome in brief

Drawing on another app I develop at my own company, I want to describe the privacy design behind a mechanism that tallies anonymous posts. The app has a feature that publishes a piece of text assembled from anonymous posts, and it had to keep the people who posted from being tracked while also preventing one device from inflating the numbers by posting over and over.

In this article I first explain how the app generates a pseudonym for each device with two layers of hashing. Next I explain the threshold that decides whether the assembled text may be published. Right after this threshold went in, it was not doing its job: it counted the number of posts while meaning to count the number of devices. Last, I describe how the design document writes out what this design cannot protect. The material is about three weeks of records of design, implementation, and fixes, along with a design document that kept being updated afterward.

Two requirements pulling in opposite directions, and the final shape

The app has a feature that assembles the anonymous posts collected about a given subject into a single piece of text and publishes it. Two requirements pull on this feature at once, in opposite directions. The people who posted must not be traceable. At the same time, the design must keep one device from posting over and over to fake the support of several devices. Not being able to identify the device is safer for preventing tracking, and being able to identify it is surer for preventing inflated numbers, so the two requirements collide head on.

Let me show the final shape first. The design settled into a form that keeps no more power to tell devices apart than detecting abuse requires. The app generates a pseudonym for each device with two layers of hashing and uses that pseudonym only to count how many distinct devices there are. The number of distinct devices supporting the text and the threshold test alone decide whether the app publishes it. The app uses the pseudonym for nothing but tallying, and what a reader sees is only the number of devices.

The two layers of hashing that generate the pseudonym

On first launch, the device generates exactly one random identifier. That identifier is a UUID v4, in which 122 of the 128 bits are random. The definition of UUID v4 is in RFC 9562. The device keeps the identifier only in its secure storage area, ties it to no account information, and never sends it to the server.

When the device sends a post, it concatenates this identifier with the ID of the subject, hashes them one way with SHA-256, and hands only the hash value to the server. The raw identifier stays inside the device alone and never leaves it. The server applies HMAC-SHA256, a keyed hash, to the value it received, using a key held only in a secret store, and saves the output as the pseudonym for tallying. The definition of HMAC is in RFC 2104. Written as formulas, the flow looks like this.

Device: p = hash(device ID + ":" + subject ID)    SHA-256 here
Server: pseudonym = keyed_hash(secret key, p)     HMAC-SHA256 here

The hash on the device hides the raw identifier, and the keyed hash on the server prevents pseudonyms from being forged. Anyone who does not know the key cannot compute the output of HMAC, so an attacker who wants to produce pseudonyms has to obtain the key itself. The key sits neither in the database nor on the device.

This division of labor borrows the salt-and-pepper pattern, a standard practice in password storage. A salt is a random value that differs for each value it is added to, and a pepper is a secret shared across the whole system and kept outside the database. OWASP's Password Storage Cheat Sheet sets out this distinction. In this design, the subject ID mixed in on the device plays the role of the salt, and the key on the server plays the role of the pepper. If the device and the server compute the derivation differently, matching the same device breaks. That is why the derivation on the server lives in a single shared module.

The design that closes the range of matching to a single subject

The key point of this design is that the device mixes in the subject ID. Even for posts that came from the same device, the pseudonym is a different value when the subject differs. So neither the server nor anyone reading the database can match a post about subject A with a post about subject B to tell whether they came from the same device. The range in which posts can be tied to the same device stays deliberately closed inside a single subject.

I did not take the option of one stable device hash shared across all subjects. With that, anyone could match the same device across subjects and assemble a history of each device's behavior out of posts that are supposed to be anonymous. On the other hand, the input to this pseudonym does not mix in a server-distributed salt that might be rotated. If the value is not stable for a given subject, counting falls apart. The implementation notes state plainly that concealment is the pepper's job.

The threshold that decides publication

The assembled text keeps a record of which devices' posts went into its material. I count those devices as the devices supporting the text. There are two conditions for publication: support from 5 or more distinct devices, and a share below 0.8 for the single device that contributes the most support. If either one is missing, the app does not publish the text. The default value of the database column is unpublished as well.

During trial operation I lower the threshold from 5 to 2. I do not lower it to 1, though. With support from a single device the share is always 1.0, so publication is structurally impossible, and it would also break the promise to publish only text gathered from several devices. I do not loosen the ceiling of 0.8 on the share even during trial operation. Beyond that, at most 3 posts per pseudonym go into the material, so that repeated posting cannot take over the content. Posts whose pseudonym the code could not compute do not count toward the number of devices, and if the pseudonym is unknown for every post, the code returns 0 devices and a share of 1.0, which always means no publication. I carved this decision out as a pure function of its inputs and outputs alone, and unit tests pin down both the default values and the trial-operation values.

The bug that counted posts while meaning to count devices

Right after it went in, this threshold was at one point effectively not doing its job. What it counted was the number of posts, not the number of distinct devices. The cause was a mix-up. The device was also sending another hash, meant for a different purpose, whose value changes with every post, and the server was using that one as the material for the pseudonym. Since the same device produced a different pseudonym every time it posted about the same subject, a single device could fake the support of several devices just by posting over and over. The fix consolidated everything into one form, in which the server applies HMAC to the stable value the device generates. I made this fix while the app was still in trial operation, before it opened to the public.

There was a reason the other hash got mixed up with this one. The same app has another hash. It mixes a server-distributed salt, a per-row random value, and the date into the device identifier, and deliberately scatters the value for every single post. It is a tool that makes any tally which groups posts by user impossible, down to the structure of the database itself. The hash that scatters its value and the pseudonym for tallying that keeps its value stable look like the same hash, yet they require exactly opposite properties. The moment one goes where the other belongs, the counting mechanism misbehaves without raising a single error.

The limits the design document spells out

This design document writes out the range it cannot protect at the same granularity as the range it can. The table below organizes this by the situation in which data leaks.

Leak situation	What the attacker gets	What they cannot do, and why
The database alone leaks	The pseudonyms output by HMAC	Without the secret key, they cannot recompute them, match them, or forge them
Both the database and the secret key leak	The pseudonyms and the key	The original input requires a 122-bit random identifier, so brute force does not work
The server runtime is compromised	Values already hashed on the device	The raw identifier does not exist outside the device, so they cannot recover it
The device itself is compromised	The data inside the device	They can read the data straight off the device, so attacking through the server is pointless

The design document is candid about why the pepper is there, too. Because the input is 122 bits of randomness, the classic motivation for a pepper, protecting a low-entropy secret, does not apply directly. It is there anyway as a defense-in-depth call: it costs only one HMAC, it gains resistance to a leak of the database alone, it keeps the secret separate, and it acts as insurance if the assumption about randomness ever falls apart.

The document names three things the design cannot protect. First, N distinct devices do not mean there are N distinct people. Second, an attack that mass-produces many devices or emulators to fake identities remains. This is the type known as a Sybil attack, in which one actor claims many identities. Weighting a new device's support low at first, together with rate limits, can raise the cost of the attack, but it cannot bring the faking itself to zero. Third, if separate real people agree among themselves and post the same falsehood, a mechanism that tallies while keeping everyone anonymous has no principle by which to tell it apart. The document accepts that the terms of service, reports from users, and human review cover this part.

The document also settles how I rotate the secret key. Rotating the key severs matching against past pseudonyms and cuts off the material for tracking. In exchange, once the key has changed, I can no longer match the same device to its past pseudonyms, so I can no longer recount the device counts for the past correctly. That is why I keep support counts already published as settled records and never withdraw them by recomputation.

The way the limits are written sits at the center of the design

What works in this design is less the choice of each hash than the way the limits are written. This design document does not declare that tracking is impossible. It splits the situations apart, asking what happens if the database alone leaks and what happens if the key leaks too, and writes out both the range it can protect and the range it cannot. That is the center of the design. Even the two lines on the side it cannot protect are in the document: the design accepts faking that uses many devices and raises its cost, and it sends coordinated falsehood to human review.

In any design, some judgment remains at the end that cannot be fully entrusted to a machine and goes back to human review. In this design it was telling coordinated falsehood apart. What I have wanted to do all along is the other thing: to teach the machine the shapes of these judgments that remain at the end, and to whittle away the role of the human reviewer itself. In time I want to leave only the data, to have even that data produced by AI, and in the end to build an AI that mirrors me. The judgment I just called one that only a person can make is no different. If I can hand over its shape one piece at a time, the line that sends it back to a person should be one I can erase someday. That is what I am working toward.

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

Detailed instructions written for an earlier generation of AI models become harmful on today's models

matsumotory — Tue, 04 Aug 2026 10:28:15 +0000

Summary

I made two passes on how to design memory and instructions for AI agents, comparing research papers with the official documentation for frontier models. Instead of taking the first pass's conclusion as it stood, I audited which generation of model each supporting paper had measured on. Not one of the five main papers measured on a frontier model, and alongside that I confirmed that the official documentation of three companies has been changing in the same direction. My sources were the notes I made reading and checking the papers' body text myself, together with the original text of each company's official documentation. I also draw on how that conclusion was worked into the operating rules of a membership platform for a technical blog.

What the official documentation says now

The model providers themselves write in their official documentation that detailed instructions aimed at an earlier generation can lower output quality on today's models. Anthropic's official documentation for Claude Fable 5 says that skills built for earlier-generation models can over-instruct and lower output quality, so if the default behavior is better, consider deleting the old instructions. It also says that you can now steer the model with short instructions, without naming and listing each behavior one at a time. Technical staff at the same provider reported that cutting more than 80 percent of the system prompt for a coding tool produced no measurable drop in their evaluations, and they characterized what they cut as constraints that had been needed to guard against the worst behavior of older models.

OpenAI points the same way. The official documentation for GPT-5.6 has a section telling you to keep instructions lean, and it gives numbers: cutting repeated instructions and examples and making tool descriptions concise raised internal evaluation scores by roughly 10 to 15 percent and cut total tokens by 41 to 66 percent. Google's developer guide says that verbose or overly complex prompting techniques written for older models can be over-analyzed, so write concisely. Only two of the three give numbers, and both are the provider's own report rather than an independent measurement, but the fact remains that all three point their guidance the same way. An instruction document that has been built up over time can, if left alone, start dragging output quality down on a newer generation of models.

Which generation of model the supporting papers measured

Let me stop here for a moment. A little while back, when I surveyed research on agent memory broadly, the first pass's conclusion leaned the other way. On the strength of papers finding that memory can degrade if you keep updating it with distilled lessons alone, and that keeping raw fragments of the conversation as memory beats processed memory, I read it as raw logs beating distillation. Before locking that conclusion into the design, I questioned a premise, namely whether that research still holds on today's models, and ran a second audit pass.

The result was clear. Of the five main papers I had leaned on, not one measures on a frontier model. The paper claiming that consolidation degrades memory uses the strongest model for its experiments, but that model, too, was from the earlier generation. The paper where raw fragments win measured only on 2024-generation models, and its authors write in their own limitations section that personalization and writing-style uses are unverified. The study showing limits on reproducing writing style measured only on two 32B open models. On top of that, the consolidation paper's numbers disagree between its abstract and its body. The abstract reads as though it fails 54 percent of the time, while the body and the figures show 54 percent as the success rate that remains. My first pass had drawn on the abstract's side. A paper's conclusion circulates through its abstract, but you cannot see the experimental setup and the limitations without reading the body.

For the record, that no one has remeasured on frontier models does not mean the papers are wrong. It states where things stand: I do not yet know whether they still hold. If that distinction is not kept, then it is my side that ends up throwing research away without grounds.

Two axes that had been mixed together

The audit showed one more thing. The very axis of the first pass, raw logs against distillation, had mixed two separate questions together. What the papers measure is the question of what to store as memory and retrieve later, while the discussion of instruction documents deals with the question of what to hand over as instructions right now. They are different questions, so they do not contradict each other. And the only layer both bodies of primary material agree in rejecting is the middle one: a normative enumeration of steps, cut loose from context and set out as bullet points. The papers say that if you keep having a machine regenerate that enumeration of steps, the process breaks down over the long run, and the providers say that layer can be cut back heavily. What neither side rejects is the record of the exchanges that actually happened, and the purpose of the work itself. This is not spelled out in any outside document. I arrived at it by comparing the papers with the official documentation.

Instructions to cut and instructions to keep

Put into practice, the right answer is not that you should make instructions abstract. What works is matching how detailed your instructions are to the generation of model you use.

Here are the kinds of instruction you can cut: instructions that specify each step one at a time, instructions that enumerate each behavior one at a time, instructions that cover every case, instructions that stack the same point in different words, and instructions that give examples on reasoning tasks. There is evidence of a generational shift here: on strong models, adding conventional examples does not raise reasoning performance. Researchers have also identified the mechanism by which fine-grained specification does harm. When domain jargon, lines stating constraints, or identifier names sit in the instruction, the model calls up a wrong answer it holds in memory. Take those out, the explanation goes, and the model has no choice but to reason from the structure of the problem itself. The same measurement also shows that on simple problems, making things vague drops accuracy by 11.8 points, so cutting is not the answer in every situation.

What you keep is the purpose and the domain background, the constraints you cannot give up, the boundary of what needs approval, the conditions that decide success, and the output format. OpenAI's official documentation, right after writing that you often do not need to prescribe every step, goes on to say that you should still supply domain background, constraints you cannot give up, approval boundaries, and the conditions that decide success. Anthropic's general guidance still tells you to be specific about the output format and the constraints you want. The same official documentation also carries concrete instructions pointing the other way: for hard tasks, give the full specification at the start, and for narrow tasks, constrain the scope explicitly. One measurement also shows that reproducibility falls apart once even the conditions of judgment are left vague. Models correctly fill in requirements you did not write only about 40 percent of the time, and the spread doubles. Along with this, write the reason for a rule rather than the rule itself, because when the reason is written, the model generalizes from the explanation. And if you want the model to question its premises, abstract instructions will not produce that on their own. Even reasoning models fail to point out somewhere between a quarter and 40 percent of false premises, by one measurement. If you want premises checked, write it explicitly as a pass-or-fail condition on the deliverable, not as a step.

I will add two caveats as well. The claim that stronger models are harmed more by concrete detail is not supported. There is a measurement showing that the effect of granularity does not correlate with model strength. The capacity to follow many instructions at once has also grown a great deal over the past year, so cutting instructions is not about a limit on volume. It narrows where the model puts its attention.

Taking stock as a procedure, not as a rule

On my own platform, I worked this conclusion into the operating rules. I did not write it as a rule saying instructions should be abstract, though. That is because the evidence here is specific to a model version, the recommendation differs from version to version even within one provider, and it will change again with the next version. What I wrote is a procedure. I wrote that I take stock of step-by-step instructions added to cover an old model's weaknesses, and delete them when the model changes. I also wrote that I remeasure how detailed the instructions are every time the generation changes.

An instruction document is an asset, but part of it is correction for the model's weaknesses, and those weaknesses fade away with the generations. In the week a model changes, try rereading your own instruction document once. The more lines you can delete, the more that document was written to fit the earlier generation of models.

Research I referred to

I list the research this article rests on, with titles and links. I checked all of them down to the body text as of 2026-07-26. Where a title is not given, I describe the contents instead.

Useful Memories Become Faulty When Continuously Updated by LLMs https://arxiv.org/abs/2605.12978
A controlled comparison of raw conversation fragments against extracted memories https://arxiv.org/abs/2601.00821
Agentic Context Engineering https://arxiv.org/abs/2510.04618
Theory-Grounded Evaluation Exposes the Authorship Gap in LLM Personalization https://arxiv.org/abs/2604.26460
When Prompt Under-Specification Improves Code Correctness https://arxiv.org/abs/2604.24712
What Prompts Don't Say https://arxiv.org/abs/2505.13360
Revisiting Chain-of-Thought Prompting: Zero-shot Can Be Stronger than Few-shot https://arxiv.org/abs/2506.14641
Evaluating Reasoning Models for Queries with Presuppositions https://arxiv.org/abs/2605.03050

Official documentation is not research, so I list it separately. The quotations in the article are as the documents read on 2026-07-26. This article organizes what public documents state, and it does not represent the views of any of the providers.

Anthropic's official documentation on prompting https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-fable-5
Anthropic's general guidance on prompting https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/claude-prompting-best-practices
An article on how 80 percent of a system prompt was cut (Anthropic) https://claude.com/blog/the-new-rules-of-context-engineering-for-claude-5-generation-models
OpenAI's official documentation on prompting https://developers.openai.com/api/docs/guides/prompt-guidance-gpt-5p6
Google's developer guide https://ai.google.dev/gemini-api/docs/gemini-3

Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.

DEV Community: matsumotory

The world as a data stream, and the human who moves to the side that makes data

Summary

Audience and takeaways

Two projects, different in field and in build, had sorted where feedback to an AI should live into the very same three layers

Summary

Feedback disappears when the session ends

Three layers and one criterion

The failure of writing general rules into memory

Shifting toward a layer that works even when it is not read

The dependence on being read that still remains

The human work left after shifting toward the machine

Matching strings do not settle the direction: which document is the original

Summary

What you can take away

What a string match does not settle

How the check that detects copies is built

A reverse-direction false alarm caused by quotations in a source document

The blind spot left by excluding my own repository

The defect in handling allowed words

Recording matches already judged, and judging the direction again

A design that revises the targets of matching without loosening the check

Research I referred to

Materials I referred to

Does shallower reasoning make an AI's Japanese more natural? I compared five drafts and the assessments split

Summary

What you can take away

How I set effort and run it now

What happened on the day I passed on the impression

Comparing five drafts written from the same material

What came of having a different AI read the five collected studies again

Why I separated the writing role from the checking role

How to run the same check in your own operation

Research I referred to

Materials referred to

Deciding the criteria for excluding an AI-collected research topic candidate

The problem and the outcome in brief

Deciding the criteria for exclusion

What the automatic extraction produced on the day I rewrote the criteria

The level of detail that makes a record usable as a candidate later

The five conditions for aggregation, and who decides adoption

What I hand to an AI agent, and what I keep with a person

Research I referred to

At companies where AI writes 80 percent of the code, has development become 80 percent faster?

Summary

What you can take away

The problem and what came of it

The hypothesis

What each company officially says

The structure where the gap between models widens the next gap

The chain I run at an individual scale

The most modest judgment came from the parties themselves

Where the hypothesis stands, neither denied nor confirmed

Research and primary sources I referred to

Rewriting prose until the tests pass: everything passed, but the check that mattered never ran once

Summary

Audience and takeaways

Bringing the code development loop into long-form writing

Dropping checks that misjudge when enforced without context

Keeping errors separate from the quality judgment

Letting assertions pass within the range of observation, suppressing variance with a majority vote

The deterministic check to CI, the reviewer to the local loop

What the machine could offer went only as far as material for judgment

Developing solo with AI: what to entrust to outside services, and what not to hold

Summary

Audience and takeaways

The gist of what happened

Before deciding what not to hold, see the whole of what has to be done

Decide up front, as a standard, what you will not hold

Verify the outdated specs the AI confidently writes against the current official documentation

Leave the research to AI, and let a person put limits on cost and permissions

Not what can be built, but what can be entrusted

The day I moved the rules to a place they can't be broken even if you try

What happened that day

Parallel-development accidents and the worktree isolation pattern

Rules written in documents stop getting read

Moving the rules into the mechanism layer

What fit the mechanism layer and what stays a matter of procedure

Rewriting research prose until the tests pass

Summary