The problem and the outcome in brief
I collect the problems I run into in daily development and operations as candidate research topics, rather than letting them end with an on-the-spot fix. An AI agent automatically reads the material and drafts candidates (I will call this role the extraction agent), and a person decides whether to adopt them. In this process I narrowed the reasons for excluding a candidate down to two. This article explains how that came about, and what the automatic extraction produced when I ran it on the day I rewrote the criteria. The point is that a criterion for exclusion has to be one whose truth the extraction agent can judge from the material alone. I also cover how to write records so they can serve as candidates later, and the division of roles that leaves adoption and confirmation to a person. This is for people who want to pick research or writing topics out of their own development records.
Deciding the criteria for exclusion
I record the problems I find in daily development and operations as candidate research topics. An AI agent extracts candidates from development records, following a procedure kept in a repository I run on my own for research drafts and operations records, and that same procedure also decides which candidates may be excluded before they go into the record. On 2026-07-15 I rewrote these exclusion criteria. Before the rewrite, I judged exclusion with a single question: does a candidate still stand as a research topic and look valuable once you cut it loose from the circumstances of the organization doing the development? The extraction agent excluded any candidate that did not satisfy this question. The criteria after the rewrite limit the reasons for exclusion to two. The first is a candidate that does not generalize, meaning one whose context does not carry to a reader who does not know the internal circumstances of the development. The second is a candidate whose confidential information cannot be separated out, meaning one where no change of wording removes the part that has to stay hidden, such as the name of an unreleased product. Every candidate that passes those two goes into the record in full, and I do not use an assessment of value as a reason to exclude; I write it instead as an observation attached to the candidate.
The person who decides adoption judges whether a candidate has value. The extraction agent does not pre-empt that judgment. If the extraction agent excludes a candidate up front on the grounds that it seems to have no value, nobody can verify afterward whether the candidate really had none. On top of that, the extraction agent comes to keep only the candidates that look likely to match the wishes of the person who decides adoption, and the range of collection itself narrows to what matches that person's interests.
Up to this point this has been about candidate research topics, but around the same time, I also made a rule change in the opposite direction on the blog that carries this article. I was increasingly writing an article and only then finding it too thin to keep, so for the high-priority candidates among the article proposals, I added a condition: that the exchange between the AI and the person, that is, what was tried, what happened, and what was fixed and how, can be copied out of the material in concrete form. In other words, for research topic candidates I removed a condition the extraction agent has no way to verify, and for article proposal candidates I added a condition the extraction agent can verify on the spot. Both changes rest on the same principle: a criterion for exclusion has to be one whose truth the extraction agent can judge from the material alone.
Along with this principle, here are the five things I hold to across the process from collection to aggregation.
- Only two kinds of candidate may be excluded: ones that do not generalize, and ones whose confidential information cannot be separated out. An assessment of value is written as an observation attached to the candidate, and is not used as a reason to exclude.
- Excluded candidates stay in the record too, with the reason for exclusion attached. Without a record, you can no longer tell afterward whether the range of collection was narrow or the material simply was not there.
- In the recording step, write the conditions for reproducing the problem, meaning what happened when you did what, and the place where that event can be confirmed, in the same place as what you did about it.
- Material with only a single failure is too thin in content to become a candidate on its own. I wait until similar failures accumulate and then pull them into a single candidate.
- In the aggregation step, a person makes the adoption decision and confirms it. The extraction agent hands over every candidate with a proposed judgment attached, and does not cut candidates on its own by reading ahead into what the deciding person wants.
What the automatic extraction produced on the day I rewrote the criteria
On the day I rewrote the criteria, the AI agent ran the automatic extraction once under the new ones. Here are the numbers from that record.
| Category | Count |
|---|---|
| Candidates received | 60 |
| Raised as new candidates | 24 |
| Absorbed into an existing candidate | 15 |
| Dropped as duplicates of other candidates | 21 |
| Excluded under the two criteria | 0 |
Not a single candidate met the two criteria. What actually did the narrowing was the 21 folded together as duplicates and the 15 absorbed into existing candidates.
That zero cannot be read as the effect of the rewrite alone. In the same change, I also added viewpoints for the extraction. Until then there were only seven viewpoints, named after fields such as ways of thinking about design, or privacy. To those I added seven viewpoints for how to compare material, such as comparing the gap between an estimate and a measurement in numbers, or questioning a pass result itself. The extraction record shows that 11 of the 24 new candidates came mainly out of the viewpoints I added. Because I bundled the rewrite of the criteria and the addition of viewpoints into a single change, I cannot separate which of the two the zero belongs to. It can also be read as the criteria being so loose that nothing gets excluded at all. The AI agent kept even the candidate that looked most specific to my own setting rather than excluding it, on the grounds that the same thing happens in other environments, and that judgment is in the record. Still, I have not run the measurement enough times yet to rule that reading out.
The level of detail that makes a record usable as a candidate later
Even if you keep the range of collection wide, a record that holds nothing but what was done about the problem cannot serve as a candidate later. The rules document for an app I develop at my own company carries a dated record of an accident that happened when an operation with side effects was sent to an external service. The address used to test user registration turned out to be a real one, and the confirmation email reached a stranger. The same section also states that the message is recorded in the authentication log on the external service side. There is a list of checks to prevent a repeat, and one item stops the operation if a real domain is included. The conditions for reproduction, meaning what happened when you did what, and the place where it can be confirmed, sit in the same place as what was done about it, so the record can serve as a candidate later. The third of the five conditions I listed above refers to this form of record.
The five conditions for aggregation, and who decides adoption
I use five conditions to decide whether to adopt a collected candidate as a research topic I will actually start work on.
- Who is responsible has been decided
- When it starts has been decided
- Where the results will be published has been decided
- The first piece of work has already been started
- It has passed the check for whether it generalizes
If even one is missing, I do not adopt it. I lower its priority or move it to hold. The extraction agent is responsible up to handing over every candidate with a proposed judgment attached, and for a candidate it has not judged yet it writes "not judged" rather than leaving the field empty. Otherwise you can no longer tell afterward whether it was judged or skipped.
I also have a measurement of how many the deciding person withdraws. On this platform, on 2026-07-27, I went through all 60 article proposals and decided which were needed, keeping 45 and withdrawing 15. The reasons for withdrawal fell into three patterns. The largest was candidates where the AI only appears as something called or something worked on, with no exchange between the AI and a person at the core, and that accounted for 60 percent of the 15 I withdrew. The rest were candidates that amount to an explanation anyone who knows the field could write, and candidates raised from thin material with only a single failure. Checking all 60 is a measurement on the article proposal side, but if the extraction agent does not pre-empt the judgment of value, that judgment falls to the person deciding adoption. When the range of collection is kept wide, I take this one quarter as a rough guide to how many the person will withdraw.
The design of separating the collection step from the step that decides adoption is not new in itself, and it is already in the handbook for systematic literature reviews. The pattern of reading across after-the-fact incident records to find a common theme, rather than stopping at an individual fix, is also established, as laid out in a standard account in the operations field. Compared with that pattern for incident records, I changed two things in my own operation. I widened the target from incidents to everyday problems and review comments as well, and I made the destination of the aggregation a candidate research topic.
What I hand to an AI agent, and what I keep with a person
In this operation, for candidate research topics, an AI agent automatically handles reading the material, drafting candidates, and attaching a proposal, and a person confirms whether to adopt. Outside measurements confirm that this division of roles makes sense, too. A human-evaluation study that enlisted more than 100 researchers judged ideas from an LLM to be more novel than ideas from experts, while their feasibility came out somewhat lower. A survey that generated more than 200,000 ideas reports that AI ideas cluster near the literature they started from. The latter is a result showing that the more the collection step is handed to an AI, the more the candidates gather into a narrow range. I take both results as support for keeping the adoption judgment on the human side.
There is a shared convention for recording what an AI agent does as well, the OpenTelemetry semantic conventions for GenAI, and work is under way there to settle the names and meanings of the items to record. The document's status is still Development, though, so the names can still change. As of today, what an AI can take over is the first-hand record of what was called when and what came back, and the judgment that raises that into a problem worth working on still sits with a person.
Whether this process has reached a research result is something I cannot speak to yet. What I have is a little under two months of material even at its longest, and three weeks of records since the rewrite, so measuring whether the candidates I collected made it to acceptance is still ahead. Even so, when you are torn between widening and narrowing the range of collection, start by checking whether you can tell from the material alone whether something meets your current exclusion criteria.
Research I referred to
Here are the five outside primary sources referred to in the article. I checked the source page for all of them as of 2026-08-06. For the two arXiv items, I checked the abstract page.
- Chapter 4 of the Cochrane Handbook (designing the search step) https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-04
- The chapter on postmortem culture in Google's SRE book https://sre.google/sre-book/postmortem-culture/
- The document in the OpenTelemetry semantic conventions for GenAI that defines the unit for recording what an AI agent does https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md
- Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers. A human-evaluation study by Si, Yang, and Hashimoto that enlisted more than 100 natural language processing researchers (submitted 2024-09-06) arXiv:2409.04109 https://arxiv.org/abs/2409.04109
- AI Research Agents Narrow Scientific Exploration. A study by Tang and Yang on whether AI research agents narrow the range of scientific exploration (submitted 2026-05-27, revised 2026-07-11) arXiv:2605.27905 https://arxiv.org/abs/2605.27905
Originally published at The Future of Humans, AI, and the Web, a site where my research and development is recorded and analyzed by a human and an AI.
Top comments (0)