🙋‍
I’m Luhui Dev, a developer who has been breaking down Agent engineering and exploring how AI can be applied in education.
I focus on Agent Harness, LLM application engineering, AI for Math, and the productization of education SaaS.
Imagine this.
You give AI a research problem: find a new way to reduce the inference cost of large models.
A few hours later, it returns a complete paper, a set of experimental code, a collection of benchmark data, and what appears to be a novel algorithm.
The paper is well structured, the charts look polished, and the experimental result is striking: inference speed improves by 35%.
Now comes the question: do you believe it?
You might first open the code to confirm that it actually implements the method described in the paper. Then you inspect the experiment logs to see whether the 35% was genuinely produced by a run or cherry-picked by the model from an intermediate experiment. Finally, you verify every citation to make sure those convincingly named papers really exist.
Then you discover that the algorithm is not in the code, the experimental results cannot be reproduced, and some of the cited papers cannot even be found.
The most awkward part is that the paper may still read as entirely professional.
ScientistOne Is Not Trying to Make AI Better at Writing Papers
In May 2026, the Google Cloud AI Research team released an arXiv paper: ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence.
The focus of this work is not merely to build a stronger AI Scientist. It pays closer attention to how AI-generated research can be audited.
The team audited 75 papers produced by five automated research systems across five categories of frontier systems research tasks. The results are revealing:
- In some systems, the hallucinated-citation rate reached 21%;
- In some systems, only 42% of reported paper scores passed re-verification;
- Method-code alignment across systems ranged from only 20% to 80%.
In other words, a paper may be well written, and its algorithm may even produce good results, but the numbers, methods, code, and citations in the paper do not necessarily belong to the same chain of facts.
It is a little like a programmer delivering elegant technical documentation, a complete test report, and working code, with just one problem: the three describe different systems.
ScientistOne's core proposal is called the Chain-of-Evidence (CoE).
The basic idea is that every important Claim must be traceable through a recorded chain of evidence back to its original basis.
From “Generating Answers” to “Generating Verifiable Knowledge”
A typical Agent output process looks like this:
User question -> Agent execution -> Final answer
For example, it tells you that a new algorithm improves accuracy by 10%. At that point, the task is complete.
CoE is not concerned with whether that sentence sounds enough like a conclusion. It asks whether a complete chain stands behind it:
Claim: Accuracy improves by 10%
↓
Evidence: Metric in the evaluation log
↓
Artifact: Evaluation script, training code, data version
↓
Grounding Source: Actual experimental run
↓
Verification: Rerun and calculate the metric
The final deliverable of a CoE Agent should therefore be more than a passage of text. It should be a knowledge unit that can be inspected.
People do not need to trust it because it “sounds true.”
They can follow the evidence chain and check for themselves whether it is true.
How CoE Works: First Classify the Claims in the Research
The paper divides scientific claims into four categories:
Citation Claim
Numerical Claim
Methodological Claim
Conclusion Claim
Different Claims require different evidence.
If AI says, “A certain study proposed a certain method,” it must link to a real paper that the system actually read.
If AI says, “Accuracy improved by 10%,” it must link to evaluation output and experiment logs.
If AI describes a new Attention optimization method, it must link to the module in the code that actually implements it.
If AI reaches a research conclusion, it must explicitly identify which verified numerical and methodological claims the conclusion depends on.
This step may look like nothing more than structuring natural language, but it is critical.
“Our method works very well” cannot be directly verified, but the following claim can:
{
"type": "numerical",
"claim": "Model accuracy increased from 81.2% to 89.4%",
"evidence": "runs/exp_042/evaluation.log#L118",
"metric": "top-1 accuracy"
}
Only when a Claim is structured does verification have a clearly defined object.
ScientistOne Does Not Add Citations at the End; It Records Evidence from the Beginning
This is what I find most noteworthy about ScientistOne.
Many Agent designs wait until a paper is nearly finished, then ask another Agent to check citations, correct numbers, and add evidence.
That is like adding tests the day before code goes live: it is not entirely useless, but much of the context has already been lost, leaving the system to guess.
ScientistOne's approach is to let evidence emerge alongside the research process.
Its system is roughly divided into three stages:
Problem Investigator
Literature retrieval, full-text reading, and sourced research briefs
↓
Discovery Engine
Parallel exploration, experiment execution, and preservation of evaluations and logs
↓
Paper Writer + Claim Verifier
Writing from existing artifacts and verifying each Claim
During the literature stage, it retrieves papers from academic databases, reads the complete PDFs, and records source information.
During the experimental stage, it preserves the code, Evaluator scores, execution logs, and ablation results together.
During the writing stage, every sentence containing a number or citation must be bound to specific evidence in advance. The Claim Verifier then checks whether numbers in the paper can be found in the logs, whether citations support the original sentences, and whether method descriptions match the experimental records.
Claims without a source, or with invalid evidence markers, are deleted directly.
This is not AI casually pasting a few links after an answer.
It is the creation of an evidence ledger, alongside the research itself, that can be replayed and inspected.
Verification, Not Just Provenance
Recording a source is not the same as completing verification.
The existence of a file does not mean the data inside it is correct.
The existence of a real paper does not mean it supports the current statement.
The fact that code can run does not mean it implements the method claimed in the paper.
ScientistOne therefore also introduces a CoE Integrity Audit that checks four categories of problems:
Score Verification
Can the scores reported in the paper be reproduced?
Specification Violation
Does the approach exploit loopholes in the evaluation rules?
Reference Verification
Do the cited references actually exist?
Method-Code Alignment
Does the method described in the paper actually appear in the code?
According to the project's published data, ScientistOne achieved the following in this set of experiments:
- Zero hallucinated citations across 337 references;
- All 12 reproducibly evaluated papers passed score verification;
- 14 of 15 papers passed the method-code alignment check.
These figures come from the paper team's own experiments, and the paper is still a preprint, so they should not be interpreted directly as proof that “autonomous AI research is now reliable.”
They do, however, point to an important direction: reliability cannot depend on a model's self-discipline; it must become a checking mechanism inside the system.
This is similar to software engineering. Code cannot go live simply because a developer says, “I think it is fine.”
AI-generated research should not automatically earn trust merely because its language is fluent and its charts look polished.
What Is the Difference Between CoE and RAG?
At this point, many people may ask: isn't this just RAG with citations?
Not exactly.
RAG primarily addresses where a model should find external knowledge.
CoE primarily addresses how a Claim generated by the model can be proven.
RAG:
Document → Retrieval → Answer
CoE:
Claim → Evidence → Artifact → Verification
RAG can provide a model with real documents and help attach citations to an answer, but it does not inherently guarantee that:
- The citation truly supports the current Claim;
- The numbers in the paper come from actual runs;
- The method description matches the submitted code;
- The final conclusion can be traced through intermediate artifacts.
CoE is therefore not a replacement for RAG.
It is more like a layer of trust infrastructure placed over RAG, tool use, code execution, and paper writing.
My summary is:
RAG explains where the material comes from.
CoE goes on to ask what makes the conclusion valid.
Final Thoughts
For the past few years, we have focused on how to make models know more.
That gave us RAG, Vector Databases, Knowledge Graphs, and longer contexts.
Now that Agents have started searching, writing code, running experiments, and generating reports on their own, a new bottleneck has appeared: they must not only produce results, but also preserve how those results were produced.
The value of ScientistOne and Chain-of-Evidence is not that they create yet another AI that is better at writing papers. They push the standard for evaluating AI research one step forward: from whether the result looks real to whether the conclusion can be verified.
This may also be a lesson that Agents must learn before entering the real world. Trustworthiness does not mean being correct forever. It means that when an Agent is wrong, we can follow the evidence to find the problem; when it is right, we do not have to trust it on intuition alone.
A truly trustworthy Agent does not merely provide answers. It lets people inspect at any time why it is correct and where it might be wrong.
Top comments (0)