A research system called Spark-to-Paper generates complete papers end to end and, on audit, got its citations right 99.5 percent of the time across 384 references. Built as thirteen composable skills running inside an existing coding assistant, it separates model judgment from deterministic execution at every stage. Its most striking measured effect is on fabrication: detection of invented claims rose from 14 percent to 92 percent when the full stack was enabled.
Key facts
- Implemented as 13 composable skills inside a coding assistant, with explicit separation between judgment and deterministic execution.
- Evaluated across 8 controlled topics, with 384 references audited for citation validity and roughly 1,900 figure elements analysed for editability. Blinded raters scored review quality.
- Reported results: 99.5 percent citation validity, 96.4 percent figure editability, fabrication detection improving from 14 percent to 92 percent under the full stack, and 74 percent adversarial-review precision.
- Primary source: Spark-to-Paper: End-to-End Research Paper Generation as a Composable Skill, with full text on arXiv.
Every previous attempt at an automated researcher has died on the same hill. The prose is fluent, the structure is convincing, and the citations are fiction -- real-sounding author lists attached to papers that do not exist, or real papers that say nothing like what the text claims. That failure is not incidental; it is the hallucination problem in its most consequential form, because a fabricated citation is specifically designed to look checkable while not being checked.
Spark-to-Paper's answer is architectural rather than a matter of better prompting. The system is decomposed into thirteen skills, and the load-bearing decision is which parts of the pipeline are allowed to involve a model's judgment at all. Choosing a research question is judgment. Resolving a citation to a real record is not -- that is a lookup, and it runs as code. Checking that a figure's elements are editable rather than flattened into an image is not judgment either. The paper's mechanisms follow that split: evidence-gated generation, pre-committed experiment design, deterministic integrity checks, self-critique, and a bounded self-refutation loop in which the system argues against its own findings before finalising them.
Pre-committing to an experiment design before running it is the most quietly important of those. It is the machine version of pre-registration, and it closes the most common route to a dishonest result: deciding what you were testing after seeing which test came out well. A system that writes its own analysis plan and then runs it cannot rationalise its way to a finding as easily as one that writes the plan afterwards.
The evaluation is unusually concrete for this genre. Rather than asking whether the output "reads like a paper," the authors audited 384 references for validity, analysed roughly 1,900 figure elements for editability, and used blinded raters to score review precision. Those are boring, checkable quantities, and boring checkable quantities are exactly what has been missing from claims about automated science. The 14-to-92 percent jump in fabrication detection is the number that best captures the design philosophy: the improvement comes from the surrounding machinery, not from the model becoming more honest.
The caveats are equally clear, and the authors do not obscure them. Eight controlled topics is a small evaluation. The team built the system and designed the audit, which is the standard conflict in agentic-systems papers and one reason the reliability of AI judges is itself now under scrutiny. Citation validity is a measure of bookkeeping, not of insight: a paper can have perfect references and be worthless, and nothing here shows the generated work is novel or important. The comparison is against human-written preprints and prior autonomous research systems, and the paper makes no claim that any output was accepted anywhere.
It also arrives into a literature that is not obviously in good health. An audit this year found NeurIPS papers averaging six objective mistakes each, up from four, and a separate replication effort reproduced 105 ICML orals and found only 34 mostly held up. Against that baseline, a system with a 99.5 percent citation-validity floor is not competing with an idealised standard of rigour. It is competing with the actual one.
The transferable lesson is narrower than "AI can write papers," and more useful. The design principle here -- identify every step that can be verified mechanically and refuse to let the model do it -- is not specific to research writing. It is the same principle behind routing unstable reasoning into deterministic code and behind the argument that agent safety belongs in the runtime rather than the weights. Packaging that discipline as portable skill files rather than a monolithic application is what makes it reusable, and probably matters more than the specific application to papers.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)