If you build tools for researchers, you learn quickly that "summarise this paper" is the easy part. The hard part is everything a researcher does after reading the summary: checking whether it is right, finding the sentence that supports it, citing it correctly, and coming back to it six months later when a reviewer asks a pointed question. This post is a set of design notes on what an evidence-traceable research agent needs to do, written from the perspective of someone thinking about the architecture rather than the marketing. I use Yila AI as a running example because it is built around exactly this problem, but the patterns apply to anything you build in this space.
The core requirement: every claim has an address
In a normal chatbot, the output is a stream of text with no internal structure. In a research tool, every factual statement in the output needs an address: which document, which page or section, which passage. Without that address, the user has two options. They can trust the output, which is unacceptable in research, or they can redo the work to verify it, which removes most of the value.
Architecturally, this means the system has to keep provenance through every stage of the pipeline. If you chunk a PDF for retrieval, the chunks need stable identifiers that map back to page and paragraph. If you summarise several chunks, the summary needs to carry the identifiers of the chunks it drew on. If you then rewrite the summary into a draft paragraph, the citations need to survive the rewrite. Losing provenance at any stage is a bug, not a cosmetic issue.
Parsing is where most quality is lost
Academic PDFs are hostile documents. Two-column layouts, footnotes, figures with embedded text, tables split across pages, equations rendered as images, supplementary materials in separate files. A retrieval system that treats a paper as a flat string will regularly attribute a sentence from a footnote to the main text or merge two columns into nonsense.
Practical lessons:
- Preserve the section structure. Knowing that a sentence comes from "Limitations" rather than "Results" changes how it should be used.
- Treat tables as data, not as text. Extract them into rows and columns so numbers can be quoted exactly and compared.
- Keep figure captions attached to figures. Captions often contain the clearest statement of what a figure shows.
- Record page numbers. Researchers cite page numbers, and reviewers check them.
Retrieval needs to be scoped to the user's library
General web search is useful for discovery, but once a user is working on a review, answers should come from the set of papers they have chosen. Mixing in unvetted sources is how fabricated or low-quality citations creep into a draft. A research workspace should make the scope explicit: this answer is based on these twelve papers in your project, and here is which ones contributed.
In Yila, a project holds the papers and datasets for one piece of work, and questions are answered against that project. That makes it possible to ask comparative questions ("which of these studies used a randomised design?") and get answers that are both complete within the library and traceable.
Generation should be constrained by evidence
Language models are fluent by default and cautious only when forced. For research writing, the generation step should be constrained so that each sentence that makes a factual claim is grounded in retrieved passages, and sentences that cannot be grounded are either flagged or omitted. A useful pattern is to generate claims and citations together, then run a verification pass that checks each claim against its cited passage and marks mismatches.
This verification pass is cheap relative to the cost of a wrong citation in a submitted paper. It also gives you a natural place to show the user what was checked.
Show the evidence in the interface
The best provenance in the world is useless if the interface hides it. Every claim in the output should be clickable, opening the source passage with the relevant sentence highlighted. Hover previews help for quick checks. For tables and figures, the user should be able to jump to the exact table cell or figure.
This design choice changes user behaviour. When checking is a click away, users check. When checking requires opening a PDF and searching, they do not.
Data analysis needs the same discipline
Many research questions end in a dataset rather than a paper. The same principles apply: every chart should carry the code that produced it, every number should be traceable to a computation, and cleaning steps should be recorded rather than applied invisibly. Yila runs analysis in code that the user can read and rerun, which keeps charts reproducible and lets a reviewer's request for a robustness check be answered in minutes.
For developers, the lesson is to treat generated code as a first-class artefact. Store it with the output, version it, and make it easy to edit and rerun.
Outputs researchers actually need
Researchers do not stop at answers. They produce literature matrices, draft sections, slide decks for lab meetings, posters for conferences. Each of these output formats needs to preserve citations. A slide deck generated from a paper should link each number back to the table it came from. A literature matrix should link each cell to the passage that justifies it. If the output format drops provenance, the user will have to rebuild it by hand.
Evaluation: test the boring failures
When evaluating a research agent, it is tempting to focus on impressive capabilities. The failures that hurt users are more mundane:
- Citing the wrong paper for a correct claim.
- Citing the right paper but the wrong page or section.
- Quoting a number that was rounded or transposed.
- Presenting an author's hypothesis as their finding.
- Omitting a limitation that the authors stated explicitly.
Build test sets around these. Take a set of papers, write questions whose answers you know, and check not just whether the answer is correct but whether the citation is precise. Track citation precision as a primary metric alongside answer accuracy.
Privacy and trust
Researchers often work with unpublished manuscripts, confidential data and embargoed results. A research tool needs clear guarantees about where documents are stored, who can see them and whether they are used for training. These guarantees should be visible in the product, not buried in a policy page.
Summary for builders
If you are building in this space, the short version is: keep provenance through every stage, scope retrieval to the user's library, constrain generation by evidence, show the evidence in the interface, and test the boring failures. Tools that get these right earn trust slowly and keep it. Tools that skip them produce impressive demos and unusable drafts.
You can see one implementation of these ideas at https://yila.ai, where literature review, data analysis and writing share a single evidence-linked workspace.
Top comments (0)