DEV Community: bao han

A paper-to-tool interface needs a scientific output contract

bao han — Tue, 03 Nov 2026 06:17:40 +0000

A tool can accept an input, finish without an exception, and return a convincing explanation while leaving the most important scientific question unresolved: what exactly was predicted, using which materials and method?

Paper2Agent makes that boundary worth examining. The authors connect research papers and their original code to callable tools, tests and natural-language use. Their public TISSUE example names three input files and requests a 95% prediction interval for Acta2. That is a concrete task with a reviewable chain, rather than an invitation to ask an unconstrained chatbot for a biological conclusion.

This article explains that example and gives a small metadata checker you can run locally. It does not install Paper2Agent, generate an MCP server, read research matrices, train a model or reproduce the authors’ result. The Python example checks our own teaching contract, not the implementation or validity of TISSUE.

Begin with the materials, not the prose

The repository’s example uses Spatial_count.txt, Locations.txt and scRNA_count.txt. Those names alone do not establish that a new dataset is suitable. A file can have the expected extension while using a different gene naming system, an incompatible orientation or locations that no longer match the count matrix.

An adequate handoff therefore includes the role of each file, its provenance and permission, the relevant units or representation, and how identifiers line up. Preserve the exact filenames from an author example when explaining it. For your own work, preserve the actual file references; do not rename unrelated files to make an interface appear compatible.

The target also needs two parts. Acta2 identifies a gene in this example. It does not, by itself, identify every quantity that a method could predict about that gene. The requested prediction target and its unit or scale still need an explicit definition that matches the original method.

Separate a contract check from an analysis

Here is an intentionally incomplete, author-created teaching contract. The paper date and repository link identify the source we are discussing. The missing revision and prediction target represent unresolved review work; they are not values we can infer from a nice-looking answer.

contract = {
    "repository": "https://github.com/jmiao24/Paper2Agent",
    "repository_revision": None,
    "inputs": {
        "spatial_counts": "Spatial_count.txt",
        "spatial_locations": "Locations.txt",
        "single_cell_reference": "scRNA_count.txt",
    },
    "gene": "Acta2",
    "prediction_target": None,
    "interval": {"kind": "prediction", "level": 0.95},
}


def review_gaps(c):
    gaps = []
    for key in ("repository", "repository_revision", "gene", "prediction_target"):
        value = c.get(key)
        if not isinstance(value, str) or not value.strip():
            gaps.append(f"Missing reviewed metadata: {key}")

    inputs = c.get("inputs", {})
    for role in ("spatial_counts", "spatial_locations", "single_cell_reference"):
        value = inputs.get(role)
        if not isinstance(value, str) or not value.strip():
            gaps.append(f"Missing input reference: {role}")

    interval = c.get("interval", {})
    level = interval.get("level")
    if interval.get("kind") != "prediction":
        gaps.append("Confirm interval kind against the source method")
    if isinstance(level, bool) or not isinstance(level, (int, float)) or not 0 < level < 1:
        gaps.append("Interval level must be a number strictly between 0 and 1")
    return gaps


for gap in review_gaps(contract):
    print(gap)

For this exact dictionary, the output is:

Missing reviewed metadata: repository_revision
Missing reviewed metadata: prediction_target

I checked this exact standard-library-only example locally for the article. No scientific tool or external service is called. The checker reports missing metadata; it does not validate file contents, data permission, a commit’s correctness, software dependencies or scientific assumptions. Even an empty list of gaps would mean only that these selected fields passed this small structural check.

A wrapper should preserve the chain

The next step is not to ask an AI to fill the two blanks with plausible text. Retrieve the actual repository revision used for the work and locate the prediction target in the source method and configuration. If that information is unavailable, record the limitation and retain the incomplete state.

For a real execution, retain the actual command or tool call, its arguments, the software environment and input references, and the original output files. Keep an explanation as an additional interpretation layer. A rewritten narrative should not become the only surviving record of what the method returned.

The same principle applies to tests. Record what the test supplied, what it asserted and which path it covered. A successful example can establish that a particular interface path ran under stated conditions. It cannot establish correctness for every dataset, every parameter choice or every downstream biological interpretation. Do not relabel interface success as experimental replication.

Preserve the meaning of the interval

For the TISSUE request, retain the interval type, level and specified prediction target when describing the output. A prediction interval concerns a prediction under the method’s setup and assumptions. The phrase “95%” does not transform it into a 95% probability that the gene causes a biological effect.

Do not invent interval bounds to complete a presentation, or substitute an uncertainty illustration for a returned numerical result. If no analysis was performed, a diagram can show the names of expected fields while stating that it contains no computed result. If a result exists, interpret it using the relevant method, data and validation evidence rather than the confidence of a generated explanation.

Practical handoff

Before relying on a paper-derived tool call, assemble the source paper and code revision, authorized input references and alignment notes, target and parameters, covered tests, original output, and interpretation limits. That package makes disagreements actionable: a reviewer can locate a missing dependency, a mismatched identifier or an unsupported claim without guessing which version produced the answer.

We are the RigorMed team. This is a developer-focused method note about public author materials, not a claim that we ran the example or integrated this particular project into our product. Open-source code also does not settle model/API costs, dependency availability or data permissions.

Sources read for the underlying example: the Nature paper, published September 16, 2026, and the authors’ repository, reviewed November 2, 2026 for the original video materials. This is an explanation of an existing publication, not a claim of a new release today.

A 95% accurate research classifier can miss every useful candidate

bao han — Tue, 03 Nov 2026 03:50:42 +0000

A classifier can correctly label 95 out of 100 candidates and still find none of the candidates you actually want. The first question for a research screening task is therefore more specific than “Is the accuracy high?”: which cases did the system find, and which did it miss?

Here is a complete synthetic example you can check without training a model or calling an AI service. Think of a research workflow that screens experimental conditions for a useful property. In this example, label 1 means the condition has that property and label 0 means it does not. These labels and predictions are invented teaching data. They are not experimental measurements or performance results from an actual AI model.

Keep the test denominator fixed

Our test set has 100 records: 5 positive and 95 negative. The first prediction vector labels all 100 records negative.

Actual class	Predicted positive	Predicted negative
Positive	0	5
Negative	0	95

This gives 95 correct predictions out of 100, so accuracy is 95%. But the system finds zero of the five positive candidates. Positive-class recall is 0 / 5 = 0%.

Precision asks a different question: among the candidates reported as positive, how many are actually positive? This system reports no positive candidates, so the denominator is zero. The mathematical ratio is undefined. A software library may expose a configurable convention for zero division; a report should state that convention rather than silently treating the result as perfect precision.

A lower accuracy can accompany fewer missed candidates

Now keep exactly the same records and actual labels. Construct a second prediction vector that identifies four of the five positive candidates, but also reports six negative candidates as positive.

Actual class	Predicted positive	Predicted negative
Positive	4	1
Negative	6	89

The second vector gives:

Precision: 4 / (4 + 6) = 40%.
Recall: 4 / (4 + 1) = 80%.
Accuracy: (4 + 89) / 100 = 93%.

Accuracy fell from 95% to 93%. Missed positive candidates fell from five to one, while false positives rose from zero to six. Those are different changes, with different research costs. Missing a promising condition may be expensive; so may spending scarce laboratory resources on false positives. These counts alone do not establish that the second prediction vector is preferable.

We constructed both vectors directly. This is not evidence that threshold tuning improved a trained model, or that a particular model can achieve these results in a laboratory.

Reproduce the arithmetic with standard Python

The code below keeps record identifiers, labels and predictions aligned. It does not use an external dataset, model, network request or third-party package.

from collections import Counter

ids = [f"test-{i:03}" for i in range(100)]
actual = [1] * 5 + [0] * 95
all_negative = [0] * 100
alternative = [1] * 4 + [0] + [1] * 6 + [0] * 89


def audit(record_ids, labels, predictions):
    assert len(record_ids) == len(labels) == len(predictions)
    assert len(record_ids) == len(set(record_ids))
    assert len(labels) > 0
    assert set(labels) <= {0, 1}
    assert set(predictions) <= {0, 1}

    cells = Counter(zip(labels, predictions))
    tp, fn = cells[1, 1], cells[1, 0]
    fp, tn = cells[0, 1], cells[0, 0]

    def ratio(numerator, denominator):
        return numerator / denominator if denominator else None

    return {
        "TP": tp, "FP": fp, "TN": tn, "FN": fn,
        "test_n": len(labels),
        "accuracy": ratio(tp + tn, len(labels)),
        "precision": ratio(tp, tp + fp),
        "recall": ratio(tp, tp + fn),
    }


print(audit(ids, actual, all_negative))
print(audit(ids, actual, alternative))

The first output is TP=0, FP=0, TN=95, FN=5, with accuracy 0.95, precision None and recall 0.0. The second is TP=4, FP=6, TN=89, FN=1, with accuracy 0.93, precision 0.4 and recall 0.8.

The duplicate-identifier check catches one simple input problem. It does not establish independence from training data, correct labeling, representative sampling or freedom from preprocessing leakage. Those require separate evidence about how the evaluation set and predictions were produced.

Give an AI assistant a bounded audit task

An assistant can help prepare this table, but it needs enough information to preserve the denominator. A usable request is:

Positive label 1 means the experimental condition has the target property. Using only the attached per-record labels, predictions and test identifiers, report TP, FP, TN and FN. Show the numerator and denominator for accuracy, positive-class precision and positive-class recall. Report zero denominators explicitly. Keep the two prediction versions separate. Flag duplicate identifiers or missing labels; do not silently drop rows or invent replacements. Do not train a model or choose a threshold from this test set.

For actual research, retain the prediction version, label definition, split unit and evaluation-set selection rules. If a classification threshold needs tuning, choose it within an appropriate validation or cross-validation process before final independent testing. Repeatedly selecting the threshold that looks best on the final test set changes what that test can tell you.

The high score in the opening example was arithmetically correct. The missing information was which cases that score counted as success.

Sources, read November 3, 2026:

scikit-learn 1.9.1: classification metrics, including accuracy, precision, recall and imbalanced classification.
scikit-learn: tuning the decision threshold, including positive-label definitions and validation boundaries.

Written by the RigorMed team. All example labels and predictions above are synthetic teaching data.