What AI actually speeds up in research work

john — Tue, 11 Aug 2026 08:42:49 +0000

I've been keeping a rough log for about a year of where AI tools saved me time in day-to-day research work, and where they quietly cost me time. The results were not what I expected. The wins were almost all in boring places. The losses were all in the places that felt most impressive in the demo.

There's one rule that predicts the split:

AI is worth using wherever verifying the output is cheaper than producing it.

That's it. Everything below follows from that. Writing a pandas transform is slow; checking whether a transform did the right thing is fast, because you can look at the output. Reasoning about whether a mixed-effects model is correctly specified is slow, and checking someone else's reasoning about it is just as slow. So the first one is a great use and the second one is a trap.

Here's the log.

1. Cleaning data files nobody designed

Every lab has a machine that exports CSVs written by someone in 2011. Units in the header row. Three date formats in one column because the firmware was updated halfway through the study. A stray n.d. where a float should be. Column names with trailing spaces that you will not see until you spend forty minutes on a KeyError.

The thing that made this work well was a change in what I ask for. I don't hand the model the data and ask for clean data back. I hand it a sample and ask for the transform.

What I paste is roughly:

print(df.head(20).to_markdown())
print(df.dtypes)
print(df.isna().sum())

Plus a plain description of what's wrong and what the target schema is. What comes back is a function:

def normalize_run(df: pd.DataFrame) -> pd.DataFrame:
    df = df.rename(columns=lambda c: c.strip().lower().replace(" ", "_"))

    # instrument switched date format at firmware 2.4
    df["collected_at"] = pd.to_datetime(
        df["collected_at"], format="mixed", dayfirst=True, errors="coerce"
    )

    # "n.d." and "<LOD" both mean below detection, but they are not the same
    # as missing — keep the distinction in a separate flag column
    below_lod = df["conc"].astype(str).str.strip().isin({"n.d.", "<LOD"})
    df["below_lod"] = below_lod
    df["conc"] = pd.to_numeric(df["conc"].where(~below_lod), errors="coerce")

    return df

The model never touches a single data point. It writes a transform, I read the transform, and I check it against the source file. If the transform is wrong, it's wrong loudly and identically on every row, which is exactly the failure mode you want. Silent per-row edits are the failure mode you don't.

One habit that pays for itself: ask for assertions at the end of the function, then keep them.

assert df["collected_at"].notna().all(), "unparsed dates remain"
assert df["conc"].between(0, 1e4).all() or df["conc"].isna().any()

They cost nothing and they've caught two genuine data problems for me that had nothing to do with AI.

2. First-pass literature screening

Not summarizing. Screening.

If you're starting a review and you have 800 abstracts from a Semantic Scholar or PubMed query, the expensive part isn't reading the 40 relevant ones. It's finding out which 40. That's a classification job with a rubric, and it's a good fit because verification is cheap: you spot-check.

The setup I use is boring on purpose. Write the inclusion criteria out explicitly, as if for a human RA:

INCLUDE if the paper reports an original measurement of X in a mammalian system.
EXCLUDE reviews, simulations without experimental validation, and non-mammalian systems.
If the abstract is ambiguous about the system, mark UNSURE — do not guess.

Then run each abstract through and store decision, reason, and the abstract ID. Three things matter:

Keep the UNSURE bucket. A screener that is forced to make a binary call will make one, confidently. The unsure pile is where you'll actually read.
Log the rejections with reasons. You need to be able to audit this later, and if you're writing a systematic review you need it for the methods section anyway.
Hand-screen a random 50 yourself and compare. This takes an hour and tells you whether the rubric is working. If agreement is bad, the rubric is ambiguous, not the model.

The last point is the whole ballgame. Most of the time when screening goes badly it's because the criteria were never precise enough for a human RA either, and you've just found out.

3. Code you know how to verify but hate to write

Argparse blocks. matplotlib subplot grids where you have to remember whether it's axes[i, j] or axes[j, i]. statsmodels formula syntax. Snakemake rules. The regex for parsing your instrument's filename convention.

This is the highest-return category and the least interesting one. The common feature is that I can look at the output and know immediately whether it's right, usually by running it. Nobody needs to be careful here.

The related case is the throwaway script that would otherwise not get written at all. A five-minute script that renames 400 files according to the sample sheet used to be a thing I did by hand because writing the script felt like more work. That trade has changed, and downstream it means fewer manual steps that never got documented.

4. Figures, which are a file format problem more than a drawing problem

Figures are where I expected the biggest win and got the most complicated answer.

The drawing is not the bottleneck. Ask anyone who has published: the bottleneck is revision three, when a reviewer asks you to rename one protein in panel B, and the layered source file is on a laptop that got reimaged, and the only surviving artifact is a flattened 300 dpi PNG. Now you're rebuilding an entire panel at 1 AM because of one word.

So the useful question for any figure tool, AI or not, isn't "can it produce a nice image." It's "what does the file look like six months from now." A generated PNG that can't be edited has moved the problem, not solved it. I want layered SVG that opens properly in Illustrator or Inkscape, or an editable PPTX if that's where the group's slide deck lives.

That's roughly why I use Scientific Figure for schematics and graphical abstracts rather than a general image model: it goes from a description, a notebook sketch, or a reference figure to a draft, but the output stays as editable layers you can export as SVG or PPTX. The point isn't the generation step. It's that the revision six months later costs two minutes instead of an afternoon.

Two things worth saying plainly. First, you have to check every label yourself, every time. Generated text inside images is the single least reliable thing in this whole workflow, and a transposed subscript in a figure is a correction notice. Second, check your target journal's AI disclosure policy before you submit anything, because they vary and they've been changing every few months.

5. Reverse-outlining your own writing

I don't draft prose with AI. What I did find useful is the opposite direction: paste in a section you already wrote and ask for the outline it implies. One sentence per paragraph, no commentary.

When the returned outline doesn't match the outline you had in your head, that's a real finding about your writing. Usually it means paragraph four is doing two jobs, or the argument you thought was in the discussion is actually only in the figure caption.

The adversarial version works too: "You are reviewer 2. List the three methodological objections most likely to appear in a review of this section." You'll get some noise, but the objections that make you wince are the ones a real reviewer will also raise, and it's much cheaper to hear them now.

6. Things it cost me time on

Statistics reasoning. Ask about model specification, multiple comparison correction, or what a particular p-value licenses you to claim, and you get an answer with the shape of expertise. Some of them are wrong in ways that take a domain expert to see. Verification here is as expensive as doing it yourself, so by the rule at the top, it's not worth it. I ask a statistician.

Citations. This should be a solved problem and isn't. Anything that produces a reference without retrieving it will produce references that don't exist, with plausible authors, in plausible journals, in plausible years. Every citation gets resolved by DOI or it doesn't go in. No exceptions, no "it's probably fine."

The pattern in both is the same: the output is confident, structurally correct, and expensive to check. That's the exact profile of things not to automate.

The rule again

Sort your tasks by how expensive it is to check the answer, not by how hard the task is.

Data cleaning, boilerplate, file wrangling, first-pass screening: check cheap, automate freely. Statistics, citations, anything where being subtly wrong is indistinguishable from being right until a reviewer catches it: do it yourself, or ask a human who knows.

The tooling will keep getting better and this rule won't change, because it isn't about capability. It's about who's accountable for the paper.

DEV Community: john