Search papers, PubMed and clinical trials in one query (no API keys)

#api #python #healthcare #datascience

A literature sweep usually means three tabs: Crossref for papers, PubMed for biomedical citations, ClinicalTrials.gov for studies. All three have free public APIs, and none needs a key. This post shows a short Python script that queries all three, the traps to expect, and a no-code option that returns one normalized table.

Disclosure: I maintain the Actor mentioned in the second half. The script in the first half is free and stands on its own.

What trips people up

Three response shapes. Crossref returns message.items, PubMed needs an esearch call for IDs and then esummary for details, and ClinicalTrials.gov v2 nests everything under protocolSection.
Politeness rules. Crossref wants a mailto; NCBI asks for tool and email and allows about 3 requests per second without a key.
Dates mean different things. Publication date for papers, start date for trials.
Abstracts are a copyright grey area, so this post only collects citation metadata and links.

The DIY way (free)

Standard library only. Set CONTACT_EMAIL to your own address.

import json
import os
import sys
import urllib.parse
import urllib.request

EMAIL = os.environ["CONTACT_EMAIL"]  # Crossref and NCBI ask for a contact address


def get(url):
    req = urllib.request.Request(url, headers={"User-Agent": f"research-example (mailto:{EMAIL})"})
    with urllib.request.urlopen(req, timeout=30) as r:
        return json.load(r)


def crossref(q, n=5):
    url = "https://api.crossref.org/works?" + urllib.parse.urlencode({"query": q, "rows": n, "mailto": EMAIL})
    return [{"source": "crossref", "id": w.get("DOI"), "title": (w.get("title") or [""])[0]}
            for w in get(url)["message"]["items"]]


def pubmed(q, n=5):
    base = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/"
    common = {"tool": "research-example", "email": EMAIL}
    ids = get(base + "esearch.fcgi?" + urllib.parse.urlencode(
        {**common, "db": "pubmed", "term": q, "retmax": n, "retmode": "json"}))["esearchresult"]["idlist"]
    if not ids:
        return []
    summ = get(base + "esummary.fcgi?" + urllib.parse.urlencode(
        {**common, "db": "pubmed", "id": ",".join(ids), "retmode": "json"}))["result"]
    return [{"source": "pubmed", "id": i, "title": summ[i].get("title")} for i in ids]


def trials(q, n=5):
    url = "https://clinicaltrials.gov/api/v2/studies?" + urllib.parse.urlencode({"query.term": q, "pageSize": n})
    return [{"source": "clinicaltrials", "id": s["protocolSection"]["identificationModule"]["nctId"],
             "title": s["protocolSection"]["identificationModule"].get("briefTitle")}
            for s in get(url).get("studies", [])]


if __name__ == "__main__":
    sys.stdout.reconfigure(encoding="utf-8")
    query = " ".join(sys.argv[1:]) or "CRISPR gene therapy"
    for fn in (crossref, pubmed, trials):
        for row in fn(query):
            print(row)

I ran it with CRISPR gene therapy on 2026-11-08 and got 15 rows, five per source:

Source	Example id
crossref	10.1007/978-3-030-93333-3_5
pubmed	42999773
clinicaltrials	NCT06465537

That is enough for a quick look. For deeper searches you add paging (Crossref cursors, PubMed retstart, ClinicalTrials.gov nextPageToken), retries, and filters for trial status and phase.

The scaled way: the Datawharf research-search Actor

I publish research-search on Apify. It calls the same three official APIs, paces requests to the documented limits, keeps going if one source fails, and returns one flat table with the same columns for every source (source, id, title, authors, date, url, plus doi, pmid or nctId). No abstracts, on purpose.

Python (pip install "apify-client>=3", token in APIFY_TOKEN), asking for recruiting phase 2/3 diabetes trials started since 2024:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

trials = client.actor("datawharf-tools/research-search").call(run_input={
    "mode": "clinicalTrials",
    "condition": "type 2 diabetes",
    "trialStatus": ["RECRUITING"],
    "trialPhase": ["PHASE2", "PHASE3"],
    "dateFrom": "2024-01-01",
    "maxResultsPerSource": 50,
})
for r in client.dataset(trials.default_dataset_id).iterate_items():
    print(r["nctId"], r["phase"], r["sponsor"], r["title"])

A JavaScript version and more examples are in the examples repo. Export the dataset as CSV or Excel from the run page, or put the saved input on a weekly Schedule to watch a topic.

Gotchas

all mode limit applies per source. With maxResultsPerSource: 20 you can get up to 60 rows.
Trial dates. dateFrom and dateTo filter on study start date for trials, publication date for papers.
Check the record. ClinicalTrials.gov records are sponsor-submitted and change; open the linked record before relying on it.
Not medical advice. This is a discovery tool for citations and links.

Cost

The script is free. The Actor charges $0.40 per 1,000 result rows plus standard Apify usage, so the 60-row all-mode case is about $0.024 in result charges; you can cap the charge per run.

If something looks wrong, tell me in the comments.

Disclosure: I maintain this Actor. It is not affiliated with or endorsed by Crossref, NLM, NIH or ClinicalTrials.gov.

DEV Community