Nobody in pharma will admit this out loud but a significant chunk of competitive intelligence in healthcare still happens via someone's Friday afternoon Google Scholar session and a manually updated spreadsheet that lives on one person's laptop.
This is a solved problem. Most companies just have not solved it yet.
Here is how to build something that actually works.
What We Are Actually Building
A system that watches clinical trial literature and competitor research in a specific therapeutic area automatically, then delivers structured summaries to whoever needs them without anyone having to go looking.
Think of it as a research analyst that never sleeps, never misses a paper, and does not expense airport lounges.
The use cases are obvious once you see them. A biotech tracking competitor pipeline activity in oncology. A pharma team monitoring every new publication on a drug target before a quarterly review. A VC doing diligence on a therapeutic area and needing a complete picture of the research landscape fast.
All of them have the same underlying need. Fresh, full text academic content, automatically, without manual effort.
Why This Is Hard Without the Right Tool
Academic literature is the earliest signal that exists for most developments in healthcare. A competitor's clinical approach shows up in a published trial before it shows up anywhere else. A drug mechanism gets validated in a paper before it gets validated in a press release.
The problem is access. PubMed covers medicine but its API is clunky and returns metadata, not full text. Scraping journal websites is a maintenance nightmare. Most solutions either cost a fortune in licensing fees or require a dedicated person whose entire job is reading papers.
ScholarAPI changes the math entirely. Thirty million open access papers, full text pre-extracted, new content indexed within 48 hours of publication, simple REST API. The data layer that used to be the hard part becomes the easy part.
The Build
Three components. A collector, a processor, and a delivery mechanism. Let us go through each.
The Collector
This runs on a schedule, pulls new papers matching your therapeutic area, and stores them.
import requests
import json
from datetime import datetime, timedelta, timezone
from pathlib import Path
API_KEY = "sch_xxxxxxxxx"
BASE = "https://scholarapi.net/api/v1"
HEADERS = {"X-API-Key": API_KEY}
TARGETS = [
"KRAS inhibitor non-small cell lung cancer",
"PD-L1 checkpoint immunotherapy clinical trial",
"ADC antibody drug conjugate HER2",
]
def collect_new_papers(query: str, hours_back: int = 24) -> list:
since = (datetime.now(timezone.utc) - timedelta(hours=hours_back)).isoformat()
resp = requests.get(f"{BASE}/list", headers=HEADERS, params={
"q": query,
"indexed_after": since,
"has_text": "true",
"limit": 50
})
return resp.json().get("results", [])
def fetch_full_texts(paper_ids: list) -> dict:
ids = ",".join(paper_ids[:100])
resp = requests.get(f"{BASE}/texts/{ids}", headers=HEADERS)
return resp.json()
def run_collection(output_file: str = "new_papers.jsonl"):
all_papers = []
for query in TARGETS:
papers = collect_new_papers(query)
print(f"'{query}': {len(papers)} new papers")
all_papers.extend(papers)
if not all_papers:
print("Nothing new today.")
return
ids = [p["id"] for p in all_papers]
texts = fetch_full_texts(ids)
with open(output_file, "a") as f:
for paper in all_papers:
pid = paper["id"]
text = texts.get(pid)
if not text:
continue
f.write(json.dumps({
"id": pid,
"title": paper.get("title"),
"date": paper.get("published_date"),
"url": paper.get("url"),
"text": text,
}) + "\n")
print(f"Collected {len(all_papers)} papers.")
run_collection()
Run this daily via cron. It appends only new content so your file grows automatically without duplicates.
The Processor
Raw paper text is not what your stakeholders want. They want signal. This is where an LLM earns its keep.
import anthropic
client = anthropic.Anthropic()
def extract_competitive_signal(title: str, text: str) -> dict:
prompt = f"""
You are a biotech competitive intelligence analyst.
Read this research paper and extract:
1. The main clinical or scientific finding in one sentence
2. Any competitor companies or drugs mentioned by name
3. Whether this represents positive, negative, or neutral news for the therapeutic area
4. Who should read this: researchers, BD team, executives, or investors
Paper title: {title}
Paper text (first 3000 words):
{text[:3000]}
Respond in JSON with keys: finding, competitors_mentioned, sentiment, audience
"""
message = client.messages.create(
model="claude-opus-4-6",
max_tokens=500,
messages=[{"role": "user", "content": prompt}]
)
raw = message.content[0].text
try:
return json.loads(raw)
except Exception:
return {"raw": raw}
You now have structured intelligence from unstructured papers.
The Delivery
Nobody reads a raw JSON file. Send a digest.
import smtplib
from email.mime.text import MIMEText
def send_digest(insights: list, recipient: str):
body = "COMPETITIVE INTELLIGENCE DIGEST\n"
body += f"Generated: {datetime.now().strftime('%B %d, %Y')}\n\n"
for item in insights:
body += f"PAPER: {item['title']}\n"
body += f"Finding: {item['signal'].get('finding', 'N/A')}\n"
body += f"Sentiment: {item['signal'].get('sentiment', 'N/A')}\n"
body += f"Competitors: {', '.join(item['signal'].get('competitors_mentioned', []))}\n"
body += f"For: {item['signal'].get('audience', 'N/A')}\n"
body += f"Source: {item['url']}\n"
body += "\n" + "-"*40 + "\n\n"
msg = MIMEText(body)
msg["Subject"] = f"Research Intelligence Digest — {datetime.now().strftime('%b %d')}"
msg["From"] = "intel@yourcompany.com"
msg["To"] = recipient
with smtplib.SMTP("smtp.yourprovider.com", 587) as s:
s.starttls()
s.login("user", "password")
s.send_message(msg)
print(f"Digest sent to {recipient}")
Point it at Slack instead of email. Or Notion. Or a dashboard. The digest is just structured text. Send it wherever your stakeholders already live.
What This Actually Costs
Running this daily across three therapeutic areas, pulling 30 papers, reading full text on all of them:
ScholarAPI: roughly 500 credits per day. The $149 pack lasts about six months of daily monitoring. Under $25 a month for a pipeline that would cost a full time analyst salary to replicate manually.
That is not a rounding error. That is a different category of tool.
The Honest Limitation
Open access only. Papers behind Elsevier, Wiley, or Taylor and Francis paywalls are not in the index. In oncology and immunology this matters less than you might think because open access publishing has become the norm for high impact clinical research. In some other areas it matters more. Know your field before assuming full coverage.
New content appears within 48 hours, not instantly. For competitive intelligence purposes that is fine. For breaking news it is not the right tool.
Who Actually Builds This
Not just engineers. A BD analyst who knows basic Python can have this running in a weekend. A research operations team can automate what currently takes a person two days a week. A startup with no budget for enterprise intelligence tools can have something better than what most large pharma teams are using.
The papers have always been public. The access has always been free for open access content. The missing piece was a clean API that made it programmable.
That part is solved now.
ScholarAPI (1,000 free credits on signup).
Top comments (0)