DEV Community

Alex Spinov
Alex Spinov

Posted on

I Built a Research Paper Finder That Actually Works (OpenAlex + Python)

#api #python #machinelearning #research

I was tired of Google Scholar.

No API. No export. No way to filter by citation count. And half the results were behind paywalls.

So I built a research paper finder using OpenAlex — an open database of 250M+ academic works. Here's what happened.

The Problem

I needed to find the 50 most-cited papers on "transformer attention mechanism" published after 2020. Google Scholar can't do this. Semantic Scholar has rate limits. Web of Science costs $thousands.

OpenAlex does it in 3 lines:

import requests

resp = requests.get("https://api.openalex.org/works", params={
    "search": "transformer attention mechanism",
    "filter": "from_publication_date:2020-01-01",
    "sort": "cited_by_count:desc",
    "per_page": 50
})

papers = resp.json()["results"]
for p in papers[:5]:
    print(f"{p['cited_by_count']:>5} citations | {p['title'][:60]}")
Enter fullscreen mode Exit fullscreen mode

Output:

12847 citations | Attention Is All You Need
 8234 citations | BERT: Pre-training of Deep Bidirectional Transformers
 6521 citations | An Image is Worth 16x16 Words: Transformers for Image R...
 4102 citations | Language Models are Few-Shot Learners
 3891 citations | Training language models to follow instructions with hum...
Enter fullscreen mode Exit fullscreen mode

What I Built

A Python tool that:

  1. Searches 250M+ papers across all fields
  2. Filters by year, citation count, open access status, institution
  3. Exports to CSV for analysis
  4. Tracks citation trends over time 
from openalex_toolkit import OpenAlexClient

client = OpenAlexClient()

# Find most-cited ML papers from 2023
papers = client.search(
    "large language models",
    sort="cited_by_count:desc",
    from_date="2023-01-01",
    per_page=20
)

# Export to CSV
client.export_csv("large language models", limit=100)

# Find papers by institution
mit_papers = client.search(
    "machine learning",
    institution="Massachusetts Institute of Technology",
    per_page=10
)
Enter fullscreen mode Exit fullscreen mode

3 Things I Learned

1. Citation counts are power-law distributed

The top 1% of papers get 50%+ of all citations. In ML specifically, "Attention Is All You Need" has more citations than the bottom 10,000 transformer papers combined.

2. China is catching up — fast 

# Count ML papers by country in 2024
for country in ["US", "CN", "GB", "DE", "JP"]:
    resp = requests.get("https://api.openalex.org/works", params={
        "search": "machine learning",
        "filter": f"from_publication_date:2024-01-01,institutions.country_code:{country}",
        "per_page": 1
    })
    count = resp.json()["meta"]["count"]
    print(f"{country}: {count:,} papers")
Enter fullscreen mode Exit fullscreen mode

China publishes nearly as many ML papers as the US now. And the gap is closing every year.

3. Most "breakthrough" papers are incremental

I analyzed the titles of the top 100 most-cited ML papers from 2023. 73% were variations on existing architectures. True novel approaches were ~15% of the list.

The Free API Ecosystem for Research

OpenAlex isn't the only free academic API. Here's the full landscape:

API Records Best For
OpenAlex 250M+ Bibliometrics, citation analysis
Crossref 150M+ DOI metadata, journal data
PubMed 36M+ Medical/biomedical research
arXiv 2M+ Preprints (AI, physics, math)
CORE 300M+ Open access full text

All free. All open source toolkits on my GitHub.

What's your go-to tool for finding research papers? And what field would you want me to analyze next?

Top comments (0)